Methods, devices, intelligent blackboard writing extraction systems, and storage media for displaying blackboard writing

Abstract

This application discloses a method, apparatus, intelligent blackboard writing extraction system, and storage medium for displaying blackboard writing. The method includes: performing blackboard recognition on the first video frame of the original blackboard writing video stream based on a target detection model, and using the obtained blackboard area as the blackboard writing background image; performing target recognition on each video frame of the original blackboard writing video stream based on a modified image segmentation model to obtain an electronic screen and a foreground occluding target; using the electronic screen as a shielding area to mask the foreground occluding target, obtaining a masking image for each video frame; and fusing and updating the blackboard writing background image based on the masking images of each video frame to obtain a completed blackboard writing video stream. This application improves the efficiency and accuracy of blackboard writing completion and enhances the matching degree of subsequent viewers' understanding of the teacher's lecture content and the blackboard writing content.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and data processing technology, and in particular to a method, apparatus, intelligent whiteboard writing extraction system, and storage medium for displaying whiteboard writing. Background Technology

[0002] With the development of multimedia technology, more and more smart interactive whiteboards are appearing in classrooms. In the classroom, teachers can choose to use an electronic screen to display courseware or write on the blackboard by hand, depending on the content of the lesson. However, when teachers choose to write on the blackboard or stand in front of the blackboard to explain, they inevitably obscure the content on the board.

[0003] In existing technologies, pre-calibration methods are typically used to test electronic screens. This approach requires manual intervention and confirmation of the equipment's position before each installation, which lacks flexibility. Furthermore, in scenarios where the blackboard can be pushed or pulled, the visible portion of the electronic screen often changes, leading to matching failures. Moreover, this method requires significant computing power and inference time. Summary of the Invention

[0004] In view of this, this application provides a method, apparatus, intelligent whiteboard writing extraction system, and storage medium for displaying whiteboard writing, in order to overcome or at least partially overcome the shortcomings of the prior art.

[0005] According to a first aspect of this application, embodiments of this application provide a method for displaying whiteboard writing, including:

[0006] Based on the object detection model, blackboard recognition is performed on the first video frame of the original blackboard video stream, and the obtained blackboard area is used as the blackboard background image.

[0007] Based on the modified image segmentation model, target recognition is performed on each video frame of the original whiteboard video stream to obtain the electronic screen and the foreground occlusion target; and the electronic screen is used as a shielding area to mask the foreground occlusion target to obtain the masking image of each video frame.

[0008] Based on the masking images of each video frame, the background image of the whiteboard is fused and updated to obtain the completed whiteboard video stream.

[0009] According to a second aspect of this application, embodiments of this application also provide a whiteboard display device, characterized in that the device comprises:

[0010] The blackboard segmentation unit is used to perform blackboard recognition on the first video frame of the original blackboard video stream based on the target detection model, and use the obtained blackboard area as the blackboard background image.

[0011] The masking detection unit is used to perform target recognition on each video frame of the original whiteboard video stream based on the modified image segmentation model to obtain the electronic screen and the foreground occlusion target; and to use the electronic screen as a shielding area to mask the foreground occlusion target to obtain the masking image of each video frame.

[0012] The occlusion completion unit is used to fuse and update the whiteboard background image based on the masking images of each video frame to obtain a completed whiteboard video stream.

[0013] According to a third aspect of this application, embodiments of this application also include an intelligent whiteboard extraction system, wherein the intelligent whiteboard extraction system is equipped with the aforementioned whiteboard display device.

[0014] According to another aspect of this application, a storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the above-described method for displaying blackboard writing.

[0015] According to another aspect of this application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the above-described method for displaying whiteboard writing.

[0016] Using the above technical solution, this application provides a method for displaying whiteboard writing. This method modifies the image segmentation model, enabling it to recognize electronic screens. Without increasing computational power consumption, it can simultaneously identify foreground occlusion targets such as people and teaching aids, as well as the electronic screen. When processing the whiteboard writing video, firstly, a target detection model is used to process the first frame of the video, separating the blackboard portion and caching it as the whiteboard writing background image. Then, based on the modified image segmentation model, foreground occlusion targets and the electronic screen are identified in each video frame of the original whiteboard writing video stream. Foreground occlusion targets include objects such as people and teaching aids that obstruct the whiteboard writing. The electronic screen area is set as a masking area. After masking the foreground occlusion targets using the masking area, a masked image for each video frame is obtained. Furthermore, based on the masked images of each video frame, the aforementioned whiteboard writing background image is fused and updated to obtain a complete, unobstructed whiteboard writing video stream. This application modifies the image segmentation model to simultaneously identify foreground occlusion targets and electronic screens, thereby enabling rapid segmentation of the electronic screen region and timely and flexible acquisition of occluded images. Furthermore, by using the electronic screen as a shielding area, it can block at least some targets such as human bodies and teaching aids, reducing the interference of foreground occlusion targets within the shielding area on blackboard extraction. This significantly reduces the computational power and time consumption of subsequent processing, improving the overall efficiency and accuracy of blackboard completion, and enhancing the matching degree of subsequent viewers' understanding of the teacher's lecture content and the blackboard content.

[0017] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1-a A schematic diagram of a blackboard according to this application is shown (the blackboard is a fixed blackboard containing an electronic screen).

[0020] Figure 1-b A schematic diagram of another blackboard according to this application is shown (the blackboard is a push-pull blackboard containing an electronic screen);

[0021] Figure 2 A flowchart illustrating a method for displaying whiteboard text according to an embodiment of this application is shown;

[0022] Figure 3 A schematic diagram illustrating the annotation of training samples for an object detection model according to an embodiment of this application is shown;

[0023] Figure 4 A schematic diagram illustrating the annotation of training samples for an image segmentation model according to an embodiment of this application is shown;

[0024] Figure 5-a A schematic diagram of a scenario according to this application is shown;

[0025] Figure 5-b It shows Figure 5-a The effect of the masked image;

[0026] Figure 6-a A schematic diagram of another scenario according to this application is shown;

[0027] Figure 6-b It shows Figure 6-a The effect of the masked image;

[0028] Figure 7 A schematic diagram illustrating the data flow of an image segmentation model processing data according to an embodiment of this application is shown;

[0029] Figure 8 A schematic diagram is shown illustrating the process of merging mask images of multiple targets to obtain a final mask image according to an embodiment of this application.

[0030] Figure 9 A schematic diagram of the structure of a whiteboard display device according to an embodiment of this application is shown;

[0031] Figure 10 A schematic diagram of the structure of a computer device according to an embodiment of this application is shown. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] With the development of multimedia technology, more and more intelligent interactive whiteboards are appearing in classrooms. Currently, the most common blackboard formats in classrooms include: the first type... Figure 1-a As shown in the image, there is a smart teaching electronic screen in the middle and a fixed blackboard on each side; the second type is shown in the attached image. Figure 1-b As shown, this is a four-panel side-by-side blackboard setup, with the two middle blackboards able to be pushed and pulled left and right, and a smart teaching electronic screen located beneath each blackboard.

[0034] In the classroom, teachers can choose to use an electronic screen to present courseware or write on the blackboard by hand, depending on the content of the lesson. However, when teachers choose to write by hand or stand in front of the blackboard to explain, they will inevitably obscure the content. This problem can be solved by using a smart blackboard extraction method to remove the obscured area.

[0035] Existing methods for detecting electronic screens often employ pre-calibrated areas, comparison of captured images with images recorded by smart teaching all-in-one machines, or object detection models. These methods require manual intervention and confirmation of device positions before each installation, lacking flexibility. Furthermore, in scenarios where the blackboard can be moved, the visible portion of the electronic screen often changes, leading to matching failures. This approach also requires significant computing power and inference time.

[0036] In response, this application proposes a method for displaying blackboard writing. Figure 2 This diagram illustrates a flowchart of a whiteboard display method according to an embodiment of this application. Figure 2 As can be seen, the embodiments of this application include steps S210 to S230:

[0037] Step S210: Based on the object detection model, the first video frame of the original blackboard video stream is segmented into blackboard areas, and the resulting blackboard area is used as the blackboard background image.

[0038] Using the first frame of the original blackboard video stream as input, an object detection model is used for inference to determine the location of the blackboard region in the first video frame. The image corresponding to the location of the blackboard region is cropped to obtain the blackboard region, which is then cached as the blackboard background image.

[0039] The object detection model can be trained based on labeled classroom scene data, including blackboard rectangles. This object detection model can obtain the background image of the writing on the blackboard in a timely and convenient manner.

[0040] In some embodiments, the blackboard area may refer to the entire blackboard area, including the physical blackboard portion and the electronic screen portion. Please refer to [reference needed]. Figure 1-a and Figure 1-b In both cases, the background image of the blackboard can be the entire blackboard, including the electronic screen. In subsequent processing, an image segmentation model can be used to detect the electronic screen and process it as a shielded area.

[0041] In other embodiments, the blackboard and electronic screen in the video frame can be directly segmented using an object detection model. Specifically, the step of performing blackboard recognition on the first video frame of the original blackboard video stream based on the object detection model includes: modifying the object detection model so that it can recognize the electronic screen; using the modified object detection model to perform category and position reasoning on the first video frame of the original blackboard video stream to obtain the overall blackboard area and the electronic screen area; and cropping the electronic screen area from the overall blackboard area to obtain the blackboard area of ​​the first video frame.

[0042] Alternatively, you can add the ability to identify the category and location of electronic screens to the object detection model. Please refer to [reference needed]. Figure 3 , Figure 3 The diagram illustrates the annotation of training samples for an object detection model according to an embodiment of this application. The original training samples of the object detection model only had annotations for blackboards. In this application, annotations for electronic screens are added to the original training samples. The parameters of the object detection model are then fine-tuned using training samples with added electronic screen annotations, enabling the model to recognize electronic screens. The regions identified as electronic screens by the object detection model are used as masked regions. The resulting blackboard image, after cropping the masked regions, is then processed in subsequent steps. In this case, there is no need to use an image segmentation model to detect electronic screens later; the result of this step can be directly used.

[0043] In some embodiments, the method further includes: performing the step of blackboard recognition on the first video frame of the original blackboard video stream based on the target detection model at a preset frequency; and updating the blackboard area of ​​the first video frame when it is determined that the position of the electronic screen area has changed.

[0044] In other words, a preset frequency can be set, such as the interval between one or several video frames. At this frequency, the target detection model is called to perform blackboard recognition on the first video frame of the original blackboard video stream. Once a change in the position of the electronic screen area is detected, the blackboard area in the first video frame is updated. This relatively frequent calling of the target detection model to perform blackboard recognition on the first video frame of the original blackboard video stream ensures that the position of the shielded area is updated in a timely manner, especially when the blackboard can be pushed and pulled, thus avoiding the loss of blackboard content.

[0045] In some embodiments, on devices with limited computing power, a lightweight object detection model, such as an object detection model based on the YOLO series, can be used.

[0046] In addition, in some embodiments, the background image of the whiteboard writing can be further enhanced to improve the clarity of the whiteboard writing image.

[0047] Step S220: Based on the modified image segmentation model, target recognition is performed on each video frame of the original whiteboard video stream to obtain the electronic screen and the foreground occlusion target; and the electronic screen is used as a shielding area to mask the foreground occlusion target to obtain the masking image of each video frame.

[0048] In some embodiments, the image segmentation model is modified according to the following method: annotating electronic screens on multiple training samples in an existing classroom scene dataset; training and fine-tuning the existing image segmentation model based on the annotated training samples so that the image segmentation model can recognize electronic screens in video frames, thereby obtaining the modified image segmentation model.

[0049] Specifically, the image segmentation model in this application is based on labeled classroom scene data. The labeled information includes the outline polygon points of objects (including people, teaching aids, etc.). The main function of this image segmentation model is to detect objects and output their masked images. Building upon this, this application adds the identification of an electronic screen category, whose annotation method is as follows: Figure 4 As shown, Figure 4 This illustration shows a schematic diagram of the annotation of data samples of an image segmentation model according to an embodiment of this application. Figure 4As can be seen, the human body annotations already exist. This application only needs to add the annotations for the electronic screens to the original annotations. Since the electronic screens are regular shapes (usually squares or rectangles), the workload of adding the annotations for the electronic screens is not large. After annotation, the modified image segmentation model used in this application can be obtained through simple training and parameter fine-tuning. In the image segmentation model, the location of the shielded area (electronic screen) can be obtained through the electronic screen category.

[0050] The advantage of using an image segmentation model is that the object mask image obtained by the image segmentation model can accurately obtain the region of the object (including people and teaching aids, etc.) for subsequent extraction and completion of intelligent whiteboard writing. The difficulty lies in the fact that the human body shape is varied and the corresponding outline polygon point annotation form is relatively complex, and the annotation time cost is high. However, since the shape of the newly added electronic screen region category in this application is relatively fixed, it is only necessary to annotate the target box of the electronic screen (usually a rectangle), so the annotation time cost is low.

[0051] After modifying and training the image segmentation model, the trained image segmentation model is used to perform target recognition on each video frame of the original blackboard video stream to obtain the foreground occlusion target and the electronic screen in each video frame. The foreground occlusion target refers to the target that obstructs the blackboard area, such as people, teaching aids, etc. These people or objects are usually in front of the blackboard, so they are referred to as foreground occlusion targets. Then, the electronic screen area is used as the shielding area to mask the foreground occlusion target. The so-called "masking" refers to the operation of ignoring the foreground occlusion target in the shielding area. After "masking", the image segmentation model outputs a mask image, which is usually a black and white binary image.

[0052] Please refer to Figure 5-a , Figure 5-b as well as Figure 6-a , Figure 6-b , Figure 5-a A schematic diagram of a scenario according to this application is shown. Figure 5-b It shows that according to Figure 5-a The effect of the masked image; Figure 6-a A schematic diagram of another scenario according to this application is shown. Figure 6-b It shows that according to Figure 6-a The effect of the masked image.

[0053] Figure 5-b and Figure 6-bThe masking image shown is a binary image. The image segmentation model performs target recognition on each video frame. Foreground occlusion targets include objects such as people and teaching aids. If the electronic screen was not detected in step S310, its category and location are also detected in this step; if the electronic screen has already been detected in step S310, it can be directly retrieved. After obtaining this information, the electronic screen is used as a masking area to cover the foreground occlusion targets such as people and teaching aids it covers. Please also refer to... Figure 5-a and Figure 5-b The entirety of human body (3) and parts of human body (1) and human body (2) are obscured, resulting in the following: Figure 5-b The masking image shown. Please also refer to... Figure 6-a and Figure 6-b Because of Figure 6-a The video area does not contain any foreground occluding target, therefore the resulting masking image is as follows: Figure 6-b As shown.

[0054] Specifically, in some embodiments, the image segmentation model includes: an instance inference module, a bounding box post-processing module, and a masking post-processing module; the step of performing target recognition on each video frame of the original whiteboard video stream based on the image segmentation model includes: determining the blackboard region of each video frame based on the position information of the whiteboard background image; for a video frame, taking the corresponding blackboard region as input, performing forward inference based on the instance inference module to obtain multiple segmentation sub-images and dense prediction information of at least one target, wherein the target includes at least an electronic screen; processing the dense prediction information based on the bounding box post-processing module to obtain the category information, position information, and segmentation harmonic parameters of each target; inputting the position information, segmentation harmonic parameters, and multiple segmentation sub-images of each target into the masking post-processing module for processing to obtain a masked image of each target.

[0055] Please refer to Figure 7 , Figure 7 This diagram illustrates the data flow of an image segmentation model processing data according to an embodiment of this application. Figure 7 As can be seen, the image segmentation model 700 includes an instance inference module 710, a rectangular box post-processing module 720, and a mask post-processing module 730. In some embodiments, the instance inference module 710 can be relatively lightweight, such as a module composed of the YOLO instance segmentation model.

[0056] After the target detection model identifies the blackboard region in the video frame, the coordinate position information is obtained. Based on this coordinate position information, the blackboard region in each video frame can be determined. The blackboard region in each video frame is used as the input of the image segmentation model 700. The instance inference module 710 performs forward inference to obtain a set of densely predicted object information. The object information here includes information on foreground occluded targets such as human bodies and teaching aids, as well as information on electronic screens. The obtained target information is recorded as dense prediction information. At the same time, the instance inference module 710 will segment each video frame into multiple segmentation sub-images for subsequent inference and prediction.

[0057] The dense prediction information of the targets is then input into the rectangular bounding box post-processing module 720. After processing by the rectangular bounding box post-processing module 720, the category, location information, and segmentation harmonic parameters of each target can be obtained. In some embodiments, the location information, segmentation harmonic parameters, and segmentation sub-images are each in a preset format, wherein the location information is the coordinate information of the minimum bounding rectangle of the target; the segmentation harmonic parameters are a first preset number of floating-point numbers; and the segmentation sub-images are a second preset number of floating-point images.

[0058] The location information of a target includes the coordinates of the object's smallest bounding rectangle; the segmentation harmonic parameter is 32 floating-point numbers; the segmentation sub-image is a 32-float image. Taking a 640*640 blackboard image as an input, the floating-point image size is 160*160, which is 1 / 4*1 / 4 of the input image size. The mask image is a binary image. In some embodiments, the size of the mask image is consistent with the size of the blackboard area in the video frame. A value of 0 in the image represents a non-object area (i.e., the black part in the image, which does not include targets such as human bodies and teaching aids, but the electronic screen is included in the non-object area as a shielding area), and a value of 1 represents an object area (i.e., the white part in the image, which includes targets other than the electronic screen).

[0059] To obtain the final masked image for each target, the target's location information, segmentation harmonic parameters, and segmentation sub-image need to be simultaneously input into the masking post-processing module 730 for calculation.

[0060] The masking post-processing module 730 can be implemented on a cloud processor or on the CPU. The computation time of the masking post-processing module 730 is significant and positively correlated with the number of objects. Therefore, within the steps executed by the masking post-processing module 730, filtering targets within the masked area can reduce the overall inference time to some extent. For example... Figure 5-a and Figure 5-b In the illustrated embodiment, the human body (3) is shielded by a shielding area, and the human body (3) does not participate in the subsequent processing at all, saving computing power and time.

[0061] Furthermore, since electronic screens are generally used for presenting courseware, they often contain cartoon characters, poster figures, etc. These human-like objects may be incorrectly detected by the model, that is, they may be mistakenly identified as foreground occlusion targets. In this application, the electronic screen is used as a shielding area to directly filter these objects, which reduces the interference with the subsequent blackboard writing extraction method and improves the accuracy of blackboard writing completion.

[0062] When processing data, the masking post-processing module 730 uses the electronic screen area as a masking region. For foreground occlusion targets, the masking region is used for masking. Specifically, for a foreground occlusion target, masking can be performed based on its relative position to the masking region. More specifically, using the electronic screen area as a masking region and masking foreground occlusion targets with the masking region includes: if the minimum bounding rectangle of a foreground occlusion target is completely within the masking region formed by the electronic screen, then the foreground occlusion target is filtered out; if the minimum bounding rectangle of a foreground occlusion target is partially within the masking region, then the minimum bounding rectangle is shrunk, retaining only the portion of the target outside the masking region; if the minimum bounding rectangle of a foreground occlusion target is completely outside the masking region, then the foreground occlusion target is retained.

[0063] like Figure 7 As shown, after passing through the rectangular frame post-processing module 720, the position coordinate information (x, y, x) of the electronic screen (i.e., the shielding area) can be obtained. 11 ,y 11 ,x 12 ,y 12 ) and the position coordinates (x) of the target object occluded by the foreground. 21 ,y 21 ,x 22 ,y 22 ). Where x 11 ,y 11 Let x and y represent the x-coordinate and y-coordinate of the top-left vertex of the smallest bounding rectangle of the shielded region, respectively. 12 ,y 12 Let x and y represent the x and y coordinates of the bottom right vertex of the smallest bounding rectangle of the shielded region, respectively. Similarly, x 21 ,y 21 Let x and y represent the x and y coordinates of the top-left vertex of the smallest bounding rectangle of any foreground occluded object, respectively; 22 ,y 22 Let x and y represent the x and y coordinates of the bottom right vertex of the smallest bounding rectangle of any foreground occluded target, respectively.

[0064] For targets that are completely within the shielded area, see attached Figure 5-aThe human body (3) shown is completely within the shielded area, that is, it satisfies the following positional relationship: and If the target is not selected, it will be directly filtered out and ignored, meaning that the target does not need to participate in the generation of the mask image.

[0065] For targets located within the shielded area, there are two scenarios: attached... Figure 5-a The examples shown are human body (1) and human body (2). For human body (1), that is, if a target satisfies the following relational position: and Then the position coordinates of the minimum bounding rectangle of the shrinking target are (x... 21 ,y 21 ,x 11 ,y 22 For the human body (2), that is, if a target satisfies the following relational position: and Then the position coordinates of the minimum bounding rectangle of the shrinking target are (x... 12 ,y 21 ,x 22 ,y 22 Then, the mask image is calculated for the shrunken rectangle, thus obtaining the mask image.

[0066] It should be noted that typically one mask image is generated for each target. Therefore, during the process of generating the final mask image, the mask image for each target needs to be saved, which consumes a significant amount of memory. For example, if the image size is w*h and the number of objects is n, then n*w*h bytes of data need to be saved. In intelligent whiteboard scenarios, image sizes are often quite large, and when the number of objects is also large, this can lead to substantial memory overhead. To address this issue, this application has made further optimizations. (See attached image.) Figure 8 As shown, in some embodiments of this application, if multiple targets exist in a video frame, the masking images corresponding to the multiple targets are merged to form the final masking image. Continuing from the previous point, a single w*h masking image is used to merge and save the masking results of all targets on the electronic screen, thereby saving (n-1) times the memory overhead. Please refer to... Figure 8 , Figure 8 This diagram illustrates a final mask image obtained by merging mask images of multiple targets according to an embodiment of this application. Merging the multiple targets yields a mask image containing the multiple targets. Figure 5-a Multiple targets in the image are merged to obtain the final mask image, as shown below. Figure 5-b As shown.

[0067] For objects completely outside the shielded area, such as Figure 6-aThe human figures (1) and (2) shown are preserved according to conventional methods. After calculating and merging the masked images, the final masked image is as follows: Figure 6-b As shown.

[0068] Step S230: Based on the masking images of each video frame, the whiteboard background image is fused and updated to obtain the completed whiteboard video stream.

[0069] After obtaining the masking image, the occluded areas of each video frame can be filled in based on the whiteboard background image and each video frame to obtain the unoccluded image of each video frame.

[0070] Specifically, in some embodiments, the process of fusing and updating the blackboard background image based on the masking images of each video frame to obtain a completed blackboard video stream includes: for a video frame, based on the masking image, dividing the blackboard area of ​​the video frame into an object area and a non-object area, wherein the non-object area includes a shielding area identified as formed by an electronic screen; for the non-object area, updating the blackboard background image based on the masking image and the blackboard area of ​​the video frame according to a preset linear combination coefficient; merging the updated non-object area with the object area of ​​the video frame to obtain the unobstructed blackboard image.

[0071] Taking a video frame as an example, for its blackboard area, the mask image is used to divide it into object area and non-object area. As mentioned earlier, the size of the mask image is the same as the size of the blackboard area in the video frame. The value of 0 in the image represents the non-object area (that is, the black part in the picture, which does not include the human body, teaching aids and other targets, but the electronic screen is included in the non-object area as a shielding area), and 1 represents the object area (that is, the white part in the picture, which includes the targets other than the electronic screen).

[0072] Then, for non-object regions, the blackboard background image is updated based on the mask image and the blackboard region of the video frame according to a preset linear combination coefficient. Specifically, the non-object regions can be updated according to the following formula (1):

[0073] Equation (1);

[0074] Where B' is the unobstructed whiteboard image, A is the blackboard region of the video frame, B is the masked image corresponding to A, and α is the linear combination coefficient; It is a background image of the blackboard without any objects obstructing it, obtained from the debinarized image of the masking image; It is a real-time whiteboard image without object occlusion, obtained from the debinarized image of the mask image.

[0075] For object regions, no special processing is needed; simply display the content recorded in the original video frame. Merging non-object regions with object regions will result in a complete, unobstructed whiteboard image. The specific merging can be performed based on the values ​​and corresponding positions of each pixel.

[0076] Depend on Figure 2 As shown in the method, this application provides a method for displaying whiteboard writing. This method modifies the image segmentation model, enabling it to recognize electronic screens. Without increasing computational power consumption, it can simultaneously identify foreground occlusion targets such as people and teaching aids, as well as the electronic screen. When processing the whiteboard writing video, a target detection model is first used to process the first frame of the video, separating the blackboard portion and caching it as the whiteboard writing background image. Then, based on the modified image segmentation model, foreground occlusion targets and the electronic screen are identified in each video frame of the original whiteboard writing video stream. Foreground occlusion targets include objects such as people and teaching aids that obstruct the whiteboard writing. The electronic screen area is set as the masking area. After masking the foreground occlusion targets using the masking area, a masked image for each video frame is obtained. Furthermore, based on the masked images of each video frame, the aforementioned whiteboard writing background image is fused and updated to obtain a complete, unobstructed whiteboard writing video stream. This application modifies the image segmentation model to simultaneously identify foreground occlusion targets and electronic screens, enabling rapid segmentation of the electronic screen region and timely and flexible acquisition of occluded images. Furthermore, by using the electronic screen as a shielding area, it can block at least some targets such as human bodies and teaching aids, reducing the interference of foreground occlusion targets within the shielding area on the blackboard extraction method. This significantly reduces the computational power and time consumption of subsequent processing, improving the overall efficiency and accuracy of blackboard completion, and enhancing the matching degree of subsequent viewers' understanding of the teacher's lecture content and the blackboard content.

[0077] Figure 9 A schematic diagram of a whiteboard display device according to an embodiment of this application is shown. Figure 9 It can be seen that the blackboard display device 900 includes:

[0078] The blackboard segmentation unit 910 is used to perform blackboard recognition on the first video frame of the original blackboard video stream based on the target detection model, and use the obtained blackboard area as the blackboard background image.

[0079] The mask detection unit 920 is used to perform target recognition on each video frame of the original whiteboard video stream based on the modified image segmentation model to obtain the electronic screen and the foreground occlusion target; and to use the electronic screen as a shielding area to mask the foreground occlusion target to obtain the mask image of each video frame.

[0080] The occlusion completion unit 930 is used to fuse and update the whiteboard background image based on the masking images of each video frame to obtain a completed whiteboard video stream.

[0081] In some embodiments of this application, the whiteboard display device further includes: a training unit, used to annotate the electronic screen on multiple training samples in an existing classroom scene dataset; and to train and fine-tune an existing image segmentation model based on the annotated multiple training samples, so that the image segmentation model can recognize the electronic screen in the video frame, thereby obtaining a modified image segmentation model.

[0082] In some embodiments of this application, the blackboard segmentation unit 910 is used to modify the target detection model so that the target detection model can recognize the electronic screen; the modified target detection model is used to perform category and position reasoning on the first video frame of the original blackboard video stream to obtain the overall blackboard area and the electronic screen area; the electronic screen area is cropped from the overall blackboard area to obtain the blackboard area of ​​the first video frame.

[0083] In some embodiments of this application, the mask detection unit 920 is further configured to perform the step of performing blackboard recognition on the first video frame of the original blackboard video stream based on the target detection model at a preset frequency; and to update the blackboard area of ​​the first video frame when it is determined that the position of the electronic screen area has changed.

[0084] In some embodiments of this application, the image segmentation model includes: an instance inference module, a bounding box post-processing module, and a mask post-processing module; the mask detection unit 920 is further configured to determine the blackboard region of each video frame based on the position information of the blackboard background image; taking the blackboard region of each video frame as input, performing forward inference based on the instance inference module to obtain multiple segmentation sub-images and dense information of at least one target, the target including at least an electronic screen; processing the dense information based on the bounding box post-processing module to obtain prediction information for each target, the prediction information including: category information, position information, and segmentation harmonic parameters; inputting the position information of each target, the segmentation harmonic parameters, and the multiple segmentation sub-images into the mask post-processing module for processing to obtain the minimum bounding box of each target.

[0085] In some embodiments of this application, the mask detection unit 920 is configured to: filter out the foreground occlusion target if the minimum bounding rectangle of a foreground occlusion target is completely within the occlusion area formed by the electronic screen; shrink the minimum bounding rectangle of a foreground occlusion target if a portion of the minimum bounding rectangle of a foreground occlusion target is within the occlusion area to retain the portion of the foreground occlusion target outside the occlusion area; and retain the foreground occlusion target if the minimum bounding rectangle of a foreground occlusion target is completely outside the occlusion area.

[0086] In some embodiments of this application, the location information is the coordinate information of the minimum bounding rectangle of the target; the segmentation harmonic parameter is a first preset number of floating-point numbers; and the segmentation subgraph is a second preset number of floating-point graphs.

[0087] In some embodiments of this application, the mask detection unit 920 is used to merge the mask images corresponding to the multiple foreground occlusion targets in a video frame to form a final mask image if there are multiple foreground occlusion targets.

[0088] In some embodiments of this application, the occlusion completion unit 930 is used to, for a video frame, divide the blackboard area of ​​the video frame into an object area and a non-object area based on the masking image, wherein the non-object area includes a shielding area identified as formed by an electronic screen; for the non-object area, update the blackboard background image based on the masking image and the blackboard area of ​​the video frame according to a preset linear combination coefficient: merge the updated non-object area with the object area of ​​the video frame to obtain the unoccluded blackboard image.

[0089] In some embodiments of this application, the occlusion completion unit 930 is used to update the non-object region according to the following formula:

[0090] ;

[0091] Where B' is the unobstructed blackboard background image, A is the blackboard area, B is the obstructed image corresponding to A, and α is the linear combination coefficient; It is a background image of the blackboard without any objects obstructing it, obtained from the debinarized image of the masking image; It is a real-time whiteboard image without object occlusion, obtained from the debinarized image of the mask image.

[0092] In some embodiments of this application, the blackboard segmentation unit 910 is also used to perform image enhancement processing on the blackboard background image.

[0093] In some embodiments of this application, the target detection model is constructed using a YOLO convolutional neural network, and the image segmentation model is constructed based on the YOLO instance segmentation model.

[0094] It should be noted that the aforementioned display devices for blackboard writing can implement the aforementioned blackboard writing display methods one by one, and will not be described in detail here.

[0095] Figure 10 This application shows a schematic diagram of the structure of a computer device according to one embodiment of the present application. Figure 10 As shown, the internal structure of this computer device may include a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of the whiteboard display method.

[0096] In one embodiment, the computer device provided in this application includes a memory and a processor. The memory stores a database and a computer program that can run on the processor. When the processor executes the computer program, it implements the steps of the whiteboard display method of this application.

[0097] In one embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the whiteboard display method of this application.

[0098] It should be noted that the functions or steps that can be implemented by the computer device or computer-readable storage medium described above can be referred to the relevant descriptions in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0099] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0100] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0101] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for displaying blackboard writing, characterized in that, include: Based on the object detection model, blackboard recognition is performed on the first video frame of the original blackboard video stream, and the obtained blackboard area is used as the blackboard background image. Based on the modified image segmentation model, target recognition is performed on each video frame of the original whiteboard video stream to obtain the electronic screen and the foreground occlusion target; and the electronic screen is used as a shielding area to mask the foreground occlusion target to obtain the masking image of each video frame. Based on the masking images of each video frame, the whiteboard background image is fused and updated to obtain the completed whiteboard video stream; The image segmentation model includes: an instance inference module, a rectangular box post-processing module, and a mask post-processing module; The modified image segmentation model performs target recognition on each video frame of the original whiteboard video stream, including: Based on the position information of the background image of the blackboard writing, the blackboard area of ​​each video frame is determined; Using the blackboard area of ​​each video frame as input, forward reasoning is performed based on the instance reasoning module to obtain multiple segmented sub-images and dense information of at least one target, wherein the target includes at least an electronic screen. The dense information is processed by the rectangular box post-processing module to obtain the prediction information of each target. The prediction information includes: category information, location information, and segmentation harmonic parameters. The location information, segmentation harmonic parameters, and multiple segmented sub-images of each target are input into the masking post-processing module for processing to obtain the minimum bounding rectangle of each target. The step of using the electronic screen as a shielding area to cover the foreground occlusion target includes: If the smallest bounding rectangle of a foreground occluder is completely within the occlusion area formed by the electronic screen, then the foreground occluder is filtered out. If the minimum bounding rectangle of a foreground occluded target is partially within the occlusion area, then the minimum bounding rectangle is shrunk to preserve the portion of the foreground occluded target outside the occlusion area. If the smallest bounding rectangle of a foreground occluded target is completely outside the occlusion area, then the foreground occluded target is retained.

2. The method according to claim 1, characterized in that, The method further includes: Label the electronic screens on multiple training samples in the existing classroom scene dataset; The existing image segmentation model is trained and fine-tuned based on multiple labeled training samples so that the image segmentation model can recognize electronic screens in video frames, resulting in a modified image segmentation model.

3. The method according to claim 1, characterized in that, The method of performing blackboard recognition on the first video frame of the original blackboard video stream based on the target detection model includes: The target detection model is modified to enable it to recognize electronic screens; The modified target detection model is used to perform category and position reasoning on the first video frame of the original blackboard video stream to obtain the overall blackboard area and the electronic screen area; The electronic screen area is cropped from the entire blackboard area to obtain the blackboard area of ​​the first video frame.

4. The method according to claim 3, characterized in that, The method further includes: The step of performing blackboard recognition on the first video frame of the original blackboard video stream based on the target detection model is executed at a preset frequency. When it is determined that the position of the electronic screen area has changed, the blackboard area of ​​the first video frame is updated.

5. The method according to claim 1, characterized in that, The location information is the coordinate information of the minimum bounding rectangle of the target; The segmentation harmonic parameter is a first preset number of floating-point numbers; The segmented subgraphs are a second preset number of floating-point graphs.

6. The method according to claim 1 or 5, characterized in that, If there are multiple foreground occlusion targets in a video frame, the mask images corresponding to the multiple foreground occlusion targets are merged to form the final mask image.

7. The method according to claim 1, characterized in that, The masking images based on each video frame are used to fuse and update the whiteboard background image to obtain the completed whiteboard video stream, including: For a video frame, based on the masking image, the blackboard area of ​​the video frame is divided into an object area and a non-object area, wherein the non-object area includes the shielding area identified as formed by the electronic screen. For non-object areas, the blackboard background image is updated based on the masking image and the blackboard area of ​​the video frame according to a preset linear combination coefficient: The updated non-object region is merged with the object region of the video frame to obtain an unobstructed whiteboard image.

8. The method according to claim 7, characterized in that, For non-object regions, the blackboard background image is updated based on the mask image and the blackboard region of the video frame according to a preset linear combination coefficient, including: The non-object region is updated according to the following formula: ； Where B' is the unobstructed blackboard background image, A is the blackboard area, B is the obstructed image corresponding to A, and α is the linear combination coefficient; It is a background image of the blackboard without any objects obstructing it, obtained from the debinarized image of the masking image; It is a real-time whiteboard image without object occlusion, obtained from the debinarized image of the mask image.

9. The method according to any one of claims 1 to 5 or claims 7 to 8, characterized in that, The method further includes: performing image enhancement processing on the blackboard background image.

10. The method according to any one of claims 1-5 or claims 7-8, characterized in that, The target detection model is constructed using a YOLO convolutional neural network, and the image segmentation model is constructed based on the YOLO instance segmentation model.

11. A display device for blackboard writing, characterized in that, The device includes: The blackboard segmentation unit is used to perform blackboard recognition on the first video frame of the original blackboard video stream based on the target detection model, and use the obtained blackboard area as the blackboard background image. The masking detection unit is used to perform target recognition on each video frame of the original whiteboard video stream based on the modified image segmentation model to obtain the electronic screen and the foreground occlusion target; and to use the electronic screen as a shielding area to mask the foreground occlusion target to obtain the masking image of each video frame. The occlusion completion unit is used to fuse and update the whiteboard background image based on the masking images of each video frame to obtain the completed whiteboard video stream. The image segmentation model includes: an instance inference module, a rectangular box post-processing module, and a mask post-processing module; The modified image segmentation model performs target recognition on each video frame of the original whiteboard video stream, including: Based on the position information of the background image of the blackboard writing, the blackboard area of ​​each video frame is determined; Using the blackboard area of ​​each video frame as input, forward reasoning is performed based on the instance reasoning module to obtain multiple segmented sub-images and dense information of at least one target, wherein the target includes at least an electronic screen. The dense information is processed by the rectangular box post-processing module to obtain the prediction information of each target. The prediction information includes: category information, location information, and segmentation harmonic parameters. The location information, segmentation harmonic parameters, and multiple segmented sub-images of each target are input into the masking post-processing module for processing to obtain the minimum bounding rectangle of each target. The step of using the electronic screen as a shielding area to cover the foreground occlusion target includes: If the smallest bounding rectangle of a foreground occluder is completely within the occlusion area formed by the electronic screen, then the foreground occluder is filtered out. If the minimum bounding rectangle of a foreground occluded target is partially within the occlusion area, then the minimum bounding rectangle is shrunk to preserve the portion of the foreground occluded target outside the occlusion area. If the smallest bounding rectangle of a foreground occluded target is completely outside the occlusion area, then the foreground occluded target is retained.

12. An intelligent whiteboard writing extraction system, characterized in that, The intelligent whiteboard extraction system is equipped with the whiteboard display device as described in claim 11.

13. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the whiteboard display method according to any one of claims 1 to 10.

14. A computer device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the program, it implements the whiteboard display method according to any one of claims 1 to 10.

Images