A method and device for automatically generating blackboard content

Abstract

The invention discloses a method and device for automatically generating blackboard writing content. The method comprises the following steps: obtaining a blackboard area image of each frame image in a teaching video, using a scale-invariant feature transformation technology to calculate the number of matching points between blackboard area images at one moment and the next moment in time sequence, and grouping the blackboard area images according to the number of matching points; using a scale-invariant feature descriptor to calculate the matching points between any two blackboard area images in each group of blackboard area images, building a target optimization function based on the matching points, and solving the mapping relationship between each pixel in the blackboard area image by using a Gauss-Newton iteration method; obtaining pixel points containing text in each group of blackboard area images, obtaining a text area by using a region growing algorithm based on the pixel points, calculating a pixel change area of ​​the text area based on an initial image, cutting and filling the pixel change area into the initial image by using a mapping relationship, and outputting the final blackboard writing content.

Description

Technical Field

[0001] The present invention relates to the field of video text content generation, and in particular to a method and device for automatically generating blackboard writing content. Background Art

[0002] Extracting the blackboard content in teaching videos is the key to understanding the content of teaching videos. When the teacher is teaching on the blackboard, the complete blackboard content can be extracted from the teaching video as a summary image of the video content.

[0003] At present, the generation of teaching video content summaries is mainly through video key frame technology. The advantage of using key frame technology is that the method is simple and the implementation speed is fast. The disadvantage is that when analyzing and grasping the content of the teaching video, the teacher may block the blackboard.

[0004] In view of this, it is very meaningful to establish a method and device for automatically generating blackboard content. Summary of the invention

[0005] In view of the above-mentioned problems in the prior art that it is difficult to automatically generate complete blackboard content as a video summary based on a teaching video, the embodiment of the present application aims to propose a method and device for automatically generating blackboard content to solve the technical problems mentioned in the above background technology section.

[0006] In a first aspect, an embodiment of the present application provides a method for automatically generating blackboard content, comprising the following steps:

[0007] The blackboard area grouping step is to obtain the blackboard area image of each frame in the teaching video, use the scale-invariant feature transformation technology to calculate the number of matching points between the blackboard area image at one moment and the next moment in chronological order, and group the blackboard area images according to the number of matching points;

[0008] A matching point mapping step, using a scale-invariant feature descriptor to calculate the matching points between any two images in each group of blackboard area images, constructing a target optimization function based on the matching points, and using a Gauss-Newton iteration method to solve the pairwise mapping relationship between each pixel in the blackboard area image; and

[0009] The step of outputting the blackboard content is to obtain the pixel points containing text in each group of blackboard area images, and obtain the text area through the region growing algorithm based on the pixel points, select the first image in each group of blackboard area images as the initial image of the output result, calculate the pixel change area of ​​the text area based on the initial image, cut and fill the pixel change area into the initial image through the mapping relationship, repeat the above steps to traverse the blackboard area images in the same group, and output the final blackboard content.

[0010] In some embodiments, the step of obtaining a blackboard area image of each frame of the teaching video in the blackboard area grouping step specifically includes the following steps:

[0011] Establish and train a deep learning target detection model, and detect the blackboard area and human body area in each frame image;

[0012] The values ​​of all pixels in the human body area in each frame image are set to 0, and the blackboard area in each frame image is cut and arranged in chronological order to form a blackboard area image.

[0013] In some embodiments, the deep learning object detection model includes YOLO v3.

[0014] In some embodiments, if the number of matching points in the blackboard area grouping step is less than a set threshold, all blackboard area images at a time and before a time are classified into one group.

[0015] In some embodiments, the threshold is set to 20% of the blackboard area image width.

[0016] In some embodiments, the objective optimization function is constructed using a nonlinear least squares method, and the formula is as follows:

[0017]

[0018]

[0019]

[0020]

[0021] Where I represents the grayscale value of the image, θ is the affine transformation matrix, θ = (a, b, c, d, e, f), a, b, c, d, e, f are the six parameters of the affine transformation, p i and p j are respectively the pixel points of the blackboard area image.

[0022] In some embodiments, the step of outputting the blackboard content specifically includes:

[0023] The images in each group of blackboard area images are binarized, where the binarized value 1 represents the pixel point of the text;

[0024] Taking the pixel point in the blackboard area image as the initial point, perform m dilation operations to obtain text area A and text area B respectively, where the initial image corresponds to text area A, and any other blackboard area image except the initial image corresponds to text area B;

[0025] A set difference operation is performed on the text area B and the text area A to obtain the pixel change area C.

[0026] In a second aspect, an embodiment of the present application further provides a device for automatically generating blackboard content, comprising:

[0027] The blackboard area grouping module is configured to obtain the blackboard area image of each frame in the teaching video, use the scale-invariant feature transformation technology to calculate the number of matching points between the blackboard area image at one moment and the next moment in time sequence, and group the blackboard area images according to the number of matching points;

[0028] A matching point mapping module is configured to calculate the matching points between any two images in each group of blackboard area images using a scale-invariant feature descriptor, construct a target optimization function based on the matching points, and solve the pairwise mapping relationship between each pixel in the blackboard area image using a Gauss-Newton iteration method; and

[0029] The blackboard content output module is configured to obtain the pixel points containing text in each group of blackboard area images, and obtain the text area through the region growing algorithm based on the pixel points, select the first image in each group of blackboard area images as the initial image of the output result, calculate the pixel change area of ​​the text area based on the initial image, cut and fill the pixel change area into the initial image through the mapping relationship, repeat the above steps to traverse the blackboard area images in the same group, and output the final blackboard content.

[0030] In a third aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement a method as described in any implementation method in the first aspect.

[0031] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0032] The present invention proposes a method for automatically generating blackboard writing content, which comprises the following steps: obtaining a blackboard region image of each frame image in a teaching video, using a scale-invariant feature transformation technology to sequentially calculate the number of matching points between a blackboard region image at a moment and the blackboard region image at the next moment in chronological order, and grouping the blackboard region images according to the number of matching points; using a scale-invariant feature descriptor to calculate the matching points between any two blackboard region images in each group of blackboard region images, constructing a target optimization function based on the matching points, and using a Gauss-Newton iteration method to solve the mapping relationship between each pixel in the blackboard region image; obtaining pixel points containing text in each group of blackboard region images, and obtaining a text region based on the pixel points by using a region growth algorithm, selecting the first image in each group of blackboard region images as the initial image of the output result, calculating the pixel change region of the text region based on the initial image, cutting and filling the pixel change region into the initial image by using a mapping relationship, repeating the above steps to traverse the blackboard region images in the same group, and outputting the final blackboard writing content. Therefore, the text in the blackboard region of each frame image of the teaching video can be fused, and the complete blackboard writing information in the teaching video can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 is an exemplary device architecture diagram to which an embodiment of the present application may be applied;

[0035] Figure 2 A schematic diagram of a process for automatically generating blackboard writing content according to an embodiment of the present invention;

[0036] Figure 3 It is a flow chart of obtaining a blackboard area image of each frame of a teaching video in step S1 of the method for automatically generating blackboard writing content in an embodiment of the present invention;

[0037] Figure 4 A schematic flow chart of step S3 of the method for automatically generating blackboard writing content according to an embodiment of the present invention;

[0038] Figure 5 A schematic diagram of a device for automatically generating blackboard writing content according to an embodiment of the present invention;

[0039] Figure 6 It is a schematic diagram of the structure of a computer device suitable for implementing the electronic device of the embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] Figure 1 An exemplary device architecture 100 is shown to which the method or device for automatically generating blackboard content according to the embodiment of the present application can be applied.

[0042] like Figure 1 As shown, the device architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0043] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various applications, such as data processing applications, file processing applications, etc., can be installed on terminal devices 101, 102, 103.

[0044] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or they can be implemented as a single software or software module. No specific limitation is made here.

[0045] The server 105 may be a server that provides various services, such as a background data processing server that processes files or data uploaded by the terminal devices 101, 102, and 103. The background data processing server may process the acquired files or data and generate processing results.

[0046] It should be noted that the method for automatically generating blackboard content provided in the embodiment of the present application can be executed by the server 105 or by the terminal devices 101, 102, 103. Accordingly, the device for automatically generating blackboard content can be set in the server 105 or in the terminal devices 101, 102, 103.

[0047] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided as required. In the case where the processed data does not need to be obtained remotely, the above device architecture may not include a network, but only a server or a terminal device.

[0048] Figure 2 The embodiment of the present application discloses a method for automatically generating blackboard content, comprising the following steps:

[0049] Step S1, obtain the blackboard area image of each frame in the teaching video, use the scale-invariant feature transformation technology to calculate the number of matching points between the blackboard area image at one moment and the next moment in chronological order, and group the blackboard area images according to the number of matching points.

[0050] In a specific embodiment, Figure 3 As shown, obtaining the blackboard area image of each frame image in the teaching video in step S1 specifically includes the following steps:

[0051] S11, establish a deep learning target detection model and perform training, and detect the blackboard area and human body area in each frame image;

[0052] S12, setting the values ​​of all pixels in the human body area in each frame image to 0, cutting the blackboard area in each frame image, and arranging them in chronological order to form a blackboard area image.

[0053] In a specific embodiment, the steps of training the deep learning target detection model are as follows:

[0054] Sample collection: Collect a large number of pictures containing blackboards or human bodies, and mark the rectangular areas of the blackboard and the human body in each picture.<x1,y1,x2,y2,label> Indicates that<x1,y1> Indicates the coordinates of the upper left corner,<x2,y2> Indicates the coordinates of the lower right corner, label=1 represents a blackboard, and label=2 represents a human body.

[0055] Model training: Since the rectangular frames of the blackboard and the human body are relatively large, a one-stage deep learning target detection is used here. The size of objects in the teaching video scene is relatively fixed, and the number of anchors can be set to 1 during the algorithm parameter setting process.

[0056] In a preferred embodiment, the deep learning object detection model includes YOLO v3.

[0057] For each frame of the video, use the trained deep learning target detection model to output the upper left corner coordinates and lower right corner coordinates of the blackboard area in each frame. If each frame contains a human body, the upper left corner coordinates and lower right corner coordinates of the human body area are also output. Set the values ​​of all pixels in the human body rectangular area detected in each frame to 0. Cut out the corresponding area of ​​the blackboard area detected in each frame and arrange them in chronological order, denoted as black_board_region_i, i=1, 2, ..., n.

[0058] In a specific embodiment, starting from i=1, the scale-invariant feature transformation technology is used to calculate the number of matching points between black_board_region_i and black_board_region_j; if the number of matching points between two adjacent images is less than a set threshold (the set threshold can be set to 20% of the width of the rectangular region), black_board_region_1, ..., black_board_region_i are classified into one group; then starting from i+1, the above process is repeated. After the blackboard region images are grouped, the content of each blackboard writing can be distinguished, and the blackboard region images corresponding to different blackboard writing contents form a group, which is convenient for the subsequent automatic generation of the blackboard writing content of each group of blackboard region images.

[0059] Step S2, using a scale-invariant feature descriptor to calculate the matching points between any two images in each group of blackboard area images, constructing a target optimization function based on the matching points, and using the Gauss-Newton iteration method to solve the pairwise mapping relationship between each pixel in the blackboard area image.

[0060] In a specific embodiment, the target optimization function is constructed using the nonlinear least squares method, and the formula is as follows:

[0061]

[0062]

[0063]

[0064]

[0065] Where I represents the grayscale value of the image, θ is the affine transformation matrix, θ = (a, b, c, d, e, f), a, b, c, d, e, f are the six parameters of the affine transformation, p i and p j are respectively the pixel points of the blackboard area image.

[0066] Among them, the Gauss-Newton iteration method is an iterative method for finding the least squares of regression parameters in a nonlinear regression model. It mainly uses the Taylor series expansion to approximately replace the nonlinear regression model, and then through multiple iterations, the regression coefficient is corrected multiple times, so that the regression coefficient continuously passes through the optimal regression coefficient of the nonlinear regression model, and finally the residual square sum of the original model is minimized. Establishing a pairwise mapping relationship between each pixel in each group of blackboard area images is conducive to content cutting and fusion.

[0067] Step S3, obtain the pixel points containing text in each group of blackboard area images, and obtain the text area through the region growing algorithm based on the pixel points, select the first image in each group of blackboard area images as the initial image of the output result, calculate the pixel change area of ​​the text area based on the initial image, cut and fill the pixel change area into the initial image through the mapping relationship, repeat the above steps to traverse the blackboard area images in the same group, and output the final blackboard content.

[0068] In a specific embodiment, Figure 4 As shown, step S3 specifically includes:

[0069] S31, performing binarization processing on each group of images in the blackboard area, wherein a binarization value of 1 represents a pixel point of text;

[0070] S32, taking a pixel point in the blackboard area image as an initial point, performing m dilation operations to obtain a text area A and a text area B respectively, wherein the initial image corresponds to the text area A, and any other blackboard area image except the initial image corresponds to the text area B; and

[0071] S33, performing a set difference operation on the text area B and the text area A to obtain the pixel change area C.

[0072] Further references Figure 5 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a device for automatically generating blackboard content, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0073] An automatic blackboard writing content generation device proposed in an embodiment of the present application includes:

[0074] The blackboard area grouping module 1 is configured to obtain the blackboard area image of each frame in the teaching video, use the scale-invariant feature transformation technology to calculate the number of matching points between the blackboard area image at one moment and the next moment in chronological order, and group the blackboard area images according to the number of matching points.

[0075] In a specific embodiment, the blackboard area grouping module 1 obtains the blackboard area image of each frame of the teaching video, specifically including:

[0076] Establish and train a deep learning target detection model, and detect the blackboard area and human body area in each frame image;

[0077] The values ​​of all pixels in the human body area in each frame image are set to 0, and the blackboard area in each frame image is cut and arranged in chronological order to form a blackboard area image.

[0078] In a specific embodiment, the deep learning target detection model is trained in the following manner:

[0079] Sample collection: Collect a large number of pictures containing blackboards or human bodies, and mark the rectangular areas of the blackboard and the human body in each picture.<x1,y1,x2,y2,label> Indicates that<x1,y1> Indicates the coordinates of the upper left corner,<x2,y2> Indicates the coordinates of the lower right corner, label=1 represents a blackboard, and label=2 represents a human body.

[0080] Model training: Since the rectangular frames of the blackboard and the human body are relatively large, a one-stage deep learning target detection is used here. The size of objects in the teaching video scene is relatively fixed, and the number of anchors can be set to 1 during the algorithm parameter setting process.

[0081] In a preferred embodiment, the deep learning object detection model includes YOLO v3.

[0082] For each frame of the video, use the trained deep learning target detection model to output the upper left corner coordinates and lower right corner coordinates of the blackboard area in each frame. If each frame contains a human body, the upper left corner coordinates and lower right corner coordinates of the human body area are also output. Set the values ​​of all pixels in the human body rectangular area detected in each frame to 0. Cut out the corresponding area of ​​the blackboard area detected in each frame and arrange them in chronological order, denoted as black_board_region_i, i=1, 2, ..., n.

[0083] In a specific embodiment, starting from i=1, the scale-invariant feature transformation technology is used to calculate the number of matching points between black_board_region_i and black_board_region_j; if the number of matching points between two adjacent images is less than a set threshold (the set threshold can be set to 20% of the width of the rectangular region), black_board_region_1, ..., black_board_region_i are classified into one group; then starting from i+1, the above process is repeated. After the blackboard region images are grouped, the content of each blackboard writing can be distinguished, and the blackboard region images corresponding to different blackboard writing contents form a group, which is convenient for the subsequent automatic generation of the blackboard writing content of each group of blackboard region images.

[0084] The matching point mapping module 2 is configured to use a scale-invariant feature descriptor to calculate the matching points between any two images in each group of blackboard area images, construct a target optimization function based on the matching points, and use the Gauss-Newton iteration method to solve the pairwise mapping relationship between each pixel in the blackboard area image.

[0085] In a specific embodiment, the target optimization function is constructed using the nonlinear least squares method, and the formula is as follows:

[0086]

[0087]

[0088]

[0089]

[0090] Where I represents the grayscale value of the image, θ is the affine transformation matrix, θ = (a, b, c, d, e, f), a, b, c, d, e, f are the six parameters of the affine transformation, p i and p j are respectively the pixel points of the blackboard area image.

[0091] Among them, the Gauss-Newton iteration method is an iterative method for finding the least squares of regression parameters in a nonlinear regression model. It mainly uses the Taylor series expansion to approximately replace the nonlinear regression model, and then through multiple iterations, the regression coefficient is corrected multiple times, so that the regression coefficient continuously passes through the optimal regression coefficient of the nonlinear regression model, and finally the residual square sum of the original model is minimized. Establishing a pairwise mapping relationship between each pixel in each group of blackboard area images is conducive to content cutting and fusion.

[0092] The blackboard content output module 3 is configured to obtain the pixel points containing text in each group of blackboard area images, and obtain the text area through the region growing algorithm based on the pixel points, select the first image in each group of blackboard area images as the initial image of the output result, calculate the pixel change area of ​​the text area based on the initial image, cut and fill the pixel change area into the initial image through the mapping relationship, repeat the above steps to traverse the blackboard area images in the same group, and output the final blackboard content.

[0093] In a specific embodiment, the blackboard content output module 3 specifically includes:

[0094] The images in each group of blackboard area images are binarized, where the binarized value 1 represents the pixel point of the text;

[0095] Taking the pixel point in the blackboard area image as the initial point, performing m dilation operations to obtain text area A and text area B respectively, wherein the initial image corresponds to the text area A, and any other blackboard area image except the initial image corresponds to the text area B; and

[0096] A set difference operation is performed on the text area B and the text area A to obtain the pixel change area C.

[0097] The present invention proposes a method for automatically generating blackboard writing content, which comprises the following steps: obtaining a blackboard region image of each frame image in a teaching video, using a scale-invariant feature transformation technology to sequentially calculate the number of matching points between a blackboard region image at a moment and the blackboard region image at the next moment in chronological order, and grouping the blackboard region images according to the number of matching points; using a scale-invariant feature descriptor to calculate the matching points between any two blackboard region images in each group of blackboard region images, constructing a target optimization function based on the matching points, and using a Gauss-Newton iteration method to solve the mapping relationship between each pixel in the blackboard region image; obtaining pixel points containing text in each group of blackboard region images, and obtaining a text region based on the pixel points by using a region growth algorithm, selecting the first image in each group of blackboard region images as the initial image of the output result, calculating the pixel change region of the text region based on the initial image, cutting and filling the pixel change region into the initial image by using a mapping relationship, repeating the above steps to traverse the blackboard region images in the same group, and outputting the final blackboard writing content. Therefore, the text in the blackboard region of each frame image of the teaching video can be fused, and the complete blackboard writing information in the teaching video can be obtained.

[0098] Reference below Figure 6 , which shows an electronic device (eg Figure 1 A schematic diagram of the structure of a computer device 600 (a server or terminal device as shown). Figure 6The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0099] like Figure 6 As shown, the computer device 600 includes a central processing unit (CPU) 601 and a graphics processing unit (GPU) 602, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 603 or the program loaded from the storage part 609 to the random access memory (RAM) 604. In the RAM 604, various programs and data required for the operation of the device 600 are also stored. The CPU 601, GPU 602, ROM 603 and RAM 604 are connected to each other through a bus 605. An input / output (I / O) interface 606 is also connected to the bus 605.

[0100] The following components are connected to the I / O interface 606: an input section 607 including a keyboard, a mouse, etc.; an output section 608 including a display such as a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 609 including a hard disk, etc.; and a communication section 610 including a network interface card such as a LAN card, a modem, etc. The communication section 610 performs communication processing via a network such as the Internet. A drive 611 may also be connected to the I / O interface 606 as needed. A removable medium 612, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 611 as needed so that a computer program read therefrom is installed into the storage section 609 as needed.

[0101] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part 610, and / or installed from a removable medium 612. When the computer program is executed by a central processing unit (CPU) 601 and a graphics processing unit (GPU) 602, the above-mentioned functions defined in the method of the present application are executed.

[0102] It should be noted that the computer-readable medium described in the present application may be a computer-readable signal medium or a computer-readable medium or any combination of the above two. The computer-readable medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device, device or component, or any combination of the above. More specific examples of computer-readable media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution device, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable medium that can send, propagate or transmit a program for use by or in conjunction with an instruction execution device, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0103] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0104] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the device, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based device that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0105] The modules involved in the embodiments described in this application may be implemented by software or hardware, and the modules described may also be set in a processor.

[0106] As another aspect, the present application further provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains the blackboard area image of each frame in the teaching video, uses the scale-invariant feature transformation technology to calculate the number of matching points between the blackboard area image at one moment and the next moment in chronological order, and groups the blackboard area images according to the number of matching points; uses the scale-invariant feature descriptor to calculate the matching points between any two images in each group of blackboard area images, constructs a target optimization function based on the matching points, and uses the Gauss-Newton iteration method to solve the pairwise mapping relationship between each pixel in the blackboard area image; obtains the pixel points containing text in each group of blackboard area images, and obtains the text area through the region growing algorithm based on the pixel points, selects the first image in each group of blackboard area images as the initial image of the output result, calculates the pixel change area of ​​the text area based on the initial image, cuts and fills the pixel change area into the initial image through the mapping relationship, repeats the above steps to traverse the blackboard area images in the same group, and outputs the final blackboard content.

[0107] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A method for automatically generating blackboard content, characterized in that: The following steps are involved: The blackboard area grouping step is to obtain the blackboard area image of each frame in the teaching video, use the scale-invariant feature transformation technology to calculate the number of matching points between the blackboard area image at one moment and the next moment in chronological order, and group the blackboard area images according to the number of matching points; A matching point mapping step, for any two images in each group of the blackboard area images, a scale-invariant feature descriptor is used to calculate the matching points between the blackboard area images, a nonlinear least squares method is used to construct a target optimization function based on the matching points, and a Gauss-Newton iteration method is used to solve the pairwise mapping relationship between each pixel in the blackboard area image; as well as The step of outputting the blackboard writing content is to obtain the pixel points containing text in each group of the blackboard area images, and obtain the text area through the region growing algorithm based on the pixel points, select the first image in each group of the blackboard area images as the initial image of the output result, calculate the pixel change area of ​​the text area based on the initial image, cut and fill the pixel change area into the initial image through the mapping relationship, repeat the above steps to traverse the blackboard area images in the same group, and output the final blackboard writing content.

2. The method for automatically generating blackboard writing content according to claim 1, characterized in that: The step of obtaining the blackboard area image of each frame in the teaching video in the blackboard area grouping step specifically includes the following steps: Establishing and training a deep learning target detection model, and detecting a blackboard area and a human body area in each frame of the image; The values ​​of all pixels in the human body area in each frame of the image are set to 0, the blackboard area in each frame of the image is cut, and the blackboard area images are formed by arranging them in chronological order.

3. The method for automatically generating blackboard writing content according to claim 2, characterized in that: The deep learning target detection model includes YOLO v3.

4. The method for automatically generating blackboard writing content according to claim 1, characterized in that: In the blackboard area grouping step, if the number of matching points is less than a set threshold, all blackboard area images at and before the moment are classified into one group.

5. The method for automatically generating blackboard writing content according to claim 4, characterized in that: The threshold is set to be 20% of the width of the blackboard area image.

6. The method for automatically generating blackboard writing content according to claim 1, characterized in that: The formula of the objective optimization function constructed using the nonlinear least squares method is as follows: Where I represents the grayscale value of the image, θ is the affine transformation matrix, θ = (a, b, c, d, e, f), a, b, c, d, e, f are the six parameters of the affine transformation, p i and p j are respectively the pixel points of the blackboard area image.

7. The method for automatically generating blackboard writing content according to claim 1, characterized in that: The blackboard content output step specifically includes: Binarization is performed on each group of images in the blackboard area, wherein a binarization value of 1 represents a pixel point of text; Taking the pixel point in the blackboard area image as the initial point, performing m dilation operations to obtain a text area A and a text area B respectively, wherein the initial image corresponds to the text area A, and any other blackboard area image except the initial image corresponds to the text area B; A set difference operation is performed on the text area B and the text area A to obtain the pixel change area C.

8. A device for automatically generating blackboard content, characterized in that: include: The blackboard area grouping module is configured to obtain the blackboard area image of each frame in the teaching video, use the scale-invariant feature transformation technology to calculate the number of matching points between the blackboard area image at one moment and the next moment in chronological order, and group the blackboard area images according to the number of matching points; A matching point mapping module is configured to calculate the matching points between any two images in each group of the blackboard area images using a scale-invariant feature descriptor, construct a target optimization function based on the matching points using a nonlinear least squares method, and solve the pairwise mapping relationship between each pixel in the blackboard area image using a Gauss-Newton iteration method; as well as The blackboard content output module is configured to select pixel points containing text in each group of the blackboard area images, and obtain the text area through a region growing algorithm based on the pixel points, select the first image in each group of the blackboard area images as the initial image of the output result, calculate the pixel change area of ​​the text area based on the initial image, cut and fill the pixel change area into the initial image through the mapping relationship, repeat the above steps to traverse the blackboard area images in the same group, and output the final blackboard content.

9. An electronic device, comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images