Composite course recording method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a composite course recording method and device, electronic equipment and a storage medium. A first-path picture and a second-path picture which are synchronously generated are obtained, the first-path picture is obtained by shooting a camera towards a teacher lecture background, the second-path picture is a content picture displayed by an interaction panel in the shooting process of the camera, and the interaction panel is located in the teacher lecture background. The area outside the interaction panel in the teaching background of the teacher is covered with a curtain; confirming a foreground image picture of a region corresponding to the curtain from the first path picture; and synthesizing the second path of picture and the foreground image picture according to a preset relative position relation and a time axis to obtain a composite course recording picture. According to the embodiment of the invention, it is ensured that effective information is presented as much as possible in the obtained composite lesson recording picture, complete teaching links can be presented to students in a simplified and condensed picture, and the effect of online teaching is effectively improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image technology, and in particular to a composite course recording method, device, electronic device and storage medium. Background Art

[0002] With the continuous development and popularization of information technology, in the field of teaching, computer-assisted teaching, distance learning and autonomous online learning, which rely on information technology, have made great progress. Autonomous online learning is to record the teacher's teaching scene and then publish the recorded teaching video to the Internet, so that students can watch the teaching video and learn independently.

[0003] In the existing recording scene, the first method is to set the podium background as a green screen, with interactive tablet teaching courseware arranged on the podium, and the green screen covers the interactive tablet. The teacher views the content and prompts displayed on the interactive tablet through the prompt tablet opposite the podium. The camera is facing the green screen to shoot. In the process of generating the recording file, the teacher part in the camera image is cut out based on the green screen, and the courseware part is obtained based on the interactive tablet image. The teacher part and the courseware part are superimposed to obtain the micro-class image. In this way, the teacher can only make monotonous voice output based on the content of the prompt tablet, and cannot perform the teaching content on the blackboard.

[0004] In the existing recording scene, the second method is to adjust the main color of the interactive tablet to achieve a virtual green screen. During the teaching process, the teacher can operate on the interactive tablet. The camera is facing the interactive tablet to shoot. In the process of generating the recorded lesson file, the teacher part of the camera image is cut out based on the green screen, and the courseware part is obtained based on the interactive tablet image. The teacher part and the courseware part are superimposed to obtain the micro-class image. This method can only output the image within the range of the interactive tablet as a micro-class image.

[0005] In the existing recording scenarios, what is recorded is not a comprehensive and authentic teaching process. The output recording files mainly show the teacher's grasp of knowledge points, but cannot show teaching skills and teaching effects. When the recording files are used for online self-study, students' sense of immersion and interaction is poor, and they cannot learn at a normal teaching rhythm. Directly recording the teaching scene requires a large area of ​​screen content that is not related to teaching to be presented in the video. Overall, the existing recording methods cannot present the complete teaching process to students with concise and concise pictures, and the effect of online teaching is quite different from that of offline teaching. Summary of the invention

[0006] The present invention provides a composite course recording method, device, electronic device and storage medium to solve the technical problems that the existing course recording method cannot present the complete teaching process to students with concise and concise pictures, and the effect of online teaching is quite different from that of offline teaching.

[0007] In a first aspect, an embodiment of the present application provides a composite course recording method, the composite course recording method comprising:

[0008] Acquire a first picture and a second picture generated synchronously, wherein the first picture is obtained by shooting with a camera toward the teacher's lecture background, and the second picture is a picture of the content displayed on the interactive flat panel during the camera shooting process, the interactive flat panel is located in the teacher's lecture background, and the area outside the interactive flat panel in the teacher's lecture background is covered with a curtain;

[0009] Confirm the foreground image of the area corresponding to the screen from the first image;

[0010] The second channel image and the foreground image are synthesized by aligning the time axis with a preset relative position relationship to obtain a composite recorded class image.

[0011] In a second aspect, an embodiment of the present application further provides a composite course recording device, the composite course recording device comprising:

[0012] A picture acquisition unit is used to acquire a first picture and a second picture generated synchronously, wherein the first picture is obtained by shooting with a camera toward the teacher's lecture background, and the second picture is a picture of the content displayed on the interactive tablet during the camera shooting process, the interactive tablet is located in the teacher's lecture background, and the area outside the interactive tablet in the teacher's lecture background is covered with a curtain;

[0013] A foreground confirmation unit, used to confirm the foreground image picture of the area corresponding to the curtain from the first picture;

[0014] The picture synthesis unit is used to synthesize the second picture and the foreground image picture in a preset relative position relationship and align the time axis to obtain a composite recorded class picture.

[0015] In a third aspect, an embodiment of the present application further provides an electronic device, the electronic device comprising:

[0016] one or more processors;

[0017] a memory for storing one or more computer programs;

[0018] When one or more computer programs are executed by one or more processors, the electronic device implements the composite course recording method as in the first aspect.

[0019] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the composite course recording method of the first aspect is implemented.

[0020] The above-mentioned composite class recording method, device, electronic device and storage medium, in this method, obtain the first channel of pictures and the second channel of pictures generated synchronously, the first channel of pictures is obtained by shooting with a camera toward the teacher's lecture background, the second channel of pictures is the content picture displayed on the interactive flat panel during the camera shooting process, the interactive flat panel is located in the teacher's lecture background, and the area outside the interactive flat panel in the teacher's lecture background is covered with a curtain; confirm the foreground image picture of the area corresponding to the curtain from the first channel of pictures; align the second channel of pictures and the foreground image picture with a preset relative position relationship and synthesize them on the time axis to obtain a composite class recording picture. By extracting the foreground of the pictures recorded in actual teaching, aligning the extracted pictures with the pictures displayed on the interactive flat panel with a preset relative position relationship and synthesizing them on the time axis, it is ensured that as much effective information as possible is presented in the obtained composite class recording picture, and the complete teaching process can be presented to students with a concise and condensed picture, effectively improving the effect of online teaching. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A method flow chart of a composite course recording method provided in an embodiment of the present application.

[0022] Figure 2 This is a schematic diagram of the first existing class recording scenario.

[0023] Figure 3 This is a schematic diagram of the second existing class recording scenario.

[0024] Figure 4 This is a schematic diagram of the first type of first screen in the composite course recording method provided in an embodiment of the present application.

[0025] Figure 5 This is a schematic diagram of the second first-channel screen in the composite course recording method provided in an embodiment of the present application.

[0026] Figure 6 Based on Figure 4 and Figure 5 Schematic diagram of the generated composite course recording screen.

[0027] Figure 7 A schematic diagram of the structure of the composite course recording device provided in an embodiment of the present application.

[0028] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0030] It should be noted that due to space limitations, this application description does not list all optional implementation methods. After reading this application description, those skilled in the art should be able to understand that as long as the technical features do not contradict each other, any combination of technical features can constitute an optional implementation method.

[0031] The following describes in detail the embodiments of the present invention.

[0032] With the development of information technology, the forms of computer-assisted teaching are constantly enriched and the effects are constantly improved. For example, an interactive tablet is configured in the teaching space to realize the electronic display and electronic recording of teaching information based on the interactive tablet. The interactive tablet mentioned in this solution can be an integrated device that controls the content displayed on the display tablet and realizes human-computer interaction through touch technology. It integrates one or more functions such as a projector, an electronic whiteboard, a screen, a sound system, a television, and a video conferencing terminal.

[0033] Generally speaking, an interactive tablet includes at least one screen. For example, the interactive tablet is equipped with a screen with a touch function, and the screen can be a capacitive screen, a resistive screen, or an electromagnetic screen. In this embodiment, further, the user can implement touch operation by touching the screen with a finger or a stylus, and accordingly, the interactive tablet detects the touch position and responds according to the touch position to implement the touch function. Typically, the interactive tablet is installed with at least one operating system, wherein the operating system includes but is not limited to the Android system, the Linux system, and the Windows system.

[0034] In the teaching scenario, the interactive tablet is an interactive device mainly used to support the electronic display and electronic recording of teaching information, and the software therein is mainly used to realize the display and recording function requirements in the teaching scenario. For example, the interactive tablet can install at least one application with a document presentation function, wherein the application can be an application that comes with the operating system; at the same time, an application downloaded from a third-party device or server is also installed. Optionally, in addition to the basic function of displaying the edited content, the program with the document presentation function also has other editing functions in the presentation process, such as inserting tables, inserting pictures, inserting graphics, drawing tables, drawing graphics, etc., to realize the real-time display of user information input and input content during the information presentation process, thereby improving the information presentation and interaction effect during the teaching-related document presentation process. For example, the interactive tablet can install at least one application with a whiteboard function, and teachers can realize real-time writing, recording, modification and saving in the teaching scenario based on this application.

[0035] When the interactive tablet is used alone, it is mainly suitable for teachers to electronically display teaching information to students in the same teaching space during offline teaching, such as teaching courseware, simulation animations, simulation software, etc.; as well as electronically record the blackboard writing, notes, annotations, etc. generated by teachers during teaching activities.

[0036] On the basis of the existing interactive tablet in the teaching space, a camera and a recording and broadcasting host can be additionally set up in the teaching space, and both the camera and the interactive tablet are connected to the recording and broadcasting host. The lens of the camera faces the side where the interactive tablet is located. From the relative position of the relevant elements in the teaching process, an important shooting target of the camera is the teacher. The wall where the interactive tablet is located is the background of the teacher's lecture. The picture captured by the camera is transmitted to the recording and broadcasting host in real time through the corresponding connection. When live online teaching and / or recording classes to generate video teaching resources, the interactive tablet displays the screen according to the teacher's operation, and transmits the displayed picture to the recording and broadcasting host in real time through the corresponding connection. Based on the above picture transmission, the recording and broadcasting host receives the original data of the interactive tablet display picture from the picture received by the camera from the interactive tablet synchronously (or at a very small interval, which is equivalent to synchronization) received from the interactive tablet.

[0037] To facilitate the later image processing, in the specific teaching space, you can Figure 2 As shown, the podium background is set to a single-color curtain 10, such as a green curtain, a red curtain, etc.; the interactive tablet 20 arranged on the podium displays teaching courseware, and the curtain 10 covers the interactive tablet 20 ( Figure 2 The interactive tablet 20 is not visible through a dotted line), and the teacher 30 views the content and prompts displayed on the interactive tablet 20 through the prompt tablet opposite the podium. The camera shoots toward the curtain 10. In the process of generating the recorded class file, the teacher part in the camera image is cut out based on the curtain 10, and the courseware part is obtained based on the interactive tablet 20 image. The teacher part and the courseware part are superimposed to obtain the micro-class image. In this way, the teacher can only perform monotonous voice output based on the content of the prompt tablet, and cannot perform the teaching content on the blackboard.

[0038] The second way to record a class is as follows Figure 3 As shown, the main color of the interactive tablet 20 is adjusted to realize a virtual curtain. During the teaching process, the teacher 30 can operate the interactive tablet 20. The camera is directed toward the interactive tablet 20 for shooting. In the process of generating the recorded class file, based on the effect of the virtual curtain of the interactive tablet 20, the teacher part in the camera picture is cut out, and the courseware part is obtained based on the content picture of the interactive tablet. The teacher part and the courseware part are superimposed to obtain the micro-class picture. This method can only output the picture within the range of the interactive tablet as a micro-class picture. As an important element of teaching, the teaching behavior of the teacher 30 cannot be effectively displayed.

[0039] In the existing methods of recording lessons, what is recorded is not a comprehensive and true teaching process. The output recording files mainly show the teacher's grasp of knowledge points, but cannot show teaching skills and teaching effects. When the recording files are used for online self-study, students' sense of immersion and interaction is poor, and they cannot learn at a normal teaching rhythm. Directly recording the teaching scene requires a large area of ​​screen content that is not related to teaching to be presented in the video screen. Overall, the existing recording methods cannot present the complete teaching process to students with concise and concise pictures, and the effect of online teaching is quite different from that of offline teaching.

[0040] In view of the above technical problems, the embodiment of the present application proposes a composite class recording method, which extracts the foreground of the screen recorded from the actual teaching, and synthesizes the extracted screen and the screen displayed on the interactive flat panel according to the preset relative position relationship and aligns the time axis, so as to ensure that as much effective information as possible is presented in the obtained composite class recording screen, and can present the complete teaching link to students with a concise and condensed screen, effectively improving the effect of online teaching. The various embodiments of the present invention are described in detail below.

[0041] Figure 1 This is a method flow chart of a composite course recording method provided in an embodiment of the present application. The composite course recording method is applied to an electronic device that processes relevant images of a teaching scene, and is implemented by an electronic device, such as a recording and broadcasting host. Figure 1 As shown, the composite course recording method includes steps S110 to S130:

[0042] Step S110: Acquire the first and second images that are generated synchronously, where the first image is shot by a camera toward the teacher's lecture background, and the second image is the content displayed on the interactive tablet during the camera shooting process. The interactive tablet is located in the teacher's lecture background, and the area outside the interactive tablet in the teacher's lecture background is covered with a curtain.

[0043] Compared to Figure 2 The curtain 10 shown completely covers the teacher's lecture background and Figure 3 Only the area where the interactive tablet 20 is located is set as a virtual screen. In the embodiment of the present application, please refer to Figure 4 The area outside the interactive tablet 20 is covered with a curtain 10, that is, during the teaching process, the teacher 30 can complete the teaching-related demonstration and blackboard writing on the interactive tablet 20. Figure 4 The screen 10 shown can obtain a first channel of images and a second channel of images that are generated synchronously.

[0044] The first picture is taken by shooting with the camera toward the teacher's lecture background. In terms of the structure of the picture content, the first picture includes the teacher as the foreground and the teacher's lecture background composed of a curtain and an interactive flat panel. The second picture is the content displayed on the interactive flat panel during the camera shooting process. If the picture is acquired by the recording and broadcasting host, the recording and broadcasting host can transmit data to the interactive flat panel via an HDMI (High Definition Multimedia Interface) cable or other multimedia protocol connection cable, that is, the second picture can be sent from the interactive flat panel to the recording and broadcasting host via any of the various multimedia protocol connection cables. The camera and the recording and broadcasting host can use the same or different connection cable as the recording and broadcasting host and the interactive flat panel for data transmission.

[0045] In another specific implementation, the interactive tablet displays a transparent mask of the same color as the curtain, such as a green transparent mask, on the upper layer of the content screen. If the interactive tablet has only one system module that provides an operating system, the content screen and the transparent mask are generated by the same operating system. If the interactive tablet has multiple system modules that provide multiple operating systems, the content screen and the transparent mask can be generated by the operating system corresponding to the content screen, or the content screen and the transparent mask can be generated by different operating systems. For example, there is a system module of the Windows system and a system module of the Android system in the interactive tablet, and the display of the interactive tablet can be controlled by the Android system. The content screen of the PPT used for teaching is generated by the Windows system. When the interactive tablet displays the PPT, the system module of the Windows system sends the content screen to the system module of the Android system, and the Android system adds a transparent mask on the content screen, and then controls the interactive tablet to display the content screen (i.e., the PPT) with the transparent mask added.

[0046] Step S120: confirming the foreground image of the area corresponding to the screen from the first image.

[0047] According to the specific structure of the teacher's lecture background, the method of confirming the foreground image of the corresponding area of ​​the curtain from the first picture can have corresponding implementation details.

[0048] If the interactive flat panel displays a normal picture (i.e., it is not configured as a virtual screen), a rectangle that meets the preset size in the first picture is identified to obtain a regional image outside the rectangle in the first picture; the HSV color gamut parameters of the pixels in the regional image are analyzed according to the color of the screen to confirm the foreground pixels in the regional image; based on the foreground pixels and the background pixels of the preset color, a foreground image screen with the same size as the circumscribed rectangle of the foreground pixels is obtained.

[0049] Detecting the rectangle in the first picture can be based on Hough transform detection or using the rectangle detection interface provided in OpenCV. In the teaching scene, the largest rectangle in the first picture is generally the rectangle formed by the outer contour of the interactive flat panel. When multiple rectangles are identified during rectangle recognition, the largest rectangle can be used as the rectangular area in the first picture, that is, the area corresponding to the screen of the interactive flat panel. The area outside the rectangular area is the area corresponding to the actual curtain. The image color collected in the area corresponding to the curtain is relatively single, and even if it is affected by ambient light, the color deviation is small. Based on this basic display rule, the HSV color gamut parameters of the pixels in the area outside the rectangle are analyzed, and the actual curtain corresponding color pixels are confirmed according to the overall display parameters. In other words, what is actually to be confirmed is not the pixels in the area outside the rectangle that completely match the display parameters of the curtain color, but the pixels that meet the general rules of curtain color display from the display rule. Even if this type of pixel is not the curtain corresponding color pixel, it can also be considered as the curtain corresponding color pixel. For example, due to the influence of ambient light, the actual image color in some areas of the screen in the first picture may deviate from the HSV color gamut parameters corresponding to the screen, but the overall color should be close to the HSV color gamut parameters corresponding to the screen color. Pixels corresponding to this color can be confirmed as pixels corresponding to the color of the screen.

[0050] HSV is a color model. In HSV model, hue H is measured by angle, ranging from 0° to 360°. Starting from red and counting in a counterclockwise direction, red is 0°, green is 120°, and blue is 240°. Their complementary colors are: yellow is 60°, cyan is 180°, and purple is 300°. Saturation S indicates the degree to which a color is close to a spectral color. A color can be regarded as the result of a mixture of a certain spectral color and white. The greater the proportion of the spectral color, the higher the degree to which the color is close to the spectral color, and the higher the saturation of the color. The higher the saturation, the deeper and brighter the color. The white light component of the spectral color is 0, and the saturation reaches the highest. The value range is usually 0% to 100%. The larger the value, the more saturated the color. Brightness V indicates the brightness of the color. For light source color, the brightness value is related to the brightness of the light source; for object color, this value is related to the transmittance or reflectance of the object. The value range is usually 0% (black) to 100% (white). Different colors have corresponding parameters in the HSV color mode. For example, when the color of the curtain is green, the corresponding image in the first channel should be green pixels. In the actual processing process, the production color and acquisition of the curtain may have deviations. Therefore, according to the statistical results, all pixels that meet the pixel distribution state are regarded as green pixels. In the specific implementation process, the analysis of the HSV color gamut parameters of the pixels in the area outside the rectangle can be global analysis or local analysis.

[0051] For global analysis, the area outside the rectangle is taken as the target statistical area, and the HSV color gamut parameters of the pixels in the target statistical area are counted to obtain the HSV distribution results of the pixels.

[0052] For local analysis, the preset area is taken as the target statistical area, and the HSV color gamut parameters of the sample pixels in the target statistical area are counted to obtain the HSV distribution results of the sample pixels. The preset area is the pixels within a certain height or width range in the area outside the rectangle; for example, the speed limit is at a height that teachers generally cannot reach.

[0053] If the interactive tablet displays a transparent mask on the upper layer of the content screen, that is, the interactive tablet is configured as a virtual screen, and the interactive tablet and the screen form a complete teacher's lecture background, then the foreground image corresponding to the comprehensive area composed of the screen and the interactive tablet can be confirmed from the first screen. This is equivalent to being able to directly confirm the foreground image, that is, the image with the teacher as the main body, from the first screen.

[0054] Step S130: Synthesize the second channel image and the foreground image in a preset relative position relationship and align the time axis to obtain a composite course recording image.

[0055] In the field of video processing, each frame in the video is displayed as a rectangle in units of frames. Based on the basic design of the video, when the second channel picture and the foreground image are synthesized to obtain a composite recording picture, a rectangular picture should be generated in the end. That is to say, the relative position relationship between the second channel picture and the foreground image is at least based on the basic condition of not overlapping and being displayed within the range of a rectangular picture. In the specific synthesis process, the minimum image frame for fully displaying the second channel picture and the foreground image and having an aspect ratio that meets a preset ratio can be determined based on the size of the second channel picture and the foreground image. The preset ratio is the aspect ratio of a common video window. Then, the second channel picture and the foreground image are completely added to the minimum image frame, and the area outside the second channel picture and the foreground image in the minimum image frame is filled with background pixels of a preset color. In another case where synthesis adjustment can be made, the teacher blocks the interactive tablet in the first picture. The teacher may be operating the interactive tablet. From the perspective of the relationship between the contents in the first picture, the teacher cannot be separated from the interactive tablet. At this time, the teacher and the interactive tablet should be synthesized as a whole, that is, when it is confirmed that the foreground image overlaps with the area where the interactive tablet is located, align the time axis and superimpose the second picture on the area where the lower interactive tablet of the foreground image is located to obtain the foreground main picture; according to the circumscribed rectangle of the foreground main picture, a composite class recording picture that meets the preset aspect ratio is obtained, and the composite class recording picture is the same as the length and / or width of the circumscribed rectangle. Of course, the changes in pixels during the specific synthesis process can refer to the pixel processing process in the existing cutout synthesis process, which will not be explained in detail here.

[0056] It should be understood that in order to ensure the synchronization of audio and video, the video is aligned with the time axis while the audio is also aligned with the time axis. The final composite recorded video can be used for synchronous remote teaching or as a subsequent online teaching resource.

[0057] For teaching activities that take place in the teaching space, there are two sources that have the greatest impact on the transmission of teaching information. The first is the content displayed on the interactive flat panel, and the second is the various information generated by the teacher, including body movements, voice expressions, blackboard writing, etc. In the recording of lessons in the prior art, usually only one of the sources of information can be realized, or the pictures in the entire teaching space are recorded without distinguishing between the primary and the secondary. These recording methods directly lead to the loss of teaching information, or the playback of the recorded lesson files displays a large amount of irrelevant information (such as a large area of ​​blank wall on the side of the podium), and the playback of recorded lesson files is usually completed on a terminal device of a smaller size, and the display effect of the actual effective information on the terminal device is even worse. Through the embodiments of the present application, as much effective information as possible can be presented in the recorded lesson files, and the complete teaching process can be presented to students with concise and concise pictures, effectively improving the effect of online teaching.

[0058] The embodiment of the present application is designed for a special case, that is, in the process of implementing the composite class recording method, when it is confirmed that the proportion of the foreground image screen in the first screen is greater than the preset threshold value, the second screen is used as the composite class recording screen. From the teaching scene, if the teacher's image area accounts for a high proportion or zero proportion in the first screen, it means that the teacher is away from the podium and close to the camera or has entered the blind spot of the camera. In this state, the teacher's behavior is usually not highly correlated with the content displayed in the interactive tablet. At this time, only the screen of the interactive tablet can be retained, that is, the second screen is directly used as the composite class recording screen, thereby increasing the proportion of effective information in the composite class recording screen.

[0059] In another specific state, when it is confirmed that the duration of the state in which there is no voice in the first channel of the picture reaches a preset duration, the second channel of the picture and the foreground image are deleted according to the preset deletion rule, and then the time axis is aligned and synthesized with a preset relative position relationship to obtain a simplified recorded class picture. The simplified recorded class picture can be used for conducting teaching and research activities to evaluate the class, reduce the silent state when playing the recorded class file during the evaluation process, and realize the output of multiple teaching-related electronic resources from one classroom.

[0060] The above processing can refer to Figure 4-Figure 6 Further highlight the advantages of this solution compared with the existing technology. Figure 4 and Figure 5 A schematic diagram of two first-channel images in the composite course recording method provided in an embodiment of the present application, Figure 4In the figure, the camera captures the complete curtain 10, the interactive tablet 20 and the teacher 30, wherein the teacher 30 is close to the interactive tablet 20 to operate. Figure 5 In the video, the camera captures the complete screen 10, the interactive tablet 20 and the teacher 30, wherein the teacher 30 is explaining at a position far away from the interactive tablet 20. According to the processing process of the prior art, either all the screen contents are retained, or only the contents displayed in the interactive tablet 20 are retained (plus at most the teacher's bust).

[0061] based on Figure 4 and Figure 5 According to this solution, the foreground image with the teacher 30 as the main body can be extracted from the first picture. No matter what relative position relationship the teacher 30 and the interactive tablet 20 are actually in, the foreground image and the second picture corresponding to the interactive tablet 20 can be synthesized together in a compact manner to obtain the following: Figure 6 The composite course recording screen shown in Figure 6 In the composite course recording screen shown, compared with Figure 4 and Figure 5 , removes external areas that are not related to information transmission in the teaching process, compresses the imaging distance between the interactive tablet 20 and the teacher 30, and at the same time retains the information in the interactive tablet 20 and all information related to the teacher 30, improves the efficiency of information transmission, and presents the complete teaching process to students with a concise and concise picture, which can effectively improve the effect of online teaching.

[0062] exist Figure 6 In the composite recorded lesson screen shown, the interactive tablet 20 and the teacher 30 can be considered as horizontal movement, and the surrounding areas can be considered as removed, and finally the core screen shown in the dotted box is obtained. The aspect ratio of the core screen may be inconsistent with the aspect ratio of general video encoding. At this time, background pixels of preset colors can be filled on both sides to obtain a screen consistent with the aspect ratio of the currently used video.

[0063] The above-mentioned composite class recording method obtains the first channel of images and the second channel of images generated synchronously. The first channel of images is obtained by shooting with a camera toward the teacher's lecture background, and the second channel of images is the content displayed on the interactive flat panel during the camera shooting process. The interactive flat panel is located in the teacher's lecture background, and the area outside the interactive flat panel in the teacher's lecture background is covered with a curtain; the foreground image of the area corresponding to the curtain is confirmed from the first channel of images; the second channel of images and the foreground image are aligned with the time axis in a preset relative position relationship for synthesis to obtain a composite class recording image. By extracting the foreground of the images recorded in actual teaching, the extracted images are aligned with the images displayed on the interactive flat panel in a preset relative position relationship for synthesis, ensuring that as much effective information as possible is presented in the obtained composite class recording image, and being able to present the complete teaching process to students with a concise and condensed image, effectively improving the effect of online teaching.

[0064] Figure 7 This is a schematic diagram of the structure of a composite course recording device provided in an embodiment of the present application. Figure 7 As shown, the composite course recording device includes a picture acquisition unit 210, a foreground confirmation unit 220 and a picture synthesis unit 230.

[0065] Among them, the picture acquisition unit 210 is used to acquire the first picture and the second picture generated synchronously, the first picture is obtained by shooting with a camera toward the teacher's lecture background, and the second picture is the content picture displayed on the interactive tablet during the camera shooting process, the interactive tablet is located in the teacher's lecture background, and the area outside the interactive tablet in the teacher's lecture background is covered with a curtain; the foreground confirmation unit 220 is used to confirm the foreground image picture of the corresponding area of ​​the curtain from the first picture; the picture synthesis unit 230 is used to synthesize the second picture and the foreground image picture with a preset relative position relationship and align the time axis to obtain a composite recorded class picture.

[0066] Based on the above embodiment, the foreground confirmation unit 220 includes:

[0067] A region recognition module, used to recognize a rectangle that meets a preset size in the first picture, and obtain an image of a region outside the rectangle in the first picture;

[0068] A pixel analysis module is used to analyze the HSV color gamut parameters of the pixels in the regional image according to the color of the curtain, and confirm the foreground pixels in the regional image;

[0069] The image generation module is used to obtain a foreground image frame having the same size as the circumscribed rectangle of the foreground pixels according to the foreground pixels and the background pixels of a preset color.

[0070] Based on the above embodiment, the interactive tablet displays a green transparent mask on the upper layer of the content screen;

[0071] Accordingly, the foreground confirmation unit 220 includes:

[0072] The comprehensive extraction module is used to confirm the foreground image corresponding to the comprehensive area composed of the curtain and the interactive flat panel from the first picture.

[0073] Based on the above embodiment, the picture synthesis unit 230 includes:

[0074] A direct overlay module is used to align the time axis and overlay the second channel image onto the area where the lower interactive tablet of the foreground image is located to obtain the foreground main image when confirming that the foreground image screen overlaps with the area where the interactive tablet is located;

[0075] The length and width confirmation module is used to obtain a composite class recording screen that meets the preset length and width ratio based on the circumscribed rectangle of the foreground main screen, and the length and / or width of the composite class recording screen is the same as the circumscribed rectangle.

[0076] Based on the above embodiment, the composite class recording device also includes a direct output unit, which is used to confirm the foreground image of the screen corresponding to the area in the first picture, and when it is confirmed that the proportion of the foreground image in the first picture is greater than a preset threshold value, use the second picture as the composite class recording picture.

[0077] On the basis of the above embodiment, the composite class recording device also includes a streamlined output unit, which is used to confirm the foreground image picture of the corresponding area of ​​the curtain from the first picture, and when it is confirmed that the duration of the state in which there is no voice in the first picture reaches a preset time, delete the second picture and the foreground image picture according to a preset deletion rule, and then align the time axis with a preset relative position relationship to synthesize them to obtain a streamlined class recording picture.

[0078] The composite course recording device provided in the embodiment of the present application is included in an electronic device and can be used to execute the corresponding composite course recording method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0079] It is worth noting that in the embodiment of the above-mentioned composite course recording device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0080] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 8As shown, the electronic device includes a processor 310 and a memory 320, and may also include an input device 330, an output device 340 and a communication device 350; the number of the processor 310 in the electronic device may be one or more, Figure 8 A processor 310 is taken as an example; the processor 310, the memory 320, the input device 330, the output device 340 and the communication device 350 in the electronic device can be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0081] The memory 320, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the composite course recording method in the embodiment of the present application. The processor 310 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the memory 320, that is, realizing the above-mentioned composite course recording method.

[0082] The memory 320 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 320 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 320 may further include a memory remotely arranged relative to the processor 310, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0083] The input device 330 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device. The output device 340 may include a display device such as a display screen.

[0084] The above-mentioned electronic device includes a composite course recording device, which can be used to execute any composite course recording method and has corresponding functions and beneficial effects.

[0085] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it is used to perform relevant operations in the composite course recording method provided in any embodiment of the present application, and has corresponding functions and beneficial effects.

[0086] Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products.

[0087] Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0088] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory. The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0089] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0090] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0091] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A composite course recording method, characterized in that: include: Acquire a first picture and a second picture generated synchronously, wherein the first picture is obtained by shooting with a camera toward a teacher's lecture background, and the second picture is a picture of content displayed on an interactive tablet during the camera shooting process, the interactive tablet is located in the teacher's lecture background, and an area outside the interactive tablet in the teacher's lecture background is covered with a curtain; Confirming the foreground image of the area corresponding to the screen from the first image; The second channel image and the foreground image are synthesized by aligning the time axis with a preset relative position relationship to obtain a composite recorded class image.

2. The composite course recording method according to claim 1, characterized in that: The step of confirming the foreground image of the area corresponding to the curtain from the first image includes: Identify a rectangle that meets a preset size in the first picture, and obtain an image of a region outside the rectangle in the first picture; Analyzing HSV color gamut parameters of pixels in the regional image according to the color of the curtain to identify foreground pixels in the regional image; A foreground image frame having the same size as a circumscribed rectangle of the foreground pixels is obtained according to the foreground pixels and background pixels of a preset color.

3. The composite course recording method according to claim 1, characterized in that: The interactive tablet displays a transparent mask of the same color as the curtain on the upper layer of the content screen; Correspondingly, the step of confirming the foreground image of the area corresponding to the curtain from the first image includes: The foreground image corresponding to the integrated area composed of the screen and the interactive flat panel is confirmed from the first picture.

4. The composite course recording method according to claim 3, characterized in that: The step of synthesizing the second channel image and the foreground image in a preset relative position relationship and aligning the time axis to obtain a composite course recording image includes: When it is confirmed that the foreground image screen overlaps with the area where the interactive tablet is located, align the time axis to superimpose the second channel image on the area where the interactive tablet is located in the lower layer of the foreground image screen to obtain the foreground main image; A composite class recording screen satisfying a preset aspect ratio is obtained according to the circumscribed rectangle of the foreground main screen, and the composite class recording screen has the same length and / or width as the circumscribed rectangle.

5. The composite course recording method according to any one of claims 1 to 4, characterized in that: After confirming the foreground image of the area corresponding to the curtain from the first picture, the method further includes: When it is confirmed that the proportion of the foreground image in the first picture is greater than a preset threshold value, the second picture is used as a composite recording picture.

6. The composite course recording method according to any one of claims 1 to 4, characterized in that: After confirming the foreground image of the area corresponding to the curtain from the first picture, the method further includes: When it is confirmed that the duration of the absence of voice in the first picture reaches the preset duration, the second picture and the foreground image are deleted according to the preset deletion rules, and then synthesized with the preset relative position relationship and aligned with the time axis to obtain a streamlined recorded class picture.

7. A composite course recording device, characterized in that: include: A picture acquisition unit is used to acquire a first picture and a second picture generated synchronously, wherein the first picture is obtained by shooting with a camera toward a teacher's lecture background, and the second picture is a content picture displayed on an interactive tablet during the camera shooting process, wherein the interactive tablet is located in the teacher's lecture background, and an area outside the interactive tablet in the teacher's lecture background is covered with a curtain; A foreground confirmation unit, used to confirm the foreground image picture of the area corresponding to the curtain from the first picture; The picture synthesis unit is used to synthesize the second picture and the foreground image picture in a preset relative position relationship and align the time axis to obtain a composite recording picture.

8. The composite course recording device according to claim 7, characterized in that: The prospect confirmation unit comprises: A region recognition module, used for recognizing a rectangle satisfying a preset size in the first picture, and obtaining a region image outside the rectangle in the first picture; A pixel analysis module, used to analyze HSV color gamut parameters of pixels in the regional image according to the color of the curtain, and confirm foreground pixels in the regional image; The image generation module is used to obtain a foreground image frame having the same size as the circumscribed rectangle of the foreground pixels according to the foreground pixels and background pixels of a preset color.

9. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more computer programs; When the one or more computer programs are executed by the one or more processors, the electronic device implements the composite course recording method as described in any one of claims 1-6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the composite course recording method as described in any one of claims 1 to 6 is implemented.

Images