Image processing method, device, electronic device and storage medium

By identifying and analyzing the HSV color gamut parameters of the rectangular area of ​​the interactive tablet screen and performing binary processing, the problem of abnormal image processing during the teaching video picture synthesis process is solved, real integration of the teaching site and the interactive tablet display screen is realized, and the Quality of video recording.

CN116883542BActive Publication Date: 2024-11-19GUANGZHOU KINDLINK SOFTWARE TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211466372.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-11-22
Publication Date
2024-11-19
Estimated Expiration
2042-11-22

AI Technical Summary

Technical Problem

During the synthesis of existing teaching videos, due to factors such as shooting angle, range, and camera color restoration capabilities, the image processing is abnormal, and the synthesized images are not realistically integrated with the pictures displayed on the teaching site and the interactive flat panel.

Method used

By real-time reception of the camera on the first screen of the interactive flat screen and the second screen of the screen content, identify the rectangular area, analyze the HSV color gamut parameters to confirm the green pixels, perform binarization processing, synthesize the comprehensive picture, and eliminate image processing abnormalities. , realize the real integration of the teaching site and the interactive flat panel display screen.

Benefits of technology

Eliminate image processing abnormalities during teaching picture synthesis, ensure that the combined image is a true fusion of the teaching site and the interactive flat panel display screen, avoiding temporary data replacement during processing, and improving the quality of video recording.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116883542B_ABST
    Figure CN116883542B_ABST
Patent Text Reader

Abstract

The embodiments of the present invention disclose an image processing method, device, electronic device and storage medium. Rectangular recognition is performed on the recorded class screen captured by the camera in the teaching scene to confirm the area where the interactive flat panel is located, and the pixels in the area where the interactive flat panel is located are statistically analyzed to confirm the green pixels, and then binarization is performed based on the confirmation results of the green pixels within the interactive flat panel range to synthesize multiple images to obtain a comprehensive image. Abnormalities in image processing during the synthesis of teaching images are eliminated, and the synthesized image is a true fusion of the teaching scene image and the interactive flat panel display image, and the actual collected image will not be replaced by temporary data generated during the processing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of image technology, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] With the continuous development and popularization of information technology, in the field of teaching, computer-assisted teaching, distance learning and autonomous online learning, which rely on information technology, have made great progress. Autonomous online learning is to record the teacher's teaching scene and then publish the recorded teaching video to the Internet, so that students can watch the teaching video and learn independently.

[0003] In order to improve the video quality and reduce invalid information, the existing teaching videos are usually not recorded directly by the camera outside the podium facing the podium, but an interactive tablet is arranged in the information classroom. The teaching courseware can be displayed on the interactive tablet, and the blackboard can also be displayed in the annotation mode or whiteboard application. When recording teaching videos in the information classroom, the recording and broadcasting host is connected to the camera and the interactive tablet, and the interactive tablet is superimposed with a green transparent layer when displayed. The recording and broadcasting host obtains a picture from the camera, and the display interface of the interactive tablet is captured in this picture; the recording and broadcasting host obtains the actual content picture from the interactive tablet. Based on the display of the green transparent layer, the recording and broadcasting host will show a part of the content captured by the camera with a green screen as the background. Based on the existence of the green screen, the display interface of the interactive tablet in the corresponding picture of the camera can be quickly cut out, and the content picture obtained from the interactive tablet can be synthesized into the cutout area in the corresponding picture of the camera, so as to achieve clear and high-quality courseware and blackboard content display based on the video recorded at the teaching site. This teaching video processing method can also be used for remote online teaching. The recording and broadcasting host will transmit the synthesized images in real time, allowing the teaching audience to get a feeling closer to on-site teaching.

[0004] In the process of synthesizing teaching videos by the recording and broadcasting host, the differences in shooting angle, range, color reproduction ability of the camera, main color of the content screen in the interactive flat panel, etc. may cause abnormalities in image processing during the screen synthesis process. The synthesized image is not a true fusion of the teaching scene screen and the interactive flat panel display screen. Summary of the invention

[0005] The present invention provides an image processing method, device, electronic device and storage medium to solve the technical problem that abnormalities occur in image processing during the existing teaching screen synthesis process, and the synthesized image is not a true fusion of the teaching scene screen and the interactive flat panel display screen.

[0006] In a first aspect, an embodiment of the present invention provides an image processing method, including:

[0007] Receive the first and second images in real time. The first image is obtained by shooting the screen of the interactive flat panel with a camera, and the second image is the content currently displayed on the screen. The screen displays a green transparent mask on the upper layer of the content;

[0008] Recognize the rectangle in the first picture and obtain the rectangular area in the first picture;

[0009] Analyze the HSV color gamut parameters of the pixels in the rectangular area to confirm the green pixels in the rectangular area;

[0010] According to the confirmation result of the green pixel, the rectangular area is binarized to obtain a black and white image;

[0011] The pixels of the second picture, the black-and-white picture and the rectangular area in the first picture are synthesized to obtain a comprehensive picture.

[0012] In a second aspect, an embodiment of the present invention further provides an image processing device, including:

[0013] The picture receiving unit is used to receive a first picture and a second picture in real time, wherein the first picture is obtained by shooting the screen of the interactive flat panel with a camera, and the second picture is the content picture currently displayed on the screen, and the screen displays a green transparent mask on the upper layer of the content picture;

[0014] A rectangle recognition unit, used for recognizing a rectangle in the first picture, and obtaining a rectangular area in the first picture;

[0015] A pixel confirmation unit, used to analyze the HSV color gamut parameters of the pixels in the rectangular area and confirm the green pixels in the rectangular area;

[0016] A binarization unit, used for binarizing the rectangular area according to the confirmation result of the green pixel to obtain a black and white image;

[0017] The picture synthesis unit is used to synthesize the pixels of the rectangular area in the second picture, the black and white picture and the first picture to obtain a comprehensive picture.

[0018] In a third aspect, an embodiment of the present invention further provides an electronic device, including:

[0019] one or more processors;

[0020] a memory for storing one or more computer programs;

[0021] When one or more computer programs are executed by one or more processors, the electronic device implements the image processing method of the first aspect.

[0022] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the image processing method of the first aspect is implemented.

[0023] The above-mentioned image processing method, device, electronic device and storage medium, in which the first channel of picture and the second channel of picture are received in real time, the first channel of picture is obtained by shooting the screen of the interactive flat panel by a camera, the second channel of picture is the content picture currently displayed on the screen, and the screen displays a green transparent mask on the upper layer of the content picture; the rectangle in the first channel of picture is identified to obtain the rectangular area in the first channel of picture; the HSV color gamut parameters of the pixels in the rectangular area are analyzed to confirm the green pixels in the rectangular area; the rectangular area is binarized according to the confirmation result of the green pixels to obtain a black and white picture; the second channel of picture, the black and white picture and the pixels in the rectangular area in the first channel of picture are synthesized to obtain a comprehensive picture. By performing rectangular recognition on the recorded class picture collected by the camera in the teaching scene, the area where the interactive flat panel is located is confirmed, and the pixels in the area where the interactive flat panel is located are statistically analyzed to confirm the green pixels, and the multiple channels of pictures are synthesized according to the confirmation result of the green pixels within the interactive flat panel range after binarization to obtain a comprehensive picture. The abnormality of image processing in the synthesis process of the teaching picture is eliminated, and the synthesized image is a true fusion of the teaching scene picture and the interactive flat panel display picture, and the actual collected picture will not be replaced by temporary data generated during the processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A method flow chart of an image processing method provided by an embodiment of the present invention.

[0025] Figure 2 A schematic diagram of the device architecture of a class recording system that applies the image processing method provided in an embodiment of the present invention.

[0026] Figure 3 This is a schematic diagram of the picture structure of the picture captured by the camera in the class recording system.

[0027] Figure 4 A schematic diagram of a first channel of images in an image processing method provided by an embodiment of the present invention.

[0028] Figure 5 A schematic diagram of the second picture in the image processing method provided by an embodiment of the present invention.

[0029] Figure 6 For the existing recording system Figure 4 The schematic diagram shown is the first channel image after processing.

[0030] Figure 7 Based on the existing course recording system Figure 5 and Figure 6Schematic diagram of the generated comprehensive screen.

[0031] Figure 8 The image processing method provided by the embodiment of the present invention is Figure 4 The schematic diagram shown is the first channel image after processing.

[0032] Fig. 9 The image processing method provided by the embodiment of the present invention is based on Figure 5 and Figure 8 Schematic diagram of the resulting composite screen.

[0033] Fig.10 A schematic diagram of the structure of an image processing device provided by an embodiment of the present invention.

[0034] Fig.11 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0036] It should be noted that due to space limitations, this application description does not list all optional implementation methods. After reading this application description, those skilled in the art should be able to understand that as long as the technical features do not contradict each other, any combination of technical features can constitute an optional implementation method.

[0037] The following is a detailed description of the various embodiments of the present invention.

[0038] With the development of information technology, the forms of computer-assisted teaching are constantly enriched and the effects are constantly improved. For example, an interactive tablet is configured in the teaching space to realize the electronic display and electronic recording of teaching information based on the interactive tablet. The interactive tablet mentioned in this solution can be an integrated device that controls the content displayed on the display tablet and realizes human-computer interaction through touch technology. It integrates one or more functions such as a projector, an electronic whiteboard, a screen, a sound system, a television, and a video conferencing terminal.

[0039] Generally speaking, the interactive tablet includes at least one screen. For example, the interactive tablet is equipped with a screen with a touch function, and the screen can be a capacitive screen, a resistive screen, or an electromagnetic screen. In this embodiment, further, the user can implement touch operation by touching the screen with a finger or a stylus, and accordingly, the interactive tablet detects the touch position and responds according to the touch position to implement the touch function. Typically, the interactive tablet is installed with at least one operating system, wherein the operating system includes but is not limited to the Android system, the Linux system, and the Windows system.

[0040] In the teaching scenario, the interactive tablet is an interactive device mainly used to support the electronic display and electronic recording of teaching information, and the software therein is mainly used to realize the display and recording function requirements in the teaching scenario. For example, the interactive tablet can install at least one application with a document presentation function, wherein the application can be an application that comes with the operating system; at the same time, an application downloaded from a third-party device or server is also installed. Optionally, in addition to the basic function of displaying the edited content, the program with the document presentation function also has other editing functions in the presentation process, such as inserting tables, inserting pictures, inserting graphics, drawing tables, drawing graphics, etc., to realize the real-time display of user information input and input content during the information presentation process, thereby improving the information presentation and interaction effect during the teaching-related document presentation process. For example, the interactive tablet can install at least one application with a whiteboard function, and teachers can realize real-time writing, recording, modification and saving in the teaching scenario based on this application.

[0041] When the interactive tablet is used alone, it is mainly suitable for teachers to electronically display teaching information to students in the same teaching space during offline teaching, such as teaching courseware, simulation animations, simulation software, etc.; as well as electronically record the blackboard writing, notes, annotations, etc. generated by teachers during teaching activities.

[0042] Please refer to Figure 2On the basis of the interactive tablet 10 already in the teaching space, a camera 20 and a recording and broadcasting host 30 can be additionally set in the teaching space, and both the camera 20 and the interactive tablet 10 are connected to the recording and broadcasting host 30. The lens of the camera 20 faces the screen of the interactive tablet 10, and the picture captured by the camera 20 is transmitted to the recording and broadcasting host 30 in real time through the corresponding connection. When live online teaching and / or recording classes to generate video teaching resources, the interactive tablet 10 displays the displayed picture on the screen according to the teacher's operation, and transmits the displayed picture to the recording and broadcasting host 30 in real time through the corresponding connection. Based on the above picture transmission, the recording and broadcasting host 30 receives the original data of the interactive tablet 10 display picture in the picture received from the camera 20 synchronously (or at a very small interval, which is equivalent to synchronization) from the interactive tablet 10. For the convenience of later image processing, when the interactive tablet 10 displays the picture of teaching-related content, a green transparent layer is displayed on the top, so that the screen area of ​​the interactive tablet 10 is green as a whole, and the picture captured by the camera 20 has a partial effect of a green screen as the background.

[0043] The picture structure diagram of the picture captured by the camera 20 is as follows Figure 3 As shown, in the teaching scenario, the teacher usually conducts teaching activities in front of the interactive tablet 10. Then, taking the lens of the camera 20 as a reference, in front of the lens, there are mainly the teacher closest to the lens and the interactive tablet 10 farthest from the lens. On the screen of the interactive tablet 10, the actual content picture and the green transparent layer are displayed on the same plane, but the green transparent layer displayed on top is closer to the lens than the content picture in imaging logic. Based on the above scenario, if the content picture in the interactive tablet 10 is a PPT, and the green transparent layer is displayed on top of the PPT, and there is a teacher conducting teaching activities in front of the interactive tablet 10, the teacher may be blocked between the lens and the screen of the interactive tablet 10 during the movement. At this time, in the picture captured by the camera 10, the main picture structure from the outside to the inside is the portrait, the transparent green screen and the PPT, among which the transparent green screen serves as the background of the picture, and the other parts serve as the foreground of the picture. After receiving the two-channel picture, the core purpose of the recording and broadcasting host 30 is to paste the foreground of the picture captured by the camera 10 onto the picture received from the interactive tablet 10, corresponding to Figure 3 In the picture shown, the portrait in the transparent green screen area is cut out and pasted on the picture received from the interactive tablet 10. The final picture can be sent to the remote end for live online teaching, or it can be recorded and saved to generate video teaching resources.

[0044] During the actual processing of two-channel images by the recording and broadcasting host 30, if the camera 20 is not placed in the right position and captures the area outside the screen of the interactive tablet 10, the image captured by the camera 20 will introduce excessive non-green screen pixels, affecting the effect and functional stability based on the green screen cutout. For the interactive tablet 10, the content image to be displayed on the screen during the teaching process may be a color similar to the green transparent mask, and the influence between these colors will also cause abnormal image processing. In addition, the camera 20 itself may also have insufficient image acquisition capabilities, such as poor white balance, poor single color shooting effect, low color reproduction, etc. The randomly matched recording and broadcasting host 30 and camera 20 cannot be optimized and solved one by one through targeted debugging. In addition, the ambient light during actual shooting will also affect the image captured by the camera 20.

[0045] The above situations may cause abnormal image processing in the recording and broadcasting host during the picture synthesis process. The synthesized image is not a true fusion of the teaching site picture and the interactive flat panel display picture, but unnecessary or erroneous temporary data generated in the processing process. These temporary data may be retained until the final comprehensive picture.

[0046] In view of the above technical problems, the present invention proposes an image processing method, and various embodiments of the present invention are described in detail below.

[0047] Figure 1 The method flow chart of an image processing method provided by an embodiment of the present invention is applied to an electronic device for processing relevant images of a teaching scene, and is implemented by an electronic device, such as a director host, so that the director host eliminates image processing anomalies during the process of processing and synthesizing multiple images, and the synthesized image is a true fusion of the teaching scene image and the interactive flat panel display image, and the actual collected image will not be replaced by temporary data generated during the processing. Figure 1 As shown, the image processing method includes steps S101 to S105:

[0048] Step S101: receiving a first channel of images and a second channel of images in real time.

[0049] The first picture is taken by the camera shooting the screen of the interactive tablet, and the second picture is the content picture currently displayed on the screen. The screen displays a green transparent mask on the upper layer of the content picture. If the interactive tablet has only one system module that provides an operating system, the content picture and the green transparent mask are generated by the same operating system. If the interactive tablet has multiple system modules that provide multiple operating systems, the content picture and the green transparent mask can be generated by the operating system corresponding to the content picture, or the content picture and the green transparent mask can be generated by different operating systems. For example, if the interactive tablet has a system module for Windows system and a system module for Android system, the display of the interactive tablet can be controlled by the Android system. Figure 3 The content screen of the PPT is generated by the Windows system. When the interactive flat panel displays the PPT, the system module of the Windows system sends the content screen to the system module of the Android system. The Android system adds a green transparent mask on the content screen and then controls the interactive flat panel to display the content screen (i.e., PPT) with the green transparent mask added. The recording and broadcasting host can transmit data to the interactive flat panel via an HDMI (High Definition Multimedia Interface) cable or other multimedia protocol connection cables, that is, the second channel of the picture can be sent from the interactive flat panel to the recording and broadcasting host via any of the various multimedia protocol connection cables. The camera and the recording and broadcasting host can use the same or different connection cables as those between the recording and broadcasting host and the interactive flat panel for data transmission.

[0050] Step S102: Recognize a rectangle in the first picture to obtain a rectangular area in the first picture.

[0051] The detection of rectangles in the first channel of the picture can be implemented based on Hough transform detection or using the rectangle detection interface provided in OpenCV. Before performing rectangle detection, the first channel of the picture can be corrected for color cast using white balance processing and other methods. When specifically identifying rectangles, the rectangle in the first channel of the picture after color cast correction is identified, and the corresponding rectangle area in the first channel of the picture after color cast correction is obtained. In the teaching scene, the largest rectangle in the first channel of the picture is generally the rectangle formed by the outer contour of the interactive flat panel. When multiple rectangles are identified during rectangle identification, the largest rectangle can be used as the rectangular area in the first channel of the picture, that is, the area corresponding to the screen of the interactive flat panel. In addition, considering that the green transparent mask displayed on the screen of the interactive flat panel has special color characteristics, the color inside the identified rectangle can also be used as a reference to confirm the final rectangular area. Through the identification of the rectangular area, the scope of foreground extraction can be constrained, and when there is an object with a similar screen display color outside the interactive flat panel in the teaching scene, the corresponding part of the object in the first channel of the picture is treated as the background, which avoids the loss of information in the first channel of the picture during the processing process and avoids the appearance of picture loopholes in the final comprehensive picture.

[0052] Step S103: Analyze the HSV color gamut parameters of the pixels in the rectangular area to confirm the green pixels in the rectangular area.

[0053] The rectangular area itself corresponds to the screen area that displays the green transparent mask as a whole. Even if the content displayed therein may deviate from the green color, it still has the characteristics of being affected by the green transparent mask as a whole. Based on this basic display rule, the HSV color gamut parameters of the pixels in the rectangular area are analyzed, and the actual green pixels are confirmed according to the overall display parameters. That is to say, what is actually to be confirmed in step S103 is not the pixels in the rectangular area that completely match the display parameters of the green pixels, but the pixels that meet the general rules of the green transparent mask display from the display rule. Even if this type of pixels is not a green pixel, it can be confirmed as a green pixel. For example Figure 3 The main color displayed in the PPT is blue. After adding a green transparent mask, the overall displayed color may deviate from the HSV color gamut parameters corresponding to green. However, the rectangular area as a whole should display a color close to the HSV color gamut parameters corresponding to green. Pixels corresponding to this color can be confirmed as green pixels.

[0054] HSV is a color model. In HSV model, hue H is measured by angle, ranging from 0° to 360°. Starting from red and counting in a counterclockwise direction, red is 0°, green is 120°, and blue is 240°. Their complementary colors are: yellow is 60°, cyan is 180°, and purple is 300°. Saturation S indicates the degree to which a color is close to a spectral color. A color can be regarded as the result of a mixture of a certain spectral color and white. The greater the proportion of the spectral color, the higher the degree to which the color is close to the spectral color, and the higher the saturation of the color. The higher the saturation, the deeper and brighter the color. The white light component of the spectral color is 0, and the saturation reaches the highest. The value range is usually 0% to 100%. The larger the value, the more saturated the color. Brightness V indicates the brightness of the color. For light source color, the brightness value is related to the brightness of the light source; for object color, this value is related to the transmittance or reflectance of the object. The value range is usually 0% (black) to 100% (white). Different colors have corresponding parameters in the HSV color mode. For the green pixels targeted by this solution, the display and acquisition of colors may deviate during the actual processing. Therefore, according to the statistical results, all pixels that meet the pixel distribution status are regarded as green pixels. In the specific implementation process, the analysis of the HSV color gamut parameters of the pixels in the rectangular area can be global analysis or local analysis.

[0055] For global analysis, the rectangular area is used as the target statistical area, and the HSV color gamut parameters of the pixels in the target statistical area are counted to obtain the HSV distribution results of the pixels. The display area of ​​the green transparent mask covers the entire rectangular area, and the statistics of the entire rectangular area are based on comprehensive information.

[0056] For local analysis, the preset area is used as the target statistical area, and the HSV color gamut parameters of the sample pixels in the target statistical area are counted to obtain the HSV distribution results of the sample pixels. The preset area is the area corresponding to the preset height ratio of the upper part of the rectangular area. In the teaching scenario, the interactive tablet has a large size and is also installed at a higher height. From the perspective of the camera, teachers usually do not block the upper part of the interactive tablet when conducting teaching activities. Corresponding to the rectangular area, it is the upper part. Therefore, when performing HSV color gamut parameter analysis, the area corresponding to the preset height ratio of the upper part of the rectangular area can be taken. For example, the upper part of the rectangular area occupies one-third or one-quarter of the height of the entire rectangular area. In this area, it is basically the display area corresponding to the green transparent mask, and the pixel statistics are more targeted.

[0057] Regardless of whether it is a global analysis or a local analysis, after obtaining the HSV distribution results in the corresponding area, the HSV parameter range of the standard green is corrected according to the HSV distribution results to obtain the reference HSV parameter range; the pixels in the rectangular area whose HSV color gamut parameters are within the reference HSV parameter range are confirmed as green pixels. The initial statistical results are based on the HSV parameter range of the standard green as a reference, and the number of pixels in different HSV parameter ranges is counted according to the HSV color gamut parameters of the pixels. The specific correction process is to adjust the HSV parameter range to ensure that the appropriate proportion of pixels are in the range corresponding to green. In this way, it is possible to adapt to various display changes, acquisition errors or other factors that cause imaging deviations in the first-channel image.

[0058] In the calibration process based on the HSV parameter range of standard green, the HSV parameter range of standard green is used as a reference for stepwise adjustment, and the distribution of pixels in the target statistical area after each adjustment is counted; when the distribution of pixels in the target statistical area within and outside the adjusted HSV parameter range satisfies the normal distribution, the HSV parameter range is confirmed to be the reference HSV parameter range. Table 1 is a reference table of HSV parameter ranges for standard pure colors, in which the HSV parameter range corresponding to green is defined as the HSV parameter range of standard green. In the calibration process, a temporary HSV parameter range can be obtained by gradually fine-tuning the HSV parameter range of standard green until the pixels falling into the temporary HSV parameter range satisfy the normal distribution, and the temporary HSV parameter range is confirmed as the reference HSV parameter range. It should be noted that in the calibration process, multiple temporary HSV parameter ranges may correspond to the normal distribution, but only the narrowest range is selected as the reference HSV parameter range. Moreover, the calibration process should meet the general rules of color deviation. For example, if the HSV color gamut parameter corresponding to a certain pixel is obviously the HSV parameter range corresponding to purple, the temporary HSV parameter range should not be expanded unlimitedly at this time.

[0059]

[0060] Table 1 HSV parameter range reference table for standard pure colors

[0061] For example, if the color of a target statistical area is lighter green, then after correction based on H(35,77) in the HSV parameter range of standard green, H(33,76) in the reference HSV parameter range may be obtained. For another example, if the color of a target statistical area is darker green, then after correction based on H(35,77) in the HSV parameter range of standard green, H(38,82) in the reference HSV parameter range may be obtained. In addition, if the color of a target statistical area is close to skin color, if the quasi-green HSV parameter range is adjusted without limit, the reference HSV parameter range may be expanded to the HSV parameter range corresponding to yellow, which obviously does not conform to the law of color deviation. At this time, an error message may be prompted and the status of all hardware must be manually confirmed.

[0062] Step S104: binarizing the rectangular area according to the confirmation result of the green pixels to obtain a black and white image.

[0063] When binarization is performed, the green pixels in the rectangular area are assigned a value of 0, and the remaining pixels in the rectangular area are assigned a value of 1. Of course, the assignment can also be reversed, but the subsequent synthesis process must be adjusted accordingly. Both assignment methods can produce black and white images. Before the final image synthesis, the black and white image can be filtered first, so that the foreground and background in the final synthesized comprehensive image are more integrated and the transition of the fused edge is more natural.

[0064] Step S105: synthesize the pixels of the second picture, the black-and-white picture and the rectangular area in the first picture to obtain a composite picture.

[0065] Because only the rectangular area is processed in this scheme, the specific synthesis process only integrates the pixels in the rectangular area. If the binarization process assigns the green pixels in the rectangular area to 0 and the remaining pixels in the rectangular area to 1, then the new image value of each pixel in the rectangular area in the integrated picture can be calculated as follows:

[0066] New image value = second image value*(black and white image value) / 255+first image value*(1-(black and white image value) / 255).

[0067] This synthesis method ensures that the image outside the rectangular area, that is, the image outside the screen of the interactive flat panel in the first image, will not change, and the second image is accurately replaced in the screen area of ​​the interactive flat panel, and the final comprehensive image is more consistent with the original data. The final comprehensive image can be sent as a live broadcast image for real-time online teaching, or saved as a video teaching resource, without specific restrictions.

[0068] The above processing can refer to Figure 4-Figure 9Further highlight the advantages of this solution compared with the existing technology. Figure 4 A schematic diagram of a first channel of images in an image processing method provided by an embodiment of the present invention, Figure 5 The schematic diagram of the second screen in the image processing method provided by the embodiment of the present invention is actually the source data for image processing in the prior art. In the first screen 21, the screen image 211 of the interactive tablet, the teacher's portrait 212, the wall clock 213 hanging next to the interactive tablet, and the globe 214 placed on the podium are captured, wherein the screen image 211 displays a PPT corresponding to a geography class, the upper layer of the PPT has a green transparent mask displayed on the top, the clock face of the wall clock 213 is green, and the part of the globe 214 near the North Pole is painted green; the second screen 11 corresponds to the content screen of the PPT obtained from the interactive tablet.

[0069] based on Figure 4 and Figure 5 According to the prior art, firstly, the foreground of the first image 21 is extracted directly based on the HSV parameter range of the standard green. At this time, the green areas on the screen image 211, the wall clock 213 and the globe 214 are processed as green screens. The corresponding processing results are as follows: Figure 6 As shown, at this time, the content on the screen, the content in the wall clock 213 and the green area on the globe 214 are no longer there. Figure 6 The results are as follows Figure 7 As shown, it is obvious that image information is lost. If the screen image 211 is processed inaccurately due to the above-mentioned influence, problems will also occur in the corresponding area of ​​the interactive flat panel in the final integrated image.

[0070] based on Figure 4 and Figure 5 According to this solution, the first image 21 is firstly subjected to rectangular recognition to obtain a rectangular area, and only the rectangular area is subjected to statistical foreground extraction. At this time, the wall clock 213 and the globe 214 are not within the processing range and are not affected. The processing of the screen image 211 is based on statistics, is not affected by external interference, and can achieve accurate extraction. The corresponding processing result is as follows: Figure 8 As shown, at this time, only the captured screen image 211 is processed, and the second image 11 is synthesized into Figure 8 The results are as follows Fig. 9 As shown, in the integrated picture, the second picture 11 is equivalent to being accurately replaced according to the shooting effect in the first picture 21, thereby ensuring the picture quality of the integrated picture and avoiding the loss of information in the original picture.

[0071] The above-mentioned image processing method receives the first channel of images and the second channel of images in real time, wherein the first channel of images is obtained by shooting the screen of the interactive flat panel by a camera, and the second channel of images is the content image currently displayed on the screen, and the screen displays a green transparent mask on the upper layer of the content image; identifies the rectangle in the first channel of images to obtain the rectangular area in the first channel of images; analyzes the HSV color gamut parameters of the pixels in the rectangular area to confirm the green pixels in the rectangular area; binarizes the rectangular area according to the confirmation result of the green pixels to obtain a black and white image; synthesizes the pixels of the second channel of images, the black and white image and the rectangular area in the first channel of images to obtain a comprehensive image. By performing rectangular recognition on the recorded class images captured by the camera in the teaching scene, the area where the interactive flat panel is located is confirmed, and the green pixels are confirmed after statistical analysis of the pixels in the area where the interactive flat panel is located, and binarizes the multiple channels of images according to the confirmation result of the green pixels within the range of the interactive flat panel to obtain a comprehensive image. The abnormality of image processing in the synthesis process of the teaching image is eliminated, and the synthesized image is a true fusion of the teaching scene image and the interactive flat panel display image, and the actual collected image will not be replaced by temporary data generated during the processing.

[0072] Fig.10 FIG. 1 is a schematic diagram of the structure of an image processing device provided by an embodiment of the present invention. Fig.10 As shown, the image processing device includes a picture receiving unit 210, a rectangle identifying unit 220, a pixel confirming unit 230, a binarizing unit 240 and a picture synthesizing unit 250.

[0073] Among them, the picture receiving unit 210 is used to receive the first picture and the second picture in real time, the first picture is obtained by shooting the screen of the interactive flat panel with a camera, and the second picture is the content picture currently displayed on the screen, and the screen displays a green transparent mask on the upper layer of the content picture; the rectangle recognition unit 220 is used to identify the rectangle in the first picture and obtain the rectangular area in the first picture; the pixel confirmation unit 230 is used to analyze the HSV color gamut parameters of the pixels in the rectangular area and confirm the green pixels in the rectangular area; the binarization unit 240 is used to binarize the rectangular area according to the confirmation result of the green pixels to obtain a black and white picture; the picture synthesis unit 250 is used to synthesize the pixels of the second picture, the black and white picture and the rectangular area in the first picture to obtain a comprehensive picture.

[0074] Based on the above embodiment, the pixel confirmation unit 230 includes:

[0075] The first statistical module is used to take the rectangular area as the target statistical area, count the HSV color gamut parameters of the pixels in the target statistical area, and obtain the HSV distribution results of the pixels;

[0076] A parameter correction module is used to correct the HSV parameter range of the standard green according to the HSV distribution result to obtain a reference HSV parameter range;

[0077] The parameter comparison module is used to confirm that the pixels whose HSV color gamut parameters in the rectangular area are within the reference HSV parameter range are green pixels.

[0078] Based on the above embodiment, the pixel confirmation unit 230 includes:

[0079] The second statistical module is used to take a preset area as a target statistical area, count the HSV color gamut parameters of the sample pixels in the target statistical area, and obtain the HSV distribution results of the sample pixels. The preset area is an area corresponding to a preset height ratio at the upper part of the rectangular area;

[0080] A parameter correction module is used to correct the HSV parameter range of the standard green according to the HSV distribution result to obtain a reference HSV parameter range;

[0081] The parameter comparison module is used to confirm that the pixels whose HSV color gamut parameters in the rectangular area are within the reference HSV parameter range are green pixels.

[0082] Based on the above embodiment, the parameter correction module includes:

[0083] The parameter adjustment submodule is used to gradually adjust the HSV parameter range of the standard green color and count the distribution of pixels in the target statistical area after each adjustment;

[0084] The distribution confirmation submodule is used to confirm that the HSV parameter range is a reference HSV parameter range when the distribution of pixels in the target statistical area within and outside the adjusted HSV parameter range satisfies the normal distribution.

[0085] Based on the above embodiment, the binarization unit 240 includes:

[0086] The pixel assignment module is used to assign the green pixels in the rectangular area to 0 and the remaining pixels in the rectangular area to 1.

[0087] On the basis of the above embodiment, it also includes a filtering unit, which is used to perform filtering on the black and white image after binarizing the rectangular area according to the confirmation result of the green pixels to obtain a black and white image, before synthesizing the pixels of the rectangular area in the second image, the black and white image and the first image.

[0088] Based on the above embodiment, a color cast correction unit is further included, which is used to identify the rectangle in the first picture, and before obtaining the rectangular area in the first picture, perform color cast correction on the first picture;

[0089] Accordingly, the rectangle recognition unit 220 includes:

[0090] The first rectangle recognition module is used to recognize a rectangle in the first channel of images after color cast correction, and obtain a rectangular area in the first channel of images after color cast correction.

[0091] The image processing device provided in the embodiment of the present invention is included in an electronic device and can be used to execute the corresponding image processing method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0092] It is worth noting that in the embodiment of the above-mentioned image processing device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0093] Fig.11 FIG. 1 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Fig.11 As shown, the electronic device includes a processor 310 and a memory 320, and may also include an input device 330, an output device 340 and a communication device 350; the number of the processor 310 in the electronic device may be one or more, Fig.11 A processor 310 is taken as an example; the processor 310, the memory 320, the input device 330, the output device 340 and the communication device 350 in the electronic device can be connected via a bus or other means. Fig.11 The example of connecting through bus is taken in the following.

[0094] The memory 320 is a computer-readable storage medium that can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the image processing method in the embodiment of the present invention. The processor 310 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the memory 320, that is, implements the above-mentioned image processing method.

[0095] The memory 320 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 320 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 320 may further include a memory remotely arranged relative to the processor 310, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0096] The input device 330 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device. The output device 340 may include a display device such as a display screen.

[0097] The electronic device mentioned above includes an image processing device, which can be used to execute any image processing method and has corresponding functions and beneficial effects.

[0098] An embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it is used to perform relevant operations in the image processing method provided in any embodiment of the present application, and has corresponding functions and beneficial effects.

[0099] Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products.

[0100] Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions in the process. Figure 1 A process or multiple processes and / or boxes Figure 1These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0101] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory. The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0102] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0103] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0104] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. An image processing method, It is characterized in that include: receiving a first picture and a second picture in real time, wherein the first picture is obtained by shooting the screen of the interactive flat panel by a camera, and the second picture is a content picture currently displayed on the screen, and the screen displays a green transparent mask on the upper layer of the content picture; Identify a rectangle in the first picture to obtain a rectangular area in the first picture; Analyze the HSV color gamut parameters of the pixels in the rectangular area to identify the green pixels in the rectangular area; Binarization is performed on the rectangular area according to the confirmation result of the green pixel to obtain a black and white image; The pixels of the rectangular area in the second picture, the black-and-white picture and the first picture are synthesized to obtain a composite picture; The step of analyzing the HSV color gamut parameters of the pixels in the rectangular area to confirm the green pixels in the rectangular area includes: Taking the rectangular area or the preset area as the target statistical area, counting the HSV color gamut parameters of the pixels in the target statistical area to obtain the HSV distribution result of the pixels, wherein the preset area is an area corresponding to a preset height ratio above the rectangular area; Correcting the HSV parameter range of the standard green according to the HSV distribution result to obtain a reference HSV parameter range; Pixels in the rectangular area whose HSV color gamut parameters are within the reference HSV parameter range are confirmed as green pixels.

2. The image processing method according to claim 1, It is characterized in that The HSV parameter range of the standard green is corrected according to the HSV distribution result to obtain a reference HSV parameter range, including: The HSV parameter range of the standard green is used as a reference for stepwise adjustment, and the distribution of pixels in the target statistical area after each adjustment is counted; When the distribution of pixels in the target statistical area within and outside the adjusted HSV parameter range satisfies a normal distribution, the HSV parameter range is confirmed to be a reference HSV parameter range.

3. The image processing method according to claim 1, It is characterized in that The binarization process is performed on the rectangular area according to the confirmation result of the green pixel to obtain a black and white image, comprising: The green pixels in the rectangular area are assigned a value of 0, and the remaining pixels in the rectangular area are assigned a value of 1.

4. The image processing method according to claim 1, It is characterized in that After binarizing the rectangular area to obtain a black-and-white picture according to the confirmation result of the green pixel, and before synthesizing the pixels of the rectangular area in the second picture, the black-and-white picture and the first picture, the method further includes: The black and white picture is subjected to filtering processing.

5. The image processing method according to claim 4, It is characterized in that Before identifying the rectangle in the first picture and obtaining the rectangular area in the first picture, the method further includes: Correcting the color cast of the first-channel image; Correspondingly, the identifying a rectangle in the first picture to obtain a rectangular area in the first picture includes: Rectangles in the first channel of images after color cast correction are identified to obtain rectangular areas in the first channel of images after color cast correction.

6. An image processing device, It is characterized in that include: A picture receiving unit is used to receive a first picture and a second picture in real time, wherein the first picture is obtained by shooting the screen of the interactive flat panel with a camera, and the second picture is a content picture currently displayed on the screen, and the screen displays a green transparent mask on the upper layer of the content picture; A rectangle recognition unit, used for recognizing a rectangle in the first picture, and obtaining a rectangular area in the first picture; A pixel confirmation unit, used to analyze the HSV color gamut parameters of the pixels in the rectangular area to confirm the green pixels in the rectangular area; A binarization unit, used for binarizing the rectangular area according to the confirmation result of the green pixel to obtain a black and white image; A picture synthesis unit, used for synthesizing the pixels of the rectangular area in the second picture, the black-and-white picture and the first picture to obtain a composite picture; The HSV color gamut parameters of the pixels in the rectangular area are analyzed to confirm the green pixels in the rectangular area, including: Taking the rectangular area or the preset area as the target statistical area, counting the HSV color gamut parameters of the pixels in the target statistical area to obtain the HSV distribution result of the pixels, wherein the preset area is an area corresponding to a preset height ratio above the rectangular area; Correcting the HSV parameter range of the standard green according to the HSV distribution result to obtain a reference HSV parameter range; Pixels in the rectangular area whose HSV color gamut parameters are within the reference HSV parameter range are confirmed as green pixels.

7. An electronic device, It is characterized in that include: one or more processors; a memory for storing one or more computer programs; When the one or more computer programs are executed by the one or more processors, the electronic device implements the image processing method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 5 is implemented.