Apparatus and method for enhancing whiteboard images
An image processing system enhances whiteboard visibility in remote meetings by applying keystone correction, noise filtering, and color adjustment to improve the clarity of whiteboard images captured within a wide conference view.
Patent Information
- Application Number
- JP2024537162
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-12-20
- Filing Date
- 2022-12-19
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2042-12-19
AI Technical Summary
Existing remote meeting solutions struggle to provide clear visibility of whiteboard content in conference rooms due to noise, lighting effects, and trapezoidation issues, especially when using a single camera for a wide field of view, without the need for additional dedicated whiteboard cameras.
An image processing system that captures high-resolution video of a writing surface, applies keystone correction, generates a binary mask to filter noise, adjusts saturation and brightness, and performs alpha blending to enhance the whiteboard image, transmitting it separately from the full room view.
Enhances the readability of whiteboard images in remote meetings by reducing noise and lighting artifacts, ensuring clear visibility of the whiteboard content without requiring additional dedicated cameras.
Smart Images

Figure 2024545461000001_ABST
Abstract
Description
[Technical field]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 291650, filed December 20, 2021, which is incorporated herein by reference in its entirety.
[0002] The present disclosure relates to image processing techniques. [Background technology]
[0003] Meeting rooms and conference halls usually have writing surfaces on which meeting participants can write with a writing implement, allowing participants to see information that one or more participants consider important. One such writing surface is the whiteboard, where participants can write on it, for example with erasable whiteboard markers, which is a valuable collaboration tool for participants to use.
[0004] In some cases, it is desirable to enable people who cannot physically attend a conference room or meeting venue to participate remotely. There are multiple remote online meeting solutions that can realize this process. Furthermore, during these meetings where one or more participants are participating remotely, it is desirable for the participants to be able to see what is written on the writing surface so that they feel as if they are part of the collaboration. To enable this, there are tools such as electronic whiteboards that digitize the written information, but they are expensive and difficult to integrate into IT networks. There are also other mechanisms such as image capture systems where an image capture device can capture an image of the writing surface and transmit that image to the remote user. However, in remote meetings where one camera is used to view the conference room, it is difficult for the remote user to see or read the contents of the whiteboard that is usually displayed by the single camera. A possible solution to this problem is to add a dedicated camera focused on the whiteboard, but this incurs additional costs for the conference room set up for remote meetings. The system and method according to the present disclosure improves on the above identified shortcomings. Summary of the Invention
[0005] An image processing device having one or more processors and one or more memories for holding instructions, which, when executed, configure the one or more processors to receive video captured from a camera capturing an image of a conference room, extract a predetermined area of the video and store it as extracted image data, perform a first image correction process on the extracted image data to generate a first corrected image, correct noise and generate a binary mask of the first corrected image, generate a filtered image based on the binary mask of the first corrected image and the first corrected image, perform a second image correction process on the filtered image to generate a second corrected image, and perform a blending process to combine the second corrected image and the first corrected image to generate a final corrected image.
[0006] In another embodiment, the method is configured to store in a memory a predetermined number of masks of the first corrected image and calculate an average of the stored predetermined number of masks to generate an average mask, and the filtered image is generated using the average mask.
[0007] These and other objects, features and advantages of the present disclosure will become apparent from the following detailed description of illustrative embodiments thereof, taken in conjunction with the accompanying drawings and claims. [Brief description of the drawings]
[0008] [Figure 1] FIG. 1 shows a system configuration of the present disclosure.
[0009] [Diagram 2] FIG. 2 shows the writing surface of the present disclosure.
[0010] [Figure 3A] 3A-3G show the image processing algorithm in this disclosure. [Figure 3B] 3A-3G show the image processing algorithm in this disclosure. [Figure 3C] 3A-3G show the image processing algorithm in this disclosure. [Figure 3D] 3A-3G show the image processing algorithm in this disclosure. [Figure 3E] 3A-3G show the image processing algorithm in this disclosure. [Figure 3F] 3A-3G show the image processing algorithm in this disclosure. [Figure 3G] 3A-3G show the image processing algorithm in this disclosure.
[0011] [Figure 4] FIG. 4 shows a hardware configuration in this disclosure.
[0012] [Diagram 5] FIG. 5 illustrates an algorithm that is executed as part of the image processing algorithm of FIGS. 3A-3G.
[0013] Throughout the figures, the same reference numerals and characters, unless otherwise noted, are used to denote like features, elements, components or portions of the illustrated embodiments. Moreover, while the subject disclosure will now be described in detail with reference to the figures, it is done so in connection with the exemplary embodiments. It is intended that changes and modifications can be made to the exemplary embodiments described without departing from the true scope and spirit of the subject disclosure, as defined by the appended claims. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0014] Detailed Description Throughout the figures, the same reference numerals and characters, unless otherwise noted, are used to denote like features, elements, components or portions of the illustrated embodiments. Moreover, while the subject disclosure will now be described in detail with reference to the figures, it is done so in connection with the exemplary embodiments. It is intended that changes and modifications can be made to the exemplary embodiments described without departing from the true scope and spirit of the subject disclosure, as defined by the appended claims.
[0015] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. Note that the following embodiment is merely an example for carrying out the present invention, and can be appropriately modified or changed according to the individual configuration and various conditions of the device to which the present invention is applied. Therefore, the present invention is not limited to the following embodiment, and according to the drawings and the embodiment described below, the described embodiment can be applied and carried out in situations other than those exemplified below.
[0016] In an online conference environment where one or more participants in a conference room use a writing surface such as a whiteboard, it is important that information written on the writing surface can be clearly seen by people participating in the conference from a remote location, i.e., online. One way to achieve this is to use an image capture system, such as a camera, capable of capturing high-resolution video image data of the writing surface, so that the resulting image can be communicated over a network to the remote participants for display on a computing device, such as a laptop, tablet, and / or phone. However, in an exemplary environment, such as that shown in FIG. 1 below, where there is a single image capture device configured to capture a wide field of view of the entire conference room space, it can be particularly difficult to obtain an image of the writing surface that is of sufficient quality for the remote conference participants. A whiteboard image extracted from an image captured by a camera that captures the entire room is very difficult to read due to noise, lighting effects, and keystone problems. To improve the readability of a whiteboard captured in such an environment (without an additional camera dedicated to the whiteboard), the systems and methods described below advantageously obtain a high-quality image of the writing surface extracted from a wide field of view of the conference area and transmitted to the remote users, such that an image with an enhanced writing surface can be transmitted to the remote users using a data stream separate from the data stream containing the full field of view of the conference environment. In one embodiment, the transmission of the enhanced whiteboard image may occur over a transmission channel that is different from the channel that communicates the video image of the entire conference area.
[0017] FIG. 1 illustrates a system configuration according to an exemplary embodiment. The system according to the present disclosure is located in a conference room 101. The conference room 101 may be a conference room or the like. However, this is not limited to one dedicated room. The system may be located in any defined area, provided that it includes the components illustrated in FIG. 1 and operates as described below. The conference room 101 includes a participant area 105 where one or more participants may sit or gather to exchange information. As illustrated herein, the participant area includes a conference table and chairs for two participants of the in-room conference. This is illustrated for illustrative purposes only and may be set up in any manner to allow the in-room participants to gather and exchange information. The conference room 101 also includes a writing surface 106 where one or more participants in the room may write information for other participants to view. In one embodiment, the writing surface is a whiteboard where writing may be done using an erasable marker. The system further comprises an imaging device 102 (e.g., a camera configured to sequentially capture video image data or a series of still images where, when each still image is played back, the image data appears as if it were video image data) arranged and positioned at a predetermined location within the conference room such that the image data captured by the imaging device 102 represents a predetermined field of view 104 (shown as the area between the dashed lines in FIG. 1) of the conference room. In one embodiment, the predetermined field of view includes a participant area 105 and a writing surface 106. The imaging device is configured to capture video data of the conference room in real time by generating an all-room view of everything within the predetermined field of view 104. This real-time captured video data is referred to as a room data stream.
[0018] The imaging device 102 is controlled by the controller 110 to capture the room data stream. The controller 110 is a computing device that may be located locally in the conference room or as a server in communication with the imaging device 102. The controller 110 is hardware as described below with reference to FIG. 4. The controller executes one or more sets of instructions stored in a memory to perform the operations and tasks described below. In one embodiment, the controller 110 controls the imaging device 102 to capture room video image data representing the field of view 104 of FIG. 1 during a conference between the room participants and one or more remote participants connected to and viewing the video data being captured by the imaging device 102. In accordance with the present disclosure, an algorithm is implemented to highlight a predetermined area of the room video data being captured in real time. The highlighted predetermined area includes the writing surface and the surrounding area.
[0019] The controller 110 is further configured to transmit the video image data representing the real-time indoor video over a communication network 120 to which at least one remote client using a computing device 130 is connected. In one embodiment, the communication network 120 is a wide area network (WAN) or a local area network (LAN) further connected to a WAN such as the Internet. The remote client device 130 can selectively access the indoor video data using a conference application that controls an online conference between participants in the room 101 and the at least one remote client device 130. The remote client device 130 may use a defined access link to at least obtain the indoor video data captured by the imaging device 102 via the controller 110. In one embodiment, the access link allows the at least one remote client device 130 to obtain both the indoor video data and a predefined region of the indoor video data enhanced according to an image processing algorithm described below.
[0020] In an exemplary operation, the present disclosure advantageously enhances a writing surface 106 (e.g., a whiteboard image) by selecting a writing surface area where a first image correction is to be performed and generating and storing in memory a first corrected image. In one embodiment, the first image correction is a keystone correction. A mask is then calculated based on the first corrected image and stored in a mask queue in memory. The mask queue is set to store a (configurable) predetermined number of calculated masks. When the mask queue is full, the oldest mask is removed and a new mask is added to the end of the queue. The calculated mask image used in performing the enhancement of the remaining images of the writing surface is calculated based on all masks in the mask queue at a given time. The purpose of this process is to reduce noise / compression induced variations across successive frames. Finally, a mask is applied to the first corrected image to remove unwanted artifacts to generate a second corrected image, which is further color enhanced to generate a third corrected image. This algorithm is realized by one or more processors (CPU 501) of the control device 103 reading and executing a predetermined program stored in the memory (ROM 503), and will be described in more detail below.
[0021] An exemplary image processing algorithm is described below that improves the appearance of a predetermined region of the video data stream extracted from the video data stream and is executed by one or more processors executing a stored set of instructions (e.g., a program). In one embodiment, the predetermined region includes a writing surface. The exemplary algorithm includes obtaining information representing the locations of predefined corners of the writing surface to be modified, as shown below. These corner locations can be input by user selection using a user interface for the user to select the corner locations. In another embodiment, the writing surface (whiteboard) is automatically detected using known whiteboard detection processes. For example, a user can look at an image of a room showing the field of view 104 and identify points that represent the four corners of the whiteboard. This can be done using an input device such as a mouse, or via a touch screen if the device displaying the video data is capable of accepting touch input.
[0022] A first image correction process is then performed on the data extracted from the above identified area, which is a keystone correction for the whiteboard area based on the defined four corners to calculate the smallest rectangle that contains the four corners, as shown in FIG.
[0023] The perspective transformation is computed using the four user-defined corners as sources and the four corners of the computed rectangle as targets. The perspective transformation is computed from four pairs of corresponding points, and the function computes the 3x3 map_matrix of the perspective transformation in equation (1). TIFF2024545461000002.tif1886Here, src represents the four defined corners, and dst represents the four corners of the smallest rectangle, as shown in the following formula. dst(i) = (x' i xy' i ), src(i) = (x i , y i ), i = 1,2,3,4 …(2) The algorithm obtains the coefficients of map_matrix(Cij) which are calculated according to the algorithm shown in Figure 5. Once the map_matrix values are obtained, a perspective transformation using the inverse of the map_matrix transformation calculated on the source image of the whiteboard is applied to obtain a keystone corrected whiteboard image (KC image). The perspective transformation transforms the source image using the matrix shown in Equation 3. TIFF2024545461000003.tif10100
[0024] A binary mask is created to filter out noise / lighting artifacts, and the threshold is the average of the neighborhood. The KC image is converted to grayscale, and a binary mask is created by applying an adaptive threshold operation to the grayscale image. Adaptive thresholding is a method of calculating the threshold for a smaller region, so the threshold for different regions may be different). In one embodiment, the threshold is the average of the neighborhood, and pixel values above the threshold are set to 1 and pixel values below the threshold are set to 0 (e.g., the neighborhood has a block / neighborhood size of 21). The created mask is added to a queue of masks.
[0025] An updated binary mask is generated using the masks in the mask queue, whereby the pixel value for each pixel in the updated binary mask is determined, and if the sum of the values for that pixel in all masks in the queue is greater than or equal to the number of masks in the queue divided by 2, then the pixel value in the updated mask is set to 1 (valid), otherwise the pixel value is set to 0. The calculations performed for each pixel in the updated mask are performed using Equation 4. TIFF2024545461000004.tif1371 where N is the number of masks in the queue, p q xy is the value of each pixel (x,y) of mask q, which is either 0 or 1, xy is the value of pixel (x,y) in the final mask.
[0026] Next, the saturation and lightness are adjusted based on the user settings to adjust the saturation / lightness of the colors more or less. This adjustment is performed for each pixel of the KC image while applying the updated binary mask. To do this, the color space of the image is converted from RGB to HSV to adjust the saturation and lightness values. After the conversion, for every pixel of the HSV image, if the pixel's mask value is 1 (enabled), the value of pixel S is updated using the configured saturation setting and the value of pixel V is updated using the configured lightness setting. On the other hand, if the pixel's mask value is 0 (disabled), the pixel is set to the white HSV value (0,0,255). Once the settings for each pixel of the HSV image have been applied, the HSV image is converted back to the RGB color space as an updated RGB image. An alpha blend is applied to the updated RGB image with the KC image (background). The alpha value of the blending is configurable, which has the advantage that it allows you to control how strongly the unfiltered frame is merged into the filtered frame. For example, if the alpha value is set to 0, only the filtered frame is displayed, and if the alpha value is set to 1, only the unfiltered frame is displayed. This allows the user to set the final blending operation to be performed. The calculation shown in Equation 5 is u ) and KC image (p kc ) to obtain the result image (p r ) to obtain the resulting updated RGB image. TIFF2024545461000005.tif786
[0027] 3A-3G are exemplary flow diagrams of the above-mentioned algorithm, described with respect to a configuration performing certain operations in FIG. 1. The description and examples provided herein implement the above-mentioned algorithm. At 302, the controller 110 causes the imaging device 102 to capture video image data, which captures an image representing the field of view 104. The entire field of view 104 is captured, including the participant area 105 and the writing surface 106, but the image 302 in FIG. 3A shows an area surrounding the writing surface 106. As shown here, the writing surface is identified based on four points preselected by the user and based on the camera position. Thus, the writing surface area is predefined and set in all cases where the image capture device is capturing the field of view 104 of the conference room 101. In other embodiments, a writing surface detection process can be performed, whereby the imaging device 102 can be moved to various positions in the conference room, and the writing surface area can be detected and processed as described below.
[0028] At 303, an area of the writing surface is extracted from the image 302. The extracted writing surface area is defined using the points identified in the image 302, with a point located at each corner of the writing surface. A first image correction process 303 is performed on the extracted writing surface area to generate a first corrected image 304 of FIG. 3B. In one embodiment, the first correction process is a keystone correction process that produces a clear rectangular image. The first corrected image 304 of FIG. 3B is stored in memory and used as follows: A copy of the first corrected image 304 is masked at 305 using an adaptive thresholding technique to produce a binary mask image 306 shown in FIG. 3C. Because the image processing described here is performed using video data, additional image frames are provided over time and masking is performed on each extracted writing surface area in each frame. For each received image frame, a new mask 306 is generated and the set of masks is added to a mask queue at 307 of FIG. 3D. The mask queue represents a set of binary masks at each point in time based on the video frame rate. The mask queue is pre-configured to store a predetermined number of masks, which can be averaged to generate an average mask image at 308 in FIG. 3D. At 309 in FIG. 3D, the average mask image is filtered along with the first correction 304 shown in FIG. 3A, where a copy of the image is left in memory. The filtering at 309 generates a second corrected image 310 in FIG. 3E, which is based on combining the first corrected image with the average mask image obtained from the mask queue.
[0029] The second corrected image 310 of FIG. 3E is subjected to a second image correction process at 311 to correct the color and brightness of the second corrected image 310. The second image correction process 311 results in a third corrected image 312 of FIG. 3F that has been keystone corrected and filtered to remove light glare and other artifacts from the original image 302. The third corrected image is provided at 313 of FIG. 3G where an alpha blending process is performed using the third corrected image 312 and a copy of the first corrected image 304 of FIG. 3B stored in memory. The alpha blending process 313 produces a final corrected image 314. The final corrected image 314 is acquired by the controller 110 and transmitted over the network 120 for receipt and display at the remote client computing device 130.
[0030] In operation, the algorithm is executed in real time as new video image data representing the room video stream is received by the control device 110. During an online conference, the room video stream is transmitted in a first format over a first communication path (e.g., channel) and displayed on the display of the remote computing device. The extracted area representing the writing surface is not transmitted in the same first format. Once the algorithm extracts data from the video frame, the highlighted writing surface area is transmitted in a second format. In one example, the second format is still image data transmitted at a certain rate such that the transmitted highlighted writing surface area appears to be a video, but is actually a series of continuously processed still images transmitted to the remote client device over a second, different communication path (channel). This provides the advantage that the control device can simultaneously display both the live video data captured by the image capture device and the enhanced area of video data generated according to the algorithm described herein. The algorithm creates a binary mask based on the keystone corrected image (based on the number of previous masks) to filter out noise, applies the mask to the original image, then performs saturation and brightness enhancement, and alpha blends the keystone corrected image with the enhanced image to produce the final result.
[0031] FIG. 4 shows hardware representing a control device 110 that can be used in implementing the above disclosure. The device includes a CPU 401, a RAM 402, a ROM 403, an input section, an external interface, and an output section. The CPU 401 uses a computer program (a series of one or more stored instructions executable by the CPU 401) and data stored in the RAM 402 or the ROM 403 to control the device. Here, the device includes one or more dedicated hardware or a graphics processing unit (GPU) different from the CPU 401, and the GPU or dedicated hardware can execute a part of the processing by the CPU 401. Examples of dedicated hardware include an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), etc. The RAM 402 temporarily stores computer programs and data read from the ROM 403, data supplied from the outside via an external interface, etc. The ROM 403 stores computer programs and data that do not need to be updated and can control the basic operation of the device. The input unit is composed of, for example, a joystick, a jog dial, a touch panel, a keyboard, a mouse, etc., and receives user operations and inputs various instructions to the CPU 401. The external interface communicates with external devices such as a PC, a smartphone, a camera, etc. The communication with the external device may be performed by wire using a local area network (LAN) cable, a serial digital interface (SDI) cable, a WIFI connection, etc., or may be performed wirelessly via an antenna. The output unit is composed of, for example, a display unit such as a display and an audio output unit such as a speaker, and displays a graphical user interface (GUI) and outputs guide sounds so that the user can operate the device as necessary.
[0032] The scope of the present invention includes non-transitory computer-readable media that store instructions that, when executed by one or more processors, cause the one or more processors to perform one or more embodiments of the present invention described herein. Computer-readable media include, for example, hard disks, floppy disks, magneto-optical disks (MO), compact disk read-only memory (CD-ROM), compact disk recordable (CD-R), CD-rewritable (CD-RW), digital versatile disk ROM (DVD-ROM), DVD-RAM, DVD-RW, DVD+RW, magnetic tape, non-volatile memory card, and ROM. Computer-executable instructions can also be provided to the computer-readable storage medium by being downloaded over a network.
[0033] Use of the terms "a" and "an" and "the" and similar referents in the context of this disclosure (particularly in the context of the claims below) describing one or more aspects of the invention should be construed to include both the singular and the plural, unless otherwise noted herein or clearly contradicted by context. The terms "comprises," "having," "including," and "containing" should be construed as open-ended terms (i.e., meaning "including, but not limited to"), unless otherwise noted. Recitation of ranges of values herein is intended merely as a shorthand method of individually referring to each individual value falling within the range, unless otherwise noted herein, and each individual value is incorporated into the specification as if it were individually set forth herein. All methods described herein can be performed in any suitable order, unless otherwise noted herein or clearly contradicted by context. The use of all examples or exemplary language (e.g., "etc.") provided herein is intended only to more clearly describe the subject matter disclosed herein, and does not limit the scope of the invention that may be derived from the disclosure, unless otherwise claimed. No language in the specification should be construed as indicating any non-claimed element as essential.
[0034] It will be understood that the present disclosure can be incorporated in the form of various embodiments, only a few of which are disclosed herein. Variations of these embodiments will be apparent to those skilled in the art upon reading the foregoing description. Accordingly, the present disclosure and any inventions derived therefrom include all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Moreover, the present disclosure includes all combinations of all possible variations of the above-described elements unless otherwise specified herein or clearly contradicted by context.
Claims
1. A control device for performing image processing, one or more processors; one or more memories for holding instructions that, when executed, configure the one or more processors; Receive video from a camera capturing the conference room, extracting a predetermined region of the video and storing it as extracted image data; performing a first image correction process on the extracted image data to generate a first corrected image and correcting noise to generate a binary mask of the first corrected image; generating a filtered image based on the binary mask of the first corrected image and the first corrected image; performing a second image correction process on the filtered image to generate a second corrected image; A blending process is performed to combine the second corrected image and the first corrected image to generate a final corrected image. one or more memories; A control device comprising:
2. Execution of the instructions further causes the one or more processors to: storing in a memory a predetermined number of masks of the first corrected image; Calculating an average of said stored predetermined number of masks; configured to generate an average mask, the filtered image being generated using the average mask. The control device according to claim 1 .
3. 2. The control device of claim 1, wherein the first image correction process is a keystone correction that produces a substantially rectangular image of the extracted predetermined area.
4. Execution of the instructions further causes the one or more processors to: storing a copy of the first corrected image in a memory; Obtaining a copy of the stored first corrected image prior to performing the blending process.
2. The control device according to claim 1, characterized in that:
5. The control device according to claim 1 , wherein the second image correction process corrects a color and a brightness of the first corrected image.
6. Execution of the instructions further causes the one or more processors to: Transforming the first corrected image from a first color space to a second color space; applying a predetermined saturation setting and a predetermined lightness setting to each pixel of the first corrected image having a first value; setting each pixel that does not have the first value to white 6. The control device according to claim 5, characterized in that said control device is configured as follows.
7. An image processing method executed by a control device, comprising: Receive video from a camera capturing the conference room, extracting a predetermined region of the video and storing it as extracted image data; performing a first image correction process on the extracted image data to generate a first corrected image and correcting noise to generate a binary mask of the first corrected image; generating a filtered image based on the binary mask of the first corrected image and the first corrected image; performing a second image correction process on the filtered image to generate a second corrected image; A blending process is performed to combine the second corrected image and the first corrected image to generate a final corrected image.
13. An image processing method comprising:
8. storing in a memory a predetermined number of masks of the first corrected image; Calculating an average of said stored predetermined number of masks; configured to generate an average mask, the filtered image being generated using the average mask.
8. The method of claim 7.
9. 8. The method of claim 7, wherein the first image correction process is a keystone correction that produces a substantially rectangular image of the extracted predetermined area.
10. storing a copy of the first corrected image in a memory; Obtaining a copy of the stored first corrected image prior to performing the blending process.
8. The method of claim 7.
11. 8. The method of claim 7, wherein the second image correction process corrects color and brightness of the first corrected image.
12. Transforming the first corrected image from a first color space to a second color space; applying a predetermined saturation setting and a predetermined lightness setting to each pixel of the first corrected image having a first value; setting each pixel that does not have the first value to white 12. The method of claim 11, wherein the color and brightness of the corrected image is corrected by:
Citation Information
Patent Citations
Image processor, image processing method and image processing program
JP2016038849A
Intelligent whiteboard collaboration system and method
JP2020525962A
