Systems and methods for detecting and displaying whiteboard text and / or active speakers

Abstract

The present disclosure provides systems and methods for identifying and displaying whiteboard text and / or active speakers in video-based presentations (e.g., video conferences). A video image of an environment including a whiteboard can be captured by a video camera system. The video image can be analyzed to detect at least one area in the environment that contains text. Each area containing text can be analyzed to determine whether the area is an area of ​​a whiteboard. When an area containing text is identified as a whiteboard area, a view area including the whiteboard area containing text can be selected for display, such as a subset of a full frame captured by the video system. A video source from the video camera system can be controlled to display the selected view area at a client device to provide a useful view of whiteboard text and / or a speaker located near the whiteboard text.

Description

Technical Field

[0001] The present invention relates to systems and methods for detecting text on a whiteboard based on video and controlling a displayed video stream (e.g., in a video conference or other video presentation) based on the detected whiteboard text to, for example, display a view area that includes the detected whiteboard text and / or a nearby active speaker. Background Art

[0002] A whiteboard is a common tool used by a speaker to present text or other information to an audience, such as a live audience or a remote audience viewing a video feed of the presentation in real time (e.g., in a video conferencing environment or other live video stream) or by playing back a recorded video stream. As used herein, "text" may include words, numbers, alphanumeric characters, symbols, shapes, drawings, or any other information that is handwritten or otherwise written, printed, or displayed on a physical medium.

[0003] Existing systems for displaying a video feed of a presentation (live or recorded) are generally unable to effectively display text on a whiteboard that may be relevant to the presentation, such as text written by the speaker before or during the presentation. Some existing systems are configured to track the position of a speaker (or more specifically the speaker's face) in real time and pan and zoom a camera or displayed camera view to focus on the tracked speaker. However, such systems may omit information presented on the whiteboard used by the speaker.

[0004] Other existing systems maintain a full view of the room or environment the speaker is in. However, it is often difficult or impossible for a viewer to recognize or decipher the information on the whiteboard based on the full room view because the text on the whiteboard may appear too small and / or low resolution on the viewer's display device.

[0005] Detecting whiteboards in video images is difficult due to their various shapes and sizes, and may be further complicated by the presence of other objects including text in the camera view (e.g., televisions or other screens, posters, signs, etc.).

[0006] Figure 1 A conventional method 100 for automatic video-based detection of a whiteboard and detection of text on the whiteboard is shown. At 102, a conventional whiteboard detection analysis is performed on a video image to identify the location of the whiteboard within the view of a camera. After the location of the whiteboard is determined, a conventional text detection analysis is performed at 104 to identify and / or decrypt the text on the whiteboard. However, in conventional methods, the whiteboard detection step 102 is often inaccurate or unsuccessful, as indicated by an "X" after step 102. For example, conventional methods may fail to locate a whiteboard or may incorrectly identify a non-whiteboard object (e.g., a television or other display screen, a poster, a sign, etc.) as a whiteboard.

[0007] Therefore, based on the deficiencies of conventional automatic whiteboard detection techniques, some video systems use manually defined whiteboard positions to define the displayed area of ​​the video source. Of course, this requires manual human intervention, which may be inconvenient, impractical, or impossible, depending on the specific situation. In addition, a clean whiteboard or its blank areas are usually irrelevant to the presentation. Therefore, displaying the entire whiteboard regardless of the portion containing text may result in an unnecessarily large viewing area, which unnecessarily reduces the size of the whiteboard text on the video display size, possibly making it difficult for the viewer to read the text.

[0008] Accordingly, there is an object for improved systems and methods for video-based automated whiteboard detection during video presentations and for providing useful whiteboard displays in a video feed displayed to viewers of the presentation (e.g., in a video conference), e.g., displaying a selected portion of the whiteboard including text or other information. Summary of the invention

[0009] This object is solved by the subject-matter of the independent claims.The dependent claims and the following description describe various embodiments of the invention.

[0010] In one aspect, a system for identifying and displaying whiteboard text and / or active speakers is provided. The system may include: a video camera system including at least one camera; and a controller that communicatively couples the video to the camera system. The camera of the video camera system is configured to capture video images showing an environment including at least one whiteboard. The controller may include at least one processor. The controller is configured to: receive a series of video images captured by at least one camera of the video camera system; analyze one or more such video images to detect at least one text-containing area in the environment; analyze each text-containing area to determine whether the corresponding text-containing area is a whiteboard area containing text; and select a view area for display that includes the identified whiteboard area containing text. The controller is then configured to generate a video source from the video images captured by the video camera system for display on a remote client device, and control the video source to display the selected view area including the whiteboard area containing text.

[0011] The basic idea of ​​the invention relates to detecting text on a whiteboard, including distinguishing the whiteboard text from other text displayed on other objects in the camera view (e.g., text on a TV or monitor, text on a sign or poster, or text on a wall), and automatically controlling the displayed video stream based on the detected whiteboard text, for example, displaying only the portion of the whiteboard that includes the text and / or the person speaking.

[0012] The inventors have found that detecting all areas containing text within a camera view, and then determining whether each area containing text is an area of ​​a whiteboard, provides improved results over conventional systems that attempt to first detect a whiteboard in a camera view and then recognize text on the whiteboard. For example, although deep learning techniques have improved the accuracy of object detection in recent years, there are not enough whiteboard detection object data sets to train models to effectively detect whiteboards, such as distinguishing them from all other possible white or light-colored objects in a room. In contrast, the inventors found that detecting text in a real-world environment is much more accurate than detecting whiteboards. However, text detection alone is not enough because a room may include a variety of other objects containing text (e.g., televisions or other displays, signs, posters, etc.). Therefore, the inventors invented a technique that first detects all areas containing text within a camera view (e.g., a conference room), and then determines whether each area containing text is a whiteboard area.

[0013] The present invention can provide new and powerful features for video conferencing systems. For example, using improved whiteboard text detection, a camera or feed from a camera can be automatically panned and / or zoomed to focus on the area of ​​the whiteboard that includes text and ignore blank areas of the whiteboard. In some systems, the whiteboard text detection of the present invention can be combined with a speaker tracking system to detect or track the position of the active speaker (i.e., the presenter generating or interacting with the whiteboard text) so that the system can control the displayed video source to show a selected area of ​​the room, including the whiteboard text and the active speaker.

[0014] In another aspect, a method for identifying and displaying whiteboard text is provided. A series of video images captured by a video camera system including at least one camera is collected, wherein each video image displays an environment including at least one whiteboard. A controller communicatively coupled to the video camera system analyzes at least one video image in the series of video images to detect at least one region in the environment containing text. After detecting at least one region in the environment containing text, the controller analyzes each region containing text to determine whether the corresponding region containing text is a whiteboard region containing text. In response to determining that the first region containing text is a whiteboard region containing text, the controller automatically selects a view area for display including the whiteboard region containing text. The controller generates a video source from the video images captured by the video camera system for display on a remote client device, and controls the video source to display the selected view area including the whiteboard region containing text.

[0015] The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present invention will be readily understood through the following detailed description taken in conjunction with the accompanying drawings, in which like reference numerals denote like structural elements.

[0017] Figure 1 Conventional methods for automatic video-based detection of whiteboards and detection of text on whiteboards are shown;

[0018] Figure 2 An exemplary system for recognizing and displaying whiteboard text on a client device according to an embodiment of the present invention is shown;

[0019] Figure 3A An exemplary method for recognizing and displaying whiteboard text on a client device according to an embodiment of the present invention is shown;

[0020] Figure 3B , 3C and 3D is shown Figure 3A exemplary images of the results of each step in the exemplary method shown;

[0021] Figure 4A , 4B 4C shows exemplary selection of display areas for three different scenarios according to an example embodiment of the present invention; and

[0022] Figure 5 An exemplary method for recognizing and displaying whiteboard text on a client device is shown, in an exemplary environment including an active speaker presenting information using text on a whiteboard. DETAILED DESCRIPTION

[0023] Below, specific embodiments of the present invention are described in detail herein. In the following description of the embodiments of the present invention, specific details are described in order to provide a thorough understanding of the present invention. However, it is apparent to one of ordinary skill in the art that the present invention can be practiced without these specific details. In other cases, well-known features are not described in detail to avoid unnecessarily complicating the current description.

[0024] In the following explanation of the invention according to the described embodiments, the term "connected to" or "connected thereto" is used to indicate a data and / or audio (signal) connection between at least two components, devices, units, processors or modules. Such a connection may be made directly between the respective components, devices, units, processors or modules; or indirectly, i.e. via an intermediate component, device, unit, processor or module. The connection may be permanent or temporary; wireless or conductor-based.

[0025] For example, the data and / or audio connection may be provided via a direct connection, a bus, or a network connection using a corresponding suitable communication protocol (such as a WAN (Wide Area Network), a LAN (Local Area Network), a PAN (Personal Area Network), a BAN (Body Area Network), including, for example, the Internet, Ethernet, a cellular network (e.g., LTE), a Bluetooth (Classic, Smart, or Low Energy) network, a DECT network, a ZigBee network, and / or a Wi-Fi network). In some embodiments, a USB connection, a Bluetooth network connection, and / or a DECT connection are used to transmit audio and / or data.

[0026] In the following description, ordinal numbers (e.g., first, second, third, etc.) may be used as adjectives of elements (i.e., any noun in the application). The use of ordinal numbers does not imply or create any particular order of elements, nor does it limit any element to only a single element, unless explicitly disclosed, such as by the use of the terms "before," "after," "single," and other similar terms. Instead, ordinal numbers are used to distinguish elements with similar names. For example, the first element is different from the second element, and the first element may contain more than one element and be after (or before) the second element in the ordering of the elements.

[0027] In a video-based presentation or communication system (e.g., a video conferencing system), for example, there is a need for improved techniques for automatically detecting and displaying text on a whiteboard. For example, there is a need for improved detection of whiteboard text to be distinguished from blank areas of the whiteboard and from other objects in a camera view that include text, such as a television or other monitor, signs, posters, etc. Accurate detection of whiteboard text allows a video source (e.g., a video stream) to be controlled to display only selected portions of the whiteboard that include text (e.g., excluding blank areas of the whiteboard), thereby increasing the display size and / or clarity of the text displayed on a client device.

[0028] In one aspect, a system for recognizing and displaying whiteboard text is provided. The system may include: a video camera system and a controller, the video camera system is arranged in a presentation environment, the controller is communicatively coupled to the video camera system and is configured to control a video source from the video camera system. The system may also include one or more client devices, the client devices are remote from the presentation environment and are configured to display a video stream controlled by the controller.

[0029] The video camera system may include a single camera or multiple cameras, each camera configured to capture video images showing an environment including at least one whiteboard, such as a video conference room or other room including at least one active speaker (e.g., a presenter) using the whiteboard to communicate information to at least one remote client, such as other participants in the video conference.

[0030] In some embodiments, the video camera system may include multiple cameras spaced apart from each other in one or more directions to provide different viewing angles. For example, the video camera system may include multiple cameras housed in a common housing. Each camera may be set or controlled to capture an entire area of ​​the presentation environment, or each camera may be set or controlled to capture a different area of ​​the presentation environment. In addition, each camera may be controlled (e.g., pan, tilt, and / or zoom) independently of each other camera.

[0031] In one embodiment, a video camera system may include (a) a first camera configured to capture an entire area of ​​a presentation environment for detecting a whiteboard text area and / or an active speaker in the environment, and (b) a second camera controlled (e.g., panned, tilted, and / or zoomed) to capture a selected portion of the environment (including the whiteboard text area and / or the active speaker) for use as a video source for communicating with a remote client.

[0032] The controller may include at least one processor, at least one memory or other data storage device, and software, firmware or other logic instructions executable by the at least one processor to perform any of the various functions disclosed herein. The controller may be embodied in a single computer, or may include multiple computers located together or remotely from each other.

[0033] In some embodiments, the controller is located in a demonstration environment. For example, the controller and the video camera system can be arranged in a common housing located on (e.g., configured to be mounted on) a wall or placed on a table or desk in the demonstration environment. In other embodiments, the controller is located in the demonstration environment, but is located in a housing separated from the video camera system. For example, the controller can be embodied as an additional device that is configured to be connected to an existing video camera system via a wired or wireless connection. In other embodiments, the controller is located at a position away from the demonstration environment. For example, the controller can be arranged in a client device that is away from the demonstration environment and is connected to the video camera system via the Internet or other suitable communication networks. As another example, the controller can be away from both the video camera system and the client device, wherein the remote system components can be interconnected via the Internet or other suitable communication networks.

[0034] The client device may include any computer device with a screen or other display device configured to display a video source received from the controller. For example, the client device may include one or more televisions, monitors, desktop computers, laptop computers, tablet computers, smart phones, smart watches, or any other device configured to display a video source. As described above, in some embodiments, the controller may be provided in the client device. In such an embodiment, the controller may receive a video image stream of the entire area of ​​the presentation environment from the video camera system, and generate a video source including only the relevant portion of the complete presentation environment area for display on the client device, for example, the presentation environment area including a selected portion of a whiteboard area containing text identified by the controller and / or an active speaker near the whiteboard area containing text.

[0035] The controller may be configured to receive a series of video images captured by a camera of a video camera system, analyze one or more of the video images to detect at least one region containing text in an environment, and then analyze each region containing text to determine whether the corresponding region containing text is a whiteboard region containing text. After identifying the whiteboard region containing text, the controller may select a view area including the identified whiteboard region containing text for display (at a client device). The controller is further configured to generate a video source from the video images captured by the video camera system for display at a client device connected to the controller (via the Internet or other suitable network), and control the video source to display the selected view area including the whiteboard region containing text.

[0036] The controller can use any suitable tool or algorithm to detect one or more areas in the environment that contain text. For example, the controller can utilize a text detection tool such as AdvancedEAST (https: / / github.com / huoyijie / AdvancedEAST). In one embodiment, the controller uses a high-speed feature extractor such as MobilenetV2. MobilenetV2 (https: / / arxiv.org / abs / 1801.04381) is a lightweight network architecture that uses deep convolutions instead of original convolutional layers to filter features, which allows the development of efficient models on mobile devices or other suitable platforms. Another embodiment uses the VGG16 tool for feature extraction.

[0037] The controller may also use any suitable tool or algorithm to determine whether each identified region containing text is a region of a whiteboard, e.g., as distinguished from text on a television / monitor, poster, sign, etc. For example, the controller may utilize a deep learning model of the MobilenetV2 tool that has been trained using a collection of images of whiteboards and non-whiteboard objects (e.g., including objects that typically contain text or otherwise appear similar to a whiteboard).

[0038] In some embodiments, the controller may first apply a color filter to eliminate any text-containing areas with a non-white background so that each remaining text-containing area has a white background, and then determine whether each white background text-containing area is a whiteboard area, for example using the deep learning model of the MobilenetV2 tool discussed above. The color filter can identify and remove non-white areas based on statistics of the brightness and saturation of each text area. Using a color filter reduces the number of candidate text areas to be analyzed by the deep learning model, and can use much fewer resources and / or costs than whiteboard recognition using a deep learning model.

[0039] To control the video source displayed on the client device, the controller can (a) control optical parameters of one or more cameras, or (b) digitally select and control the portion of the video image displayed on the client device. With respect to the first option, the controller can, for example, control each camera to perform at least one of a physical pan, a physical tilt, or an optical zoom to capture a selected area of ​​the presentation environment including a whiteboard area containing text and / or an active speaker. In such an embodiment, the video camera system can include (a) a first camera that continues to capture an area of ​​the presentation environment that the controller can use to detect the whiteboard text area and / or the active speaker, and (b) a second camera that is controlled (e.g., panned, tilted, and / or zoomed) by the controller to focus on the selected area including the whiteboard text area and / or the active speaker, wherein the video source of the second camera is transmitted to the remote client for display.

[0040] In some embodiments, in addition to detecting a whiteboard area containing text, the controller is also configured to identify speakers (also referred to herein as active speakers) in the environment and automatically select a view area for display that includes both (a) at least a selected portion of the identified speaker and (b) the whiteboard area containing text. In some embodiments, if the controller identifies both a whiteboard area containing text and an active speaker, the controller may determine, for example based on a distance between the speaker and the whiteboard text, whether to include both the whiteboard text area and the active speaker in the selected area for display or to include only the active speaker (and not the whiteboard text area) in the selected area. For example, the controller may be configured to determine a distance between a detected speaker and a whiteboard text area; compare the determined distance to a defined threshold distance, and determine to include the whiteboard text area in the selected area for display only if the determined distance is less than the defined threshold distance. Figures 4A to 4C Examples of these features are shown in , as described below.

[0041] In one embodiment, if the controller identifies both a whiteboard area containing text and an active speaker, the controller may determine a temporary view area that includes both the whiteboard text area and the detected speaker, and analyze the temporary view area to determine whether to display both the whiteboard text area and the speaker, or only the speaker. For example, the controller may analyze the temporary view area to determine the text size or whiteboard area size of the whiteboard text area in the temporary view area, and may then select between a temporary view area (including both the whiteboard text area and the speaker) or a view area that only includes the speaker based on the determined text size or whiteboard area size of the whiteboard text area.

[0042] The controller may be configured to use any suitable input data and analysis techniques to identify active speakers in the presentation environment. In some embodiments, the controller may detect active speakers by analyzing video images from a video camera system. In other embodiments, the controller may detect active speakers based on sound data received from one or more microphones, either alone or in combination with video analysis. In one embodiment, the system includes a microphone array that includes a plurality of microphones spaced apart from each other and / or arranged in different directions, and the controller is configured to automatically detect active speakers based at least on the sound data received from the microphone array. In one embodiment, the controller may include any system and method and provide any functionality for detecting and tracking active speakers based on video analysis, such as disclosed in U.S. Patent No. 8,248,448, entitled "Automatic Camera Framing for Videoconferencing," filed on May 18, 2010, the entire contents of which are incorporated herein by reference for all purposes.

[0043] In another embodiment, the controller may include any systems and methods and provide any functionality for detecting and tracking active speakers based on analysis of ultrasonic signals and / or sounds in the audible spectrum detected by one or more microphones, including any algorithm or technique disclosed in U.S. Patent No. 8,249,298, entitled “Ultrasonic Camera Tracking System and Associated Methods,” filed on October 15, 2007, the entire contents of which are incorporated herein by reference.

[0044] In another embodiment, the controller may include any of the systems and methods and provide any functionality for detecting and tracking active speakers using video analysis and microphone signal analysis (eg, detected ultrasonic signals and / or detected sounds in the audible spectrum).

[0045] In another aspect, a method for identifying and displaying whiteboard text is provided. A series of video images captured by a video camera system is collected, wherein each video image displays an environment including at least one whiteboard. A controller analyzes at least one video image in the series of video images to detect at least one region containing text in the environment. After detecting at least one region containing text in the environment, the controller analyzes each region containing text to determine whether the corresponding region containing text is a whiteboard region containing text. In response to determining that the first region containing text is a whiteboard region containing text, the controller automatically selects a view area for display including the whiteboard region containing text, generates a video source from the video images captured by the video camera system for display on a remote client device, and controls the video source based on the selected view area for display including the whiteboard region containing text.

[0046] Figure 2 An exemplary system 200 for recognizing and displaying whiteboard text on a client device 250 according to an embodiment of the present invention is shown. The system 200 includes a controller 202, a video camera system 204 including one or more video cameras, and a microphone array 206 including one or more microphones. The video camera system 204 and the microphone array 206 are located in a presentation environment 210, which includes a whiteboard 212 containing text 214, a plurality of non-whiteboard objects 216A, 216B containing text, and an active speaker 218. The video camera system 204 is arranged to capture a video image of the presentation environment 210. A full frame captured by the video camera system 204 is indicated at 220. The microphone array 206 is arranged to capture wireless signals from the environment 210, including ultrasonic signals (e.g., from an ultrasonic transmitter worn or carried by the active speaker 218) and / or sounds in the audible spectrum (e.g., speech or other audio from the active speaker 218), and to generate signals representing or derived from such detected wireless signals and transmit these signals to the controller 202. Wireless signals (including ultrasonic signals, sounds in the audible spectrum, and / or other wireless signals) detected by the microphone array 206 are referred to herein as “sound signals,” and data transmitted from the microphone array 206 to the controller 202 that represents or originates from the sound signals detected by the microphone array 206 is referred to herein as “sound data.”

[0047] The controller 202 is connected to the video camera system 204 and the microphone array 206 via any suitable wired or wireless link for receiving video and sound data from the video camera system 204 and the microphone array 206. The controller 202 may also be connected to the client device 250 via any suitable wired or wireless link, for example, via a network 240 (e.g., the Internet or other communication network).

[0048] The controller 202 can be integrated with the video camera system 204 and / or the microphone array 206 in a single device housing, or can be separate from the video camera system 204 and the microphone array 206 and can be located in the presentation environment 210, or set on the client device 250, or can be remote from the presentation environment 210 and the client device 250.

[0049] The controller 202 may be configured to provide any of the controller functions discussed herein. The controller 202 may include a whiteboard recognition unit 230, a speaker tracking unit 232, and a video control unit 234. The whiteboard recognition unit 230 includes hardware, software, and / or firmware for detecting areas of a whiteboard containing text, such as to perform the following discussion. Figure 3A In the example shown, whiteboard recognition unit 230 is configured to recognize text 214 on whiteboard 212 and text on non-whiteboard objects 216A and 216B, and then analyze each of the three text-containing regions to determine that only text 214 is on the whiteboard.

[0050] Speaker tracking unit 232 includes hardware, software, and / or firmware for identifying an active speaker 218 (or multiple speakers) in presentation environment 210 based on video images received from video camera system 204 and / or sound data received from microphone array 206. For example, speaker tracking unit 232 may include any of the systems and methods disclosed in U.S. Pat. No. 8,248,448, the entire contents of which are hereby incorporated by reference.

[0051] The video control unit 234 includes hardware, software, and / or firmware for selecting an area of ​​the environment 210 for display and controlling a video source transmitted to the client device 250 based on the area selected for display. In the example shown, the video control unit 234 selects an area 260 that includes the identified whiteboard text area 214 and the identified active speaker 218. To control the video source to display the selected area 260 at the client device 250, the video control unit 234 can (a) control optical parameters of the video camera system 204 (e.g., optical pan, tilt, and / or zoom control), or (b) digitally select and control a portion of the complete camera frame 220 displayed at the client device 250 (e.g., digital pan, tilt, and / or zoom). The video source including the selected area 260 can be displayed on a screen or other display device 252 of the client device 250.

[0052] Figure 3A An exemplary method 300 for recognizing and displaying whiteboard text on a client device according to an embodiment of the present invention is shown. Figure 3B ,3C and 3D are exemplary images showing the results of each step in method 300. For example, method 300 may be performed by Figure 2 The controller 202 of the illustrated system 200 executes.

[0053] In step 310 , all text regions in the frame are identified, for example, using a text detection tool AdvancedEAST (https: / / github.com / huoyijie / AdvancedEAST). Figure 3B A box around each of the three text regions detected in step 310 is shown, including a first text region on the whiteboard, a second text region on the poster, and a third text region on the TV. In some embodiments, a high-speed feature extractor can be used, such as MobilenetV2 (https: / / arxiv.org / abs / 1801.04381), which is a lightweight network architecture that uses deep convolution to replace the original convolution layer to filter features.

[0054] At step 320, a color filter is applied to the three detected text regions to eliminate the text regions that are not on a white background. For example, the color filter can be based on brightness and saturation statistics. In the example shown, the color filter excludes the TV text region, such as Figure 3C shown.

[0055] At step 330, a whiteboard recognition analysis is performed on each remaining white background text region to determine that each region is on a whiteboard. This step may utilize a deep learning model of MobilenetV2 (or other suitable tool) that is trained using a collection of images of whiteboards and non-whiteboard objects for whiteboard recognition. In the example shown, the whiteboard recognition analysis excludes the poster text, leaving only the whiteboard text region, such as Figure 3D shown.

[0056] Figure 4A , 4B 4C show exemplary selections of display areas for three different scenarios according to an example embodiment of the present invention. Figure 4A An exemplary scene includes whiteboard text, but no active speaker in the camera frame. Thus, the disclosed systems and methods can generate a video source of a magnified view (using optical or digital zoom) of the whiteboard text.

[0057] Figure 4B An exemplary scene includes whiteboard text and an active speaker very close to the whiteboard. The disclosed systems and methods can determine that the distance between the active speaker and the whiteboard text is less than a threshold distance, and thus generate a video source including a zoomed-in view of both the whiteboard text and at least the head of the active speaker.

[0058] Figure 4C An exemplary scene includes whiteboard text and an active speaker far from the whiteboard. The disclosed systems and methods can determine that the distance between the active speaker and the whiteboard text exceeds a threshold distance, thereby generating a video feed of a zoomed-in view of only the speaker, excluding the whiteboard.

[0059] Figure 5 An exemplary method 500 for recognizing and displaying whiteboard text on a client device in an exemplary environment including an active speaker (person) presenting information using text on a whiteboard according to an embodiment of the present invention is shown. All or part of the method 500 may be performed, for example, by Figure 2 The system 200 shown in the figure or a similar system is executed. In this example, the system 200 includes: a controller 202 connected to a video camera 204; and a microphone array 206 including a plurality of microphones. Therefore, the description of the method 500 provided below can refer to Figure 2 Get the best understanding.

[0060] At 502, the video camera 204 and the microphone array 206 can generate and transmit video and sound signals to the controller 202. The video camera 204 can generate and transmit to the controller 202 a series of video images of a presentation environment 210, the presentation environment including a whiteboard 212 containing text 214, a plurality of non-whiteboard objects 216A, 216B containing text, and an active speaker 218. For example, each video image can have a frame indicated at 220. At the same time, the microphone array 206 can generate and transmit to the controller 202 sound signals representing sounds in the environment 210, including ultrasonic signals and / or audible sounds from the active speaker 218 and / or other audio (e.g., audio from the television 216B, ambient noise, etc.). The plurality of microphones of the microphone array 206 can be spaced apart from each other (in a common housing or in different spaced apart housings) and / or oriented at different angles, so that the sound signals generated by the plurality of microphones can be used to determine the location of a noise source, for example, to determine the location of the active speaker 218 (e.g., using sound source localization techniques).

[0061] At 504, the controller 202 may analyze the video image received from the camera 204 to detect each region containing text in the video frame 220 of the video image, including the poster text 216A, the text displayed on the TV 216B, and the text region 214 on the whiteboard 212. In some embodiments, the controller 202 may use a text detection tool, such as AdvancedEAST (https: / / github.com / huoyijie / AdvancedEAST), and / or a high-speed feature extractor, such as MobilenetV2 (https: / / arxiv.org / abs / 1801.04381), as described above.

[0062] After detecting multiple text-containing regions in the video frame 220, the controller 202 can analyze each text-containing region to determine whether the corresponding text-containing region is on a whiteboard or on a non-whiteboard object. For example, at 506, the controller 202 can use a color filter to eliminate each text-containing region with a non-white background so that each remaining text-containing region has a white background, and at 508, the controller 202 can determine whether each remaining text-containing region with a white background is a whiteboard region.

[0063] For example, the color filter applied at 506 may be based on a statistical analysis of brightness and saturation for each text region. Figure 2 In the example shown, the color filter can eliminate the TV text area, but leave the white background poster text area 216A and the whiteboard text area 214. The analysis of each remaining white background text area (in this example, the poster text area 216A and the whiteboard text area 214) at 508 can utilize a deep learning model, such as a MobilenetV2 model, which is trained using a collection of images of whiteboards and non-whiteboard objects (stored in a memory accessible to the controller 202) for whiteboard recognition, or utilize any or other suitable analysis tools. In the example shown, the whiteboard recognition analysis does not include the poster text area 216A, leaving only the whiteboard text area 214.

[0064] In addition to identifying the whiteboard text area 214, the controller 202 may also automatically detect the presence and / or location of the active speaker 218 in the video frame 220 (e.g., simultaneously with, before, after, or in an alternating manner with the whiteboard text recognition). In one embodiment, at 510, the controller 210 may analyze both (a) the video image from the camera 204 and (b) the sound signal from the microphone array 206 to detect the presence and / or location of the active speaker 218 in the frame 220. For example, controller 202 may detect, locate and / or track active speaker 218 by: (a) applying a suitable object recognition and / or motion detection algorithm to video images from camera 204, such as, for example, any algorithm or technique disclosed in U.S. Patent No. 8,248,448, the entire contents of which are hereby incorporated by reference, and (b) applying an audio-based position detection algorithm to acoustic signals from multiple microphones of microphone array 206, such as, for example, any algorithm or technique disclosed in U.S. Patent No. 8,249,298, entitled “Ultrasonic Camera Tracking System and Associated Methods,” filed on October 15, 2007, the entire contents of which are hereby incorporated by reference.

[0065] The controller 202 may then select a view area, e.g., a subset of the full video image frame 220, to be displayed by the client device 250 based at least on (a) the presence of an active speaker 218 in the frame 220, (b) the location of the active speaker 218 (if present), and / or (c) the location and / or size of the identified whiteboard text area 214. In one embodiment, the controller 202 may determine whether to (a) select a view area that includes only the whiteboard text area 214 but not the active speaker (e.g., if no active speaker is detected), (b) select a view area that includes both the whiteboard text area 214 and the detected active speaker 218 (or at least the speaker's head), or (c) select a view area that includes the detected active speaker 218 (or at least the speaker's head) but not the whiteboard text area 214 based on one or more defined view selection factors.

[0066] The view selection factors may include one or more of the following, for example, (a) whether an active speaker 218 is present in the frame 220, (b) a determined distance between the whiteboard text area 214 and the active speaker 218 (e.g., the distance between the defined midpoints of each object, or the distance between the nearest edge or point of the whiteboard text area 214 and the active speaker 218, or the distance between the farthest edge or point of the whiteboard text area 214 and the active speaker 218, (c) a size or scaling factor of a minimum view area that includes both the whiteboard text area 214 and the active speaker 218, (d) a percentage of the view area filled by the whiteboard text area 214 and the active speaker 218 in the minimum view area that includes both the whiteboard text area 214 and the active speaker 218, (e) a determined size, clarity, focus, or other quality of the whiteboard text in the minimum view area that includes both the whiteboard text area 214 and the active speaker 218, or any other suitable factor.

[0067] In one embodiment, at 512, the controller 202 may first determine whether an active speaker 218 is detected in the environment 210. If not, at 514, the controller 202 may select a minimum view area that includes the detected whiteboard text area 214 and a defined buffer area around the whiteboard text area 214, such as Figure 4A As shown in the exemplary view of FIG.

[0068] Alternatively, if the controller 202 detects an active speaker 218 in the environment 210 , the method may proceed to steps 516 - 522 to select a view area that includes the active speaker 218 (or at least a portion thereof) and, if appropriate, the whiteboard text area 214 .

[0069] In the illustrated embodiment, at 516, the controller 202 may determine a distance between the closest point of the active speaker 218 and the whiteboard text area 214, for example, as defined by a number of pixels or a percentage of the full horizontal width of the video frame 220. At 518, the controller 202 may compare the determined distance to a defined threshold distance. If the distance determined between the active speaker 218 and the whiteboard text area 214 is less than the defined threshold distance, then at 520, the controller 202 selects a view area that includes both the whiteboard text area 214 and the active speaker 218 (or at least the speaker's head) using any rules for defining the selected area. For example, the controller 202 may select a minimum view area that includes the entire whiteboard text area 214, the active speaker's head 218, and a defined buffer area around the whiteboard text area 214 and the speaker's 218 head.

[0070] If the distance determined between the active speaker 218 and the whiteboard text area 214 is greater than or equal to the defined threshold distance, then at 522, the controller 202 uses any rules for defining the selected area to select a view area that includes the active speaker 218 (or at least the speaker's head), excluding the whiteboard text area 214. For example, the controller 202 may select a minimum view area that includes the speaker's head and shoulders and a defined buffer area around the head and shoulders.

[0071] After selecting a view area to be displayed at the remote client devices 250, the controller 202 may generate a video feed based on the selected view area at 524 and transmit the video feed to the client devices 250 at 526. Each client device 250 may then display the video feed on the display screen 252 of the respective client device 250, the video feed showing the selected view area.

[0072] Depending on the particular embodiment, the controller 202 may generate a video feed based on the selected view area at 524 in various ways. In one embodiment, the controller 202 may receive a full frame image from the video camera 204 and perform digital pan, tilt, and / or zoom functions to select (i.e., digitally focus on) a selected view area, e.g., including the active speaker 218 or including both the active speaker 218 and the whiteboard text area 214, as described above. In another embodiment, the controller 202 may control the video camera 204 to perform optical pan, tilt, and / or zoom functions to focus on the selected view area so that the video frame captured by the camera 204 matches the selected view area.

[0073] The method may return to 502 to dynamically adjust the selected view area, for example, based on movement of the active speaker 218 and / or changes in the whiteboard text area 214 (eg, when the speaker adds more text or erases text on the whiteboard).

[0074] Although exemplary embodiments of the present invention are described and illustrated herein, it should be understood that these embodiments are merely illustrative and that modifications may be made to these embodiments without departing from the spirit and scope of the present invention. The actions described herein may be computer-readable and executable instructions that may be implemented by one or more processors and stored on a computer-readable memory or article. Computer-readable and executable instructions may include, for example, applications, program modules, routines and subroutines, execution threads, and the like. In some cases, not all actions may need to be implemented in the manner described herein.

[0075] Terms such as "component," "module," "unit," and "system" are intended to encompass hardware, software, firmware, or any combination thereof. For example, a system or component may be a process, a process executing on a processor, or a processor. Furthermore, functions, components, or systems may be located on a single device or distributed across multiple devices. The described subject matter may be implemented as an apparatus, method, or article of manufacture using standard programming or engineering techniques to produce software, firmware, hardware, or any combination thereof to control one or more computing devices.

[0076] Accordingly, it is intended that the scope of the invention be defined solely by the following claims, as they may be amended, each of which is expressly incorporated into this Detailed Description as an embodiment of this invention.

Claims

1. A system for recognizing whiteboard text, comprising: a video camera system including at least one camera, each camera configured to capture a video image showing an environment including at least one whiteboard; a controller communicatively coupled to the video camera system, the controller comprising at least one processor, the at least one processor being configured to: receiving a series of video images captured by the at least one camera of the video camera system; analyzing at least one video image in the series of video images to detect at least one region of the environment containing text; After detecting at least one area containing text in the environment, analyzing each area containing text to determine whether the corresponding area containing text is a whiteboard area containing text; In response to determining that the first text-containing area is a whiteboard area containing text, selecting a view area for display including the whiteboard area containing text; generating a video feed from video images captured by said video camera system for display on a remote client device; as well as The video source is controlled based on the selected view area for display including the whiteboard area containing text.

2. The system according to claim 1, wherein: The controller is configured to control the video source by controlling operation of a camera of the video camera system to capture the selected view area for display.

3. The system according to claim 1 or 2, wherein: The controller is configured to control the camera to perform at least one of physical panning, physical tilting, or optical zooming.

4. The system according to claim 1, wherein: The controller is configured to control the video source by performing at least one of a digital pan, a digital tilt, or a digital zoom within a video image captured by the video camera system.

5. The system according to claim 1, wherein: The controller is configured to analyze each text-containing region to determine whether the corresponding text-containing region is a text-containing whiteboard region by: using a color filter to eliminate any text-containing regions having a non-white background so that each remaining text-containing region has a white background; and Determine whether each remaining text-containing region with a white background is a text-containing whiteboard region.

6. The system according to claim 1, wherein: The controller is also configured to: automatically detecting speakers in the environment; and The view area for display is automatically selected by selecting a view area of ​​the environment that includes both at least a selected portion of the speaker and the whiteboard area containing text.

7. The system according to claim 6, wherein: The controller is configured to automatically detect the speaker in the environment by analyzing the at least one video image in the series of video images to detect the speaker in the environment.

8. The system of claim 6, further comprising at least one microphone; and in, The controller is configured to automatically detect the speaker in the environment by: receiving sound data from the at least one microphone; and The speaker in the environment is detected based at least on the received sound data from the at least one microphone.

9. The system of claim 6, comprising a microphone array having a plurality of microphones; and in, The controller is configured to automatically detect the speaker in the environment by: receiving sound data from the microphone array; as well as A location of the speaker in the environment is detected based at least on the received sound data from the microphone array.

10. The system according to claim 1, wherein: The controller is configured to: Automatically detecting a speaker in the environment; determining a relative distance between the detected speaker and the whiteboard area containing text; comparing the relative distance between the detected speaker and the whiteboard region containing text to a defined threshold distance; as well as In response to determining that the relative distance between the detected speaker and the whiteboard area containing text is less than the defined threshold distance, determining to include the whiteboard area containing text in the view area selected for display.

11. The system according to claim 1, wherein: The controller is configured to: Automatically detecting a speaker in the environment; determining a temporary view area including the whiteboard area containing text and at least a portion of the detected speaker; as well as Analyzing the temporary view area to determine a text size or a whiteboard area size of the whiteboard area containing text; and The selecting the view area for display comprises: selecting the temporary view area as the view area for display based at least on the determined text size or whiteboard area size of the whiteboard area containing text.

12. A method for recognizing and displaying whiteboard text, comprising: collecting a series of video images captured by a video camera system including at least one camera, each video image showing an environment including at least one whiteboard; analyzing, by a controller communicatively coupled to the video camera system, at least one video image in the series of video images to detect at least one region of the environment containing text; After detecting at least one area containing text in the environment, the controller analyzes each area containing text to determine whether the corresponding area containing text is a whiteboard area containing text; In response to determining that the first area containing text is a whiteboard area containing text, automatically selecting, by the controller, a view area for display including the whiteboard area containing text; generating, by the controller, a video feed from video images captured by the video camera system for display on a remote client device; as well as The video source is controlled by the controller based on a selected view area for display including the whiteboard area containing text.

13. The method according to claim 12, comprising: The video source is controlled by controlling operation of a camera of the video camera system to capture the selected view area for display.

14. The method of claim 12 or 13, comprising controlling the camera to perform at least one of a physical pan, a physical tilt or an optical zoom.

15. The method according to claim 12, comprising: The video source is controlled by performing at least one of a digital pan, a digital tilt, or a digital zoom within a video image captured by the video camera system.

16. The method according to claim 12, wherein: Analyzing each text-containing region to determine whether the corresponding text-containing region is a text-containing whiteboard region includes: using a color filter to eliminate any text-containing regions having a non-white background so that each remaining text-containing region has a white background; and Determine whether each remaining text-containing region with a white background is a text-containing whiteboard region.

17. The method of claim 12, further comprising: Automatically detecting a speaker in the environment; and The step of automatically selecting a viewing area for display includes selecting a viewing area in the environment that includes at least a selected portion of the speaker and the whiteboard area containing text.

18. The method according to claim 17, wherein: Automatically detecting a speaker in the environment includes analyzing the at least one video image in the series of video images to detect the speaker in the environment.

19. The method according to claim 17, wherein: Automatically detecting speakers in the environment includes: receiving audio data from at least one microphone; and The speaker in the environment is detected based at least on the received sound data from the at least one microphone.

20. The method according to claim 19, wherein: Automatically detecting a speaker in the environment further comprises: receiving sound data from a microphone array comprising a plurality of microphones; and A location of the speaker in the environment is detected based at least on the received sound data from the microphone array.

21. The method of claim 12, further comprising: Automatically detecting a speaker in the environment; determining a relative distance between the detected speaker and the whiteboard area containing text; comparing the relative distance between the detected speaker and the whiteboard region containing text to a defined threshold distance; as well as In response to determining that the relative distance between the detected speaker and the whiteboard area containing text is less than the defined threshold distance, determining to include the whiteboard area containing text in the view area selected for display.

22. The method of claim 12, further comprising: Automatically detecting a speaker in the environment; determining a temporary view area including the whiteboard area containing text and at least a portion of the detected speaker; Analyzing the temporary view area to determine a text size or a whiteboard area size of the whiteboard area containing text; and The selecting the view area for display comprises: selecting the temporary view area as the view area for display based at least on the determined text size or whiteboard area size of the whiteboard area containing text.

Images