Information presentation system, area extraction device and program

Abstract

To make items written on a blackboard easy to see by eliminating interference between a projected image and items written on the blackboard.SOLUTION: An information presentation system 1 presents a projected image formed by exiting light from an electronic educational material image e onto a blackboard 8, extracts a blackboard area R' from a captured video C including the projected image and items w written on the blackboard, and applies white illumination light to an area of the written items w on the blackboard 8. The information presentation system 1 includes: a projector 3 for projecting the exiting light from the electronic educational material image e to the blackboard 8 through a light control device 5; a white illumination device 4 for applying white illumination light to the blackboard 8 through the light control device 5; a light control device 5 for applying white illumination light instead of the exiting light from the electronic educational material image e on the basis of written item coordinate information R' (x, y); a camera 6 for capturing the captured image C including the projected image and the written items w; and an area extraction device 2 which extracts a projected image c from the captured image C, extracts a written item area R' by comparing the captured image c and the electronic educational material image e, and outputs the written item coordinate information R' (x, y) to the light control device 5.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information presentation system that optically blocks any emitted light and irradiates the blocked area with white light, and to an area extraction device and program that extracts the area irradiated with white light. [Background technology]

[0002] In the past, the use of ICT (Information and Communication Technology) devices has been increasing in the field of education, and in addition to paper media and blackboard writing, lessons are now being conducted using electronic teaching materials. Lessons using such electronic teaching materials can improve the quality of education.

[0003] For example, students can individually view electronic learning materials using a terminal at hand, or view electronic learning materials projected onto a blackboard, whiteboard, electronic blackboard, or the like installed in the classroom using a projector.

[0004] As an example of a system for projecting electronic teaching materials onto a blackboard, an electronic blackboard system has been proposed (see, for example, Patent Documents 1 and 2).

[0005] The electronic whiteboard system in Patent Document 1 consists of a projector, a streaming device, and a smartphone. In this electronic whiteboard system, the projector and streaming device are connected by an HDMI (registered trademark) cable, and the streaming device and smartphone are connected by Wi-Fi (registered trademark), so that electronic learning materials are transmitted from the smartphone to the projector via the streaming device, and images of the electronic learning materials are projected onto the whiteboard.

[0006] In addition, the electronic blackboard system of Patent Document 2 wirelessly connects a smartphone and a projector, and transmits electronic teaching materials from the smartphone to the projector, thereby projecting an image of the electronic teaching materials onto the blackboard. For example, a teacher operates the smartphone to project the image of the electronic teaching materials onto the blackboard. Then, the teacher conducts the lesson while writing on the blackboard with chalk while the image is projected onto the blackboard. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Publication No. 2023-147131 [Patent Document 2] Japanese Patent Application Publication No. 2024-8542 Summary of the Invention [Problem to be solved by the invention]

[0008] However, in the electronic whiteboard systems of Patent Documents 1 and 2, the image of the electronic teaching material is projected directly onto the blackboard from a projector. This causes interference between the projected image and the characters, diagrams, etc. written on the blackboard (hereinafter referred to as "written matters"), making it difficult to see the written matters. In particular, the colored light contained in the projected image interferes, making it difficult to see the written matters.

[0009] Therefore, the present invention has been made to solve the above-mentioned problems, and its purpose is to provide an information presentation system, an area extraction device, and a program that eliminate interference between the projected image and the written matter on the blackboard, making the written matter easier to see. [Means for solving the problem]

[0010] In order to solve the above-mentioned problem, the information presentation system of claim 1 presents a projected image formed by emitted light of a predetermined image on a blackboard, extracts an area of ​​the written matter as a blackboard writing area from a captured video including the projected image and the written matter written on the blackboard, and irradiates the area of ​​the written matter on the blackboard with white illumination light, and the information presentation system has a projector that projects the emitted light of the predetermined image onto the blackboard via a digital micromirror device to form the projected image, a white illumination device that irradiates the white illumination light onto the blackboard via the digital micromirror device, and the digital micromirror device is made up of a predetermined number of mirrors, and a light control device that tilts all of the mirrors so that all of the emitted light of the specified image from the mirrors is projected onto the blackboard, and tilts some of the mirrors so that, based on coordinate information of the blackboard writing area, the white illumination light from the white illumination device that corresponds to the blackboard writing area is irradiated onto the blackboard; a camera that captures an image including the projected image and the written matter as the captured image; and an area extraction device that extracts an image of the projected image area from the captured image, compares the image of the projected image area with the specified image, extracts an area with a difference as the blackboard writing area, and outputs coordinate information of the blackboard writing area to the light control device.

[0011] The information presentation system of claim 2 is the information presentation system of claim 1, wherein the light control device outputs coordinate information of the illumination area to the area extraction device, and the area extraction device has an input unit that inputs the captured image, an output unit that outputs the predetermined image to the projector, an illumination presence / absence determination unit that receives the coordinate information of the illumination area from the light control device and determines whether or not illumination indicating that the white illumination light is being irradiated onto the blackboard based on the coordinate information of the illumination area, and when it is determined that illumination is not present by the illumination presence / absence determination unit, extracts an image of the area of ​​the projection image from the captured image input by the input unit, and outputs the image of the area of ​​the projection image. an unilluminated blackboard writing area extraction unit that compares the image with the predetermined image to extract an area with a difference as the blackboard writing area and outputs coordinate information of the blackboard writing area to the light control device; and when the illumination presence / absence determination unit determines that illumination is present, extracts an image of the projection image area from the captured image input by the input unit, and compares an image obtained by excluding the illumination area from the image of the projection image area with an image obtained by excluding the illumination area from the predetermined image to extract an area with a difference as a first blackboard writing area, extracts an image of the illumination area from the image of the projection image area as a first image, and extracts an image of the illumination area from an image of the same area as the projection image, which is an image obtained by irradiating the white illumination light onto the blackboard, as a second image; The device is characterized by comprising an illuminated blackboard area extraction unit that compares the first image with the second image to determine whether the blackboard matter is included in the illuminated area within the area of ​​the projected image, and if it is determined that the blackboard matter is not included, treats the first blackboard area as the blackboard area and outputs coordinate information of the blackboard area to the light control device, and if it is determined that the blackboard matter is included, treats the first blackboard area and the illuminated area as the blackboard area and outputs coordinate information of the blackboard area to the light control device.

[0012] Furthermore, the area extraction device of claim 3 is an area extraction device that inputs a captured image that includes a projected image formed on a blackboard by emission light of a predetermined image from a projector and that includes matters written on the blackboard by a user, extracts an area including the matters written on the blackboard as a blackboard area based on the captured image, and irradiates white illumination light toward the area of ​​the matters written on the blackboard, and is equipped with an input unit that inputs the captured image, an output unit that outputs the predetermined image to the projector, and a blackboard area extraction unit that extracts an image of the area of ​​the projected image from the captured image, compares the image of the area of ​​the projected image with the predetermined image, thereby extracting an area with a difference as the blackboard area, and outputs coordinate information of the blackboard area to a light control device, and is characterized in that the light control device irradiates the white illumination light toward the area of ​​the matters written on the blackboard based on the coordinate information of the blackboard area instead of projecting a portion of the emission light of the predetermined image.

[0013] Furthermore, the program of claim 4 is characterized in that the computer constituting an area extraction device inputs a captured image including a projected image formed on a blackboard by emission light of a predetermined image from a projector and including a matter written on the blackboard by a user, extracts an area including the matter written on the blackboard as a blackboard area based on the captured image, and irradiates white illumination light toward the area of ​​the matter written on the blackboard, the computer functioning as an input unit that inputs the captured image, an output unit that outputs the predetermined image to the projector, and a blackboard area extraction unit that extracts an image of the area of ​​the projected image from the captured image, compares the image of the area of ​​the projected image with the predetermined image, extracts an area with a difference as the blackboard area, and outputs coordinate information of the blackboard area to a light control device, and causes the light control device to irradiate the white illumination light toward the area of ​​the matter written on the blackboard based on the coordinate information of the blackboard area instead of projecting a portion of the emission light of the predetermined image toward the area of ​​the matter written on the blackboard. [Effects of the Invention]

[0014] As described above, according to the present invention, interference between the projected image and the matter written on the board can be eliminated, making the matter written on the board easier to see. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a schematic diagram illustrating an example of the overall configuration of an information presentation system according to an embodiment of the present invention. [Figure 2] FIG. 1 is a block diagram illustrating an example of the configuration of a region extraction device. [Figure 3] 10 is a flowchart illustrating an example of processing by the area extraction device. [Figure 4] (1) A diagram showing an example of an image A captured before the projector and white illumination device are turned on. (2) A diagram showing an example of an image B1 captured using emitted light after the projector is turned on. (3) A diagram showing an example of an image B2 captured using white illumination light after the white illumination device is turned on. [Figure 5] (1) A diagram showing an example of a captured image C onto which electronic learning materials are projected when there is no lighting. (2) A diagram showing an example of a captured image C onto which electronic learning materials are projected when there is lighting. [Figure 6] (1) A diagram showing an example of projection image coordinate information b1'(x,y). (2) A diagram showing an example of illumination coordinate information Q'(x,y). (3) A diagram showing an example of blackboard coordinate information R'(x,y). [Figure 7] FIG. 2 is a block diagram showing an example of the configuration of an initial processing unit. [Figure 8] 10 is a flowchart illustrating an example of processing by an initial processing unit. [Figure 9] 10A to 10C are diagrams illustrating an example of processing by a homography transformation coordinate information generating unit. [Figure 10] 10A and 10B are diagrams illustrating an example of processing by a projection image coordinate information generating unit. [Figure 11] 10A to 10C are diagrams illustrating an example of processing by a white illumination rectangular image generating unit. [Figure 12] FIG. 2 is a block diagram showing an example of the configuration of a blackboard area extraction unit. [Figure 13] 10 is a flowchart showing an example of processing by a blackboard area extraction unit; [Figure 14] FIG. 13 is a diagram illustrating an example of the blackboard area extraction process (step S1303) when there is no illumination. [Figure 15]FIG. 13 is a diagram illustrating an example of the blackboard area extraction process (step S1304) when illumination is on. [Figure 16] FIG. 16 is a diagram illustrating the continuation of FIG. 15. DETAILED DESCRIPTION OF THE INVENTION

[0016] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. [Information presentation system] First, an information presentation system according to an embodiment of the present invention will be described. Fig. 1 is a schematic diagram showing an example of the overall configuration of an information presentation system according to an embodiment of the present invention.

[0017] This information presentation system 1 is composed of an area extraction device 2, a projector 3, a white lighting device 4, a light control device 5, a camera 6, and lenses 7-1, 7-2, and 7-3. The information presentation system 1 presents a projected image formed by the light of an electronic teaching material image e emitted from the projector 3 via a digital micromirror device (DMD) included in the light control device 5 onto a blackboard 8. The information presentation system 1 also extracts an area including the written matter as a blackboard writing area from a captured video including the projected image and matters written by a user (in the area of ​​the projected image on the blackboard 8), and irradiates white illumination light from the white lighting device 4 toward the area of ​​the written matter on the blackboard 8 via the digital micromirror device corresponding to the blackboard writing area.

[0018] In the initial processing described later, the area extraction device 2 inputs the captured image from the camera 6, generates projection image coordinate information b1'(x,y) indicating the coordinate information of the projection image based on the captured image, and outputs the projection image coordinate information b1'(x,y) to the light control device 5.

[0019] This projected image coordinate information b1'(x, y) is information indicating xy coordinates that reflect the resolution of the digital micromirror device included in the light control device 5. The projected image coordinate information b1'(x, y) will be described in detail later.

[0020] Furthermore, the area extraction device 2 outputs the electronic learning material image e to the projector 3. Then, the area extraction device 2 inputs a captured image including a projected image of the electronic learning material image e and matters written on the blackboard from the camera 6, extracts the projected image of the electronic learning material image e from the captured image, and compares the projected image with the electronic learning material image e to extract a blackboard area from the area of ​​the projected image. Then, the area extraction device 2 outputs coordinate information of the blackboard area to the light control device 5. Details of the area extraction device 2 will be described later.

[0021] Here, the projected image is an image formed by projecting light emitted from the projector 3 onto the blackboard 8 via the light control device 5, and is an image that can be seen by users (e.g., students and teachers) looking at the blackboard 8. The coordinate information is information indicating x and y coordinates based on the resolution of the digital micromirror device included in the light control device 5.

[0022] The projector 3 receives the electronic learning material image e from the region extraction device 2, and outputs the emitted light of the electronic learning material image e to the light control device 5 through the lens 7-1.

[0023] The white illumination device 4 is a white LED or a white light bulb, and outputs white illumination light to the light control device 5 through a lens 7-2.

[0024] The light control device 5 has a digital micromirror device and has the function of projecting the light emitted from the projector 3 of the electronic teaching material image e onto the blackboard 8 through the lens 7-3, and the function of irradiating the white illumination light from the white illumination device 4 onto the blackboard 8 through the lens 7-3. The digital micromirror device is composed of a predetermined number of mirrors arranged in an array.

[0025] The projected image of the electronic teaching material image e by the emitted light from the projector 3 is formed on the digital micromirror device via the lens 7-1, and then formed on the blackboard 8 via the lens 7-3. This allows the emitted light of the electronic teaching material image e to be projected onto the blackboard 8 with high quality.

[0026] The light control device 5 receives the projection image coordinate information b1'(x,y) from the area extraction device 2 and forms an xy plane based on the projection image coordinate information b1'(x,y). This allows the area indicated by the projection image coordinate information b1'(x,y) to correspond to the area of ​​all light emitted from the projector 3. In other words, the area indicated by the projection image coordinate information b1'(x,y) can correspond to a predetermined number of mirrors (all mirrors) that make up the digital micromirror device.

[0027] The light control device 5 then inputs coordinate information of the blackboard writing area from the area extraction device 2, and changes the tilt of each of the multiple mirrors constituting the digital micromirror device, for example, on a pixel-by-pixel basis (mirror-by-mirror basis), based on the coordinate information of the blackboard writing area. In other words, the light control device 5 switches between the emitted light of the electronic teaching material image e from the projector 3 and the white illumination light from the white illumination device 4 on a pixel-by-pixel basis, based on the coordinate information of the blackboard writing area on the xy plane.

[0028] Specifically, when the light control device 5 does not receive coordinate information of the blackboard writing area from the area extraction device 2 (when it receives coordinate information of a null blackboard writing area), it tilts all of the mirrors constituting the digital micromirror device so that all of the light emitted from the projector 3 of the electronic teaching material image e is projected onto the blackboard 8. This forms an optical path from the projector 3 to the blackboard 8 via the light control device 5.

[0029] On the other hand, when the light control device 5 receives coordinate information of the blackboard writing area from the area extraction device 2, it tilts some of the mirrors constituting the digital micromirror device that correspond to the blackboard writing area so that the white illumination light from the white illumination device 4 that corresponds to the blackboard writing area is irradiated onto the blackboard 8, and the area of ​​the mirrors that correspond to the blackboard writing area (the blackboard writing area) is set as the illumination area. In this case, the remaining mirrors that correspond to areas other than the blackboard writing area are tilted so that the light emitted from the projector 3 of the electronic teaching material image e is projected onto the blackboard 8. In other words, for the blackboard writing area, an optical path is formed from the white illumination device 4 to the blackboard 8 via the light control device 5, and for areas other than the blackboard writing area, an optical path is formed from the projector 3 to the blackboard 8 via the light control device 5.

[0030] As a result, the light emitted from the electronic teaching material image e does not reach the area on the blackboard 8 where the written matter is written, but the white illumination light reaches it.

[0031] Furthermore, the light control device 5 outputs coordinate information of the illumination area on the xy plane to the area extraction device 2. Here, if the white illumination light corresponding to the blackboard writing area is not irradiated onto the blackboard 8, that is, if some of the mirrors for irradiating the white illumination light onto the blackboard 8 are not tilted, the light control device 5 sets coordinate information of a null illumination area and outputs it to the area extraction device 2.

[0032] In this way, when the light control device 5 receives coordinate information of the null blackboard writing area from the area extraction device 2, a projected image reflecting all of the emitted light of the electronic learning material image e is formed on the blackboard 8. On the other hand, when the light control device 5 receives coordinate information of the blackboard writing area, white illumination light is irradiated onto the blackboard writing area instead of projecting the emitted light of the electronic learning material image e. Therefore, a projected image reflecting the emitted light of the electronic learning material image e excluding a portion (the blackboard writing area) is formed on the blackboard 8, and the blackboard writing area is irradiated with white illumination light.

[0033] The light control device 5 may perform the above-described processing based on the coordinate information of the blackboard area input from the area extraction device 2 at a cycle longer than the cycle at which the coordinate information of the blackboard area is generated by the area extraction device 2. This reduces flickering of the projected image on the blackboard 8 caused by changes in the tilt of the mirror.

[0034] The camera 6 is installed facing the blackboard 8, and when the projector 3 and the white lighting device 4 are not activated, the camera 6 captures an image including the blackboard 8 (photographed image A shown in FIG. 4(1) to be described later). Furthermore, when all mirrors provided in the light control device 5 are tilted so that an optical path from the projector 3 to the blackboard 8 via the light control device 5 is formed, and the projector 3 is activated but no light of the electronic teaching material image e is output from the projector 3, or when the projector 3 is released from its sleep state, the camera 6 captures an image including a white projection image b1 on the blackboard 8 (photographed image B1 shown in FIG. 4(2) to be described later). Furthermore, when all mirrors provided in the light control device 5 are tilted so that an optical path from the white lighting device 4 to the blackboard 8 via the light control device 5 is formed, and the white lighting device 4 is activated, the camera 6 captures an image including a white lighting image b2 on the blackboard 8 (photographed image B2 shown in FIG. 4(3) to be described later).

[0035] Furthermore, when the projector 3 is activated and emits light of the electronic teaching material image e, forming a projected image on the blackboard 8, the camera 6 captures an image including the projected image c on the blackboard 8 (a captured image C shown in FIGS. 5(1) and (2) described later). Then, the camera 6 outputs the captured images A, B1, B2, and C to the area extraction device 2.

[0036] The emitted light of the electronic teaching material image e is projected as a projected image from the projector 3 via the light control device 5 onto the blackboard 8, and the user writes on the blackboard, thereby writing items. The items are written within the area of ​​the projected image.

[0037] The blackboard 8 includes not only a general blackboard but also a whiteboard, an electronic blackboard, or the like equivalent to a blackboard used in an educational setting.

[0038] In this way, in the information presentation system 1, when the projected image from the projector 3 on the blackboard 8 and the written matter overlap, instead of irradiating the emitted light of the electronic teaching material image e from the projector 3 in an arbitrary shape corresponding to the blackboard writing area, white illumination light from the white illumination device 4 is irradiated.

[0039] As a result, the light emitted from the projector 3 does not reach the area of ​​the blackboard 8 where the written matter is written, and therefore the written matter on the blackboard 8 is not affected by the light emitted from the projector 3. Furthermore, the white illumination light from the white illumination device 4 reaches the area of ​​the blackboard 8 where the written matter is written, and therefore the written matter on the blackboard 8 becomes even easier to see.

[0040] Therefore, the user can clearly distinguish between the image projected onto the blackboard 8 and the written matter written on the blackboard 8. In other words, interference between the projected image and the written matter can be eliminated, making it easier to see the written matter when the electronic teaching material image e is projected.

[0041] [Region extraction device 2] Next, a detailed description will be given of the region extraction device 2 shown in Fig. 1. Fig. 2 is a block diagram showing an example of the configuration of the region extraction device 2, and Fig. 3 is a flowchart showing an example of processing by the region extraction device 2.

[0042] The area extraction device 2 includes a video input unit 10, memory units 11 and 14, an electronic teaching material output unit 12, and an information recognition unit 13. The information recognition unit 13 includes an initial processing unit 20 and a blackboard area extraction unit 21.

[0043] 2 and 3, the processing of the area extraction device 2 will be described below. In the initial processing for calibration, the video input unit 10 inputs the captured video A from the camera 6 before the projector 3 and the white lighting device 4 are started up, and also inputs the captured videos B1 and B2 after the projector 3 is started up (step S301), and outputs the captured videos A, B1, and B2 to the initial processing unit 20.

[0044] The initial processing unit 20 receives the captured images A, B1, and B2 from the image input unit 10. The initial processing unit 20 then performs initial processing based on the captured images A, B1, and B2 to generate transformation coordinate information cv, projection image coordinate information b1(X,Y), b1'(x,y), brightness correction information br, and a white-illuminated rectangular image b2', and stores these parameters in the storage unit 14 (step S302). The initial processing unit 20 also outputs the projection image coordinate information b1'(x,y) to the light control device 5 (step S303).

[0045] In this case, before inputting the captured image B1 from the image input unit 10, the initial processing unit 20 outputs mirror direction instruction information indicating the projector side to the light control device 5 in order to tilt all of the mirrors provided in the light control device 5 so that an optical path is formed from the projector 3 to the blackboard 8 via the light control device 5. When the mirror direction instruction information indicating the projector side is input, the light control device 5 tilts all of the mirrors so that an optical path is formed from the projector 3 to the blackboard 8 via the light control device 5.

[0046] Furthermore, before inputting the captured image B2 from the image input unit 10, the initial processing unit 20 outputs mirror direction instruction information indicating the white illumination side to the light control device 5 in order to tilt all of the mirrors provided in the light control device 5 so that an optical path is formed from the white illumination device 4 to the blackboard 8 via the light control device 5. Upon receiving the mirror direction instruction information indicating the white illumination side, the light control device 5 tilts all of the mirrors so that an optical path is formed from the white illumination device 4 to the blackboard 8 via the light control device 5. Details of the initial processing by the initial processing unit 20 will be described later.

[0047] 4(1) is a diagram showing an example of a captured image A before the projector 3 and the white lighting device 4 are activated. The captured image A is an image including the blackboard 8 before the projector 3 and the white lighting device 4 are activated, and does not include a projected image due to the light emitted from the projector 3 or an image due to the white illumination light from the white lighting device 4. It is also assumed that no matter w is written on the blackboard 8.

[0048] FIG. 4(2) is a diagram showing an example of a captured image B1 of emitted light from the projector 3 after it is started up. The captured image B1 is an image including a white projection image b1 projected onto the blackboard 8 after the projector 3 is started up (at startup) or when it is woken up from sleep mode, with an optical path from the projector 3 to the blackboard 8 formed via the light control device 5. The white projection image b1 is a white projection image of emitted light from the projector 3, and is not a projection image c of an electronic teaching material image e, which will be described later. Since all mirrors provided in the light control device 5 are tilted so that an optical path from the projector 3 to the blackboard 8 is formed via the light control device 5, the white projection image b1 of all emitted light from the projector 3 appears on the blackboard 8. It is also assumed that no matter w is written on the blackboard 8.

[0049] FIG. 4(3) is a diagram showing an example of a captured image B2 using white illumination light after the white illumination device 4 is turned on. The captured image B2 includes a white illumination image b2 formed on the blackboard 8 by irradiating the blackboard 8 with white illumination light from the white illumination device 4 while an optical path is formed from the white illumination device 4 via the light control device 5 to the blackboard 8. The white illumination image b2 is a white image formed by the white illumination light from the white illumination device 4, and its size is the same as the white projection image b1 shown in FIG. 4(2). Since all mirrors provided in the light control device 5 are tilted so that an optical path is formed from the white illumination device 4 via the light control device 5 to the blackboard 8, the white illumination image b2 formed by all the white illumination light from the white illumination device 4 appears on the blackboard 8. It is also assumed that no matter w is written on the blackboard 8.

[0050] The conversion coordinate information cv generated by the initial processing is coordinate information for converting a distorted projection image or area into a distortion-free projection image or area that reflects the resolution of the digital micromirror device included in the light control device 5 in order to eliminate distortion in the white projection image b1 included in the captured image B1 and the projection image c included in the captured image C described below.

[0051] The projection image coordinate information b1(X, Y) corresponds to the area where all the light emitted from the projector 3 is projected onto the blackboard 8, and is coordinate information of the area of ​​the distorted white projection image b1 included in the captured image B1.

[0052] The projection image coordinate information b1'(x,y) corresponds to the area where all the light emitted from the projector 3 is projected onto the blackboard 8, and as described above, is coordinate information that reflects the resolution of the digital micromirror device included in the light control device 5. The projection image coordinate information b1'(x,y) is obtained by using the conversion coordinate information cv to remove distortion from the area of ​​the distorted white projection image b1 included in the captured video B1 (the area indicated by the projection image coordinate information b1(X,Y)).

[0053] 6(1) is a diagram showing an example of projection image coordinate information b1'(x,y). As mentioned above, the projection image coordinate information b1'(x,y) is the coordinate information of the white rectangular projection image b1' obtained by eliminating the distortion of the white projection image b1 included in the captured image B1 shown in FIG. 4(2). The rectangular area indicated by this projection image coordinate information b1'(x,y) corresponds to the rectangular area of ​​all the outgoing light output by the projector 3 in the light control device 5, and also corresponds to the multiple mirrors that project all the outgoing light output by the projector 3 onto the blackboard 8.

[0054] 2 and 3, the brightness correction information br is information for correcting the brightness value of the electronic teaching material image e to the brightness value of the projected image c of the electronic teaching material image e. The brightness correction information br is generated from the brightness value b1'(BR) of the white rectangular projected image b1' (described later) and a preset brightness value PBR of white in the emitted light of the projector 3.

[0055] The white illumination rectangular image b2' is an image of the inside of the blackboard 8 illuminated by white illumination light from the white illumination device 4 after the white illumination device 4 is turned on, and is a rectangular image converted using the conversion coordinate information cv from an image of the same area as the white projection image b1 obtained after the projector 3 is turned on.

[0056] The details of the conversion coordinate information cv, the projection image coordinate information b1(X,Y), b1'(x,y), the brightness correction information br, and the method for generating the white-illuminated rectangular image b2' will be described later.

[0057] In the blackboard area extraction process, the electronic learning material output unit 12 and the blackboard area extraction unit 21 read out the electronic learning material image e from the storage unit 11. Then, the electronic learning material output unit 12 outputs the electronic learning material image e to the projector 3 (step S304). As a result, the emitted light of the electronic learning material image e is output from the projector 3 to the light control device 5, and a projected image is formed on the blackboard 8 via the light control device 5.

[0058] Furthermore, in the blackboard writing area extraction process, the video input unit 10 inputs the captured video C from the camera 6 at regular time intervals, and the blackboard writing area extraction unit 21 inputs the illumination coordinate information Q'(x,y) from the light control device 5 (step S305). The video input unit 10 then outputs the captured video C to the blackboard writing area extraction unit 21. The blackboard writing area extraction unit 21 inputs the captured video C from the video input unit 10, and reads out the transformation coordinate information cv, the projection image coordinate information b1(X,Y), b1'(x,y), the brightness correction information br, and the parameters of the white-illuminated rectangular video b2' from the storage unit 14.

[0059] The board writing area extraction unit 21 performs a board writing area extraction process based on the captured image C, the lighting coordinate information Q'(x, y) and these parameters to generate coordinate information of an area including the matter written on the board as board writing coordinate information R'(x, y) (step S306) and outputs the board writing coordinate information R'(x, y) to the light control device 5 (step S307). Details of the board writing area extraction process by the board writing area extraction unit 21 will be described later.

[0060] The area extraction device 2 repeats the processes of steps S304 to S307 unless the processing of the area extraction device 2 is completed (step S308: N), and ends the processing of the area extraction device 2 in a predetermined case (step S308: Y).

[0061] 5(1) is a diagram showing an example of a captured image C in which an electronic teaching material is projected without illumination. This captured image C is an image including a projection image c of an electronic teaching material image e projected onto the blackboard 8 when white illumination light from the white illumination device 4 is not irradiated. The captured image C includes, in the area of ​​the projection image c, the blackboard item w "XXX" written on the blackboard 8 by the user.

[0062] Since part of the projected image c and the written matter w "XXX" overlap on the blackboard 8, it is difficult for a user looking at the blackboard 8 to clearly distinguish between the projected image of the electronic teaching material image e and the written matter w "XXX".

[0063] 5(2) is a diagram showing an example of a captured image C in which an electronic learning material is projected when illumination is on. This captured image C is an image including a projected image c of an electronic learning material image e projected onto the blackboard 8 when white illumination light is irradiated from the white illumination device 4. In the projected image c, an illumination area Q is formed by the irradiation of the white illumination light, and in the illumination area Q, the emitted light of the electronic learning material image e is not projected, and the blackboard note w "XXX" written on the blackboard 8 by the user is present.

[0064] Therefore, a user looking at the blackboard 8 can clearly distinguish between the projected image of the electronic teaching material image e (the projected image of the portion other than the illumination area Q) and the written matter w "XXX".

[0065] Fig. 6(2) is a diagram showing an example of illumination coordinate information Q'(x,y). As shown in Fig. 6(2), the illumination coordinate information Q'(x,y) is coordinate information of each position of the illumination area Q' where the white illumination light from the white illumination device 4 is irradiated onto the blackboard 8 by the light control device 5 on the xy plane representing the projection image coordinate information b1'(x,y). The dotted square is the area indicated by the projection image coordinate information b1'(x,y).

[0066] Fig. 6(3) is a diagram showing an example of the board coordinate information R'(x,y). As shown in Fig. 6(3), the board coordinate information R'(x,y) is coordinate information of each position of the board area R' including the written matter w on the xy plane representing the projection image coordinate information b1'(x,y). The dotted square is the area indicated by the projection image coordinate information b1'(x,y).

[0067] (Initial processing unit 20) Next, a detailed description will be given of the initial processing by the initial processing unit 20 shown in Figures 2 and 3. Figure 7 is a block diagram showing an example of the configuration of the initial processing unit 20.

[0068] The initial processing unit 20 includes a homography conversion coordinate information generating unit 30, a projection image coordinate information generating unit 31, a luminance correction information generating unit 32, and a white illumination rectangular image generating unit 33.

[0069] Fig. 8 is a flowchart showing an example of processing by the initial processing unit 20. Fig. 9 is a diagram illustrating an example of processing by the homography transformation coordinate information generation unit 30, Fig. 10 is a diagram illustrating an example of processing by the projection image coordinate information generation unit 31, and Fig. 11 is a diagram illustrating an example of processing by the white illumination rectangular image generation unit 33.

[0070] <Homography transformation coordinate information generation unit 30> The homography transformation coordinate information generating unit 30 inputs the captured image A from the image input unit 10 (step S801). Referring to Fig. 9, the captured image A is an image before the projector 3 and the white illumination device 4 are activated, as shown in Fig. 4(1), and does not include the white projection image b1, the white illumination image b2, and the projection image c of the electronic teaching material image e. Note that the blackboard 8 is omitted from the captured image A in Fig. 9. The same applies to the captured image B1 in Fig. 9 and the captured image B2 in Fig. 11, which will be described later.

[0071] The homography transformation coordinate information generating unit 30 outputs mirror direction instruction information indicating the projector side to the light control device 5 (step S802). As a result, all mirrors provided in the light control device 5 are tilted so as to form an optical path from the projector 3 to the blackboard 8 via the light control device 5. Then, the camera 6 captures the captured image B1 shown in FIG. 4(2), and the captured image B1 is input to the image input unit 10.

[0072] The homography transformation coordinate information generating unit 30 inputs the captured image B1 from the image input unit 10 (step S803). Referring to Fig. 9, the captured image B1 is an image including a white projection image b1 projected onto the blackboard 8 after the projector 3 is started, as shown in Fig. 4(2).

[0073] The homography transformation coordinate information generating unit 30 compares the captured image A with the captured image B1, calculates the difference between the captured images A and B1, and extracts a white projection image b1 from the captured image B1 (step S804). Because the only difference between the captured images A and B1 acquired before and after the projector 3 is the white projection image b1, the white projection image b1 is extracted. As shown in FIG. 4(2), the white projection image b1 is a distorted projection image.

[0074] Specifically, the coordinate information generation unit 30 for homography transformation generates a difference image from the captured images A and B1, performs edge extraction on the difference image using a Sobel filter, and identifies areas where the change in brightness is greater than or equal to a predetermined threshold, thereby extracting a white projection image b1 from the captured image B1.

[0075] The homography transformation coordinate information generating unit 30 forms a predetermined XY plane for the captured image B1 including the white projection image b1, and generates two-dimensional coordinate information of the white projection image b1 as projection image coordinate information b1(X,Y) (step S805). Referring to Fig. 9, the projection image coordinate information b1(X,Y), which is the two-dimensional coordinate information of the white projection image b1, is generated from the difference between the captured images A and B1.

[0076] The homography transformation coordinate information generating unit 30 performs homography transformation from the distorted shape indicated by the projection image coordinate information b1(X,Y) (indicated by the area of ​​the white projection image b1) to a rectangular shape by a preset projector 3, thereby generating transformation coordinate information cv, and stores the transformation coordinate information cv in the storage unit 14 (step S806). Referring to Fig. 9, the transformation coordinate information cv is generated from the projection image coordinate information b1(X,Y).

[0077] The homography transformation for generating the transformation coordinate information cv is performed so that the projection image coordinate information b1(X,Y), which is the coordinate information of the distorted white projection image b1, corresponds to rectangular coordinate information that reflects the resolution of the digital micromirror device possessed by the light control device 5.

[0078] The conversion coordinate information cv is a parameter for converting the distorted white projection image b1 shown in Figure 4(2) into a distortion-free rectangular white rectangular projection image b1', and is also a parameter for converting the projection image c shown in Figures 5(1) and (2) into a distortion-free rectangular projection image c' described later.

[0079] <Projected image coordinate information generation unit 31> The projection image coordinate information generation unit 31 stores the projection image coordinate information b1(X,Y) generated by the coordinate information generation unit for homography transformation 30 in step S805 in the memory unit 14, and also generates rectangular projection image coordinate information b1'(x,y) by homography transforming the projection image coordinate information b1(X,Y) using the transformation coordinate information cv, outputs the projection image coordinate information b1'(x,y) to the light control device 5, and stores it in the memory unit 14 (step S807).

[0080] Referring to Figure 10, using the transformation coordinate information cv, the area indicated by the projection image coordinate information b1(X,Y), which is the area of ​​the white projection image b1, is homography transformed into the area indicated by the projection image coordinate information b1'(x,y), which is the area of ​​the white rectangular projection image b1'.

[0081] <Brightness correction information generation unit 32> The brightness correction information generation unit 32 uses the transformation coordinate information cv to perform homography transformation on the white projection image b1 extracted by the homography transformation coordinate information generation unit 30 in step S804, thereby generating a white rectangular projection image b1', and generates brightness correction information br based on the brightness value b1'(BR) of the white rectangular projection image b1' and the preset brightness value PBR of the white light emitted by the projector 3, and stores the brightness correction information br in the memory unit 14 (step S808).

[0082] For example, the luminance correction information generating unit 32 obtains the luminance correction information br by dividing the luminance value b1'(BR) of the white rectangular projected image b1' by a preset luminance value PBR.

[0083] <White illumination rectangular image generating unit 33> The white-illumination rectangular image generating unit 33 outputs mirror direction instruction information indicating the white-illumination side to the light control device 5 (step S809). As a result, all mirrors provided in the light control device 5 are tilted so as to form an optical path from the white-illumination device 4 to the blackboard 8 via the light control device 5. Then, the camera 6 captures the captured image B2 shown in FIG. 4(3), and the captured image B2 is input to the image input unit 10.

[0084] The white-illumination rectangular image generator 33 inputs the captured image B2 from the image input unit 10 (step S810). The captured image B2 is an image including a white-illumination image b2 formed by irradiating the blackboard 8 with white illumination light from the white illumination device 4, as shown in FIG. 4(3).

[0085] The white-illuminated rectangular image generation unit 33 uses the transformation coordinate information cv generated by the homography transformation coordinate information generation unit 30 in step S806 to perform homography transformation on the white-illuminated image b2 (white-illuminated image b2 of the same size as the white projection image b1) included in the captured image B2, thereby generating a white-illuminated rectangular image b2', and stores the white-illuminated rectangular image b2' in the memory unit 14 (step S811).

[0086] Referring to FIG. 11, the region of the white-illuminated image b2 is homography transformed into the region of the white-illuminated rectangular image b2' using the transformation coordinate information cv.

[0087] In this way, the initial processing unit 20 generates the transformation coordinate information cv, the projection image coordinate information b1(X,Y), b1'(x,y), the brightness correction information br, and the white illumination rectangular image b2' based on the captured images A, B1, and B2, and stores these parameters in the memory unit 14, and outputs the projection image coordinate information b1'(x,y) to the light control device 5.

[0088] (Blackboard area extraction unit 21) Next, a detailed description will be given of the blackboard area extraction process performed by the blackboard area extraction unit 21 shown in Figures 2 and 3. Figure 12 is a block diagram showing an example of the configuration of the blackboard area extraction unit 21.

[0089] The board writing area extraction unit 21 includes an illumination presence / absence determination unit 40, an unilluminated board writing area extraction unit 41, and an illuminated board writing area extraction unit .

[0090] Fig. 13 is a flowchart showing an example of processing by the blackboard area extraction unit 21. Fig. 14 is a diagram explaining an example of processing for extracting a blackboard area without illumination (step S1303), Fig. 15 is a diagram explaining an example of processing for extracting a blackboard area with illumination (step S1304), and Fig. 16 is a continuation of Fig. 15.

[0091] Hereinafter, the processing of the illumination presence / absence determining unit 40, the unilluminated blackboard writing area extracting unit 41, and the illuminated blackboard writing area extracting unit 42 will be described with reference to FIGS.

[0092] When the initial processing by the initial processing unit 20 is completed, prior to the processing by the illumination presence / absence determination unit 40, the unilluminated board writing area extraction unit 41, and the illuminated board writing area extraction unit 42, the board writing area extraction unit 21 outputs mirror direction instruction information indicating the projector side to the light control device 5. As a result, all mirrors provided in the light control device 5 are tilted so as to form an optical path from the projector 3 to the blackboard 8 via the light control device 5. Then, the camera 6 captures the captured image C shown in FIG. 5(1), and the captured image C is input to the image input unit 10.

[0093] In addition, when the light control device 5 irradiates the blackboard 8 with white illumination light from the white illumination device 4 to form an illumination area Q, the camera 6 captures the captured image C shown in Figure 5 (2), and the captured image C is input to the image processing unit 10.

[0094] <Lighting presence / absence determination unit 40> The illumination presence / absence determining unit 40 receives the captured image C from the image input unit 10 and also receives the illumination coordinate information Q'(x, y) from the light control device 5 (step S1301).

[0095] The illumination presence / absence determination unit 40 determines whether or not illumination is present, which indicates that white illumination light is being irradiated onto the blackboard 8, based on the illumination coordinate information Q'(x, y) (step S1302). Specifically, when the illumination coordinate information Q'(x, y) is null, the illumination presence / absence determination unit 40 determines that illumination is absent, which indicates that no optical path is formed from the white illumination device 4 to the blackboard 8 via the light control device 5 (the illumination area Q is not formed), and white illumination light is not being irradiated onto the blackboard 8. On the other hand, when the illumination coordinate information Q'(x, y) is not null, the illumination presence / absence determination unit 40 determines that illumination is present, which indicates that a light path is formed from the white illumination device 4 to the blackboard 8 via the light control device 5 (the illumination area Q is formed), and white illumination light is being irradiated onto the blackboard 8.

[0096] If it is determined in step S1302 that there is no illumination (step S1302: N), the illumination presence / absence determination unit 40 outputs information indicating that there is no illumination to the unilluminated blackboard writing area extraction unit 41. On the other hand, if it is determined in step S1302 that there is illumination (step S1302: Y), the illumination presence / absence determination unit 40 outputs information indicating that there is illumination to the illuminated blackboard writing area extraction unit 42.

[0097] <Unilluminated board writing area extraction unit 41> Upon receiving information indicating no illumination from the illumination presence / absence determination unit 40, the non-illuminated board writing area extraction unit 41 proceeds from step S1302(N) and performs a non-illuminated board writing area extraction process (step S1303) to generate board writing coordinate information R'(x, y).Then, the non-illuminated board writing area extraction unit 41 outputs the board writing coordinate information R'(x, y) to the light control device 5 (step S1305).

[0098] An example of the process of extracting a blackboard area without illumination by the non-illuminated blackboard area extracting section 41 will be described in detail with reference to FIG.

[0099] The unilluminated blackboard writing area extraction unit 41 reads the transformation coordinate information cv, the projection image coordinate information b1(X,Y), b1'(x,y), and the brightness correction information br from the storage unit 14. Then, the unilluminated blackboard writing area extraction unit 41 extracts the projection image c from the captured image C using the projection image coordinate information b1(X,Y) (step S1401). That is, the unilluminated blackboard writing area extraction unit 41 extracts the area indicated by the projection image coordinate information b1(X,Y) from the captured image C and obtains the projection image c, which is the image of this area. Here, the projection image c is an image that exists in the same area of ​​the captured image C as the projected image formed by the light emitted from the projector 3 of the electronic learning material image e, and includes the written matter w1 "XXX" in addition to the projected image formed by the light emitted from the projector 3 of the electronic learning material image e.

[0100] The non-illuminated blackboard region extraction unit 41 performs homography transformation on the distorted projection image c using the transformation coordinate information cv, thereby generating a rectangular projection image c' that is undistorted (step S1402).

[0101] As a result, the blackboard item w1 "XXX" contained in the projection image c is also homography transformed, and in step S1404 described below, a rectangular image containing the blackboard item w1 "XXX" is identified, and this rectangular area is generated as the blackboard area R'.

[0102] Meanwhile, the unilluminated blackboard region extraction unit 41 reads out the electronic learning material image e from the storage unit 11, corrects the luminance of the electronic learning material image e using the luminance correction information br, and generates a luminance-corrected electronic learning material image e' (step S1403).

[0103] For example, the unilluminated blackboard region extraction unit 41 multiplies the brightness value e(BR) of the electronic learning material image e by the brightness correction information br to obtain the brightness value e'(BR) of the brightness-corrected electronic learning material image e'. This generates the brightness-corrected electronic learning material image e'.

[0104] The unilluminated blackboard area extraction unit 41 compares the rectangular projection image c' generated in step S1402 with the brightness-corrected electronic learning material image e' generated in step S1403, calculates the difference between the rectangular projection image c' and the brightness-corrected electronic learning material image e', and extracts the blackboard item w1 "XXX" from the rectangular projection image c'.Then, the unilluminated blackboard area extraction unit 41 identifies a rectangular image including the blackboard item w1 "XXX", generates this rectangular area as a blackboard area R', and generates coordinate information of the blackboard area R' as blackboard coordinate information R'(x, y) (step S1404).

[0105] Specifically, the unilluminated blackboard area extraction unit 41 generates a difference image from the rectangular projection image c' and the brightness-corrected electronic teaching material image e', performs edge extraction on the difference image using a Sobel filter, and identifies areas where the change in brightness is greater than or equal to a predetermined threshold, thereby extracting the blackboard item w1 ``XXX'' from the rectangular projection image c'.

[0106] <Illuminated blackboard writing area extraction unit 42> 12 and 13, when the illuminated blackboard writing area extraction unit 42 receives information indicating that illumination is present from the illumination presence / absence determination unit 40 after proceeding from step S1302(Y), it performs an illuminated blackboard writing area extraction process (step S1304) and generates blackboard writing coordinate information R'(x, y).Then, the illuminated blackboard writing area extraction unit 42 outputs the blackboard writing coordinate information R'(x, y) to the light control device 5 (step S1305).

[0107] An example of the process of extracting a blackboard area with illumination by the illuminated blackboard area extracting unit 42 will be described in detail with reference to FIGS.

[0108] The illuminated blackboard writing area extraction unit 42 reads the transformation coordinate information cv, the projection image coordinate information b1(X,Y), b1'(x,y), the brightness correction information br, and the white-light rectangular image b2' from the memory unit 14. Then, the illuminated blackboard writing area extraction unit 42 uses the projection image coordinate information b1(X,Y) to extract the projection image c from the captured image C (step S1501). That is, the illuminated blackboard writing area extraction unit 42 extracts the area indicated by the projection image coordinate information b1(X,Y) from the area of ​​the captured image C, and obtains the projection image c, which is the image of this area.

[0109] Here, the projected image c includes the blackboard item w2 "YYY", and also includes an illumination area Q formed by irradiating it with white illumination light from the white illumination device 4. The emitted light of the electronic teaching material image e is not projected onto the illumination area Q on the blackboard 8.

[0110] The illuminated blackboard region extraction unit 42 performs homography transformation on the distorted projection image c using the transformation coordinate information cv, thereby generating a rectangular projection image c' that is undistorted (step S1502).

[0111] As a result, the blackboard item w2 "YYY" included in the projection image c is also homography transformed, and in step S1504 described later, a rectangular image including the blackboard item w2 "YYY" is identified and this rectangular area is generated as the blackboard area R1'. In addition, the illumination area Q included in the projection image c is also homography transformed, and a rectangular illumination area Q' is generated.

[0112] Meanwhile, the illuminated blackboard region extracting unit 42 reads out the electronic learning material image e from the storage unit 11, corrects the luminance of the electronic learning material image e using the luminance correction information br, and generates a luminance-corrected electronic learning material image e' (step S1503).

[0113] The illuminated blackboard area extraction unit 42 compares the image of the rectangular projection image c' generated in step S1502 other than the area indicated by the illumination coordinate information Q'(x,y) (other than the illumination area Q') with the image of the brightness-corrected electronic teaching material image e' generated in step S1503 other than the area indicated by the illumination coordinate information Q'(x,y), calculates the difference between the two images, and extracts the blackboard item w2 "YYY" from the rectangular projection image c'.The illuminated blackboard area extraction unit 42 then identifies the rectangular image containing the blackboard item w2 "YYY", generates this rectangular area as a blackboard area R1', and generates coordinate information of the blackboard area R1' as blackboard coordinate information R1'(x,y) (step S1504).

[0114] Then, the illuminated blackboard writing area extraction unit 42 extracts the image of the illuminated area Q' from the rectangular projection image c' using the illumination coordinate information Q'(x,y) (step S1505). That is, the illuminated blackboard writing area extraction unit 42 extracts the image of the area indicated by the illumination coordinate information Q'(x,y) from the rectangular projection image c' as the image of the illuminated area Q'.

[0115] For example, assume that the light control device 5 causes the white illumination light from the white illumination device 4 to be irradiated onto an area on the blackboard 8 where a written item w1 exists, thereby forming an illumination area Q. If the written item w1 is subsequently deleted from the blackboard 8, the written item w1 will no longer exist in the illumination area Q'. On the other hand, if the written item w1 remains on the blackboard 8, the written item w1 will exist in the illumination area Q'.

[0116] The illuminated blackboard writing area extraction unit 42 places the white-illuminated rectangular image b2' in the same area on the xy plane as the rectangular projected image c', and extracts an image of the white-illuminated area Q" from the white-illuminated rectangular image b2' using the illumination coordinate information Q'(x, y) (step S1506). In other words, the illuminated blackboard writing area extraction unit 42 extracts an image of the area indicated by the illumination coordinate information Q'(x, y) from the white-illuminated rectangular image b2' as the image of the white-illuminated area Q".

[0117] The illuminated blackboard region extraction unit 42 compares the image of the illuminated region Q' extracted in step S1505 with the image of the white-illuminated region Q'' extracted in step S1506, and finds the difference between the two images (step S1507).

[0118] Specifically, the illuminated blackboard area extraction unit 42 generates a difference image from the image of the illuminated area Q' and the image of the white illuminated area Q'', performs edge extraction on the difference image using a Sobel filter, and identifies areas where the change in brightness is greater than or equal to a predetermined threshold, thereby determining whether or not there is a difference.

[0119] If the threshold processing determines that there is no difference, the illuminated blackboard area extraction unit 42 determines that the illuminated area Q' does not contain the written matter w1, and sets the blackboard area R1' generated in step S1504 as the blackboard area R', and generates the blackboard coordinate information R1'(x,y) as the blackboard coordinate information R'(x,y) (step S1508). In other words, the illuminated area Q' (illuminated area Q) is determined to be an area from which the written matter w1 has been removed.

[0120] In this case, since only the blackboard item w2 "YYY" is written on the blackboard 8, the blackboard coordinate information R1'(x,y), which is the coordinate of the area where the blackboard item w2 "YYY" exists, is generated as the blackboard coordinate information R'(x,y).

[0121] On the other hand, if the threshold processing determines that there is a difference, the illuminated board writing area extraction unit 42 determines that the illuminated area Q' contains the written matter w1, and sets the area including the illuminated area Q' in addition to the board writing area R1' generated in step S1504 as the board writing area R', and generates the board writing coordinate information R1'(x,y) and the illumination coordinate information Q'(x,y) as the board writing coordinate information R'(x,y) (step S1509). In other words, the illuminated area Q' (illuminated area Q) is determined to be the area where the written matter w1 remains (the area where it has not been erased).

[0122] In this case, the blackboard item w2 "YYY" is written on the blackboard 8, and the blackboard item w1 written in the illumination area Q also exists. Therefore, the area including the illumination coordinate information Q'(x,y) which is the coordinate of the area where the blackboard item w1 exists, in addition to the blackboard coordinate information R1'(x,y) which is the coordinate of the area where the blackboard item w2 "YYY" exists, is generated as the blackboard coordinate information R'(x,y).

[0123] In step S1504, if there is no difference (a predetermined threshold or less) between the image of the rectangular projection image c' other than the area indicated by the illumination coordinate information Q'(x,y) and the image of the brightness-corrected electronic teaching material image e' other than the area indicated by the illumination coordinate information Q'(x,y), the illuminated blackboard area extraction unit 42 determines that a new blackboard item w2 does not exist. In this case, the blackboard coordinate information R1'(x,y) is not generated.

[0124] Then, in step S1507, if the illuminated blackboard writing area extraction unit 42 determines that there is no written matter w1 in the illuminated area Q', it determines that nothing has been written on the blackboard 8. In this case, the illuminated blackboard writing area extraction unit 42 generates null blackboard writing coordinate information R'(x, y) and outputs it to the light control device 5.

[0125] When the light control device 5 receives the null blackboard coordinate information R'(x,y), it tilts all of the mirrors that make up the digital micromirror device so that all of the light emitted from the projector 3 of the electronic teaching material image e is projected onto the blackboard 8. This forms a light path from the projector 3 to the blackboard 8 via the light control device 5, and the white illumination light from the white illumination device 4 is no longer irradiated onto the blackboard 8.

[0126] On the other hand, if the illuminated blackboard area extraction unit 42 determines in step S1507 that the written matter w1 is present in the illuminated area Q', it generates the illuminated coordinate information Q'(x,y) as the written matter coordinate information R'(x,y). In other words, only the written matter w1 written in the illuminated area Q' (illuminated area Q) exists on the blackboard 8.

[0127] As described above, according to the region extraction device 2 shown in FIG. 2, the homography transformation coordinate information generation unit 30 of the initial processing unit 20 compares the captured images A and B1, extracts the white projection image b1 from the captured image B1, and generates transformation coordinate information cv by homography transformation from the shape indicated by the projection image coordinate information b1(X, Y) to a rectangular shape by the preset projector 3.

[0128] The projection image coordinate information generator 31 generates projection image coordinate information b1'(x,y) by performing homography transformation on the projection image coordinate information b1(X,Y) using the transformation coordinate information cv, and outputs this to the light control device 5. As a result, an xy plane is formed by the projection image coordinate information b1'(x,y) in the light control device 5, and multiple mirrors for irradiating white illumination light corresponding to the board writing coordinate information R'(x,y) input from the board writing area extractor 21 can be identified.

[0129] The brightness correction information generation unit 32 generates a white rectangular projection image b1' by performing homography transformation on the white rectangular projection image b1 using the transformation coordinate information cv, and generates brightness correction information br based on the brightness value b'(BR) of the white rectangular projection image b1' and a predetermined brightness value PBR.

[0130] The white-illuminated rectangular image generation unit 33 extracts a white-illuminated image b2 from the captured image B2 using the projection image coordinate information b1(X,Y), and generates a white-illuminated rectangular image b2' by performing a homography transformation on the white-illuminated image b2 using the transformation coordinate information cv.

[0131] Furthermore, the illumination presence / absence determining unit 40 of the blackboard writing area extracting unit 21 determines whether illumination is present or absent based on the illumination coordinate information Q′(x, y) output from the light control device 5.

[0132] If it is determined that there is no illumination, the unilluminated blackboard region extraction unit 41 uses the projection image coordinate information b1(X,Y) to extract a projection image c from the captured video C, and generates a rectangular projection image c' by performing homography transformation on the projection image c using the transformation coordinate information cv.Then, the unilluminated blackboard region extraction unit 41 generates a brightness-corrected electronic teaching material image e' from the electronic teaching material image e using the brightness correction information br.

[0133] The unilluminated blackboard writing area extraction unit 41 compares the rectangular projection image c' with the brightness-corrected electronic teaching material image e' to extract the blackboard writing item w1 from the rectangular projection image c', generate blackboard writing coordinate information R'(x, y), and output it to the light control device 5. This allows the light control device 5 to avoid projecting the emitted light from the electronic teaching material image e onto the location of the blackboard writing item w1 written on the blackboard 8, and to irradiate it with white illumination light.

[0134] When it is determined that illumination is present, the illuminated blackboard writing area extraction unit 42 generates a rectangular projection image c' from the captured image C, similar to the unilluminated blackboard writing area extraction unit 41, and generates a brightness-corrected electronic teaching material image e' from the electronic teaching material image e.

[0135] The illuminated blackboard area extraction unit 42 compares an image obtained by excluding the area indicated by the illumination coordinate information Q'(x, y) from the rectangular projection image c' with an image obtained by excluding the area indicated by the illumination coordinate information Q'(x, y) from the brightness-corrected electronic teaching material image e', thereby extracting the blackboard item w2 from the rectangular projection image c' and generating blackboard coordinate information R1'(x, y).

[0136] The illuminated blackboard area extraction unit 42 uses the illumination coordinate information Q'(x, y) to extract the image of the illuminated area Q' from the rectangular projection image c' and also extract the image of the white-illuminated area Q" from the white-illuminated rectangular image b2'.The illuminated blackboard area extraction unit 42 then compares the image of the illuminated area Q' with the image of the white-illuminated area Q" to determine whether or not there is a written matter w1 in the illuminated area Q'.

[0137] If the illuminated blackboard area extraction unit 42 determines that there is no blackboard item w1 within the illumination area Q', it generates blackboard coordinate information R1'(x,y) as blackboard coordinate information R'(x,y), and if it determines that there is a blackboard item w1 within the illumination area Q', it generates blackboard coordinate information R1'(x,y) and illumination coordinate information Q'(x,y) as blackboard coordinate information R'(x,y).

[0138] The illuminated blackboard writing area extraction unit 42 outputs the blackboard writing coordinate information R'(x, y) to the light control device 5. As a result, based on the blackboard writing coordinate information R'(x, y), the light control device 5 can irradiate the location of the blackboard item w2 or the blackboard items w2, w1 written on the blackboard 8 with white illumination light, without projecting the emitted light of the electronic teaching material image e.

[0139] In this way, when the projected image from the projector 3 and the item w written by the user overlap on the blackboard 8, the blackboard area extraction unit 21 outputs the blackboard coordinate information R'(x, y) to the light control device 5. Then, the light control device 5 irradiates the blackboard 8 with white illumination light instead of projecting the emitted light of the electronic teaching material image e in an arbitrary shape corresponding to the area indicated by the blackboard coordinate information R'(x, y).

[0140] As a result, the light emitted from the projector 3 does not reach the area of ​​the blackboard 8 where the item w is written, and therefore the item w on the blackboard 8 is not affected by the light emitted from the projector 3. Furthermore, the white illumination light from the white illumination device 4 reaches the area of ​​the blackboard 8 where the item w is written, and therefore the item w on the blackboard 8 becomes even easier to see.

[0141] Therefore, the user can clearly distinguish between the image projected onto the blackboard 8 and the written matter w on the blackboard 8. In other words, interference between the projected image and the written matter w can be eliminated, making it easier to see the written matter w when the electronic teaching material image e is projected.

[0142] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments and can be modified in various ways without departing from the technical concept thereof.

[0143] In the above embodiment, the information presentation system 1 is a system used in an educational setting, and uses an electronic teaching material image e as the projected image, but it may also be a system used outside of an educational setting, or may use an image other than the electronic teaching material image e.

[0144] In the above embodiment, the homography transformation coordinate information generation unit 30 provided in the initial processing unit 20 of the region extraction device 2 extracts the white projection image b1 from the captured image B1 by comparing the captured images A and B1 in step S804 of Fig. 8. Alternatively, the homography transformation coordinate information generation unit 30 may extract the white projection image b1 from the captured image B1 using machine learning data.

[0145] Similarly, the unilluminated board writing area extraction unit 41 and the illuminated board writing area extraction unit 42 provided in the board writing area extraction unit 21 of the area extraction device 2 use the projection image coordinate information b1(X,Y) to extract the projection image c from the captured image C in step S1401 of Fig. 14 and step S1501 of Fig. 15. In contrast to this, the unilluminated board writing area extraction unit 41 and the illuminated board writing area extraction unit 42 may also use machine learning data to extract the projection image c from the captured image C.

[0146] Similarly, in the example shown in Fig. 14, the unilluminated blackboard writing area extraction unit 41 may use machine learning data to generate a blackboard writing area R' from the captured image C. Furthermore, in the example shown in Fig. 15, the illuminated blackboard writing area extraction unit 42 may use machine learning data to generate a blackboard writing area R1' from the captured image C.

[0147] In the above embodiment, the unilluminated board writing area extraction section 41 and the illuminated board writing area extraction section 42 provided in the board writing area extraction section 21 of the area extraction device 2 generate the board writing area R' using the captured image C input from the camera 6. However, the unilluminated board writing area extraction section 41 and the illuminated board writing area extraction section 42 may generate the board writing area R' using sensor information input from a sensor provided on the blackboard 8, instead of image recognition processing using the captured image C input from the camera 6.

[0148] In the above embodiment, the projector 3, the white lighting device 4, the light control device 5, and the lenses 7-1, 7-2, and 7-3 are configured as individual devices and components, but the projector 3 may be equipped with the white lighting device 4, the light control device 5, and the lenses 7-1, 7-2, and 7-3. In this case, the information presentation system 1 is configured with the area extraction device 2, the projector 3 (the projector 3 equipped with the white lighting device 4, the light control device 5, and the lenses 7-1, 7-2, and 7-3), and the camera 6.

[0149] A normal computer can be used as the hardware configuration of the area extraction device 2 included in the information presentation system 1 according to the embodiment of the present invention. The area extraction device 2 is configured by a computer including a CPU, a volatile storage medium such as RAM, a non-volatile storage medium such as ROM, an interface, etc.

[0150] The functions of the video input unit 10, storage units 11 and 14, electronic teaching material output unit 12 and information recognition unit 13 provided in the area extraction device 2 are each realized by causing a CPU to execute a program describing these functions.

[0151] These programs are stored in the storage medium and are read and executed by the CPU. These programs can also be stored in a storage medium such as a magnetic disk (e.g., a floppy disk, a hard disk), an optical disk (e.g., a CD-ROM, a DVD), or a semiconductor memory and distributed, or can be transmitted and received via a network. [Explanation of symbols]

[0152] 1 Information presentation system 2 Region extraction device 3. Projector 4 White lighting device 5. Light control device 6. Camera 7-1, 7-2, 7-3 Lenses 8. Blackboard 10 Video input section 11,14 Storage section 12 Electronic teaching material output section 13 Information Recognition Department 20 Initial processing section 21 Board area extraction part 30 Homography transformation coordinate information generation unit 31 Projection image coordinate information generation unit 32 Brightness correction information generation unit 33 White lighting rectangular image generator 40 Lighting presence / absence determination section 41 Unlit blackboard area extraction 42 Illuminated blackboard area extraction section A, B1, B2, C Footage b1 White projection image b1' White rectangular projection image b1(X,Y),b1'(x,y) Projected image coordinate information b1'(BR) Brightness value of white rectangular projection image b1' b2 White lighting image b2' White illumination rectangular image c Projected image c' Rectangular projection image e Electronic teaching material images e' Brightness corrected digital teaching material image e(BR) Brightness value of the electronic teaching material image e e'(BR) Brightness value of the electronic teaching material image e' after brightness correction R',R1' Board area R'(x,y),R1'(x,y) Coordinate information on the board Q,Q' illumination area Q'(x,y) Lighting coordinate information Q” White illumination area cv Transformation coordinate information br Brightness correction information w,w1,w2 Matters written on the board PBR white luminance value

Claims

1. An information presentation system that presents a projected image formed by emitted light of a predetermined image on a blackboard, extracts an area of ​​the written matter as a blackboard area from a captured image including the projected image and the written matter written on the blackboard, and irradiates white illumination light toward the area of ​​the written matter on the blackboard, a projector that projects emitted light of the predetermined image onto the blackboard via a digital micromirror device to form the projected image; a white illumination device that irradiates the white illumination light onto the blackboard via the digital micromirror device; The digital micromirror device includes a predetermined number of mirrors, and tilts all of the mirrors so that all of the light emitted from the projector for the predetermined image is projected onto the blackboard. a light control device that tilts some of the mirrors based on coordinate information of the blackboard writing area so that the white illumination light from the white illumination device that corresponds to the blackboard writing area is irradiated onto the blackboard; and a camera that captures an image including the projected image and the written matter as the captured image; an area extraction device that extracts an image of the region of the projection image from the captured image, compares the image of the region of the projection image with the predetermined image, extracts an area with a difference as the blackboard writing area, and outputs coordinate information of the blackboard writing area to the light control device; An information presentation system comprising:

2. The information presentation system according to claim 1, The area irradiated with the white illumination light corresponding to the writing area is defined as an illumination area, The light control device includes: outputting coordinate information of the illumination area to the area extraction device; The region extraction device an input unit for inputting the captured video; an output unit that outputs the predetermined image to the projector; an illumination presence / absence determination unit that receives coordinate information of the illumination area from the light control device and determines whether or not illumination is present, which indicates that the white illumination light is being irradiated onto the blackboard, based on the coordinate information of the illumination area; extracting an image of the region of the projection image from the captured image input by the input unit when the illumination presence / absence determination unit determines that illumination is absent; an unilluminated blackboard writing area extraction unit that compares a video of the region of the projection image with the predetermined image, extracts a region with a difference as the blackboard writing area, and outputs coordinate information of the blackboard writing area to the light control device; extracting an image of the region of the projection image from the captured image input by the input unit when the illumination presence / absence determination unit determines that illumination is present; extracting a difference area as a first blackboard area by comparing an image obtained by excluding the illumination area from an image of the projection image area with an image obtained by excluding the illumination area from the predetermined image; extracting an image of the illumination area as a first image from an image of the area of ​​the projection image; and extracting an image of the illumination area as a second image from an image obtained by irradiating the white illumination light onto the blackboard, the image being in the same area as the projection image; determining whether the matter written on the board is included in the illumination area within the area of ​​the projected image by comparing the first image with the second image; If it is determined that the matter written on the board is not included, the first writing area is set as the writing area, and coordinate information of the writing area is output to the light control device. an illuminated writing area extraction unit that, when it is determined that the matter written on the board is included, determines the first writing area and the illuminated area as the writing area and outputs coordinate information of the writing area to the light control device; An information presentation system comprising:

3. 1. A region extraction device that inputs a captured image including a projected image formed on a blackboard by emitted light of a predetermined image from a projector and including a matter written on the blackboard by a user, extracts an area including the matter written on the blackboard as a blackboard area based on the captured image, and irradiates white illumination light toward the area of ​​the matter written on the blackboard, an input unit for inputting the captured video; an output unit that outputs the predetermined image to the projector; a blackboard area extraction unit that extracts an image of the region of the projection image from the captured image, compares the image of the region of the projection image with the predetermined image, extracts an area with a difference as the blackboard area, and outputs coordinate information of the blackboard area to a light control device; An area extraction device characterized in that the light control device is caused to irradiate the area of ​​the blackboard writing item on the blackboard with the white illumination light instead of projecting a portion of the emitted light of the specified image toward the area of ​​the blackboard writing item on the blackboard based on coordinate information of the blackboard writing area.

4. a computer constituting an area extraction device that inputs a captured image including a projected image formed on a blackboard by light emitted from a projector of a predetermined image and including matters written on the blackboard by a user, extracts an area including the matters written on the blackboard as a blackboard area based on the captured image, and irradiates white illumination light toward the area of ​​the matters written on the blackboard; an input unit for inputting the captured video; an output unit that outputs the predetermined image to the projector; and extracting an image of the region of the projection image from the captured image, comparing the image of the region of the projection image with the predetermined image, extracting an area with a difference as the blackboard writing region, and functioning as a blackboard writing region extraction unit that outputs coordinate information of the blackboard writing region to a light control device; A program that causes the light control device to irradiate the white illumination light instead of projecting a portion of the emitted light of the specified image toward the area of ​​the blackboard writing item on the blackboard based on coordinate information of the blackboard writing area.

Images