Touch area detection system, touch area detection method, and computer program

Abstract

To enable accurate detection of the location of an object's touch, even when the target surface is curved. [Solution] A touch area detection system for detecting the touch area of ​​an object on a target surface, comprising: a plurality of imaging devices arranged to image the target surface from different viewpoints; and a control device that analyzes a plurality of images captured by the plurality of imaging devices of an object touching the target surface to detect the touch area of ​​the object on the target surface. The control device stores a grid map in which the target surface is virtually divided into a plurality of cells, and detects the touch area of ​​the object on the target surface by identifying the cell to which the object is touching based on the plurality of images from different viewpoints.

Description

Technical Field

[0001] The present invention relates to a touch area detection system, a touch area detection method, and a computer program for detecting a touch area of an object with respect to a detection target surface. Specifically, the present invention relates to a technology that can appropriately detect the touch area of an object not only when the detection target surface is a flat surface but also when the detection target surface is a curved surface. In particular, the present invention can be applied to the technical field of projection mapping.

Background Art

[0002] In recent years, projection mapping technology for projecting images onto the surfaces of buildings such as walls, ceilings, and pillars, and three-dimensional structures has become widely popular. In projection mapping technology, it is possible to deform an image according to the three-dimensional shape of the projection target and realize a visually impressive expression. In addition, technologies for detecting the touch position of an object (such as a user) with respect to the image projection surface have also been developed so that the user can interact with the image projection by projection mapping.

[0003] As a technology for detecting the touch position of an object, a method using a laser range finder is known. A laser range finder is a technology that irradiates a laser beam along a detection target surface and detects the presence of an object by the object blocking the laser beam. The technology using this laser range finder can detect a wide range with high accuracy in planar touch detection.

[0004] However, the conventional technology using a laser range finder has problems in touch detection for curved surfaces. Specifically, as shown in FIG. 10, since the laser range finder irradiates a laser beam along a flat surface, it cannot irradiate a laser beam along a curved surface. As a result, there is a problem that only a part of the curved surface can be irradiated and the laser beam does not reach the range to be detected. In order to cover the entire curved surface, a huge number of sensors are required, but there is a problem that it is practically difficult to prepare such a number of sensors.

[0005] On the other hand, a technology has also been proposed that uses multiple cameras to perform touch detection (see, for example, Patent Document 1). In this technology, a first camera and a second camera capture the display surface from different angles at approximately the same time to acquire a first image and a second image. When the first coordinate, which is the touch position of the indicator object on the display surface as seen in the first image, and the second coordinate, which is the touch position of the indicator object on the display surface as seen in the second image, approximately coincide, the approximately coincidental coordinate position is detected as the touch position of the indicator object. This technology utilizes the fact that the apparent touch coordinates of the indicator object coincide in multiple images captured from different angles to distinguish and detect touches and non-contact proximity of the indicator object. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2019-53769 [Overview of the project] [Problems that the invention aims to solve]

[0007] Incidentally, while the technology described in Patent Document 1 above works effectively when the surface to be detected for touch position is flat, it is difficult to properly detect the touch position when the surface to be detected is curved. This is because the technology in Patent Document 1 is a detection method that utilizes the matching of coordinates in images taken from different viewpoints.

[0008] To explain in more detail, the technology described in Patent Document 1 identifies the coordinate positions of the indicator object in the images captured by multiple cameras and determines whether these coordinate positions coincide. On a plane, the coordinate system viewed from each camera can be treated uniformly, so this coordinate coincidence determination works effectively. However, on a curved surface, the apparent coordinate system changes complexly depending on the viewpoint, making it difficult to accurately identify the touch position with a simple coordinate coincidence determination. Furthermore, the more complex the shape of the curved surface, the more complex the coordinate transformations at each viewpoint become, leading to problems such as increased computational load and decreased accuracy.

[0009] Therefore, the main objective of the present invention is to provide a technology that can appropriately detect the location of an object's touch even when the surface to be detected is curved. [Means for solving the problem]

[0010] Therefore, the inventors of the present invention diligently studied means to solve the problems of the above-mentioned prior art, and as a result obtained the finding that by generating a grid map in which the detection target surface is virtually divided into multiple cells, and by comprehensively determining the occupancy status of each cell in captured images from multiple viewpoints, the touch area of ​​an object can be appropriately detected even on a curved surface. Based on the above finding, the inventors realized that the problems of the prior art could be solved, and thus completed the present invention. Specifically, the present invention has the following configuration or steps.

[0011] The first aspect of the present invention is a touch area detection system. The touch area detection system according to the present invention is a touch area detection system for detecting the touch area of ​​an object on a surface to be detected. "Touch" includes not only that the object is in contact with the surface to be detected, but also that the object is close enough to the surface to be recognized as touching in the captured image, even if it is not actually touching (for example, the distance between the object and the surface to be detected is about 0 to 20 mm). The surface to be detected may be a flat surface or a curved surface. The surface to be detected may consist only of a flat surface or only of a curved surface. If the surface to be detected is a curved surface, its curvature is not a factor. The system comprises a plurality of imaging devices and a control device. The plurality of imaging devices are arranged to image the surface to be detected from different viewpoints. "Different viewpoints" means that the surface to be detected is imaged from different angles and positions, and the plurality of imaging devices can acquire three-dimensional information of the surface to be detected. The control device analyzes a plurality of captured images taken by the plurality of imaging devices of an object touching the surface to be detected, and detects the touch area of ​​the object on the surface to be detected. The control device has a storage unit and a touch area detection unit (processor, etc.). The memory unit stores a grid map in which the detection target surface is virtually divided into multiple cells. Here, "grid map" refers to virtual map information that divides the detection target surface into a grid, and "cell" refers to the individual areas that make up the grid map. The touch area detection unit detects the touch area of ​​an object on the detection target surface by identifying the cell that the object is touching based on multiple captured images from different viewpoints.

[0012] As described above, the present invention uses a grid map that virtually divides the target surface into multiple cells to perform touch area detection. This enables not only the detection of touch areas on flat surfaces but also the detection of touch areas on curved surfaces, which was difficult with conventional techniques. Specifically, conventional laser rangefinder techniques irradiate laser light along a flat surface, making it difficult to handle curved surfaces. In contrast, the present invention can generate a grid map for any shape of target surface, enabling efficient touch area detection even on curved surfaces. Furthermore, the coordinate matching detection method described in Patent Document 1 made accurate detection difficult on curved surfaces due to the complex changes in the coordinate system depending on the viewpoint. In contrast, the present invention performs occupancy determination at the cell level, avoiding the complexity of coordinate transformations and enabling stable detection even on curved surfaces. Moreover, since integrated determination is performed at the cell level based on images captured from multiple viewpoints, high-precision detection can be achieved even when a touch area cannot be detected from a single viewpoint.

[0013] In the touch area detection system according to the present invention, it is preferable that the surface to be detected includes a curved surface. A "curved surface" means a surface that is not planar, that is, a surface that has curvature in at least part of it, and includes cylindrical surfaces, spherical surfaces, free-form surfaces, etc. Specifically, a surface to be detected being a curved surface may be a surface whose horizontal cross-section is curved (horizontal curved surface), a surface whose vertical cross-section is curved (vertical curved surface), or a surface where both the horizontal cross-section and the vertical curved surface are curved (horizontal-vertical curved surface). Furthermore, the surface to be detected may include both planar and curved surfaces, or may consist only of curved surfaces. Thus, in the present invention, touch areas can be appropriately detected not only on planar surfaces, but also on curved wall surfaces, surfaces with organic shapes, surfaces of three-dimensional structures, etc., by a detection method using a grid map. This makes it possible to detect touch areas on surfaces of various shapes onto which images are projected by projection mapping, thereby providing users with an interactive experience. In addition, by supporting curved surfaces, this system can be applied to complex shapes such as columns and walls of buildings and surfaces of artworks, greatly expanding the range of applications.

[0014] In the touch area detection system according to the present invention, the control device may further include an object extraction unit. The object extraction unit extracts object areas in which objects are captured from multiple captured images. In this case, the touch area detection unit determines the occupied range of the object area in each cell of the grid map for each of the multiple captured images, and identifies cells in which an occupation of a predetermined range or more is determined in a predetermined number or more of the multiple captured images as cells that an object is touching. This enables highly reliable touch area detection integrating multiple viewpoints. Specifically, even if a false detection occurs in one viewpoint due to external factors such as noise or changes in lighting, the false detection can be effectively suppressed by integrating the detection results from other viewpoints. Furthermore, by adjusting thresholds such as "a predetermined number or more" or "a predetermined range or more," the balance between detection sensitivity and false detection suppression can be optimized according to the application.

[0015] The touch area detection system according to the present invention may further include a projection device that projects an image onto the detection target surface. In this case, the control device preferably further includes an image control unit and a grid map generation unit. The image control unit causes the projection device to project a structured light pattern onto the detection target surface. A "structured light pattern" is a light pattern with positional information designed for three-dimensional shape measurement, and includes, for example, Gray code patterns, stripe patterns, and dot patterns. The grid map generation unit generates a grid map that virtually divides the detection target surface into multiple cells based on the structured light patterns on the detection target surface captured by multiple imaging devices. Note that the generation of the grid map also includes updating the generated grid map according to the shape of the detection target surface. This makes it possible to generate (including update) a grid map automatically and with high accuracy regardless of the shape of the detection target surface. In other words, since a grid map can be generated by projecting a structured light pattern onto the detection target surface, there is no need to manually measure the shape of the detection target surface in advance or to attach physical markers to the detection target surface.

[0016] As described above, the touch area detection system according to the present invention may further include a projection device that projects an image onto the surface to be detected. In this case, it is preferable that the control device further includes an image control unit that controls the image to be projected onto the projection device based on the touch area detection result. This makes it possible to change the image content in real time according to the detected touch area, thereby providing the user with an interactive experience. Specifically, it is possible to display an image of flowers blooming in the area touched by the user on the surface to be detected, draw lines or patterns along the touch trajectory, or generate new image effects from the touch area.

[0017] A second aspect of the present invention is a touch area detection method. In the touch area detection method according to the present invention, first, multiple imaging devices capture images of the target surface from different viewpoints (imaging step). Next, a control device analyzes multiple images captured by the multiple imaging devices to detect the touch area of ​​an object touching the target surface (touch area detection step). The control device stores a grid map in which the target surface is virtually divided into multiple cells, and detects the touch area of ​​an object on the target surface by identifying the cell being touched by the object based on multiple images from different viewpoints. This method can achieve the same effects as the touch area detection system according to the first aspect described above. It is also possible to introduce the technology of the present invention as an add-on function to existing systems by implementing this method using existing imaging devices and control devices.

[0018] A third aspect of the present invention is a computer program. The computer program according to the present invention causes a computer to perform a process to detect the touch area of ​​an object on a surface to be detected. The program according to the present invention causes a computer to perform an image acquisition process, which acquires images from a plurality of imaging devices arranged to image the surface to be detected from different viewpoints, and a touch area detection process, which analyzes a plurality of images taken by the plurality of imaging devices of an object touching the surface to be detected and detects the touch area of ​​the object on the surface to be detected. The touch area detection process includes detecting the touch area of ​​an object on the surface to be detected by identifying the cell in which the object is touching the surface in a grid map that virtually divides the surface to be detected into a plurality of cells based on a plurality of images taken from different viewpoints. This program makes it possible to realize the technology of the present invention on a general-purpose computer. This makes it easy to provide the technology of the present invention as software or to incorporate it into existing systems. [Effects of the Invention]

[0019] According to the present invention, it becomes possible to appropriately detect the location of an object's touch even if the surface to be detected is curved.

Brief Description of the Drawings

[0020] [Figure 1] FIG. 1 is a conceptual diagram for explaining the basic principle of the touch area detection technology according to the present invention. [Figure 2] FIG. 2 is a schematic diagram showing the overall configuration of a touch area detection system according to an embodiment of the present invention. [Figure 3] FIG. 3 is a block diagram showing the functional configurations of a touch area detection system according to an embodiment of the present invention. [Figure 4] FIG. 4 is a flowchart showing the flow of grid map generation processing. [Figure 5] FIG. 5 is an explanatory diagram regarding the projection of a gray code pattern and the generation of a grid map. [Figure 6] FIG. 6 is a flowchart showing the flow of touch area detection processing. [Figure 7] FIG. 7 is a schematic diagram regarding an application example of touch area detection on a curved surface. [Figure 8] FIG. 8 is an explanatory diagram of the steps up to histogram creation at each viewpoint. [Figure 9] FIG. 9 is an explanatory diagram of touch area detection determination using the histograms of each viewpoint. [Figure 10] FIG. 10 schematically shows the prior art.

Embodiments for Carrying Out the Invention

[0021] Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. The present invention is not limited to the embodiments described below, and also includes those appropriately modified by those skilled in the art within an obvious range from the following embodiments.

[0022] Referring to Figure 1, the basic principle of the touch area detection technology according to the present invention will be explained. In this invention, the touch area of ​​an object such as a user's body is detected by using a grid map that virtually divides the detection target surface into multiple cells. As shown in Figure 1, the detection target surface is divided into a rectangular grid of a total of 100 cells, from grid ID 0 to grid ID 99. Each cell is assigned a unique identification number (grid ID), and this grid map is virtually applied to the detection target surface. With this configuration, any position on the detection target surface can be identified on a cell-by-cell basis.

[0023] In the example shown in Figure 1, the detection target surface is shown as a plane, and the first imaging device 20(a) and the second imaging device 20(b) are arranged to image the detection target surface from different viewpoints. The first image acquired by the first imaging device 20(a) is shown in the left-hand bubble, and the second image acquired by the second imaging device 20(b) is shown in the right-hand bubble. These imaging devices take images from different angles and positions to acquire three-dimensional information of the detection target surface. Here, the images from each imaging device 20(a,b) are pre-associated with a grid map, and the control device stores information on which grid ID cell each pixel on the image corresponds to. This correspondence allows the grid ID of the cell in which an object exists to be immediately identified from the pixel position where an object is detected on the image. In this way, the same object (such as a user) that is in contact with or close to the detection target surface is simultaneously imaged from multiple different viewpoints. This improves the detection accuracy of touch areas even in situations where detection is difficult with a single viewpoint.

[0024] In the example shown in Figure 1, in the first image captured by the first imaging device 20(a), the user's left hand (object) overlaps with pixels corresponding to grid IDs 55, 56, 65, 66, 76, 86, and 96. The control device (not shown in Figure 1) analyzes this first image and determines which cells the object region occupies. In this determination process, the occupied range of the object region in each cell is calculated, and cells where the occupancy exceeds a predetermined threshold are identified as cells where an object exists. As a result of detection from the first viewpoint, the above seven cells are extracted as cells where an object exists. On the other hand, in the second image captured by the second imaging device 20(b), the same user's left hand (object) overlaps with cells 55, 56, 65, 75, 85, and 95. The control device performs a similar analysis on the second image to identify the cells where an object exists. The control device then comprehensively analyzes the judgment results of the first and second captured images. Specifically, it determines that cells detected in common across multiple captured images are cells that an object is actually touching. In the example in Figure 1, grid ID 55 and grid ID 65 are detected in both captured images, so the control device can detect that the user is touching these cells.

[0025] Thus, the present invention achieves highly accurate touch area detection by integrating the occupancy determination results from images captured from multiple viewpoints. Compared to conventional detection methods based on coordinate matching, the determination is performed at the cell level, thus avoiding the complexity of coordinate transformations depending on the viewpoint. Furthermore, false detections can be effectively suppressed by determination based on agreement across multiple viewpoints. In addition, since the grid map can be generated to match the shape of the detection target surface, the same principle can be applied not only to the plane shown in Figure 1, but also to curved surfaces and detection target surfaces with complex shapes.

[0026] Several methods can be applied to generate grid maps. The most basic method involves physically drawing an identifier-based grid pattern on the detection target surface, or attaching a film or sheet with this grid pattern to the detection target surface, then imaging the state with each imaging device, and pre-memorizing the correspondence between each pixel on the captured image and which cell it corresponds to. A more advanced method, as described in the embodiments below, involves projecting a structured light pattern (e.g., a Gray code pattern) onto the detection target surface, and having multiple imaging devices image the projected pattern to measure the three-dimensional shape of the detection target surface and automatically generating a grid map according to that shape. This automatic generation method is effective when it is difficult to place physical markers or for complex shapes such as curved surfaces.

[0027] Next, with reference to Figures 2 to 9, a touch area detection system 100 according to one embodiment of the present invention will be described. Figure 2 shows the overall configuration of the touch area detection system 100 according to one embodiment of the present invention. In this embodiment, a substantially cylindrical structure is used as the detection target surface. This structure has a curved shape that widens downwards, and its radius changes curvilinearly, being 1 to 2 m at the top and approximately 2 to 4 m at the bottom (about twice the radius at the top). The height of this structure is not particularly limited and can be 5 m or more or 10 m or more. As shown in the horizontal cross-section at the top of Figure 2, this structure has a circular cross-section. Also, as shown in the vertical cross-section shown in the callout at the bottom of Figure 2, the radius of this structure widens curvilinearly (inversely) as it goes downwards. That is, this structure forms a horizontal-vertical curved surface in which both the horizontal and vertical cross-sections of the detection target surface are curved. This structure is large enough for a user (symbol O) to climb on, and is intended for application to interactive art works using projection mapping, etc.

[0028] Around the structure, multiple imaging devices 20 are arranged to image the detection target surface (indicated as S) of the structure from different viewpoints for touch area detection. In this embodiment, the detection target surface is the entire perimeter of the structure, at least the area that the user can touch. Therefore, the entire perimeter of this structure is imaged by multiple imaging devices 20. Specifically, four imaging devices, from 20(a) to 20(d), are arranged to image the detection target surface from an oblique overhead viewpoint or from the side. In addition, three imaging devices, from 20(e) to 20(g), are attached to the roughly cylindrical structure itself and image the detection target surface from directly above. Importantly, each imaging device 20 is arranged so that a specific point on the detection target surface of the structure is included in the shooting range of at least two imaging devices 20 from different viewpoints. In particular, the same location is imaged from different viewpoints by three or more imaging devices 20. In other words, the imaging ranges of at least two imaging devices 20 must overlap across the entire surface to be detected, and it is particularly preferable that they overlap by three or more devices. This configuration makes it possible to reliably detect touch areas even in the upper parts of structures and areas close to vertical surfaces that are difficult to capture from an overhead view.

[0029] As shown in Figure 2, in this embodiment, imaging devices 20(a) to 20(d) that capture an overhead image of a structure with a detection target surface from a distance are combined with imaging devices 20(e) to 20(g) that are attached to the structure itself with the detection target surface. This combination allows the vertical surfaces and upper areas of the structure, which tend to be blind spots from an overhead viewpoint, to be supplemented by the imaging devices attached to the structure. Furthermore, while the imaging devices positioned at the overhead location can capture a wide area, the imaging devices attached to the structure can capture detailed images at close range, thus achieving both wide-area monitoring and detailed detection. In this way, by combining imaging devices with different imaging distances and angles, touch area detection becomes possible even under lighting conditions and changes in the orientation of objects.

[0030] Four projection devices, 30(a) through 30(d), are positioned around the structure for projecting images. Although only four projection devices 30 are shown in the illustration, more projection devices 30 can be arranged depending on the size of the structure and the projection range. These projection devices 30 are positioned to project images over the entire circumference of the roughly cylindrical structure, making the entire circumference of the structure the projection surface. The projection devices 30 deform and project images to conform to the curved shape of the structure. These projection devices 30 also have the function of generating a grid map by projecting a structured light pattern. When a user (symbol O) touches the surface of the structure, the image content can be changed in real time according to the touch area, providing the user with an interactive experience.

[0031] Figure 3 shows the functional configuration of the touch area detection system 100 according to this embodiment. In particular, Figure 3 shows a block diagram of the control device 10 included in the touch area detection system 100. The control device 10 is a computer for controlling the entire touch area detection system 100, and controls multiple imaging devices 20, multiple projection devices 30, and one or more sound devices 40. A general-purpose computer can be used as the control device 10. The control device 10 is basically wiredly connected to the imaging devices 20, projection devices 30, and sound devices 40 via LAN cables, USB cables, switching hubs, etc. However, the control device 10 may also be wirelessly connected to these devices via wireless LAN or the internet.

[0032] As shown in Figure 3, the control device 10 includes a control unit 11, a storage unit 12, an input unit 13, and an output unit 14. The control unit 11 can utilize a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The control unit 11 basically reads a program stored in the storage unit 12, loads it into main memory, and executes predetermined calculations according to this program. The control unit 11 can also write and read calculation results according to the program to the storage unit 12 as appropriate. In particular, the storage unit 12 stores a grid map used for determining touch areas. A grid map is map information that virtually divides the detection target surface into multiple cells, and each cell is assigned a unique identification number (grid ID). The storage unit 12 also stores correspondence information between pixels on the captured images of each imaging device 20 and each cell in the grid map. This grid map makes it possible to comprehensively determine cells containing objects in captured images from multiple viewpoints. Details of the grid map generation method will be described later.

[0033] The memory unit 12 is an element for storing information used in calculation processing by the control unit 11 and the results of those calculations. The storage function of the memory unit 12 can be realized by non-volatile memory such as an HDD or SSD. The memory unit 12 may also have the function of main memory for writing or reading intermediate results of calculation processing by the control unit 11. The memory function of the memory unit 12 can be realized by volatile memory such as RAM or DRAM.

[0034] The input unit 13 includes pointing devices such as keyboards, mice, and touchpads, as well as audio input devices such as microphones. These input devices allow the user to perform operations such as setting the operating parameters of the touch area detection system 100, manually adjusting the grid map, and starting and stopping the system. The output unit 14 is a group of output devices for presenting the system's operating status and detection results to the user. The output unit 14 includes display devices such as liquid crystal displays and organic EL displays, as well as audio output devices such as speakers and headphones. These output devices allow the user to check the touch area detection status, system operation logs, error information, etc., in real time.

[0035] The imaging device 20 consists of multiple cameras for capturing images of the detection target surface from different viewpoints. The imaging device 20 can be a general-purpose CCD camera, CMOS camera, infrared camera, etc. The imaging device 20 is composed of, for example, a photographic lens, a mechanical shutter, a shutter driver, a photoelectric conversion element such as a CCD image sensor or CMOS image sensor, a digital signal processor (DSP) that reads the charge amount from the photoelectric conversion element and generates image data, and an IC memory. The image data acquired by the imaging device 20 is transmitted to the control device 10.

[0036] The projection device 30 is a projector that primarily projects images onto the detection target surface. The projection device 30 can be an LCD projector, DLP projector, LED projector, laser projector, etc. The projection device 30 consists of, for example, a light source (lamp, LED, laser, etc.), an optical modulation element such as a liquid crystal panel or DMD (Digital Micromirror Device), a projection lens, a cooling fan, etc. Based on control signals from the control device 10, the projection device 30 performs the projection of structured light patterns and interactive images.

[0037] The sound device 40 is a speaker that outputs sound effects, background music, and other audio. The sound device 40 can be a speaker system combining various speaker units such as woofers, tweeters, and full-range speakers. The sound device 40 also includes an amplifier, a digital-to-analog converter (DAC), and an audio signal processing circuit. For example, the sound device 40 can output sound effects corresponding to the detection results of the touch area based on an audio control signal from the control device 10, providing the user with an interactive auditory experience.

[0038] Figure 3 also shows the functional blocks of the control unit 11. The control unit 11 includes, as functional blocks, a video control unit 11a, an imaging control unit 11b, a grid map generation unit 11c, an object extraction unit 11d, a touch area detection unit 11e, and an acoustic control unit 11f. These functional blocks 11a to 11f are realized by the control unit 11 (processor) executing a predetermined program.

[0039] The video control unit 11a controls the projection of images onto the projection device 30. Specifically, in the grid map generation flow (Figure 4), the video control unit 11a controls the projection of a structured light pattern (e.g., Gray code pattern, stripe pattern, dot pattern, etc.) onto the detection target surface, and in the touch area detection flow (Figure 6), it controls the projection of interactive images based on the detection results of the touch area. When projecting a structured light pattern, the video control unit 11a reads a specific pattern image for grid map generation from the storage unit 12 and instructs the projection device 30 to output this pattern image. When projecting interactive images, it receives detection results from the touch area detection unit 11e and controls the image content to change in real time according to the touched position and touch trajectory.

[0040] The imaging control unit 11b controls the imaging operations of multiple imaging devices 20. The imaging control unit 11b synchronizes the imaging timing of each imaging device 20, controlling all imaging devices 20 to perform imaging at the same time. This allows for accurate recording of the state of the target surface at the same time from multiple viewpoints. The imaging control unit 11b may also dynamically adjust imaging parameters such as exposure time, gain, frame rate, white balance, and focal length depending on the lighting conditions and the material of the target surface. Furthermore, the imaging control unit 11b can set different imaging modes for grid map generation and touch area detection. For example, a high-resolution, low-frame-rate mode can be set for high-precision shape measurement during grid map generation, while a moderate resolution, high-frame-rate mode can be set for real-time performance during touch area detection.

[0041] The grid map generation unit 11c generates a grid map that virtually divides the target surface into multiple cells based on the structured light patterns on the target surface captured by the multiple imaging devices 20. The grid map generation method will be described in detail later, but the grid map generation unit 11c analyzes the captured images from multiple viewpoints acquired during structured light projection and reconstructs the shape of the target surface using three-dimensional shape measurement technology. Then, based on the reconstructed three-dimensional shape, it divides the target surface into a grid and assigns a unique identification number (grid ID) to each cell. The grid map generation unit 11c also calculates the correspondence between pixels on the captured images from each imaging device 20 and each cell in the grid map and stores this correspondence information in the storage unit 12.

[0042] The object extraction unit 11d performs image analysis processing to extract object regions containing objects from images captured by multiple imaging devices 20. The object extraction unit 11d identifies the areas where objects exist within the captured images using image processing techniques such as background subtraction, color information, motion information, and contour detection. The extracted object region information is sent to the touch region detection unit 11e.

[0043] The touch area detection unit 11e uses the object area extracted by the object extraction unit 11d and the grid map generated by the grid map generation unit 11c to determine object occupancy on a cell-by-cell basis in multiple captured images. The touch detection process will be described in detail later, but the touch area detection unit 11e calculates the occupied range of the object area in each cell of the grid map for each of the multiple captured images and determines the occupancy rate for each cell. Then, cells in which an occupancy of a predetermined range or more is determined in a predetermined number of captured images are identified as cells that are being touched by an object. This integrated determination effectively suppresses situations that are difficult to detect with a single viewpoint and false detections due to noise.

[0044] The sound control unit 11f controls the sound output to the sound device 40. For example, the sound control unit 11f can control sound output such as sound effects, background music, and voice guidance according to the detection result from the touch area detection unit 11e.

[0045] Next, the grid map generation process according to this embodiment will be described with reference to Figures 4 and 5. Figure 4 is a flowchart showing the flow of the grid map generation process. This grid map generation process is a pre-processing step to measure the three-dimensional shape of the surface to be detected and generate a virtual grid map according to that shape. The grid map generation process is executed when the system is initialized or when the shape of the surface to be detected is changed.

[0046] As shown in Figure 4, the grid map generation process begins with structured pattern generation (S1-1) by the grid map generation unit 11c. In structured pattern generation, the grid map generation unit 11c generates a structured light pattern for three-dimensional shape measurement. The grid map generation unit 11c may also read a pre-generated structured pattern from the storage unit 12. In this embodiment, a Gray code pattern as shown in Figure 5 is used. A Gray code pattern is a structured light pattern having multiple bit patterns in the vertical and horizontal directions, and unique code information can be embedded at each position. As shown in Figure 5, the vertical pattern is a vertical stripe pattern, and the horizontal pattern is a horizontal stripe pattern. These patterns are generated based on a binary code called Gray code and are designed to minimize the difference in code values ​​between adjacent regions.

[0047] In the example shown in Figure 5, the code values ​​read from each bit pattern are shown in a specific cell region indicated by a thick border. In this specific region, the horizontal Gray code reads horizontalCode[1] = 1, horizontalCode[2] = 0, horizontalCode[3] = 1, and horizontalCode[4] = 0. In other words, in each Gray code pattern, if light is shining on the specific region, it will be "1", and if light is not shining on it, it will be "0". Similarly, the vertical Gray code reads verticalCode[1] = 1, verticalCode[2] = 1, verticalCode[3] = 0, and verticalCode[4] = 1. In this way, the Gray code pattern assigns a unique digital code to each position on the detection target surface, which is then used as a grid ID in subsequent processing.

[0048] Next, the video control unit 11a instructs the projection device 30 to project a positive pattern onto the generated structured light pattern (S1-2), and the imaging control unit 11b instructs the multiple imaging devices 20 to take images (S1-3). The positive pattern refers to a normal black and white pattern, which is projected onto the detection target surface by the projection device 30. The multiple imaging devices 20 synchronously image the detection target surface onto which this positive pattern is projected, and acquire a positive image. Next, the video control unit 11a instructs the projection device 30 to project a negative pattern (S1-4), and the imaging control unit 11b instructs the multiple imaging devices 20 to take images (S1-5). The negative pattern is a pattern with the black and white inverted from the positive pattern, and is used to eliminate the influence of lighting conditions.

[0049] The grid map generation unit 11c performs a difference calculation (S1-6) on the positive and negative images acquired for the same viewpoint and the same pattern. In the difference calculation, the grid map generation unit 11c subtracts the negative image from the positive image to remove the effects of ambient light and the reflective properties of the detection target surface. By projecting and capturing both positive and negative patterns and performing the difference calculation described above, it is possible to remove the effects of ambient light (indoor lighting, sunlight, etc.), normalize differences in reflectivity due to the material of the detection target surface, correct brightness and color unevenness of the projection device 30 itself, and cancel out noise during imaging. As a result, in subsequent difference calculations, the effects of the external factors mentioned above can be eliminated, and only the pure components of the structured light pattern can be extracted with high accuracy.

[0050] Next, the grid map generation unit 11c performs a binarization process (S1-7). In the binarization process, the grid map generation unit 11c applies adaptive thresholding to the difference image, converting each pixel into a binary value of either white (1) or black (0). This allows for clear identification of each bit information in the Gray code. In addition, the binarization process may involve performing multiple image processing steps on the difference image. For example, first, in the threshold calculation process, the grid map generation unit 11c uses Otsu's binarization method or an adaptive thresholding algorithm to calculate the optimal threshold for each region of the difference image. This process enables dynamic threshold setting according to differences in lighting conditions and the material of the surface to be detected. Next, in the edge detection process, the grid map generation unit 11c applies differential operators such as a Sobel filter or a Canny edge detector to emphasize and clarify the boundaries of the structured light pattern. In the subsequent contour extraction process, the grid map generation unit 11c uses connected component analysis and boundary tracking algorithms to precisely identify the boundaries of each pattern region at the pixel level and correct noise and missing parts. Finally, the grid map generation unit 11c integrates these preprocessing results and performs a final conversion of each pixel to a binary value of either white (1) or black (0). This series of image processing steps eliminates the influence of external disturbances and enables high-precision and stable identification of each bit information in the Gray code.

[0051] Next, the grid map generation unit 11c checks whether the processing of all structured light patterns (each bit in the vertical and horizontal directions) has been completed in the determination of completion of all patterns (S1-8). If it has not been completed, the grid map generation unit 11c selects the next bit pattern in the next pattern selection (S1-9) and repeats the processing from S1-2. For example, when using a 5-bit Gray code pattern, projection and photography of a total of 10 patterns—5 patterns in the vertical direction and 5 patterns in the horizontal direction—are performed sequentially.

[0052] Once all patterns have been processed, the grid map generation unit 11c executes the grid ID determination process (S1-10). The grid ID determination process by the grid map generation unit 11c will be explained with reference to a specific example in Figure 5. In Figure 5, when the grid map generation unit 11c reads the value of each bit for a specific cell area indicated by a thick border, horizontal Code[1] = 1, horizontalCode[2] = 0, horizontalCode[3] = 1, and horizontalCode[4] = 0 in the horizontal direction. Combining these values, horizontalCode = 01010. Similarly, vertical Code[1] = 1, verticalCode[2] = 1, verticalCode[3] = 0, and verticalCode[4] = 1 in the vertical direction, resulting in verticalCode = 01101. Here, Gray code is a special binary code in which only one bit changes between adjacent values, and it is necessary to first convert it to a normal binary number (binary code) rather than directly converting it to a decimal number. The grid map generation unit 11c performs a conversion process from Gray code to binary code. horizontalCode = 01010 (Gray code) is converted to 01100 (binary code) by XOR operation, which becomes 12 when converted to decimal. Similarly, verticalCode = 01101 (Gray code) is converted to 01001 (binary code), which becomes 9 in decimal. Finally, the grid map generation unit 11c calculates the coordinates (h,v)=(12,9) and determines the grid ID based on these coordinates. For example, in the case shown in Figure 5, the grid ID is obtained by the formula v × number of cells in the horizontal direction + h, and in this example, the grid ID is determined as 9 × 15 + 12 = 147.

[0053] Finally, the grid map generation unit 11c executes the grid map generation process (S1-11). In the grid map generation process, the grid map generation unit 11c associates the grid ID calculated by the grid ID determination process described above with each pixel on the captured image of each imaging device 20. For example, the grid map generation unit 11c can generate a lookup table for each imaging device 20 that shows the correspondence between the pixel coordinates (x,y) of the captured image and the grid ID. This lookup table is implemented as a two-dimensional array that stores correspondences such as "pixel coordinates (100,200) → grid ID:147". In this way, by associating the same grid ID with different pixel positions on multiple imaging devices 20, it becomes possible to identify the same physical area in images from multiple viewpoints. For example, information is registered that the area with grid ID:147 on the detection target surface corresponds to pixel coordinates (100,200) on the first imaging device 20(a) and pixel coordinates (350,150) on the second imaging device 20(b). The grid map generation unit 11c stores these generated lookup tables in the storage unit 12 for each imaging device 20. This enables high-speed coordinate transformation processing in the subsequent touch area detection process. With this grid map, when a touch area is detected, the pixel position where an object is detected on the image captured from each imaging device 20 is converted into a grid ID, making it possible to identify areas corresponding to the same grid cell across multiple imaging devices 20.

[0054] Next, the touch area detection process according to this embodiment will be described with reference to Figures 6 to 9. Figure 6 is a flowchart showing the flow of the touch area detection process. This touch area detection process is a process that detects the touch area of ​​an object in real time using the grid map generated by the grid map generation process described above. The touch area detection process is mainly executed in cooperation between the object extraction unit 11d and the touch area detection unit 11e of the control device 10.

[0055] First, the basic principle of touch area detection on a curved surface will be explained with reference to Figure 7. Figure 7 shows a situation where a user is touching a roughly cylindrical curved structure with their hand and foot. In this example, imaging devices 20 are positioned at three different locations: viewpoint 1, viewpoint 2, and viewpoint 3. Each imaging device 20 captures the same area of ​​the curved structure from different angles. Here, even for the same physical touch area (hand or foot), the appearance differs significantly in the images captured from each viewpoint. For example, as shown in the lower part of Figure 7, the object is detected as a vertically elongated black area from viewpoint 1, as a different shape from viewpoint 2, and as two separate areas from viewpoint 3. Therefore, in this invention, by integrating the images captured from each viewpoint with a grid map, it becomes possible to accurately identify the same physically identical area. Specifically, in this invention, the area where the object is detected from all three viewpoints—viewpoints 1, 2, and 3, i.e., the area corresponding to the intersection of the detection results from multiple viewpoints—is determined to be the area where the object is truly touching the detection target surface of the structure. Thus, the method for detecting touch areas on curved surfaces is based on the basic principle shown in Figure 1.

[0056] Next, we will describe in detail each stage of the touch area detection process according to the flowchart in Figure 6. As shown in Figure 6, the touch area detection process begins with imaging by multiple cameras (S2-1) based on the control of the imaging control unit 11b. The imaging control unit 11b instructs the multiple imaging devices 20 to perform synchronized imaging and acquires images that include objects that may be touching the detection target surface. Unlike during grid map generation, this imaging is performed under normal lighting conditions without projecting a structured light pattern.

[0057] Next, the object extraction unit 11d performs object extraction processing (S2-2). In object extraction processing, the object extraction unit 11d applies image processing techniques such as background subtraction, color information analysis, motion detection, and contour extraction to the images captured from each imaging device 20 to generate a mask image that shows the area in which an object is captured. This mask image is a binary image in which the object area is represented by white (1) and the background area by black (0). Specifically, as shown in Figure 8, the object extraction unit 11d receives the images captured from each imaging device 20 as input and identifies the object (mainly the user's body part) touching the detection target surface. In addition to conventional image processing techniques such as background subtraction, color information analysis, motion detection, and contour extraction, the object extraction unit 11d can also apply semantic segmentation techniques using deep learning and object detection techniques. In particular, in person detection, by using a trained model using a convolutional neural network (CNN), it becomes possible to extract the person area even in complex background environments. The object extraction unit 11d generates an object extraction image (mask image) as shown in Figure 8 through these processes. This mask image is a binary image in which the object region is represented by white (value 1) and the background region by black (value 0), and contains accurate contour information of objects that are in contact with or close to the detection target surface. The generated mask image is used in the subsequent grid occupancy determination process for compositing with the grid map.

[0058] Furthermore, the object extraction unit 11d reads the grid map previously created by the grid map generation process described above and performs a synthesis process with the object extraction image. As shown in the central part of Figure 8, the touch area detection unit 11e synthesizes this grid map with the object extraction image (mask image) to generate a composite image. In the composite image generation process, only pixels corresponding to the object area retain the pixel value (grid ID) of the grid map, and pixels in the background area are set to a value of 0. This results in a composite image in which information about which grid cell corresponds to the area where the object exists is accurately recorded at the pixel level. The object extraction unit 11d also performs a histogram process on this composite image. As shown on the right side of Figure 8, the histogram is configured with the pixel value (grid ID) on the horizontal axis and the number of pixels on the vertical axis. This histogram provides statistical information on how many pixels of the object area exist in the cell corresponding to each grid ID. This statistical information serves as basic data for quantitatively evaluating the degree of object occupancy in each cell.

[0059] Next, the touch area detection unit 11e performs grid occupancy determination (S2-3) for each viewpoint. This grid occupancy determination is a process that individually determines whether or not each grid cell on the detection target surface is occupied by an object, based on the statistical information obtained in the histogram processing described above. For example, as shown in Figure 8, in the composite image of the object extraction image (mask image) and the grid map, only the pixels corresponding to the object area retain the pixel value (grid ID) of the grid map, and the pixels in the background area are 0. In the histogram of this composite image, the horizontal axis is the pixel value (grid ID), and the vertical axis is the number of pixels. Based on this histogram, the touch area detection unit 11e determines whether or not the number of pixels for each grid ID exceeds a preset threshold. If the number of pixels exceeds the preset threshold, it is determined that an object exists in that grid cell. This threshold can be freely adjusted according to the application, taking into account the cell size, the resolution of the imaging device, the required level of detection accuracy, environmental conditions, etc. In the example in Figure 8, the judgment results are represented by ×, △, and ○, where ○ indicates an area where an object exists on the detection target surface, × indicates an area where an object does not exist on the detection target surface, and △ indicates an area where the detection target surface does not exist. (That is, ○ is an area where the detection target surface and the object overlap, × is an area where the detection target surface and the object do not overlap, and △ is an area where the detection target surface does not exist at all.) As a result, information on which grid cell is occupied by an object is individually acquired from the viewpoint of each imaging device 20.

[0060] Next, the touch area detection unit 11e performs a multi-viewpoint integrated determination (S2-4). This process will be explained in detail with reference to Figure 9. The multi-viewpoint integrated determination is a process that integrates multiple determination results obtained from the grid occupancy determination at each viewpoint as described above, and identifies the grid cell that an object is actually touching based on the agreement across multiple viewpoints. For example, as shown in Figure 9, in the multi-viewpoint integrated determination, the touch area detection unit 11e integrates the histogram analysis results from multiple viewpoints (viewpoint 1, viewpoint 2, viewpoint 3, etc.). At each viewpoint, the object presence determination result (×, ○, △, etc.) for each grid ID is obtained from the grid occupancy determination described above. For each grid ID, the touch area detection unit 11e aggregates the determination results from multiple viewpoints and calculates the number of touch determinations. For example, if a particular grid ID is judged as ○ at viewpoint 1, ○ at viewpoint 2, and ○ at viewpoint 3, the number of touch determinations for that grid cell will be 3. When the number of touch determinations is 3, that grid cell is identified as being touched by an object. Furthermore, for a specific grid ID, even if two viewpoints show ○ and the remaining one shows △, that grid cell is identified as being touched by an object. On the other hand, for three viewpoints, cells with one or more ×s and cells with two or more △s are identified as cells not being touched by an object. In this way, the touch area detection unit 11e compares this number of touch detections with a preset threshold and identifies grid cells that exceed the threshold as being touched by an object. This touch detection threshold can also be freely adjusted according to the application, taking into account environmental conditions, etc. In the example in Figure 9, the final detection result is shown as ×, ○, ○, ×, ×.

[0061] The touch area detection unit 11e performs a touch detection determination (S2-5) based on the integrated determination result. In the touch detection determination, the touch area detection unit 11e analyzes the number and distribution of touch cells identified by the integrated determination and determines whether or not a valid touch area exists. If no touch area is detected (No in S2-5), the system transitions to a non-touch state (S2-6) and repeats the process from S2-1. On the other hand, if a touch area is detected (Yes in S2-5), the touch area detection unit 11e performs a touch area calculation (S2-7). In the touch area calculation, the touch area detection unit 11e calculates geometric features such as the centroid position, area, and shape of the identified group of touch cells and determines detailed information about the touch area.

[0062] Finally, the control device 10 notifies the application (S2-8). In this notification process, the control device 10 transmits the calculated touch area information to the relevant functional units such as the video control unit 11a and the sound control unit 11f. The video control unit 11a instructs the projection device 30 to project interactive images based on the touch area information. For example, it can project images such as flowers blooming in the touched area, butterflies or birds fluttering, lines or patterns drawn along the touch trajectory, effect images radiating from the touch position, images with changing colors, or images generating geometric patterns. The sound control unit 11f also instructs the sound device 40 to output sound effects. For example, it can output sounds such as bells or chimes when touched, natural sounds (wind, water droplets, birdsong, etc.), musical instrument sounds (piano, harp, xylophone, etc.), changes in ambient sound, and modulation of the tone and volume of background music. Furthermore, the detailed information of the touch area obtained in the aforementioned touch area calculation (S2-7) can be used to control the output methods of these images and sounds. For example, it is possible to adjust the size of video effects and the volume level of sound according to the area of ​​the touch region, control the projection position of the video and the direction of sound output according to the center of gravity, and select different video and sound patterns according to the shape of the touch region. This provides a real-time interactive experience that responds to the user's touch actions. After processing is complete, the system returns to S2-1 and performs continuous touch region detection.

[0063] As described above, the touch area detection process of this embodiment integrates captured images from multiple viewpoints with a grid map and performs histogram analysis to achieve multi-viewpoint integrated judgment using statistical methods. This effectively suppresses situations that are difficult to detect with a single viewpoint and false detections due to noise, enabling stable touch area detection even on curved surfaces.

[0064] In this specification, embodiments of the present invention have been described with reference to the drawings in order to express the content of the present invention. However, the present invention is not limited to the above embodiments, and includes modifications and improvements that are obvious to those skilled in the art based on the matters described in this specification.

[0065] For example, although the above embodiment shows an example using multiple imaging devices 20, the number and arrangement of imaging devices can be appropriately changed according to the shape of the surface to be detected and the required detection accuracy. Furthermore, regarding the object extraction method in the object extraction unit 11d, in addition to semantic segmentation using deep learning, conventional image processing techniques (background subtraction, color information analysis, motion detection, contour extraction, etc.) can be used individually or in combination. Moreover, in the generation of the grid map, in addition to the Gray code pattern, stripe patterns, dot patterns, random patterns, etc. can be used as structured light patterns, and various modifications are possible to the projection method and analysis method of these patterns.

[0066] Furthermore, the objects to be detected are not limited to people; the present invention can be applied to animals, robots, tools such as pointers and pens, or any other object. The surfaces to be detected can also be diverse, including not only the walls, columns, floors, and ceilings of buildings, but also the interior surfaces of vehicles, furniture surfaces, exhibit surfaces, and outdoor installation surfaces. Moreover, applications utilizing the detection results of the touch area are not limited to projection mapping for image projection or sound output; they can be applied to a variety of control systems, such as lighting control, air conditioning control, mechanical device control, and information display system control.

[0067] Furthermore, the system configuration of the present invention is not limited to a configuration in which the control device 10, imaging device 20, projection device 30, and sound device 40 are physically separated, but can also be configured in which some or all of these components are integrated. For example, a projector device that integrates the control device 10 and the projection device 30, or a projection device that incorporates the functions of the imaging device 20, are also included in the scope of the present invention. In addition, it is possible to expand the invention to distributed processing systems in which part of the processing is performed on a cloud server, or to large-scale systems that link multiple touch area detection systems. Thus, the present invention is not limited to the specific configurations shown in the above embodiments, and various modifications and applications are possible within the scope of the essential technical idea of ​​the invention. [Industrial applicability]

[0068] This invention can appropriately detect the touch area of ​​an object not only when the target surface is flat, but also when it is curved. Therefore, it can be used in a wide range of industrial fields, including projection mapping technology. In particular, it can be applied to interactive video projection systems for curved structures such as building walls and columns, digital signage systems in museums and commercial facilities, and learning support systems in educational institutions. [Explanation of Symbols]

[0069] 10…Control device 11…Control unit 11a...Video control unit 11b...Imaging control unit 11c...Grid map generation unit 11d...Object extraction unit 11e...Touch area detection unit 11f...Acoustic control unit 12...Memory unit 13...Input unit 14…Output unit 20…Imaging device 30...Projection device 40...Sound device 100... Touch area detection system S...Surface to be detected O...Object

Claims

1. A touch area detection system for detecting the touch area of ​​an object on a surface to be detected, Multiple imaging devices arranged to image the target surface from different viewpoints, The control device analyzes multiple images captured by the multiple imaging devices of the object touching the detection target surface to detect the touch area of ​​the object on the detection target surface. The control device is A storage unit that stores a grid map in which the detection target surface is virtually divided into multiple cells, The system includes a touch area detection unit that detects the touch area of ​​the object on the target surface by determining that cells detected as being touched by the object in common across multiple captured images taken from different viewpoints are cells that the object is actually touching. Touch area detection system.

2. The aforementioned detection target surface includes a curved surface. The touch area detection system according to claim 1.

3. The control device further includes an object extraction unit that extracts an object region in which the object is captured from the plurality of captured images, The touch area detection unit determines the occupied range of the object area in each cell of the grid map for each of the multiple captured images, and determines that cells in which an occupied range of a predetermined amount or more is determined to be cells that the object is actually touching. The touch area detection system according to claim 1 or claim 2.

4. The device further comprises a projection device that projects an image onto the surface to be detected. The control device is The projection device includes an image control unit that projects a structured light pattern onto the detection target surface, The system further includes a grid map generation unit that generates a grid map in which the detection target surface is virtually divided into multiple cells based on the structured light pattern on the detection target surface captured by the plurality of imaging devices. The touch area detection system according to claim 1 or claim 2.

5. The device further comprises a projection device that projects an image onto the surface to be detected. The control device further includes an image control unit that controls the image to be projected onto the projection device based on the detection result of the touch area. The touch area detection system according to claim 1 or claim 2.

6. A touch area detection method for detecting the touch area of ​​an object on a surface to be detected, A step of imaging the target surface from different viewpoints using multiple imaging devices, The control device includes the step of analyzing multiple captured images taken by the multiple imaging devices of the object touching the detection target surface, and detecting the touch area of ​​the object on the detection target surface. The control device is The system stores a grid map in which the detection target surface is virtually divided into multiple cells. By determining that cells detected as being touched by the object in multiple captured images from different viewpoints are cells that the object is actually touching, the touch area of ​​the object on the detection target surface is detected. A method for detecting a touch area.

7. A program that causes a computer to perform a process to detect the touch area of ​​an object on a target surface, The aforementioned program, Image acquisition process that acquires images from multiple imaging devices arranged to capture the target surface from different viewpoints, The computer is instructed to perform a touch area detection process, which involves analyzing multiple images captured by the multiple imaging devices of the object touching the target surface, and detecting the touch area of ​​the object on the target surface. The touch area detection process includes detecting the touch area of ​​the object on the detection target surface by determining, among the grid map obtained by virtually dividing the detection target surface into multiple cells, that the cells in which the object is detected to be touching in common across multiple captured images from different viewpoints are the cells in which the object is actually touching. Computer program.

Images