Projection system and projection method

Abstract

To more accurately identify object types in projection mapping. [Solution] The projection system 100 includes a temperature measuring device 20 that measures the temperature of an object, a projection device 40 that projects an image onto the object, and a control device 10 that controls the image projected onto the object by the projection device 40 based on the temperature of the object measured by the temperature measuring device 20.

Description

[Technical Field]

[0001] The present invention relates to a projection system and a projection method for projecting an image onto an object such as food or drink. [Background technology]

[0002] Projection mapping technology has been known for some time, in which an image of an object is captured by a camera and an image is projected onto the surface of the object so as to track the object. For example, Non-Patent Document 1 discloses a system that performs projection mapping on natural objects (such as plant leaves and flowers). This system is said to be able to recognize and track the shape of the object without a marker using an infrared camera, infrared light, and a projector. More specifically, the contour of the object is extracted from an image captured by an infrared camera, and a projected image is generated based on the contour information, and is projected while tracking the movement of the object. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Morimoto, Yuki, Plant and Animal Display, Proceedings of the International Display Workshops, December 9, 2021, Vol. 27, pp. 854-855 Summary of the Invention [Problem to be solved by the invention]

[0004] Although it is not explicitly stated in Non-Patent Document 1, it is presumed that the type of object is identified based on the captured image and then the video effect is changed depending on the type of object, as the contours of the object are extracted from the image captured by the infrared camera and the video effects differ depending on the type of object, such as ginkgo leaves, maple leaves, silkworms, etc. Note that even if Non-Patent Document 1 does not actually identify the type of object based on the captured image, it is currently a well-known technology to identify the type of object based on external feature points such as the contours and size of the object contained in this imaging device.

[0005] However, when extracting external feature points of an object from a captured image and identifying the type of object based on those feature points, the identification becomes difficult when the differences between the feature points of the object are small. Furthermore, as described below, the applicant of the present application is considering using such projection mapping technology in the food and beverage service industry to project images onto food and beverages served on tableware or their surroundings. When such tableware or food and beverages are used as the target of image projection, differences in external feature points such as outline and size are often small, making it difficult to accurately identify the object from a captured image. Furthermore, since beverages are fluids and their shape is determined by the container into which they are poured, it is impossible to identify their type based on their shape alone. Furthermore, the shape of the container is generally circular in a planar view, making it difficult to distinguish between hot and cold beverages by appearance. Furthermore, since ice cream melts and changes shape over time, it is difficult to identify the type of ice cream based on its shape. For example, a typical ice cream bowl and a tea bowl, each about 8 to 12 cm in diameter, have similar shapes and sizes in a plan view, making it difficult to distinguish them from a photographed image alone.

[0006] Furthermore, because this type of projection mapping projects image light onto an object, it is generally necessary to have a dark environment around the object (for example, an illumination level of 50 lux or less). However, in images of objects in such dark places, external features become unclear, making it more difficult to identify their type.

[0007] Therefore, the main objective of the present invention is to provide a technology that enables more accurate identification of the type of object in projection mapping, which changes the image projected onto the object depending on the type of object, even when there is little difference in the object's external features or when the object is in a dark place. [Means for solving the problem]

[0008] Therefore, the inventor of the present invention has intensively studied means for solving the above-mentioned problems of the prior art, and has come up with the idea of ​​providing a temperature measuring device for measuring the temperature of an object instead of or in addition to a photographing device for photographing the object, and changing the image projected on the object based on the measured temperature of the object. This has led to the discovery that the type of object can be identified not only by the external features of the object but also by its temperature. Based on this discovery, the inventor has come to the realization that the problems of the prior art can be solved, and has completed the present invention. Specifically, the present invention has the following configuration or steps.

[0009] A first aspect of the present invention relates to a projection system. This projection system is used for projection mapping. The projection system includes at least a temperature measuring device, a projection device, and a control device. The temperature measuring device measures the temperature of an object. The temperature measuring device is preferably a non-contact temperature measuring device such as a thermal camera capable of measuring surface distribution or an infrared temperature sensor (radiation temperature sensor) for single-point measurement, but may also be a contact temperature measuring device such as a digital thermocouple thermometer or thermistor thermometer. The projection device projects an image onto the object. The control device controls the image projected onto the object by the projection device based on the temperature of the object measured by the temperature measuring device. By measuring the temperature of the object in this way, it is possible to identify the type of the object even when there is little difference in the external features of the object or when the object is located in a dark place. As described below, an additional imaging device may be provided to photograph the object and the type of the object can be further identified from the external features, but it is also possible to identify the type of the object simply by measuring the temperature of the object with the temperature measuring device. For example, if the type of object onto which the image is intended to be projected can be clearly distinguished by temperature, such as hot tea, cold tea, or cold ice cream, it is possible to identify the type of object using only a temperature measuring device.

[0010] The projection system according to the present invention preferably further includes an imaging device for capturing an image of the object. The imaging device is preferably a night-vision camera such as an infrared camera, but a general digital camera can also be used as long as the surroundings of the object are kept relatively bright. In this case, the control device identifies the type of object's external shape based on the image captured by the imaging device. The control device then controls the image projected onto the object by the projection device based on the object's temperature and the type of object's external shape. By using both the object's temperature information and the identification information on the object's external shape type in this way, it is possible to distinguish between objects with the same external shape but different temperatures. For example, this makes it possible to distinguish between cold and hot beverages in the same container.

[0011] In the projection system according to the present invention, the control device may identify the coordinates of the object based on an image captured by the imaging device. In this case, it is preferable that the control device controls the image projected by the projection device onto the object based on the temperature of the object, the external shape of the object, and coordinate change information of the object. The coordinate change information of the object may include information on whether the coordinates of the object have changed and, if so, may also include information on the amount of change. The coordinates of the object may be identified using a two-dimensional coordinate system in the captured image or a three-dimensional coordinate system in the space in which the object exists. In this way, by changing the effect of the image projected onto the object depending on whether the coordinates of the object have changed and the amount of change, interactive video presentations can be achieved. In particular, in the projection system according to the present invention, it is preferable that the control device changes the image projected by the projection device onto the object depending on whether the coordinates of the object have changed and whether they have not.

[0012] In the projection system according to the present invention, the target object is preferably a container capable of containing the object. For example, the container is preferably tableware such as a cup, teacup, ice cream cup, soup plate, or plate, and the object is preferably food or drink contained in such tableware. In this case, the control device identifies the type of container based on the image captured by the imaging device. The control device also identifies the type of object contained in the container based on the type of container and the temperature measured by the temperature measuring device. The control device then preferably controls the image projected onto the target by the projection device based on the type of object. According to this embodiment, the identification information of the container can be used to easily distinguish between food and drink at different temperatures, such as hot drinks, cold drinks, and even cold ice cream, making this projection system suitable for use in the food service industry.

[0013] In the projection system according to the present invention, the temperature measuring device is preferably a thermal camera. A thermal camera can acquire a planar temperature distribution within a shooting range. Therefore, the control device can grasp the position and shape of the object based on the temperature distribution acquired from the thermal camera. Furthermore, since a thermal camera can continuously output temperature data as an image, the control device can easily acquire and analyze the temperature data, and can track coordinate changes even when the object moves. Therefore, it is possible to acquire the temperature of the object, its external characteristics (outline and size), and coordinate changes of the object using only the thermal camera. Furthermore, since a thermal camera can measure temperature even in complete darkness, the temperature of the object can be accurately measured even in a dark environment during image projection.

[0014] A second aspect of the present invention relates to a projection method. In the projection method according to the present invention, a temperature measuring device measures the temperature of an object (temperature measuring step), a control device controls an image to be projected onto the object based on the temperature of the object measured by the temperature measuring device (image control step), and a projection device projects an image onto the object based on the control of the control device (projection step). [Effects of the Invention]

[0015] According to the present invention, even when there is little difference in the external features of the object or when the object is in a dark place, the type of the object can be more accurately identified.

[0016] For example, by using the temperature information of the object, it is possible to reliably distinguish between hot and cold beverages contained in containers of the same shape, and project appropriate visual effects for each. Furthermore, even for objects whose shape changes over time, such as ice cream, accurate identification can be continued based on the temperature information. Furthermore, when using a thermal camera, the temperature of the object can be accurately measured even when projecting images in a dark place, and appropriate visual effects can be continuously projected based on that information. Therefore, the present invention is useful as a projection mapping system with high performance in the food and beverage service industry and other fields. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a schematic diagram showing the overall configuration of a projection system according to one embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing the functional configuration of a projection system according to an embodiment of the present invention. [Figure 3] FIG. 3 is a diagram for explaining the temperature detection pattern by the temperature measuring device, showing the change in the detected temperature over time and the corresponding characteristic temperature pattern. [Figure 4] FIG. 4 is a diagram showing an example of changes in the projected image, and (a) to (d) show the stages of image projection onto an object contained in a container. [Figure 5] FIG. 5 is a diagram illustrating an example of the relationship between object identification and video control. [Figure 6] FIG. 6 is a diagram illustrating an example of the automatic temperature threshold adjustment function. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The present invention is not limited to the embodiments described below, and includes appropriate modifications of the embodiments described below within the scope obvious to those skilled in the art.

[0019] FIG. 1 shows the overall configuration of a projection system 100 according to an embodiment of the present invention. As shown in FIG. 1, in this embodiment, the projection system 100 is designed to project images onto food and drink in containers placed on a table or floor (e.g., tatami mat) and the surrounding area, and to output sound (e.g., background music, sound effects, and commentary). In other words, this projection system 100 is used in restaurants to project images onto food and drink served to customers and to output sounds appropriate for the food and drink. The projection system 100 includes a control device 10 such as a computer, a thermal camera 20 (temperature measuring device), an infrared camera 30 (photographing device), a projector 40 (projection device), and a speaker 50 (sound emitting device), all of which are connected via a wired or wireless LAN or the like. The thermal camera 20, the infrared camera 30, the projector 40, and the speaker 50 form a single unit. One unit may be provided for each customer seat, or a single unit may be used to project images onto multiple seats. It should be noted that one control device 10 may simultaneously control a plurality of units, or one control device 10 may control one unit.

[0020] In this embodiment, it is desirable to adjust the illuminance of the projection target area to 50 lux or less, preferably 20 lux or less, in order to ensure visibility of the image projected by the projector 40. Even in such a dark environment, it is possible to measure the temperature with the thermal camera 20 and capture images with the infrared camera 30.

[0021] As shown in Fig. 1, the thermal camera 20, the infrared camera 30, and the projector 40 are basically installed so as to face a surface (projection surface) such as a table on which an object to be projected is placed. Specifically, these devices 20, 30, and 40 are installed above the table, such as on the ceiling, and oriented so as to capture or project an image of the object placed on the table in a planar view. The installation positions and orientations of the devices 20, 30, and 40 are adjusted so that an area is formed in which the imaging range of the thermal camera 20, the imaging range of the infrared camera 30, and the projection range of the projector 40 all overlap. The object to be projected (e.g., a container containing food or drink) is placed in this overlapping area.

[0022] 2 is a block diagram showing the functional configuration of the projection system 100 according to this embodiment. The control device 10 is a computer that is responsible for overall control of the projection system 100. In particular, the control device 10 analyzes the images and information acquired by the thermal camera 20 and the infrared camera 30, selects or generates an image based on the analysis results, and controls the projector 40 to project the image onto an object. The control device 10 also analyzes the images and information acquired by the thermal camera 20 and the infrared camera 30, selects or generates sound (such as background music, sound effects, and commentary audio) based on the analysis results, and controls the speaker 50 to output the sound.

[0023] While FIGS. 1 and 2 show an example in which the control device 10 is configured using a single computer, it is also possible to distribute the functions of the control device 10 among multiple computers and construct the control device 10 using these multiple computers. For example, when the control device 10 is configured in a distributed manner, an image processing computer that analyzes input information from the thermal camera 20 and the infrared camera 30 and an image generation computer that selects or generates images to be projected based on the analysis results can be provided separately. Furthermore, when performing control corresponding to multiple seats, it is also possible to provide an independent image generation computer for each seat. Similarly, a separate sound generation computer that selects or generates sounds to be output based on the analysis results can also be provided. These computers are interconnected via a network to share necessary information.

[0024] As shown in FIG. 2, the control device 10 includes at least a control unit 11 and a storage unit 12. The control unit 11 is composed of, for example, a processor and a memory. Examples of the processor include a known CPU (Central Processing Unit), GPU (Graphics Processing Unit), and other control circuits. The processor performs predetermined arithmetic processing according to a program stored in the memory, and executes various control processes while writing the results of the calculations into a working space in the memory. The memory is composed of, for example, a volatile memory such as RAM (Random Access Memory), and is used for the arithmetic processing by the processor.

[0025] The memory unit 12 is a storage element that mainly stores video content to be projected onto the object by the projector 40, audio content to be output from the speaker 50, and information used in the arithmetic processing by the control unit 11. The memory unit 12 is configured with a non-volatile memory such as a read-only memory (ROM). Examples of the video content include texture images to be projected onto the surface of the object, image sequences for animation, pattern images for generating various visual effects, and parameter information for video effects that change according to the movement of the object. Examples of the audio content include background music corresponding to the type and movement of the object, sound effects, and audio commentary about the object. The information used in the arithmetic processing by the control unit 11 may also include, for example, a trained model used to estimate the type of object based on its external features from an image captured by the infrared camera 30. Such a trained model is generated by machine learning using pre-prepared training data (pairs of captured images of the object and label information indicating the type of object corresponding to the image). For example, a convolutional neural network (CNN) suitable for image recognition can be used as a machine learning technique. Alternatively, a deep learning model combined with a Kalman filter, such as YOLO (You Only Look Once), which is excellent for object detection, a support vector machine (SVM), which can efficiently extract the contour shape features of an object, or a Kalman filter suitable for tracking the position of an object, can also be used. These machine learning models can be appropriately selected depending on the actual usage environment. Furthermore, the memory unit 12 stores temperature threshold information for determining the type of object from the temperature measurement results obtained by the thermal camera 20, tracking parameters for tracking the position and movement of the object, and the like. The memory unit 12 may also store a program used for processing by the control unit 11.For example, a group of programs including an image processing program, a temperature data analysis program, a video generation program, an audio control program, libraries for various types of arithmetic processing, etc. These programs are executed by the processor of the control unit 11 to realize various functions described below.

[0026] The thermal camera 20 is an example of a temperature measuring device that measures the temperature of an object without contact. Specifically, the thermal camera 20 detects infrared rays emitted from the object and calculates the surface temperature of the object from the intensity of the infrared rays. In this embodiment, the lowest temperature object is assumed to be ice cream at approximately -10°C, and the highest temperature object is assumed to be hot tea at approximately 90°C. For this reason, it is preferable to use a thermal camera 20 with a measurement temperature range of approximately -20°C to 120°C. The thermal resolution of the thermal camera 20 is approximately 0.1 to 1°C, and the frame rate is 30 fps or 60 fps. The thermal camera 20 can also output the temperature distribution within the imaging area as two-dimensional image data, and the position and shape of the object can be determined from this temperature distribution data. This temperature distribution data is transmitted to the control device 10 and used for identifying the type of object and determining the image projection position. Although a thermal camera 20 is most suitable as a temperature measuring device, other non-contact temperature measuring devices such as a single-point measurement infrared temperature sensor (radiation temperature sensor) or an infrared array sensor can also be used instead.

[0027] The infrared camera 30 is an example of an imaging device that captures images of an object and its surroundings. The infrared camera 30 can detect light in the near-infrared region (wavelengths of approximately 0.7 to 1.4 μm) and is suitable for imaging in dark places. The infrared camera 30 basically includes an optical lens that transmits infrared light and an imaging element (image sensor) that converts the infrared light into an electrical signal. An infrared transmission filter (IR pass filter) that cuts visible light is placed in front of the imaging element to eliminate the influence of visible light. This allows stable infrared images to be captured regardless of the presence or absence of visible light, such as lighting. In this embodiment, near-infrared light emitted from an infrared light source (not shown) is detected as light reflected by the object, allowing the contour and shape of the object to be clearly captured even in dark places. Furthermore, by using an infrared camera 30 with a resolution of 1920 × 1080 pixels or more and a frame rate of 60 fps or more, the movement of the object can be accurately tracked. Image data captured by the infrared camera 30 is transmitted to the control device 10, where image processing is performed to extract the contour and detect the position of the object. If the lighting in the shooting environment is sufficient, it is also possible to use a general digital camera that detects visible light as the shooting device instead of the infrared camera 30. However, in order to ensure the visibility of the image projected by the projector 40, the shooting environment is generally a dark place, so it is preferable to use the infrared camera 30 described here.

[0028] Projector 40 is an example of a projection device, and projects an image onto an object under the control of control device 10. Projector 40 preferably has a resolution of 1920 x 1080 pixels or more, a brightness of 4000 lumens or more, and a projection angle of view of approximately 60 to 80 degrees. Projector 40 is preferably a short-focus projector, which can ensure a wide projection range while minimizing the installation height from the ceiling. Projected images are updated at a frame rate of 30 fps or 60 fps depending on the video content, enabling smooth image display that tracks the movement of the object. The intensity of the image light projected from projector 40 is preferably automatically adjusted according to the ambient illuminance.

[0029] In this embodiment, the infrared camera 30 and the projector 40 are preferably installed adjacent to each other. This minimizes the misalignment (parallax) between the position of the object detected by the infrared camera 30 and the position of the image projected by the projector 40. For example, as shown in FIG. 1, by placing the infrared camera 30 and the projector 40 close to each other, even if a customer lifts an object, the image can be accurately projected by tracking the object's position. On the other hand, if there is a large distance between the infrared camera 30 and the projector 40, a large parallax occurs between the camera's viewpoint and the projector's projection point. This causes significant misalignment of the projected image, especially when the object is located far from the projection surface. For this reason, in this embodiment, the linear distance between the optical axes of the infrared camera 30 and the projector 40 is preferably 30 cm or less, and more preferably 20 cm or 15 cm or less. This allows for image alignment with sufficient accuracy for practical use.

[0030] It is also preferable to install the thermal camera 20 and the projector 40 as close as possible. This is to improve the accuracy of alignment between the temperature distribution of the object measured by the thermal camera 20 and the image projection position by the projector 40. If there is a large distance between the thermal camera 20 and the projector 40, a misalignment will occur between the temperature measurement position and the image projection position, and this misalignment will be particularly noticeable when the object is located far from the projection surface. For this reason, in this embodiment, the linear distance between the optical axes of the thermal camera 20 and the projector 40 is preferably 30 cm or less, and particularly preferably 20 cm or 15 cm or less. It is also important to match the lens heights of the thermal camera 20 and the projector 40. This is because different lens heights will change the relative positional relationship between the temperature measurement position and the image projection position depending on the distance to the object. Therefore, by installing the centers of the lenses of the thermal camera 20 and the projector 40 at the same height, stable alignment can be achieved even when the object moves.

[0031] The speaker 50 is an example of a sound-emitting device that outputs sound (such as background music, sound effects, and commentary) under the control of the control device 10. Any known speaker 50 can be used. For example, the speaker 50 is preferably a small, full-range speaker capable of reproducing a wide frequency range from low to high tones. The speaker 50 may be installed on the ceiling or other location near the customer's seat, similar to the infrared camera 30 and the projector 40. In this case, the speaker 50 is preferably installed with consideration given to directionality so that the sound from the speaker 50 does not excessively interfere with other seats. Specifically, by adopting a highly directional speaker as the speaker 50 and installing it with its directional axis facing the seat, sound leakage to other seats can be minimized. The volume output from the speaker 50 is preferably automatically adjusted according to the surrounding environmental sounds. The speaker 50 is not limited to being installed on the ceiling; it can also be installed, for example, on a table or embedded in the table.

[0032] Next, specific functions of the control unit 11 of the control device 10 shown in Fig. 2 will be described in detail. As shown in Fig. 2, the control unit 11 includes, as characteristic functional elements, a container identification unit 11a, a temperature identification unit 11b, a position detection unit 11c, an image control unit 11d, and an audio control unit 11e. Each of these functions can be realized by executing a program stored in the storage unit 12.

[0033] The container identification unit 11a executes a process for identifying the type of container among the target objects (containers containing contents) based on the captured image from the infrared camera 30. In this embodiment, multiple containers of different sizes (diameters) are identified. In this embodiment, it is assumed that all containers are perfectly circular in plan view and differ only in size. More specifically, two types of containers are identified: ice cream containers and tea containers. The ice cream containers are larger than the tea containers. To facilitate identification of the two types, the diameter of the ice cream container is preferably 1.2 times or more, more preferably 1.5 times or more, or even 2 times or more, the diameter of the tea container. In this manner, the container identification unit 11a extracts the outline of the container from the captured image and identifies the type based on its external characteristics, such as its shape and size. Although not shown, it is also possible to make the shape (outline) of each container different to facilitate identification of the container based on its external characteristics.

[0034] In addition to the type of container, the container identification unit 11a may also identify whether each container contains content based on the image captured by the infrared camera 30. For example, in the above example, the container identification unit 11 can identify a filled ice cream container, an empty ice cream container, a filled tea container, and an empty tea container. To determine whether a container contains content, it is preferable for the container to be transparent or translucent and the contents to be opaque. In an image captured by the infrared camera 30, a filled container displays a distinctive pattern inside the container due to the reflection and absorption of light by the contents, whereas an empty container displays only reflected light from the bottom of the container. The container identification unit 11a can determine whether a container contains content based on the difference in the image pattern inside the container. Furthermore, the amount (remaining amount) of content can be estimated by detecting the position and shape of the liquid level of the content. This allows for appropriate changes in the visual effects, for example, when the tea or ice cream has been drunk.

[0035] Preferably, the container identification unit 11a estimates the type of container and the presence or absence of contents using a trained model stored in the memory unit 12. That is, the control unit 11 inputs the container's outline information, size, shape features, etc. extracted from the captured image into the trained model, and identifies the container type based on the output result. The trained model is generated by machine learning using numerous container images captured in actual usage environments and their correct labels as training data, and its use can improve the accuracy of container identification. Note that, to prevent erroneous detection of containers, the container identification unit 11a may also identify background objects, such as the customer's arm or small items, included in the captured image.

[0036] The temperature identification unit 11a executes a process for identifying the temperature range of the object, particularly the contents of the container, based on the temperature information of the object obtained by the thermal camera 20. FIG. 3 shows an example of a temperature identification pattern by the temperature identification unit 11a. In the example shown in FIG. 3, the container contains three types of contents: hot tea, iced tea, and ice cream. Each of these three types of contents has a temperature range for identification. For example, hot tea is set to a temperature range above 35°C, iced tea is set to a temperature range of 10 to 35°C, and ice cream is set to a temperature range of -10 to 15°C. Although the temperature ranges of iced tea and ice cream overlap in part, they can be distinguished because they are placed in different containers. Therefore, it is sufficient to set the temperature range for each content so that at least contents placed in the same container (e.g., hot tea and iced tea) can be distinguished. As shown in FIG. 3, it is also assumed that the temperature of these contents changes over time. For example, hot tea starts at around 50°C and gradually decreases in temperature as it approaches room temperature (see symbols A1-A2). Cold tea starts at around 10°C and gradually increases in temperature as it approaches room temperature (see symbols B1-B3). Similarly, ice cream starts at around -10°C and gradually increases in temperature as it approaches room temperature (see symbols C1-C2). Note that temperature ranges may also be set so that human skin and a table can be distinguished, even though they are not direct projection targets.

[0037] Figure 3 shows examples of objects detected in each temperature range along with the temperature discrimination pattern. In the highest temperature range (over approximately 45°C), only hot tea (A1) is detected. In the second highest temperature range (approximately 35-45°C), hot tea (A2) that is beginning to cool is detected. However, since this temperature range is close to human body temperature, human skin is also detected as a similar temperature. Since human skin is not in a container, it can be excluded as an object. In the third highest temperature range (approximately 25-35°C), tea that was originally cold but has since warmed up (B3) is detected. However, since this temperature range is close to room temperature, a table is also detected as a similar temperature. Since the table is not in a container, it can be excluded as an object. In the fourth highest temperature range (approximately 15-25°C), cold tea (B2) is detected, and in some cases, a container of melted ice cream (C2) is also detected. The fifth warmest region (approximately 0-15°C) shows that in addition to the well-chilled tea (B1), an ice cream bowl (C1 bowl) and ice cream that has begun to melt (C2) are detected. Finally, the coldest region (below approximately 0°C) shows that only the cold ice cream (C1) that has not yet melted is detected.

[0038] Note that the types of contents listed here are just examples, and there may be four types, such as hot tea, cold tea, matcha ice cream, and yuzu ice cream. In this case, hot tea and cold tea are distinguished by temperature range as described above, but matcha ice cream and yuzu ice cream are difficult to distinguish by temperature range, so it is best to put them in containers of different sizes. In this case, different sizes of containers would be prepared for tea, matcha ice cream, and yuzu ice cream. Similarly, by adjusting the temperature range and container size, it is possible to accommodate five or more types of contents.

[0039] The coordinate determination unit 11c performs processing to determine the position coordinates and the amount of change of the object based primarily on the image captured by the infrared camera 30. Specifically, it extracts the contour of the object from the image captured by the infrared camera 30 and calculates the center of gravity position as the coordinates of the object. Furthermore, by comparing the coordinates of the object between consecutive frames, it is possible to determine the amount of change in the lateral (x-y) movement direction and movement speed of the object. Furthermore, the coordinate determination unit 11c can estimate the movement of the object in the vertical direction (z-direction) based on the change in the apparent size (diameter) of the object in the captured image. Specifically, if the area calculated from the contour of the object in the image captured by the infrared camera 30 increases, it can be determined that the object is approaching the camera (i.e., being lifted upward). Conversely, if the apparent area of ​​the object decreases, it can be determined that the object is moving downward. In this way, it is also possible to estimate the three-dimensional movement of the object from the two-dimensional captured image. The coordinate determination unit 11c outputs the determined coordinate information to the image control unit 11d, enabling image projection according to the position and movement of the object.

[0040] The position of the object can also be determined from the temperature distribution acquired by the thermal camera 20. The position of the object may be determined using only the temperature distribution acquired by the thermal camera 20, or the position of the object may be determined by combining the temperature distribution data acquired by the thermal camera 20 with the image captured by the infrared camera 30. In particular, the coordinate determination unit 11c can achieve more stable coordinate tracking by using both the temperature distribution data acquired by the thermal camera 20 and the image captured by the infrared camera 30. For example, even if the capture by the infrared camera 30 becomes temporarily unstable due to a change in lighting conditions, the coordinate determination unit 11c can continue tracking the object using the temperature distribution data acquired by the thermal camera 20. In this way, the coordinate determination unit 11c can accurately track the position and movement of the object by combining information from multiple sensors. In particular, even if a customer lifts an object, it is possible to determine its coordinates by following its three-dimensional movement.

[0041] The image control unit 11d controls the image projected by the projector 40 based on the processing results of the container identification unit 11a, the temperature identification unit 11b, and the coordinate identification unit 11c. First, with reference to FIG. 4, an example of a visual effect achieved by image projection in this embodiment will be briefly described. As shown in FIG. 4(a), assume that a container contains cold tea. When this container is placed within the projection range of the projection system 100 (the overlapping area between the shooting range and the projection range shown in FIG. 1), an image of flowers corresponding to the cold tea is projected onto the surface of the tea in the container, as shown in FIG. 4(b). Although not shown, related images (e.g., petals fluttering or ripples spreading) can also be projected onto the table surface, wall surface, or the audience's arms around the container. Furthermore, as shown in FIG. 4(c), when a customer lifts the container while the image is projected onto the tea, an effect is created in which the petals projected onto the tea scatter. Although not shown, it is also possible to project video effects (e.g., petals falling through the customer's hand or petals piling up on the table) that are linked to the surroundings of the container, such as on the table, wall, or the customer's arm. Such dynamic video effects, such as petals scattering, can create a natural presentation linked to the movement of the container. Furthermore, as shown in FIG. 4(d), when the customer finishes drinking the tea in the container and the container is empty, the video projected onto the tea disappears. Related video projected onto the environment surrounding the container also disappears. In this way, this embodiment can create an interactive presentation in which different images are projected onto different types of objects and the video changes as the object moves or its state changes. It is also possible to project video not only onto the object itself but also onto its surroundings (e.g., a table, a wall, or a human body).

[0042] Next, FIG. 5 is a schematic diagram illustrating the sequence of image control processing by the projection system 100 according to this embodiment. As shown in FIG. 5, the container identification unit 11a of the control device 10, as described above, identifies whether the object shown in the image captured by the infrared camera 30 is a filled ice cream container, a filled tea container, an empty container, or some other object. Next, the temperature identification unit 11b of the control device 10, as described above, identifies the temperature range of the contents of each container based on the temperature information from the thermal camera 20. The image control unit 11d of the control device 10 can then identify the type of contents in each container by combining the container identification information identified by the container identification unit 11a and the temperature range identification information identified by the temperature identification unit 11b. Specifically, if an ice cream container contains contents in a temperature range of approximately -10 to 15°C (see FIG. 3), the contents are determined to be ice cream. Similarly, if a tea container contains contents in a temperature range of approximately 0 to 35°C, the contents are determined to be cold tea. Although the temperature ranges of ice cream and cold tea overlap partially, they can be clearly distinguished because they are in different containers. Also, if a teacup contains contents with a temperature range of 35°C or higher, it is determined that the contents are hot tea. Although cold tea and hot tea are placed in the same container, they can be clearly distinguished because they have different temperature ranges. In this way, the image control unit 11d can use the information obtained from the thermal camera 20 and the infrared camera 30 to determine whether the contents of the container are cold ice cream, cold tea, or hot tea.

[0043] Next, the video control unit 11d reads out video content corresponding to the identified type of content from the storage unit 12. In the example shown in FIG. 5, a tea plant texture is associated with cold ice cream, a dedicated flower texture is associated with cold tea, and similarly, a dedicated flower texture is associated with hot tea. Each texture may be a still image or a moving image (animation). These video contents are prepared in advance for each type of content and stored in the storage unit 12. The video control unit 11d reads out video content corresponding to the identified type of content from the storage unit 12, and controls the projector 40 to project the video content onto the position of the object identified by the coordinate identification unit 11c.

[0044] Next, the image control unit 11d identifies the position and movement (coordinate change) of the container containing the content that is the target of the image projection, mainly based on the image captured by the infrared camera 30. At this time, the image control unit 11d may identify the position and movement of the container by referring to the temperature distribution information acquired by the thermal camera 20 in addition to the image captured by the infrared camera 30. The position information of the container is mainly used to control the projection position of the image, and the movement information of the container is mainly used to control the display mode of the video content. Specifically, the image control unit 11d controls the projection position of the image by the projector 40 based on the position information of the container identified by the coordinate identification unit 11c. This allows the image to be accurately projected (projection mapped) onto the container even if the position or orientation of the container changes. The image control unit 11d also changes the display mode of the video content according to the movement of the container. For example, for cold ice cream, when the container is stationary, an image of branches growing inside and around the container is projected, and when the container starts moving, an image of the branches scattering is projected. For both cold and hot tea, when the container is stationary, the image of a flower blooming is projected, and when the container starts moving, the image of petals scattering is projected. In this way, it is possible to create a visual effect that is linked to the movement of the container.

[0045] Furthermore, after projecting an image onto a container, the image control unit 11d terminates the projection of the image onto that container if the contents in the container are consumed or drunk up. Specifically, when the contents are gone, the container is identified as an empty container by the container identification unit 11a (see FIG. 5). Furthermore, when the contents are gone, the temperature of the container also changes, so the temperature identification unit 11b can also determine whether there is any content. For example, if a container contains hot tea, the temperature of the container will gradually decrease as the tea is drunk up, eventually dropping to about room temperature. Similarly, if a container contains ice cream, the temperature of the container will rise as the ice cream is consumed, similarly dropping to about room temperature. In particular, by determining whether there is any content based on both the container state identification result by the container identification unit 11a and the temperature change detection result by the temperature identification unit 11b, the timing to end the image projection can be controlled more accurately.

[0046] The audio control unit 11e controls the sound output from the speaker 50 based on the processing results of the container identification unit 11a, the temperature identification unit 11b, and the coordinate identification unit 11c. Specifically, similar to the video control by the video control unit 11d, the audio control unit 11e reads audio content corresponding to the identified type of content from the storage unit 12 and outputs sound based on the audio content from the speaker 50. For example, background music with a cool atmosphere is associated with cold ice cream, refreshing background music is associated with cold tea, and similarly, background music with a relaxing atmosphere is associated with hot tea. The audio control unit 11e can also output sound effects in response to the movement of the container detected by the coordinate identification unit 11c. For example, for cold tea, a sound effect linked to an image of petals scattering when the container is lifted is output, and for ice cream, a sound effect linked to an image of branches scattering when the container is moved is output. In this way, a wider range of effects can be achieved by combining audio and video effects. Furthermore, the audio control unit 11e can output audio that explains the contents and how to enjoy them. These audio explanations may be output automatically at appropriate times depending on the type of contents, or the output may be controlled manually according to the customer's request. The audio control unit 11e may also automatically adjust the volume of the audio output from the speaker 50 depending on the surrounding environmental sounds.

[0047] Next, Fig. 6 shows an example of the function of automatically adjusting the temperature threshold value of the temperature identification unit 11b. Fig. 6(a) shows the temperature distribution at a certain point in time, with the horizontal axis representing the temperature T and the vertical axis representing the frequency (number of detections). The temperature distribution within the captured range roughly follows a normal distribution, with a peak centered around the average value (average temperature). In this temperature distribution, an object in a temperature range that exceeds a threshold set at a temperature higher than the average temperature by a predetermined value (for example, +30 to +40°C) is identified as "hot tea."

[0048] On the other hand, Figure 6(b) shows a state in which the temperature distribution has shifted overall to a higher temperature due to factors such as an increase in room temperature. For example, if the temperature setting of an air conditioner is increased or the room temperature rises due to an increase in the outside air temperature, the temperatures of all objects within the shooting range will rise, causing the entire temperature distribution to shift to a higher temperature. In this case, the temperature identification unit 11b automatically corrects the threshold for determining "hot tea" according to the amount of change in the average temperature of the temperature distribution. Specifically, by shifting the threshold according to the amount of change in the average temperature, hot tea can be identified based on the same relative temperature difference as before the temperature increase. The amount of shift in the threshold may be a value equal to or proportional to the amount of change in the average temperature, or may be a value calculated using a predetermined function that uses the amount of change in the average temperature as an input value.

[0049] This automatic correction prevents the temperature from being mistakenly identified as "hot tea" even if, for example, the room temperature rises and the measured temperature of the human body becomes higher. This automatic threshold adjustment function also applies to the thresholds for all temperature ranges shown in FIG. 3. Specifically, the thresholds for the temperature ranges of hot tea (over 45°C), cold tea (10-35°C), and ice cream (-10-15°C) are simultaneously adjusted according to the amount of change in the average temperature. By automatically adjusting the temperature threshold in this way according to environmental changes, more stable temperature identification can be achieved. The automatic correction of the temperature threshold may be performed periodically at a preset time interval, or may be performed when the amount of change in the average temperature exceeds a predetermined value.

[0050] In the above description of the present invention, the embodiments of the present invention have been described with reference to the drawings in order to express the contents of the present invention. However, the present invention is not limited to the above embodiments, and includes modifications and improvements that are obvious to those skilled in the art based on the matters described in the present specification. [Explanation of symbols]

[0051] 10...Control device 11...Control unit 11a...Container identification section 11b...Temperature identification section 11c...position detection unit 11d...image control unit 11e...acoustic control unit 12...storage unit 20...Thermal camera 30...Infrared camera 40...Projector 50...Speaker 100...Projection system

Claims

1. A thermal camera that measures the temperature distribution within a shooting range including the target object; an imaging device that images the object; a projection device that projects an image onto the object; a control device; 1. A projection system comprising: the target object is a container containing an object, The control device Identifying the type of the container from the external shape of the object based on the captured image of the imaging device; Identifying the type of the object contained in the container based on the type of the container and the temperature distribution measured by the thermal camera; controlling the image projected by the projection device onto the target object based on the type of the object; A threshold value of a temperature range for identifying the type of the object is automatically adjusted according to the amount of change in the average value of the temperature distribution. Projection system.

2. The imaging device is a night vision camera.

10. The projection system of claim 1.

3. The control device Identifying the coordinates of the object based on the captured image of the imaging device; The projection device controls the image projected onto the object based on the type of the object and the coordinate change information of the object.

10. The projection system of claim 1.

4. The control device changes the image projected by the projection device onto the object depending on whether or not the coordinates of the object have changed.

4. The projection system of claim 3.

5. A step of measuring the temperature distribution within a shooting range including an object using a thermal camera; a step of photographing the object with an imaging device; a step of controlling an image projected onto the object by a control device; and a step of projecting an image onto the object by the projection device based on the control of the control device. A projection method comprising: the target object is a container containing an object, The control device Identifying the type of the container from the external shape of the object based on the captured image of the imaging device; Identifying the type of the object contained in the container based on the type of the container and the temperature distribution measured by the thermal camera; controlling the image projected by the projection device onto the target object based on the type of the object; A threshold value of a temperature range for identifying the type of the object is automatically adjusted according to the amount of change in the average value of the temperature distribution. Projection method.

Images