Aerial suspension image display device

Abstract

Provided is a more suitable aerial floating image display device. Therefore, the method is beneficial to sustainable development objectives of '3 health and welfare of all people', '9 establishment of a foundation for industrial and technical innovation' and '11 construction of a sustainable residence city'. The aerial floating image display device is provided with: an image input I / F; a video processing circuit that processes a video based on an input video input via the I / F; an image display unit that displays the processed image; and a retroreflective plate that reflects the image light emitted from the image display unit and forms an aerial floating image, the image light emitted from the display surface of the image display unit being reflected by the retroreflective plate. The optical path length from the display surface to the position of the aerial floating image by the reflection of the retroreflective plate varies according to the position of the display surface where the image light is emitted, and the image processing circuit processes the image light at a plurality of positions of the image based on the input image corresponding to a plurality of positions where the optical path lengths of the image light differ. And different image sharpening processing is carried out.

Description

Technical Field

[0001] The invention relates to an aerial suspended image display device. Background Art

[0002] Regarding the mid-air floating information display technology, for example, as disclosed in Patent Document 1, it is realized by an imaging method using retroreflection.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application No. 2018-564127 Summary of the invention

[0006] Problem that the invention aims to solve

[0007] However, in the disclosure of Patent Document 1, since the optical path length to the image forming plane is different in each area, the resolution of the image floating in the air varies.

[0008] If the resolution of each area in the image is different, based on the different clarity of the display position, there will sometimes be restrictions on the size of the text in the content, such as having to be made so that it can be visually recognized in the area with the lowest clarity. Sometimes the user will feel strong discomfort, such as the different accuracy of input operations for the floating image in each area.

[0009] The present invention has been made in view of this situation, and an object of the present invention is to provide a more appropriate mid-air floating image display device.

[0010] Solutions for solving problems

[0011] In order to solve the above-mentioned problems, one embodiment of the present invention, for example, can be configured as follows to display an image suspended in the air by an air floating image display device: comprising: an image input interface; an image processing circuit, which processes an image based on an input image input via the image input interface; an image display unit, which displays the image processed by the image processing circuit; and a retroreflective plate, which reflects image light emitted from the image display unit to form an image suspended in the air. Regarding the image light emitted from the display surface of the image display unit, the optical path length from the image light emitted from the display surface of the image display unit and reflected by the retroreflective plate to the position of the image suspended in the air is different depending on the position in the display surface of the image display unit that emits the image light. The image processing circuit performs different image clarity processing at multiple positions of the image based on the input image and corresponding to multiple positions with different optical path lengths of the image light.

[0012] Effects of the Invention

[0013] According to the present invention, a more appropriate mid-air floating image display device can be realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a diagram showing an example of how to use the mid-air floating image display device according to an embodiment of the present invention.

[0015] Figure 2 This is a diagram showing an example of the main structure of the mid-air floating image display device and the structure of the retroreflective portion according to an embodiment of the present invention.

[0016] Figure 3A It is a projection diagram of a retroreflective plate constituting an aerial floating image display device according to an embodiment of the present invention.

[0017] Figure 3B It is a top view of a retroreflective plate constituting an air-floating image display device according to an embodiment of the present invention.

[0018] Figure 4A 1 is a perspective view showing a corner reflector constituting a retroreflective plate constituting an aerial floating image display device according to an embodiment of the present invention.

[0019] Figure 4B 1 is a top view showing a corner reflector constituting a retroreflective plate constituting an aerial floating image display device according to an embodiment of the present invention.

[0020] Figure 4C 1 is a side view showing a corner reflector constituting a retroreflective plate constituting an aerial floating image display device according to an embodiment of the present invention.

[0021] Figure 5 It is a circuit block diagram of a device for displaying suspended images in the air according to an embodiment of the present invention.

[0022] Fig. 6A It is a diagram showing an imaging optical path in a configuration example of an air-floating image display device according to an embodiment of the present invention.

[0023] Figure 6B It is a diagram showing an imaging optical path in a configuration example of an air-floating image display device according to an embodiment of the present invention.

[0024] Fig. 7A FIG. 1 is a diagram showing an example of an output aerial floating image of an aerial floating image display device according to an embodiment of the present invention.

[0025] Figure 7B FIG. 1 is a diagram showing the brightness distribution of an output aerial floating image of the aerial floating image display device according to an embodiment of the present invention.

[0026] Figure 8This is a diagram showing the optical transfer characteristics of the mid-air floating image display device with respect to the spatial frequency according to one embodiment of the present invention.

[0027] Fig. 9 This is a diagram showing an example of an image processing method according to an embodiment of the present invention.

[0028] Fig. 10A FIG. 1 is a diagram showing components of various filters used in the image processing method according to an embodiment of the present invention.

[0029] Fig. 10B FIG. 1 is a diagram showing components of various filters used in the image processing method according to an embodiment of the present invention.

[0030] Fig.11 This is a diagram showing evaluation indices of waveform changes of video signals obtained by an image processing method according to an embodiment of the present invention.

[0031] Fig. 12A This is a diagram showing an example of filter setting in an image processing method according to an embodiment of the present invention.

[0032] Fig. 12B This is a diagram showing an example of filter setting in an image processing method according to an embodiment of the present invention.

[0033] Fig. 12C The figure shows the waveform change of the image signal obtained by various filter parameters used in the image processing method according to one embodiment of the present invention.

[0034] Fig.13A This is a diagram showing an example of filter setting in an image processing method according to an embodiment of the present invention.

[0035] Fig. 13B This is a diagram showing an example of filter setting in an image processing method according to an embodiment of the present invention.

[0036] Fig. 13C The figure shows the waveform change of the image signal obtained by various filter parameters used in the image processing method according to one embodiment of the present invention.

[0037] Fig.14A This is a diagram showing an example of filter setting in an image processing method according to an embodiment of the present invention.

[0038] Fig. 14B The figure shows the waveform change of the image signal obtained by various filter parameters used in the image processing method according to one embodiment of the present invention. DETAILED DESCRIPTION

[0039] Hereinafter, embodiments of the present invention will be described in detail based on the accompanying drawings. It should be noted that the present invention is not limited to the description of the embodiments, and various changes and modifications can be made by those skilled in the art within the scope of the technical ideas disclosed in this specification. In addition, in all the figures used to illustrate the present invention, the same reference numerals are marked on the components having the same functions, and their repeated descriptions are sometimes omitted.

[0040] The following embodiments relate to an image display device that can make an image based on image light from an image light source pass through a transparent component separated by a space such as glass, and display it as an image suspended in the air on the outside of the transparent component. It should be noted that in the description of the following embodiments, the term "image suspended in the air" is used to express the image suspended in the air. This term can also be replaced by "aerial image", "spatial image", "spatial suspended image", "spatial suspended optical image of displayed image", "air-suspended optical image of displayed image", etc. The term "air-suspended image" mainly used in the description of the embodiments is used as a representative example of these terms.

[0041] According to the following embodiments, it is possible to realize an image display device suitable for, for example, bank ATM machines, station ticket machines, digital signage, etc. For example, currently, in bank ATM machines, station ticket machines, etc., touch panels are usually used, and transparent glass surfaces and light-transmitting plates are used, on which high-resolution image information can be displayed in a suspended state in the air. In addition, for example, it is possible to provide an air-suspended image display device for vehicles, which performs a so-called unidirectional air-suspended image display in the vehicle that can be visually recognized inside and / or outside the vehicle. In this image display device, by applying an appropriate image processing method, the clarity of the entire surface of the air-suspended image can be corrected to the same degree. By this image processing method, the accuracy of the input operation of the air-suspended image is improved.

[0042] Figure 1 FIG. 1 is a diagram showing an example of how the air floating image display device according to an embodiment of the present invention is used, and is a diagram showing the overall structure of the air floating image display device according to this embodiment. Figure 2etc., a converging image beam based on retroreflection is emitted from the aerial floating image display device 1000, passes through the transparent member 100 (glass, etc.), and forms an aerial image (aerial floating image 3) as a real image on the outside of the glass surface. In the aerial floating image 3, the response to the input operation 4 can be reflected. It should be noted that in the following embodiments, with the end point 30 of the imaging surface as the origin, the three axes of the right direction relative to the imaging surface as the x-axis, the lower (upper) direction as the y-axis, and the depth direction as the z-axis are set as the coordinate system shared by each structural diagram. Similarly, the three axes of the width direction of the aerial floating image display device 1000 as the a-axis, the depth direction as the b-axis, and the height direction as the c-axis are set as the coordinate system shared by each structural diagram. In the illustrations of the aerial floating image display device and the output aerial floating image, the xyz direction axis and the abc direction axis are sometimes recorded as the coordinate system representing the direction corresponding to each figure.

[0043] Figure 2 This is a diagram showing an example of the main structure of the air-floating image display device and the retro-reflective structure of an embodiment of the present invention. A display device 10 for diffusing specific image light is provided in an oblique direction of a transparent member 100 such as glass. The display device 10 includes a liquid crystal display panel 11 and a light source device 13 for generating light having a unique diffusion characteristic.

[0044] The principal light ray 20 representing the light beam emitted from the display device 10 travels in the y direction and is incident on the retroreflective plate 5 at an incident angle α (e.g., 45°). The retroreflective plate 5 is an optical member having an optical property of retroreflecting light in at least a part of the direction. In addition, since the reflected light has an optical property of forming an image, the retroreflective plate 5 can also be expressed as an imaging optical member or an imaging optical plate. Regarding the specific structure of the retroreflective plate 5, use Figure 3A , Figure 3B As will be described in detail, the principal light 20 is reflected in directions a and b while traveling in the c direction by the retroreflective plate 5. Thus, the reflected light 21 travels in the z direction, passes through the transparent member 100, and forms an aerial image 3 as a real image on the imaging surface.

[0045] The light beams forming the suspended image 3 are a collection of light rays that converge from the retroreflective plate 5 toward the optical image of the suspended image 3. These light rays continue to travel straight after passing through the optical image of the suspended image 3. Therefore, the suspended image 3 is different from a diffuse image formed on a screen by a normal projector, etc., and is an image with high directivity. Figure 2In the structure, when the user visually recognizes from the direction of arrow A, the image 3 floating in the air is visually recognized as a bright image. However, when other people are visually recognized from the direction of arrow B, the image 3 floating in the air cannot be visually recognized as an image at all. This characteristic is suitable for use in a system that displays an image requiring high security or a highly confidential image that is to be kept secret from a person facing the user.

[0046] use Figure 3A , Figure 3B An example of the structure of the retroreflective sheet 5 is described. The retroreflective sheet 5 is configured by arranging a plurality of corner reflectors 40 in an array on the surface of a transparent member 50. Regarding the specific structure of the corner reflectors 40, Figure 4A , Figure 4B , Figure 4C To explain in detail, the light rays 111, 112, 113, and 114 emitted from the light source 110 are reflected twice by the two mirror surfaces 41 and 42 of the corner reflector 40, and become reflected light rays 121, 122, 123, and 124. The two reflections become retroreflections in the same direction as the incident direction (advancing in the direction rotated 180°) with respect to the a and b directions, and become regular reflections in the c direction with the incident angle and the reflection angle being consistent through total reflection. That is, the light rays 111 to 114 generate reflected light rays 121 to 124 on a straight line symmetrical with respect to the corner reflector 40 with respect to the c direction, and form an image of a real image 120 in the air. It should be noted that the light rays 111 to 114 emitted from the light source 110 are four light rays representing the diffused light from the light source 110. According to the diffusion characteristics of the light source 110, the light rays incident on the retroreflective plate 5 are not limited to this, but any incident light rays cause the same reflection and form an image of a real image 120 in the air. It should be noted that for ease of observation of the accompanying drawings, the position of the light source 110 and the position of the real image 120 in the air in direction a are staggered, but in fact the position of the light source 110 and the position of the real image 120 in the air in direction a are the same position and are overlapping when observed from direction c.

[0047] Next, use Figure 4A , Figure 4B , Figure 4C The structure and effect of the corner reflectors 40 constituting the retroreflective plate 5 are described. The corner reflectors 40 are rectangular parallelepipeds whose only two specific surfaces are mirror surfaces 41 and 42 and whose other four surfaces are formed of transparent members. The retroreflective plate 5 has a structure in which the corner reflectors 40 are arranged in an array so that their corresponding mirror surfaces face the same direction.

[0048] When observed from the upper surface (+c direction), the light 111 emitted from the light source 110 is incident on the mirror 41 (or mirror 42) at a specific incident angle, and after total reflection at the reflection point 130, it is totally reflected again at the reflection point 132 on the mirror 42 (or mirror 41). If the incident angle of the light 111 relative to the mirror 41 (or mirror 42) is set to θ, the incident angle of the first reflected light 131 reflected by the mirror 41 (or mirror 42) relative to the mirror 42 (or mirror 41) can be expressed as 90°-θ. Therefore, relative to the light 111, the second reflected light 121 obtains a rotation of 2θ through the first reflection and a rotation of 2×(90°-θ) through the second reflection, thus forming a total inverted optical path of 180°. On the other hand, when observed from the side (the middle direction between -a and -b), total reflection relative to the c direction occurs only once. Therefore, if the incident angle with respect to the mirror surface 41 or the mirror surface 42 is set to Then, relative to light 111, reflected light 121 is obtained by one reflection. rotation.

[0049] According to the above, the light incident on the corner reflector 40 generates reverse reflection in the a and b directions, and becomes regular reflection based on total reflection in the c direction. If the retroreflector 5 is considered, the same reflection is caused in each light path, so the reverse light path with convergence relative to the a and b directions forms an image at a point symmetrical with respect to the c-axis direction. The resolution of the suspended image formed by the light from the image output unit 10 depends to a large extent on the resolution of the liquid crystal display panel 11. Figure 3A , Figure 3B The diameter D and pitch P of the retroreflective part of the retroreflective sheet 5 are shown (not shown). For example, when a 7-inch WUXGA (1920×1200 pixels) liquid crystal display panel is used, even if one pixel (one triplet) is about 80 μm, for example, if the diameter D of the retroreflective part is 240 μm and the pitch P is 300 μm, one pixel of the suspended image is equivalent to 300 μm. Therefore, the effective resolution of the suspended image is reduced to about 1 / 3.

[0050] Therefore, in order to make the resolution of the airborne image equal to the resolution of the display device 10, it is desirable to make the diameter D and the spacing P of the retroreflective portion close to one pixel of the liquid crystal display panel. On the other hand, in order to suppress the generation of moiré fringes caused by the pixels of the retroreflective plate and the liquid crystal display panel, it is preferred to design the respective spacing ratios to be staggered by an integral multiple of one pixel. In addition, it is preferred that the shape is configured so that any side of the retroreflective portion does not overlap with any side of one pixel of the liquid crystal display panel.

[0051] It should be noted that the shape of the retroreflective plate (imaging optical plate) of this embodiment is not limited to the above examples. It can have various shapes that achieve retroreflection. Specifically, it can be various cubic corners, or it can be a shape in which a combination of the reflecting surfaces of the slit mirror array is periodically arranged. Alternatively, a capsule lens type retroreflective element with periodically arranged glass beads can be provided on the surface of the retroreflective plate of this embodiment. The detailed structure of these retroreflective elements can use existing technology, so the detailed description is omitted. Specifically, the technology disclosed in Japanese Patent Gazette No. 2017-33005, Japanese Patent Gazette No. 2019-133110, etc. can be used.

[0052] Next, a block diagram of the internal structure of the mid-air floating image display device 1000 will be described. Figure 5 1 is a block diagram showing an example of the internal structure of the mid-air floating image display device 1000 .

[0053] The aerial floating image display device 1000 includes a retro-reflective unit 1101, an image display unit 1102, a light guide 1104, a light source 1105, a power supply 1106, an external power input interface 1111, an operation input unit 1107, a non-volatile memory 1108, a memory 1109, a control unit 1110, an image signal input unit 1131, an audio signal input unit 1133, a communication unit 1132, an aerial operation detection sensor 1351, an aerial operation detection unit 1350, an audio output unit 1140, an image control unit 1160, a storage unit 1170, an imaging unit 1180, etc. It should be noted that a removable medium interface 1134, a posture sensor 1113, a transmissive self-luminous image display device 1650, a second display device 1680, or a secondary battery 1112, etc. may also be provided.

[0054] The components of the aerial floating image display device 1000 are disposed in a housing 1190 . It should be noted that the camera unit 1180 and the aerial operation detection sensor 1351 may also be disposed outside the housing 1190 .

[0055] Figure 5 The retro-reflective portion 1101 and Figure 2 The retroreflective portion 1101 retroreflects the light modulated by the image display portion 1102. The light output to the outside of the air-floating image display device 1000 among the reflected light from the retroreflective portion 1101 forms an air-floating image 3.

[0056] Figure 5 The image display unit 1102 corresponds to Figure 2 The liquid crystal display panel 11. Figure 5 The light source 1105 and Figure 5 The light guide 1104 becomes Figure 2 The corresponding relationship contained in the light source device 13.

[0057] The image display unit 1102 is a display unit that modulates transmitted light to generate an image based on an image signal input under the control of the image control unit 1160 described later. The image display unit 1102 corresponds to Figure 2 The image display unit 1102 may be a transmissive liquid crystal panel. Alternatively, the image display unit 1102 may be a reflective liquid crystal panel that modulates reflected light, a DMD (Digital Micromirror Device: a registered trademark) panel, or the like.

[0058] The light source 1105 generates light for the image display unit 1102, and is a solid light source such as an LED light source or a laser light source. The power supply 1106 converts AC current input from the outside through the external power input interface 1111 into DC current, and supplies power to the light source 1105. In addition, the power supply 1106 supplies the required DC current to each part in the air-suspended image display device 1000. The secondary battery 1112 stores the power supplied from the power supply 1106. In addition, the secondary battery 1112 supplies power to the light source 1105 and other structures that require power without supplying power from the outside through the external power input interface 1111. That is, when the air-suspended image display device 1000 is equipped with the secondary battery 1112, the user can use the air-suspended image display device 1000 even when power is not supplied from the outside.

[0059] The light guide 1104 guides the light generated by the light source 1105 to illuminate the image display unit 1102. The combination of the light guide 1104 and the light source 1105 may also be referred to as a backlight source of the image display unit 1102. The light guide 1104 may also be configured to be mainly made of glass. The light guide 1104 may also be configured to be mainly made of plastic. The light guide 1104 may also be configured to be a structure using a reflector. Various methods may be considered for the combination of the light guide 1104 and the light source 1105. A specific structural example of the combination of the light guide 1104 and the light source 1105 will be described in detail later.

[0060] The air operation detection sensor 1351 is a sensor for detecting the operation of the user's finger on the air floating image 3. The air operation detection sensor 1351 senses, for example, a range overlapping with the entire display range of the air floating image 3. It should be noted that the air operation detection sensor 1351 may also sense only a range overlapping with at least a portion of the display range of the air floating image 3.

[0061] As specific examples of the aerial operation detection sensor 1351, distance sensors using non-visible light such as infrared rays, non-visible light lasers, ultrasonic waves, etc. can be cited. In addition, the aerial operation detection sensor 1351 can also be configured to combine multiple sensors and detect coordinates of a two-dimensional plane. In addition, the aerial operation detection sensor 1351 can also be composed of a ToF (Time of Flight) method LiDAR (Light Detection and Ranging) or an image sensor.

[0062] The in-air operation detection sensor 1351 may be capable of detecting a touch operation performed by a user's finger on an object displayed as the in-air floating image 3. Such sensing can be performed using existing technology.

[0063] The air operation detection unit 1350 obtains a sensing signal from the air operation detection sensor 1351, and detects whether the user's finger contacts the object of the air floating image 3 based on the sensing signal, calculates the position (contact position) where the user's finger contacts the object, etc. The air operation detection unit 1350 is composed of, for example, a circuit such as an FPGA (Field Programmable Gate Array). In addition, some functions of the air operation detection unit 1350 can also be implemented by software, for example, through an air operation detection program executed by the control unit 1110.

[0064] The aerial operation detection sensor 1351 and the aerial operation detection unit 1350 may be configured to be built into the aerial floating image display device 1000, or may be disposed outside the aerial floating image display device 1000. When disposed outside the aerial floating image display device 1000, the aerial operation detection sensor 1351 and the aerial operation detection unit 1350 are configured to transmit information and signals to the aerial floating image display device 1000 via a wired or wireless communication connection path and an image signal transmission path.

[0065] In addition, the aerial operation detection sensor 1351 and the aerial operation detection unit 1350 may also be provided separately from the aerial floating image display device 1000. Thus, it is possible to construct a system in which the aerial floating image display device 1000 without the aerial operation detection function is used as the main body, and only the aerial operation detection function can be selectively added. In addition, it is also possible to have a structure in which only the aerial operation detection sensor 1351 is provided as a separate body, and the aerial operation detection unit 1350 is built into the aerial floating image display device 1000. In the case where it is desired to configure the aerial operation detection sensor 1351 more freely relative to the setting position of the aerial floating image display device 1000, the structure in which only the aerial operation detection sensor 1351 is provided as a separate body has an advantage.

[0066] The camera unit 1180 is a camera with an image sensor, which captures the space near the air-floating image 3 and / or the user's face, arms, fingers, etc. A plurality of camera units 1180 may be provided. In addition, the camera unit 1180 may also be provided with a depth sensor. By using a plurality of camera units 1180, or by using a camera unit 1180 with a depth sensor, the air operation detection unit 1350 can be assisted in the detection and processing of the user's touch operation on the air-floating image 3. The camera unit 1180 may also be provided separately from the air-floating image display device 1000. In the case where the camera unit 1180 is provided separately from the air-floating image display device 1000, it is sufficient to be configured so as to be able to transmit the camera signal to the air-floating image display device 1000 via a wired or wireless communication connection path, etc.

[0067] For example, in the case where the aerial operation detection sensor 1351 is configured to detect whether there is an object intrusion into the intrusion detection plane by taking the plane (intrusion detection plane) that includes the display surface of the aerial floating image 3 as the object, it is sometimes impossible for the aerial operation detection sensor 1351 to detect information such as how far an object that has not intruded into the intrusion detection plane (for example, a user's finger) is from the intrusion detection plane, or how close the object is to the intrusion detection plane.

[0068] In this case, the distance between the object and the intrusion detection plane can be calculated by using information such as the depth calculation information of the object based on the camera images of the plurality of camera units 1180 and the depth information of the object based on the depth sensor. Then, various information such as the distance between the object and the intrusion detection plane is used for various display controls of the aerial floating image 3.

[0069] Alternatively, instead of using the mid-air operation detection sensor 1351 , the mid-air operation detection unit 1350 may detect a touch operation performed by the user on the mid-air floating image 3 based on the captured image of the imaging unit 1180 .

[0070] Alternatively, the camera unit 1180 may capture the face of the user operating the air floating image 3, and the control unit 1110 may perform user recognition processing. Alternatively, in order to determine whether others are standing around or behind the user operating the air floating image 3, or whether others are peeping at the user's operation of the air floating image 3, the camera unit 1180 may capture a range including the user operating the air floating image 3 and the user's surrounding area.

[0071] The operation input unit 1107 is, for example, an operation button, a signal receiving unit such as a remote controller, or an infrared light receiving unit, and inputs a signal related to an operation different from the user's mid-air operation (touch operation). Unlike the aforementioned user who performs touch operation on the mid-air floating image 3, the operation input unit 1107 can also be used for, for example, a manager to operate the mid-air floating image display device 1000.

[0072] The video signal input unit 1131 is connected to an external video output device and inputs video data. Various digital video input interfaces can be considered for the video signal input unit 1131. For example, the video signal input unit 1131 can be composed of a video input interface of the HDMI (registered trademark) (High-Definition Multimedia Interface) standard, a video input interface of the DVI (Digital Visual Interface) standard, or a video input interface of the DisplayPort standard. Alternatively, an analog video input interface such as analog RGB or composite video can also be provided. The audio signal input unit 1133 is connected to an external audio output device and inputs audio data. The audio signal input unit 1133 can be composed of, for example, an audio input interface of the HDMI standard, an optical digital terminal interface, or a coaxial digital terminal interface. In the case of an interface of the HDMI standard, the video signal input unit 1131 and the audio signal input unit 1133 can also be composed of an interface in which a terminal and a cable are integrated. The audio output unit 1140 can output audio based on audio data input to the audio signal input unit 1133. The audio output unit 1140 can also be composed of a speaker. The sound output unit 1140 may also output built-in operation sounds and error warning sounds. Alternatively, the sound output unit 1140 may output digital signals to external devices, such as the Audio Return Channel function specified in the HDMI standard.

[0073] The nonvolatile memory 1108 stores various data used in the air floating image display device 1000. The data stored in the nonvolatile memory 1108 includes, for example, data for various operations displayed on the air floating image 3, display icons, data for objects for user operations, layout information, etc. The memory 1109 stores image data displayed as the air floating image 3, data for controlling the device, etc.

[0074] The control unit 1110 controls the operation of each connected unit. In addition, the control unit 1110 may cooperate with the program stored in the memory 1109 to perform calculation processing based on information obtained from each unit in the air-floating image display device 1000.

[0075] In addition, the removable medium interface 1134 is an interface for connecting a removable recording medium (removable medium). The removable recording medium may also be composed of a semiconductor device memory such as a solid state drive (SSD), a magnetic recording medium recording device such as a hard disk drive (HDD), or an optical recording medium such as an optical disc. The removable medium interface 1134 can read various information such as image data, image data, sound data, etc. recorded in the removable recording medium. The image data, image data, etc. recorded in the removable recording medium are output as an air-suspended image 3 via the image display unit 1102 and the retro-reflection unit 1101.

[0076] The storage unit 1170 is a storage device for recording various information such as various data such as image data, image data, and sound data. The storage unit 1170 may also be composed of a magnetic recording medium recording device such as a hard disk drive (HDD) or a semiconductor device memory such as a solid state drive (SSD). In the storage unit 1170, for example, various information such as various data such as image data, image data, and sound data may be pre-recorded when the product is shipped. In addition, the storage unit 1170 may also record various information such as various data such as image data, image data, and sound data obtained from an external device, an external server, etc. via the communication unit 1132.

[0077] The video data and image data recorded in the storage unit 1170 are output as the floating image 3 via the image display unit 1102 and the retro-reflection unit 1101. The video data and image data of display icons and objects for user operation displayed as the floating image 3 are also recorded in the storage unit 1170.

[0078] The storage unit 1170 also stores display icons and layout information of objects displayed as the floating image 3, and various metadata information related to the objects. The audio data stored in the storage unit 1170 is outputted as audio from the audio output unit 1140, for example.

[0079] The image control unit 1160 performs various controls related to the image signal input to the image display unit 1102. The image control unit 1160 may also be referred to as an image processing circuit, and may be composed of hardware such as an ASIC, an FPGA, or an image processor. It should be noted that the image control unit 1160 may also be referred to as an image processing unit or an image processing unit. The image control unit 1160 performs, for example, image switching control such as which image signal is input to the image display unit 1102, such as an image signal stored in the memory 1109 or an image signal (image data) input to the image signal input unit 1131.

[0080] In addition, the image control unit 1160 can also perform the following control: generate a superimposed image signal by superimposing the image signal stored in the memory 1109 and the image signal input from the image signal input unit 1131, and input the superimposed image signal to the image display unit 1102, thereby forming a composite image as an aerial floating image 3.

[0081] In addition, the image control unit 1160 may also control image processing of the image signal input from the image signal input unit 1131, the image signal stored in the memory 1109, and the like. Examples of image processing include a scaling process for enlarging, reducing, or deforming an image, a brightness adjustment process for changing the brightness, a contrast adjustment process for changing the contrast curve of an image, and a Retinex process for decomposing an image into light components and changing the weight of each component.

[0082] In addition, the image control unit 1160 may also perform special effect image processing for assisting the user's mid-air operation (touch operation) on the image signal input to the image display unit 1102. The special effect image processing is performed based on, for example, the detection result of the user's touch operation by the mid-air operation detection unit 1350 and the user's camera image obtained by the camera unit 1180.

[0083] The posture sensor 1113 is a sensor composed of a gravity sensor or an acceleration sensor, or a combination thereof, and is capable of detecting the posture in which the air-floating image display device 1000 is set. The control unit 1110 may also control the actions of the connected units based on the posture detection result of the posture sensor 1113. For example, when an unpreferable posture as a user's usage state is detected, the display of the image displayed by the image display unit 1102 may be terminated and an error message may be displayed to the user. Alternatively, when the posture sensor 1113 detects that the setting posture of the air-floating image display device 1000 has changed, the direction of the image displayed by the image display unit 1102 may be rotated.

[0084] As described above, various functions are installed in the air floating image display device 1000. However, the air floating image display device 1000 does not need to have all of these functions, and may have any structure as long as it has a function of forming the air floating image 3.

[0085] Next, the relationship between the light-emitting point position and the imaging optical distance in the display device 10 will be described. Fig. 6A 1 is a diagram showing an imaging optical path in a configuration example of the mid-air floating image display device 1000 .

[0086] Of the two light-emitting points separated by an interval d on the display device 10, the one with the larger z coordinate is set as the light-emitting point 140, and the one with the smaller z coordinate is set as the light-emitting point 150. The light rays emitted from the light-emitting points 140 and 150 have the same diffusion characteristics, so as light beams corresponding to the secondary rays 142 and 143 shifted by a certain diffusion angle ±δ with respect to the main ray 141 from the light-emitting point 140, the secondary rays 152 and 153 shifted by the diffusion angle ±δ with respect to the main ray 151 from the light-emitting point 150 can be defined.

[0087] The main light beams 141 and 151 are emitted in the direction of the incident angle α with respect to the retroreflective plate 5. The difference between the interval between the points where the secondary light beams 142 and 143 are incident on the retroreflective plate 5 and the interval between the points where the secondary light beams 152 and 153 are incident on the retroreflective plate 5 can be expressed as:

[0088] [Mathematical formula 1]

[0089]

[0090] After being incident on the retroreflective plate 5, the main light rays 141, 151 and the secondary light rays 142, 143, 152, 153 are reflected respectively to become the main light rays 161, 171 and the secondary light rays 162, 163, 172, 173. The main light rays 161, the secondary light rays 162, 163 become converging light rays, and form an optical real image in the air at the imaging point 160. Similarly, the main light rays 171, the secondary light rays 172, 173 become converging light rays, and form an optical real image in the air at the imaging point 170. These imaging points gather to form an image 3 suspended in the air. Here, the intervals between the main light rays 171, the secondary light rays 172, and the secondary light rays 173 incident on the imaging point 170 are larger than the intervals between the main light rays 161, the secondary light rays 162, and the secondary light rays 163 incident on the imaging point 160, so the imaging point 170 is more susceptible to aberration than the imaging point 160. This means that when observed from the direction of arrow A, the resolution performance of imaging point 160 is higher than that of imaging point 170 .

[0091] It should be noted that, in the optical system of the air-suspended image display device 1000 of this embodiment, the resolution performance of each imaging point is different according to the optical path length of the main light of the imaging optical path from the light-emitting point of the display device 10 to the imaging point forming the air-suspended image 3. Figure 6B In addition to the imaging point 160 and the imaging point 170, the imaging point 165 is also shown. The imaging point 165 is the middle point thereof and the position of the center of the screen of the air-floating image 3. In addition, in addition to the light-emitting point 140 and the light-emitting point 150 of the display device 10, the light-emitting point 145 is also shown. The light beam reaching the imaging point 165 is emitted from the light-emitting point 145. Thus, as Figure 6B As shown, the optical path length of the main light emitted from the light-emitting point 140 and reaching the imaging point 160 is LB1+LB2. The optical path length of the main light emitted from the light-emitting point 145 and reaching the imaging point 165 is LM1+LM2. The optical path length of the main light emitted from the light-emitting point 150 and reaching the imaging point 170 is LH1+LH2. Figure 6B As can be seen, LB1+LB2<LM1+LM2<LH1+LH2. Therefore, the optical path length of the principal ray emitted from the light-emitting point 140 and reaching the imaging point 160 is the shortest among these three points, the optical path length of the principal ray emitted from the light-emitting point 145 and reaching the imaging point 165 is the second shortest, and the optical path length of the principal ray emitted from the light-emitting point 150 and reaching the imaging point 170 is the longest among these three points. Therefore, on the aerial suspended image 3, it can be said that the resolution performance is from high to low in the order of imaging point 160, imaging point 165, and imaging point 170. In other words, it can be said that the resolution performance at imaging point 165 is lower than that at imaging point 160, and the resolution performance at imaging point 170 is lower than that at imaging point 165. The relationship between the position of the light-emitting point, the position of the imaging point, the optical path length of the principal ray, and the resolution performance in the optical system of the aerial suspended image display device 1000 of this embodiment is as described above. In the optical system of the air-suspended image display device 1000 of this embodiment, no matter where the light-emitting point of the display device 10 is or where the imaging point of the air-suspended image 3 is, the same relationship between the optical path length of the principal ray and the resolution holds true.

[0092] As explained above, when using Figure 2 In the configuration example of the retroreflective sheet 5 described in the figure, when viewed from the direction of arrow A, the resolution performance of the floating image 3 outputted by the floating image display device 1000 is reduced in the region where the y coordinate is large. The image processing method of one embodiment of the present invention aims to correct the image non-uniformity caused by the difference in resolution performance corresponding to the y coordinate in the floating image 3.

[0093] The appearance of the aerial image 3 when actually viewed from the direction of arrow A will be described. Fig. 7A , it is shown that the resolution performance is poor when outputting a specific pattern in each area of ​​the aerial floating image 3. Figure 7B In the Fig. 7A The brightness change of the display pattern in the x direction.

[0094] The display device 10 outputs the image 3 in the air in the direction of arrow A so that circular patterns 211, 212, and 213 of the same radius are displayed in three regions corresponding to the values ​​of the y coordinates. Figure 7B In the figure, the distribution of brightness changes along the straight lines 201 (H1-H'1), 202 (H2-H'2), and 203 (H3-H'3) passing through the centers of the circular patterns 211, 212, and 213 and parallel to the x-axis are respectively represented by curves 231, 232, and 233. In addition, in the following description, when the air floating image 3 is observed from the direction of arrow A, the regions 221, 222, and 223 divided into three according to the value of the y coordinate are represented as the upper part, the center, and the lower part, respectively.

[0095] As in Fig. 6A , Figure 6B As described in FIG. 1 , in the air floating image 3, the imaging property of the optical real image displayed in the center is lower than that in the lower part, and the imaging property of the optical real image displayed in the upper part is lower than that in the center. That is, the visibility (resolution, clarity) of the circular pattern 212 is lower than that of the circular pattern 213, and the visibility (resolution, clarity) of the circular pattern 211 is lower than that of the circular pattern 212. This influence is Fig. 7A The visual information observed as blur of the optical real image in Figure 7B This is qualitative information observed as a downward spread of the brightness gradient when viewed in the x-direction.

[0096] exist Figure 7B In FIG. 1 , how the definition of the image 3 floating in the air changes according to the optical distance is represented as an example by MTF (Modulation Transfer Function) as a response corresponding to the spatial frequency.

[0097] As a method for measuring MTF as a sharpness evaluation index of the aerial floating image 3, there is, for example, a rectangular wave pattern method for evaluating the transmittance of a rectangular wave pattern (rectangles filled with white and black are displayed at regular intervals). Figure 2The amplitude of the periodic measured brightness change observed in the direction of arrow A (the difference between the maximum measured brightness and the minimum measured brightness) divided by the amplitude of the periodic brightness change based on the input rectangular wave (the difference between the maximum input brightness and the minimum input brightness) is used as the MTF. The method for measuring MTF is not limited to the above-mentioned rectangular wave graph method. In addition, there are edge methods using Fourier transform, etc. The MTF response representing clarity is consistent regardless of the measurement method. In the following, the measured value of MTF is expressed as MTF, MTF response, response, etc., but its measurement method is not limited to the above-mentioned rectangular wave graph method. At this time, the interval in the actual space of the displayed rectangular wave pattern is defined as spatial frequency. Spatial frequency refers to an indicator of the resolution performance of the displayed pattern given in units such as pl / mm. It should be noted that as units of spatial frequency, LP / mm, Cycles / mm, roots / mm, lines / mm, etc. are used, but all represent the same indicator.

[0098] exist Figure 8 In the figure, reference numerals 241, 242, and 243 are examples of the sharpness characteristics measured at the upper part, the center, and the lower part of the air floating image 3 displayed by the air floating image display device 1000, respectively. In this embodiment, a curve representing the MTF characteristics is shown. Figure 7B As explained in , in the upper part where the imaging performance is reduced, the maximum brightness is reduced, so the MTF response is also reduced. Therefore, in all spatial frequency domains, it can be said that the MTF response in the center is higher than that in the upper part, and the MTF response in the lower part is higher than that in the center.

[0099] In addition, as described above, the pixel pitch of the air-suspended image 3 is reduced to about 1 / 3 of the pixel pitch of the display device 10. Since the pixel pitch corresponds to the spatial frequency, the greater the response at a higher spatial frequency, the higher the resolution performance and clarity. For example, the response characteristics of the display device 10 in the spatial frequency domain 250 are reflected as the response characteristics of the air-suspended image display device 1000 in the spatial frequency domain 251. That is, the high-resolution pattern display that can be displayed on the display device 10 and conforms to the spatial frequency domain 251 is not suitable for the air-suspended image 3.

[0100] The display pattern suitable for the aerial floating image 3 and the correction method thereof are described.

[0101] In the air floating image 3, it is desirable to create a display image within the range of the spatial frequency domain 250 that can ensure a sufficient response. In addition, regarding the response of the air floating image 3, when observing a specific spatial frequency 252, the center is higher than the upper part, and the lower part is higher than the center. In the display device 10, when outputting a display image represented by the spatial frequency 252, response differences 253 and 254 are respectively generated between the lower part and the center, and between the lower part and the upper part. In order to make the clarity of the entire display area of ​​the air floating image 3 uniform, the input image signal is corrected, such as correcting the response difference 253 in the center and the response difference 254 in the upper part. That is, in the air floating image 3, by specifying the correction amount of the correction process of the input image signal according to the MTF response of each area, the clarity of the entire surface can be corrected to the same level, and the air floating image display device 1000 can display an appropriate image. The correction process of the input image signal can be performed by an image processing circuit such as the image control unit 1160.

[0102] The image processing method using filters is described. Fig. 9 In FIG. 1 , a method of performing convolution processing using a filter is represented using a portion 300 of pixels of the liquid crystal display panel 11 , and for simplicity, convolution processing using a 3×3 filter is described.

[0103] In the liquid crystal display panel 11, light corresponding to the input value of each pixel is output, and the light is combined in the entire display area to display an image. Convolution processing in image processing refers to a calculation method for obtaining an output of a component of the surrounding pixel value according to the component of the applied filter. For example, as an applied filter, Fig. 9 In that case, we choose to represent the component of row i and column j as coefficient k ij 3×3 filter 301. In addition, the pixels in the extracted area are counted from the bottom left as A, B, C rows..., 1, 2, 3 columns..., and their pixel values ​​are expressed as a1, a2,..., b1,..., c1,... Fig. 9 , an image of the convolution operation when the filter 301 is applied to the pixel D3 (indicated by reference numeral 302) at the Dth row and the third column is shown. As the actual calculation performed, when the pixel value of the newly obtained pixel D3 is set to d'3, it can be expressed as:

[0104] [Mathematical formula 2]

[0105] d′3=k 11 ×c2+k 12 ×c3+k 13 ×c4+k 21 ×d2+k 22 ×d3+k 23 ×d4+k31 ×e2+k 32 ×e3+k 33 ×e4 …(2)

[0106] For example, as shown by arrow 303, by applying this calculation to each pixel in order from the lower left, it is possible to apply a filtering effect to the pixel values ​​in the entire plane, and output an image with the effect obtained by image processing added. The type of filter and its effect depend on the coefficient k. ij The filter and pixel value are examples, and the calculation is performed in the same way below.

[0107] The effects, production methods, and ideas of each filter are explained. Fig. 10A As an example of a filter used for image sharpening processing, a calculation formula for a 3×3 size moving average mask is shown. Similarly, Fig. 10B As an example of a filter used for the image sharpening process, a calculation formula for a Gaussian mask of 3×3 size is shown. Here, the image sharpening process also includes concepts such as edge emphasis processing for emphasizing edges in an image.

[0108] exist Figure 8 In the image processing, the required image processing is a sharpening filter that corrects the sharpness of the upper and central parts to match the lower part. Moving average filters and Gaussian filters work in the direction of removing noise and reducing the rate of change of pixel values ​​between pixels. The sharpening filter is obtained by adding a weight k to the difference between the blurred image obtained by the moving average filter, Gaussian filter, etc. and the original image and adding it to the original image. Using the difference between the blurred image obtained by the filtering process and the original image to process it into an image with emphasized contours is called unsharp masking. Similarly, the sharpening filter obtained by the difference with the moving average filter is called a moving average mask, and the sharpening filter obtained by the difference with the Gaussian filter is called a Gaussian mask.

[0109] For simplicity, Fig. 10A, the formula and effect of the moving average mask using a 3×3 filter are explained. Since all the elements weighting the pixel values ​​around the center are 0, the pass filter 311 is a filter that does not apply changes to the original image. On the other hand, the moving average filter uniformly adds the weights of each element, thereby suppressing the amount of change when observing the surrounding pixels in the application pixel. In other words, it is a filter that can output an image with blurred outlines. The result of applying the pass filter 311 is the input image itself, so the pixel value change when the rectangular wave signal is input is shown in the pixel value distribution 321. In addition, the pixel value change when the moving average filter 312 is used, the pixel value change caused by the difference between the pass filter 311 and the moving average filter 312, and the pixel value change obtained as the output of the moving average mask 314 as a result of the calculation are respectively shown in the pixel value distributions 322, 323, and 324. Observing the figure, it can be seen that the change gradient of the pixel value distribution 322 is suppressed by the moving average filter 312. Therefore, the pixel value distribution 323 obtained when the difference between the straight-through filter 311 and the moving average filter 312 is applied generates a pulse before and after the pixel value changes. The filter that gives the weight k to the difference and takes the sum with the straight-through filter 311 is the moving average mask 314. As a signal that emphasizes the contour of the image signal of the pulse shape, the result of superimposing the pixel value distribution 323 on the pixel value distribution 321 of the original image obtained by the straight-through filter 311 is the effect of the moving average mask 314. By observing the pixel value distribution 324, the amount of change and the gradient of the pixel value can be increased by using the moving average mask 314.

[0110] For simplicity, Fig. 10B In the example, the formula and effect of Gaussian masking using a 3×3 filter are explained. Fig. 10AAs described in , the straight-through filter 311 is a filter that directly outputs the pixel value of the original image. On the other hand, the Gaussian filter adds the weights of each element according to the Gaussian distribution, thereby suppressing the amount of change when observed from the surrounding pixels in the application pixel. In other words, it is a filter that can output an image with blurred contours. The result of applying the straight-through filter 311 is the input image itself, so the pixel value change when the rectangular wave signal is input is shown in the pixel value distribution 321. In addition, the pixel value change when the Gaussian filter 315 is used, the pixel value change based on the difference between the straight-through filter 311 and the Gaussian filter 315, and the pixel value change obtained as the output of the Gaussian mask 317 obtained as a result of the calculation are respectively shown in the pixel value distributions 325, 326, and 327. It can be seen from the figure that the change gradient of the pixel value distribution 325 is suppressed by using the Gaussian filter 315. Therefore, the pixel value distribution 326 obtained when the difference between the straight-through filter 311 and the Gaussian filter 315 is applied generates pulses before and after the pixel value changes. The filter that gives the difference a weight of k and takes the sum with the straight-through filter 311 is the Gaussian mask 317. As a signal that emphasizes the contour of the pulse-shaped image signal, the result of superimposing the pixel value distribution 326 on the pixel value distribution 321 of the original image obtained by the straight-through filter 311 is the effect of the moving average mask. Observing the pixel value distribution 327, the change amount and gradient of the pixel value can be increased by using the Gaussian mask 317.

[0111] Next, use Fig.11 , which explains how the appearance of the output image changes through image processing using sharpening filters such as moving average masking and Gaussian masking. Fig.11 This shows the change in the input signal obtained by passing through the sharpening filter and the distribution when the output is observed as brightness.

[0112] Consider from Fig. 10A , Fig. 10B The brightness distribution 331 obtained by the same input signal as the pixel value distribution 321 of the rectangular wave signal in . As described above, the input signal after the sharpening filter including the moving average masking and the Gaussian masking is implemented, and the brightness change is emphasized as shown in the brightness distribution 332. The MTF response is expressed by the ratio of the amplitude 341 of the input signal to the amplitude 342 of the output signal, so this value does not change before and after the image processing. However, in appearance, the clarity is felt to be improved by the amplitude 343 including the edge part emphasized by the image processing. Different from the original definition, in the following description, the clarity after the image processing is used as an indicator of the improvement in clarity obtained by the image sharpening processing, and the MTF response in the clarity is set to the ratio of the amplitude 341 of the input signal to the amplitude 343 including the edge part.

[0113] In the following, according to the weighting coefficients and filter sizes determined in the upper and lower parts, Fig. 7A The following three examples are provided for explaining the method of changing the applied filter according to the y-coordinate value of each pixel in the suspended image 3. In order to make the clarity correction amount change continuously corresponding to the y-coordinate value, the following three methods are proposed: (1) a method of changing only the weighting coefficient (weighting coefficient gradient); (2) a method of changing only the filter size (filter size gradient); (3) a method of changing the type of applied filter (matrix element gradient). These three methods are examples, and the method is not limited to these as long as it is a method of changing the image clarity processing parameters (filter size, weighting coefficient, and each matrix element) in the filter in the y-coordinate direction. Fig. 12A , Fig. 12B , Fig. 12C In the method (1), Fig.13A , Fig. 13B , Fig. 13C In the method (2), Fig.14A , Fig. 14B Method (3) is described in the following.

[0114] In the image processing of the present invention, image sharpening processing for improving clarity is performed on a plurality of pixels included in an arbitrary y-coordinate region of the suspended image 3. The maximum value in the y-coordinate range to which the image sharpening processing is applied is set to y1, and the minimum value is set to y2. In order to apply the image processing of the present invention to the entire region within the suspended image 3, it is necessary to set y1 to the upper end of the suspended image 3 and y2 to the lower end of the suspended image 3, but the application range is not limited to this. In the case where y1 is set outside the upper end of the suspended image 3 and y2 is set outside the lower end of the suspended image 3, it is desirable to apply the image sharpening processing parameters of y1 within the range of y>y1 and to apply the image sharpening processing parameters of y2 within the range of y>y2, but the present invention is not limited to this.

[0115] The purpose of the image processing of the present invention is to change the correction amount in accordance with the gradient of the change in clarity from the upper part to the lower part of the floating image 3 obtained as the characteristic of the floating image display device 1000, thereby correcting the clarity of the entire surface of the floating image 3 to the same level. An arbitrary sharpening filter is applied to y=y1 and y=y2 on the floating image 3, and the response after correction is measured at y=y1 and y=y2 on the floating image 3 by the aforementioned MTF measurement method. The sharpening parameters of the filters applied respectively are determined in such a way that the responses after correction of y=y1 and y=y2 on the floating image 3 are the same level. It should be noted that for simplicity, Fig. 12A , Fig. 12B , Fig.13A , Fig. 13B , Fig.14AThe filter inside is a sharpening filter based on unsharp masking with a size of 3×3 or 5×5, but a sharpening filter with a size different from this is also effective as a setting filter applied in the present invention.

[0116] use Fig. 12A , Fig. 12B , Fig. 12C , an example of filter setting, image processing method and its effect that match the clarity characteristics 241, 242, 243 of the air-floating image 3 is described. Fig. 12A , Fig. 12B The following shows a method of changing the weighting coefficient of each element in the y direction in order to change the amount of enhancement of the sharpness of each area of ​​the flying image 3 . Fig. 12A This is an example of how the weighting coefficient changes when moving average masking is used. Fig. 12B This is an example of how the weighting coefficient changes when Gaussian masking is used. Fig. 12C This is an example of a rectangular wave output when the weighting coefficient in the moving average mask is changed.

[0117] In the filters 402 and 412 applied at y=y2, if an arbitrary filter size and weighting coefficient are determined, then at y=y1, where the sharpness correction amount improved by the image sharpening process can be measured, by varying the weighting coefficient while measuring the sharpness correction amount, a weighting coefficient having the same correction amount as y=y2 can be obtained, and the filter applied at this time is set to the filters 401 and 411 applied at y=y2. The weighting coefficient set in the filters 401 and 411 is set to k1, and the weighting coefficient set in the filters 402 and 412 is set to k2. In the application range y2<y<y1 of the image sharpening process, the rate of change of the weighting coefficient at an arbitrary coordinate y can be obtained by dividing the difference between k1 and k2 by the number of pixels y1-y2 in the application range as follows:

[0118] [Mathematical formula 3]

[0119]

[0120] Therefore, the weighting coefficient of the filter actually applied at an arbitrary coordinate y is the coefficient obtained by multiplying the change rate by the number of pixels y-y2 from the lower end of the application range, and can be set as:

[0121] [Formula 4]

[0122]

[0123] The image sharpening process based on this parameter can continuously change the weighting coefficient k1 of y=y1 to the weighting coefficient k2 of y=y2 in the application range y2<y<y1. The above-mentioned image sharpening process can correct the sharpness of the entire surface of the air-floating image 3 to the same degree in accordance with the gradient of the sharpness change from the upper part to the lower part of the air-floating image 3 obtained as the characteristic of the air-floating image display device 1000.

[0124] use Fig.13A , Fig. 13B , Fig. 13C , an example of a filter setting and image processing method that matches the clarity characteristics 241, 242, and 243 of the air-floating image 3 is described. Fig.13A , Fig. 13B The following shows a method of changing the filter size in the y direction in order to change the amount of enhancement of the clarity of each area of ​​the flying image 3 . Fig.13A is an example of how the filter size changes when using moving average masking, Fig. 13B is an example of how the filter size changes when using Gaussian masking. Fig. 13C This is an example of a rectangular wave output when the filter size in the moving average mask is changed.

[0125] In filters 422 and 432 applied at y=y2, if an arbitrary filter size and weighting coefficient are determined, the amount of sharpness correction improved by the image sharpening process can be measured. At y=y1, by measuring the amount of sharpness correction while changing the filter size, the filter size with the same correction amount as y=y2 can be obtained, and the filter applied at this time is set to filters 421 and 431 applied at y=y2. The filter size set in filters 421 and 431 is set to s1×s1, and the filter size set in filters 422 and 432 is set to s2×s2. In the application range y2<y<y1 of the image sharpening process, the rate of change of the filter size at an arbitrary coordinate y can be obtained by dividing the difference between s1 and s2 by the number of pixels y1-y2 in the application range:

[0126] [Mathematical formula 5]

[0127]

[0128] Therefore, the weighting coefficient of the filter actually applied at an arbitrary coordinate y is the coefficient obtained by multiplying the change rate by the number of pixels y1-y2 from the lower end of the application range, and can be set as:

[0129] [Mathematical formula 6]

[0130]

[0131] The image sharpening process based on this parameter can continuously change the filter size s1 of y=y1 to the filter size s2 of y=y2 in the application range y2<y<y1. The above-mentioned image sharpening process can correct the sharpness of the entire surface of the air-floating image 3 to the same degree in accordance with the gradient of the sharpness change from the upper part to the lower part of the air-floating image 3 obtained as the characteristic of the air-floating image display device 1000.

[0132] use Fig.14A , Fig. 14B , an example of a filter setting and image processing method that matches the clarity characteristics 241, 242, and 243 of the air-floating image 3 is described. Fig.14A The following shows a method of changing the type of filter (each matrix element) applied in the y direction in order to change the amount of enhancement of the sharpness of each area of ​​the flying image 3 . Fig.14A This is an example of how each matrix element changes when using moving average masking and Gaussian masking. Fig. 14B is an example of a rectangular wave output based on moving average masking and Gaussian masking.

[0133] In the filter 442 applied at y=y2, if an arbitrary filter size and filter type (arrangement method of each matrix element) are determined, the sharpness correction amount improved by the image sharpening process can be measured. At y=y1, by measuring the sharpness correction amount while changing the weighting coefficient of the mask using a filter of a different type from the filter 442, it is possible to obtain a weighting coefficient that is the same as the correction amount at y=y2, and the filter applied at this time is set to the filter 441 applied at y=y2. The matrix element of the i-row and j-column set in the filter 441 is set to t1(i, j), and the matrix element of the i-row and j-column set in the filter 442 is set to t2(i, j). In the application range y2<y<y1 of the image sharpening process, the change rate of the matrix element of the i-row and j-column of an arbitrary coordinate y can be obtained by dividing the difference between t1(i, j) and t2(i, j) by the number of pixels y1-y2 in the application range:

[0134] [Formula 7]

[0135]

[0136] Therefore, the weighting coefficient of the filter actually applied at an arbitrary coordinate y is the coefficient obtained by multiplying the change rate by the number of pixels y-y2 from the lower end of the application range, and can be set as:

[0137] [Mathematical formula 8]

[0138]

[0139] The image sharpening process based on this parameter can continuously change from the matrix element t1(i, j) of the i-row and j-column of y=y1 to the matrix element t2(i, j) of the i-row and j-column of y=y2 in the range y2<y<y1. By the above-mentioned image sharpening process, the sharpness of the entire surface of the air-floating image 3 can be corrected to the same degree in accordance with the gradient of the sharpness change from the upper part to the lower part of the air-floating image 3 obtained as a characteristic of the air-floating image display device 1000.

[0140] In the technology of this embodiment, by displaying high-resolution and high-brightness image information in a suspended state in the air, for example, users can operate without feeling uneasy about contact infection of infectious diseases. If the technology of this embodiment is used in a system used by multiple unspecified users, the risk of contact infection of infectious diseases can be reduced, and a non-contact user interface that can be used without feeling uneasy can be provided. This contributes to the "3 Health and Well-being for All" of the Sustainable Development Goals (SDGs) advocated by the United Nations.

[0141] In addition, in the technology of this embodiment, by making the clarity of the emitted image light uniform to the same degree, a bright and clear floating image in the air can be obtained. According to the technology of this embodiment, a non-contact user interface with excellent usability and capable of significantly reducing power consumption can be provided. This contributes to the Sustainable Development Goals (SDGs: Sustainable Development Goals) advocated by the United Nations, "9 Establishing the foundation for industrial and technological innovation" and "11 Building a sustainable city".

[0142] Various embodiments have been described in detail above, but the present invention is not limited to the above embodiments, and includes various modified examples. For example, the above embodiments are examples of describing the entire system in detail in order to easily explain the present invention, and are not limited to all the structures described. In addition, a part of the structure of a certain embodiment can be replaced with the structure of another embodiment, and in addition, the structure of another embodiment can be added to the structure of a certain embodiment. In addition, for a part of the structure of each embodiment, other structures can be added, deleted, or replaced.

[0143] Description of Reference Numerals

[0144] 3…image suspended in the air; 4…input operation; 10…display device; 100…transparent member; 5…retroreflective plate; 11…liquid crystal display panel; 13…light source device; 1000…image suspended in the air display device; 1110…control unit; 1160…image control unit; 1180…camera unit; 1102…image display unit; 1350…in-air operation detection unit; 1351…in-air operation detection sensor

Claims

1. An aerial suspended image display device, which displays an aerial suspended image, wherein: The aerial suspended image display device has: Image input interface; an image processing circuit for processing an image based on an input image input via the image input interface; an image display unit, for displaying the image processed by the image processing circuit; and The retroreflective plate reflects the image light emitted from the image display unit to form an image suspended in the air. Regarding the image light emitted from the display surface of the image display unit, the optical path length from the image light emitted from the display surface of the image display unit to the position of the image floating in the air after being reflected by the retroreflective plate varies depending on the position of the image light on the display surface of the image display unit from which the image light is emitted. The image processing circuit performs different image sharpening processes at a plurality of positions of an image based on the input image corresponding to a plurality of positions having different optical path lengths of the image light.

2. The aerial floating image display device according to claim 1, wherein: The image display area of ​​the display surface of the image display unit is rectangular. The image display unit and the retroreflective plate are configured so that the optical path length of the image light is inclined along a first direction at a position on the display surface of the image display unit, and the first direction is a direction along one side of the rectangle. The image processing circuit makes the image sharpening process different stepwise so that the effect of the image sharpening process is inclined in a direction corresponding to the first direction on the display surface of the image display unit based on the position of the input image in the image.

3. The aerial floating image display device according to claim 1, wherein: The plurality of positions at which the optical path lengths of the image light are different include a first position and a second position, the optical path length of the image light at the second position being longer than the optical path length of the image light at the first position, The image processing circuit uses image sharpening processing with a stronger image sharpening effect than filtering processing at the position corresponding to the first position for image sharpening processing at the position corresponding to the second position based on the position in the image of the input image.

4. The aerial floating image display device according to claim 1, wherein: The image sharpening process performed by the image processing circuit is an image sharpening process using a sharpening filter. The different image sharpening processing is an image sharpening processing in which the weighting coefficient of the sharpening filter is different.

5. The aerial floating image display device according to claim 4, wherein: The sharpening filter is a moving average filter-based sharpening filter or a Gaussian filter-based sharpening filter.

6. The aerial floating image display device according to claim 1, wherein: The image sharpening process performed by the image processing circuit is an image sharpening process using a sharpening filter. The different image sharpening processes are image sharpening processes in which the filter sizes of the sharpening filters are different.

7. The aerial floating image display device according to claim 6, wherein: The sharpening filter is a moving average filter-based sharpening filter or a Gaussian filter-based sharpening filter.

8. The aerial floating image display device according to claim 1, wherein: The different image sharpening processes performed by the image processing circuit are image sharpening processes using different types of sharpening filters.

9. The aerial floating image display device according to claim 1, wherein: The image sharpening process performed by the image processing circuit is an image sharpening process using a sharpening filter. The different image sharpening processing is an image sharpening processing in which the type of the sharpening filter is different between a sharpening filter based on a moving average filter and a sharpening filter based on a Gaussian filter.

Images