Control device, display system, control method, and program

Abstract

This control device controls display of a non-contact display. The control device includes an accepting unit, a processing unit, and an output control unit. The non-contact display includes a touch panel, the touch panel being provided overlaid on part of a surface of an optical element included in the non-contact display. An accepting means accepts an operation performed with respect to an aerial image detected by a sensor included in the non-contact display, and an operation performed with respect to a touch panel, as each being different operations. The processing unit, upon accepting an operation performed with respect to the aerial image, performs processing in accordance with an operation performed with respect to the aerial image, and upon accepting an operation performed with respect to the touch panel, performs processing in accordance with an operation performed with respect to the touch panel. The output control unit controls display on a display, on the basis of processing results by the processing unit.

Description

Control device, display system, control method, and recording medium

[0001] The present disclosure relates to a control device and the like.

[0002] There are non-contact displays that form an image of what is displayed on a display in mid-air (see, for example, Patent Documents 1 to 3). In the non-contact displays, for example, a sensor detects an operation by a user on the mid-air image.

[0003] JP 2022-93789 A International Publication No. 2016 / 047173 International Publication No. 2022 / 045294

[0004] When using a non-contact display, it is difficult to perform various operations.

[0005] An example of an object of the present disclosure is to provide a control device or the like that enables various operations to be performed when using a non-contact display.

[0006] A control device in one aspect of the present disclosure includes a receiving means for receiving an operation on an aerial image detected by a sensor included in a non-contact display and an operation on a touch panel superimposed on a portion of the surface of an optical element included in the non-contact display as separate operations, a processing means for performing processing in accordance with the operation on the aerial image when an operation on the aerial image is received, and for performing processing in accordance with the operation on the touch panel when an operation on the touch panel is received, and an output control means for controlling the display on a display included in the non-contact display based on the processing result by the processing means.

[0007] A display system in one aspect of the present disclosure comprises a non-contact display and a control device, wherein the non-contact display comprises a touch panel, an optical element, a display, and a sensor, wherein the touch panel is positioned over a portion of the surface of the optical element, wherein the touch panel detects operations on the panel itself, and wherein the sensor detects operations on an aerial image formed by the display and the optical element, and wherein the control device comprises: a receiving means for receiving an operation on the aerial image and an operation on the touch panel as separate operations, a processing means for performing processing in accordance with the operation on the aerial image when an operation on the aerial image is received, and for performing processing in accordance with the operation on the touch panel when an operation on the touch panel is received, and an output control means for controlling the display on the display based on the processing result by the processing means.

[0008] A control device in one aspect of the present disclosure comprises: a reception means for receiving operations on an aerial image detected by a sensor included in a non-contact display installed in a vehicle and operations on a touch panel superimposed on a portion of the surface of an optical element included in the non-contact display; a processing means for, when receiving an operation on the aerial image, performing processing according to the operation on the aerial image, and, when receiving an operation on the touch panel, performing processing according to the operation on the touch panel; and an output control means for controlling display on a display included in the non-contact display based on the processing result by the processing means; wherein the optical element, the display, and the touch panel are installed so that the operation surface of the touch panel and the surface of the optical element form an acute angle with the display surface of the display; the operation on the aerial image is an operation by a passenger in the front seat of the vehicle, and the operation on the touch panel is an operation by the driver of the vehicle; and the reception means receives operations on the aerial image and does not receive operations on the touch panel while the vehicle is moving.

[0009] A control method in one aspect of the present disclosure accepts an operation on an aerial image detected by a sensor included in a non-contact display and an operation on a touch panel superimposed on a portion of the surface of an optical element included in the non-contact display as separate operations, and when an operation on the aerial image is accepted, performs processing in accordance with the operation on the aerial image, and when an operation on the touch panel is accepted, performs processing in accordance with the operation on the touch panel, and controls the display on a display included in the non-contact display based on the processing results.

[0010] A control method in one aspect of the present disclosure receives an operation on an aerial image detected by a sensor included in a non-contact display installed in a vehicle and an operation on a touch panel superimposed on a portion of the surface of an optical element included in the non-contact display; when an operation on the aerial image is received, performs processing corresponding to the operation on the aerial image; when an operation on the touch panel is received, performs processing corresponding to the operation on the touch panel; and based on the processing result, controls display on a display included in the non-contact display; the optical element, the display, and the touch panel are installed so that an operation surface of the touch panel and the surface of the optical element form an acute angle with a display surface of the display; the operation on the aerial image is an operation by a passenger in the front seat of the vehicle, and the operation on the touch panel is an operation by the driver of the vehicle; and in the reception, while the vehicle is moving, the operation on the aerial image is received and the operation on the touch panel is not received.

[0011] A program in one aspect of the present disclosure causes a computer to accept an operation on an aerial image detected by a sensor included in a non-contact display and an operation on a touch panel superimposed on a portion of the surface of an optical element included in the non-contact display as separate operations, and when an operation on the aerial image is accepted, performs processing in accordance with the operation on the aerial image, and when an operation on the touch panel is accepted, performs processing in accordance with the operation on the touch panel, and controls the display on a display included in the non-contact display based on the processing results.

[0012] A program in one aspect of the present disclosure causes a computer to receive operations on an aerial image detected by a sensor included in a non-contact display installed in a vehicle and operations on a touch panel superimposed on a portion of the surface of an optical element included in the non-contact display, and when an operation on the aerial image is received, performs processing corresponding to the operation on the aerial image, and when an operation on the touch panel is received, performs processing corresponding to the operation on the touch panel, and controls display on a display included in the non-contact display based on the processing results, wherein the optical element, the display, and the touch panel are installed so that an operation surface of the touch panel and the surface of the optical element form an acute angle with a display surface of the display, and the operation on the aerial image is an operation by a passenger in the front seat of the vehicle, and the operation on the touch panel is an operation by the driver of the vehicle, and the receiving process receives operations on the aerial image and does not accept operations on the touch panel while the vehicle is moving.

[0013] Each program may be stored in a non-transitory computer-readable recording medium.

[0014] According to the present disclosure, various operations can be performed when using a non-contact display.

[0015] 1 is an explanatory diagram showing an example of a display system; FIG. 1 is an explanatory diagram (part 1) showing a simplified view of an aerial display; FIG. 2 is an explanatory diagram (part 2) showing a simplified view of an aerial display; FIG. 3 is an explanatory diagram showing an example of the hardware configuration of a computer; FIG. 4 is a block diagram showing an example of the configuration of a control device according to a first embodiment; FIG. 5 is an explanatory diagram showing installation example 1 of an aerial display when used by multiple users; FIG. 6 is an explanatory diagram showing installation example 2 of an aerial display when used by multiple users; FIG. 7 is an explanatory diagram showing an example of a user being imaged to correct an operation position; FIG. 8 is an explanatory diagram showing an example of a screen for correcting a standing position; FIG. 9 is a flowchart showing an example of the operation of a control device when used by multiple users; FIG. 10 is an explanatory diagram showing an installation example when one user operates both a touch panel and aerial imaging; FIG. 11 is an explanatory diagram showing an example of an operation determined to be an operation on aerial imaging; FIG. 12 is an explanatory diagram showing an example of an operation determined not to be an operation on aerial imaging; FIG. 13 is an explanatory diagram showing an example of the installation of an imaging device for detecting the position of a user's hand; FIG. 14 is an explanatory diagram showing an example of an aerial display used as a POS terminal; FIG. 15 is a flowchart showing an example of the operation of a control device when used by multiple users. 1 is an explanatory diagram showing an example of mirror installation when an aerial image is superimposed on an image reflected in the mirror. FIG. 2 is an explanatory diagram showing an example of superimposed display. FIG. 3 is an explanatory diagram showing an example of installation of an imaging device for making a display follow the movement of a user. FIG. 4 is an explanatory diagram showing an example of making a display follow the movement of a user. FIG. 5 is an explanatory diagram showing an example of installation of another display when an aerial image is superimposed on an image reflected on another display. FIG. 6 is an explanatory diagram showing an example of installation of an object when an aerial image is superimposed on the object. FIG. 7 is an explanatory diagram showing an example of superimposing an aerial image on an object. FIG. 8 is a block diagram showing an example of the configuration of a control device according to a second embodiment. FIG. 9 is a flowchart showing an example of operation of a control device according to a second embodiment.

[0016] Hereinafter, embodiments of a control device, a display system, a control method, a program, and a non-transitory recording medium for recording a program according to the present disclosure will be described in detail with reference to the accompanying drawings. The disclosed technology is not limited to these embodiments.

[0017] In each embodiment, an aerial display will be described as an example of a non-contact type display, but the example of the non-contact type display is not particularly limited.

[0018] (Embodiment 1) Fig. 1 is an explanatory diagram showing an example of a display system. The display system 1 includes an aerial display 10 and a control device 11. The control device 11 controls the aerial display 10. The control device 11 accepts user operations via the aerial display 10. The control device 11 also displays information on the aerial display 10.

[0019] For example, the aerial display 10 and the control device 11 are connected via a communication network NT.

[0020] The display system 1 may further include other devices. For example, the display system 1 may include an imaging device that captures images of the user. The control device 11 and the imaging device are connected via a communication network NT. For example, the communication network NT connecting the imaging device and the control device 11 and the communication network NT connecting the aerial display 10 and the control device 11 may be the same or different, and are not particularly limited.

[0021] 2A and 2B are explanatory diagrams showing a simplified view of the aerial display 10. In FIGS. 2A and 2B, the aerial display 10 includes, for example, an optical element 101, a display 102, a sensor 105, and a touch panel 103. For ease of explanation, FIGS. 2A and 2B define the X-axis, Y-axis, and Z-axis. In FIG. 2A, the plane of the X-axis and Y-axis is, for example, approximately parallel to the surface of the optical element 101 and the operation surface of the touch panel 103. FIG. 2B shows the aerial display 10 as viewed from the positive direction of the X-axis. The aerial display 10 has display directionality. Therefore, when a user views the aerial display 10 from the positive direction of the X-axis, the aerial image 104 cannot actually be seen. However, the aerial image 104 is illustrated for ease of understanding.

[0022] In the aerial display 10, an optical element 101 passes light emitted by the display 102 and forms an identical image on the opposite side of the display 102. In Figures 2A and 2B, this image is an aerial image 104. A sensor 105 is used to operate the aerial image 104. The sensor 105 is, for example, a motion sensor or a 3D (Dimension) sensor.

[0023] The angle between the display 102 and the optical element 101 may be controllable. This changes the angle between the aerial image 104 and the optical element 101, making it possible to change the orientation of the aerial image 104. Here, the display 102 may be installed so that the optical element 101 and the aerial image 104 are approximately parallel to each other.

[0024] Next, the touch panel 103 is installed on at least a part of the surface of the optical element 101. For example, the operation surface of the touch panel 103 is approximately parallel to the surface of the optical element 101. Here, as an example in which the touch panel 103 is installed on the optical element 101, the touch panel 103 may be attached to at least a part of the optical element 101. In this case, the touch panel 103 may be integrated with a protective glass for the optical element 101. Furthermore, as another example, the touch panel 103 may be embedded in the optical element 101.

[0025] The aerial display 10 may also have a vibration generating device that generates vibrations. For example, the vibration generating device may generate vibrations in response to an operation under control of the control device 11. The vibration generating device may be a device that generates vibrations in response to an operation on the aerial image 104, and a device that generates vibrations in response to an operation on the touch panel 103, which may be different devices. The aerial display 10 may also have an audio output device that outputs audio. For example, the audio output device may output audio in response to an operation under control of the control device 11. The audio output device may be a device that outputs audio in response to an operation on the aerial image 104, and a device that outputs audio in response to an operation on the touch panel 103, which may be different devices.

[0026] Next, an example of a hardware configuration in which the control device 11 is realized by a computer will be described. Fig. 3 is an explanatory diagram showing an example of a hardware configuration of a computer. For example, some or all of the devices can be realized using any combination of a computer 80 and a program as shown in Fig. 3.

[0027] The computer 80 includes, for example, a processor 801, a ROM (Read Only Memory) 802, a RAM (Random Access Memory) 803, and a storage device 804. The computer 80 also includes a communication interface 805 and an input / output interface 806. The components are connected to each other, for example, via a bus 807. The number of each component is not particularly limited, and there may be one or more of each component.

[0028] The processor 801 controls the entire computer 80. As the processor 801, for example, a CPU (Central Processing Unit), a DSP (Digital Signal Processor), a GPU (Graphics Processing Unit), a MPU (Micro Processing Unit), an FPU (Floating point number Processing Unit), a PPU (Physics Processing Unit), a TPU (Tensor Processing Unit), a quantum processor, a microcontroller, or a combination thereof can be used.

[0029] The computer 80 also includes a ROM 802, a RAM 803, and a storage device 804 as storage units. Examples of the storage device 804 include semiconductor memory such as flash memory, a hard disk drive (HDD), and a solid state drive (SSD). For example, the storage device 804 stores an operating system (OS) program, application programs, and programs according to the embodiments. Alternatively, the ROM 802 stores application programs and programs according to the embodiments. The RAM 803 is used as a work area for the processor 801.

[0030] The processor 801 also loads programs stored in the storage device 804, ROM 802, etc. The processor 801 then executes each process coded in the program. The processor 801 may also download various programs via the communication network NT. The processor 801 also functions as a part or all of the computer 80. The processor 801 may then execute the processes or instructions in the illustrated flowchart based on the program.

[0031] The communication interface 805 is connected to a communication network NT such as a LAN (Local Area Network) or a WAN (Wide Area Network) via a wireless or wired communication line. The communication network NT may be composed of multiple communication networks NT. As a result, the computer 80 is connected to external devices and external computers 80 via the communication networks NT. The communication interface 805 serves as an interface between the communication network NT and the inside of the computer 80. The communication interface 805 also controls the input and output of data from external devices and external computers 80.

[0032] Furthermore, the input / output interface 806 is connected to at least one of an input device, an output device, and an input / output device. The connection method may be wireless or wired. Examples of the input device include a keyboard, a mouse, and a microphone. Examples of the output device include a display device, a lighting device, and an audio output device that outputs audio. Examples of the input / output device include a touch panel display. The input device, output device, and input / output device may be built into the computer 80 or may be external.

[0033] The hardware configuration of the computer 80 is an example. The computer 80 may have some of the components shown in FIG. 3 . The computer 80 may have components other than those shown in FIG. 3 . For example, the computer 80 may have a drive device or the like. The processor 801 may then read programs and data stored in a recording medium attached to the drive device or the like into the RAM 803. Examples of non-transitory tangible recording media include optical disks, flexible disks, magneto-optical disks, and USB (Universal Serial Bus) memories. As described above, the computer 80 may also have input devices such as a keyboard and a mouse. The computer 80 may also have an output device such as a display 102.

[0034] Next, a description will be given of each functional unit of the control device 11. Fig. 4 is a block diagram showing an example of a configuration of the control device 11 according to the first embodiment. The control device 11 includes a reception unit 111, a processing unit 112, an output control unit 113, a correction unit 114, a determination unit 115, and a detection unit 116.

[0035] First, the reception unit 111 receives an operation on the aerial image 104 detected by the sensor 105 and an operation on the touch panel 103. For example, the reception unit 111 may receive an operation on the aerial image 104 detected by the sensor 105 and an operation on the touch panel 103 as different operations. Here, the operation on the aerial image 104 and the operation on the touch panel 103 being different are not particularly limited. Taking a button operation as an example, the operation on the aerial image 104 and the operation on the touch panel 103 being different may mean that even if the same button is pressed, the processing performed by the processing unit 112 after the button is pressed is different. Taking a button operation as an example, the operation on the aerial image 104 and the operation on the touch panel 103 being different may mean that a button is valid only on one of the two. For example, the reception unit 111 may perform control so as to accept an operation on the aerial image 104 and an operation on the touch panel 103 as the same operation or as different operations depending on the position of the operation. More specifically, for example, the reception unit 111 accepts an operation on the aerial image 104 and an operation on the touch panel 103 as the same operation in a predetermined area, and accepts an operation on the aerial image 104 and an operation on the touch panel 103 as different operations outside the predetermined area.

[0036] Next, when the processing unit 112 receives an operation on the aerial image 104, it performs processing according to the operation on the aerial image 104. Furthermore, when the processing unit 112 receives an operation on the touch panel 103, it performs processing according to the operation on the touch panel 103.

[0037] The output control unit 113 also controls the display on the display 102. The output control unit 113 controls the display on the display 102 based on the processing result by the processing unit 112, for example.

[0038] Furthermore, the output control unit 113 changes the color of the position where an operation on the aerial image 104 has been performed. Furthermore, the output control unit 113 changes the color of the position where an operation on the touch panel 103 has been performed. For example, the output control unit 113 may change the color of the position where an operation on the touch panel 103 has been performed so that it continues to change until the finger is removed. For example, the output control unit 113 may restore the color of the position where the color has been changed when the finger is removed. Furthermore, the color of the position where an operation on the aerial image 104 has been performed and the color of the position where an operation on the touch panel 103 has been performed may be different. This allows the user to distinguish whether the aerial image 104 or the touch panel 103 is being operated by the color.

[0039] The output control unit 113 may also control audio output. For example, the output control unit 113 may control the sound output when an operation is performed on the aerial image 104 so that the sound output when an operation is performed on the touch panel 103 is different from the sound output. This allows each user to determine whether they are operating the aerial image 104 or the touch panel 103 by the audio.

[0040] Furthermore, the output control unit 113 may control the generation of vibrations. For example, the output control unit 113 may control the vibrations generated when an operation is performed on the aerial image 104 so that they are different from the vibrations generated when an operation is performed on the touch panel 103. For example, the vibrations generated when an operation is performed on the aerial image 104 may be aerial haptics using ultrasound or the like. For example, the vibrations generated when an operation is performed on the touch panel 103 may be an existing technology such as haptics used in smartphones or the like. The vibrations generated when an operation is performed on the aerial image 104 may be different enough to be distinguishable from the vibrations generated when an operation is performed on the touch panel 103. This allows each user to determine whether they are operating the aerial image 104 or the touch panel 103 based on the vibrations.

[0041] Here, the number of users may be multiple or may be one. An example of a case where there are multiple users will be explained first, followed by an example of a case where there is one user.

[0042] <Operation by Multiple Users> First, an example of installation of the aerial display 10 when there are multiple users will be described with reference to FIGS. 5 and 6. FIG.

[0043] 5 is an explanatory diagram showing an installation example 1 of the aerial display 10 when used by multiple users. In FIG. 5, for example, the surface of the optical element 101 of the aerial display 10 and the operation surface of the touch panel 103 are installed approximately parallel to a wall, a vehicle dashboard, etc. In FIG. 5, the state is such that the user's head is being viewed.

[0044] 5, when multiple users operate the aerial display 10 at the same time, the display 102 is installed so that the operation surface of the touch panel 103 and the operation surface of the aerial image 104 are not parallel. The optical element 101, the display 102, and the touch panel 103 are installed so that the operation surface of the touch panel 103 and the surface of the optical element 101 form an acute angle with the display surface of the display 102.

[0045] In Fig. 5, a first user views the display 102 via the aerial image 104. Meanwhile, a second user views the display 102 directly. By installing the display 102 as shown in Fig. 5, the orientation of the screens seen by the first and second users when they view the aerial image 104 and the display 102 from the front will be the same. However, the aerial image 104 may appear brighter than the display 102.

[0046] (Example of Use in a Vehicle) For example, when the display system 1 is installed in a vehicle, the angle between the display 102 and the optical element 101 and the touch panel 103 may be determined based on the position of the driver's face. The position of the driver's face may be the position of an average driver's face or the position of an individual driver's face. The position of the individual driver's face may be detected from an image captured by an imaging device installed in the vehicle, for example. Specifically, for example, the imaging device captures an image of the driver's face. Then, the detection unit 116 detects the position of the driver's face from the captured image. The display system 1 may determine the angle between the display 102 and the optical element 101 based on the detected position of the face, and control the position of the display 102 to achieve the determined angle.

[0047] The reception unit 111 receives an operation on the aerial image 104 and an operation on the touch panel 103. For example, the operation on the touch panel 103 may be an operation by the driver of the vehicle, and the operation on the aerial image 104 may be an operation by a passenger other than the driver. The passenger other than the driver may be, for example, a passenger sitting in the passenger seat.

[0048] For example, the reception unit 111 receives an operation on the touch panel 103 and receives an operation on the aerial image 104 when the vehicle is not in motion. On the other hand, the reception unit 111 does not receive an operation on the touch panel 103 and receives an operation on the aerial image 104 when the vehicle is in motion. Although the example is given of the vehicle being in motion, in a situation that may impede safety even when the vehicle is not in motion, the reception unit 111 may not receive an operation on the touch panel 103, which is an operation by the driver, and may receive an operation on the aerial image 104, which is an operation by a passenger other than the driver. This allows the driver and the passenger in the front passenger seat to operate the same screen using the touch panel 103 and the aerial image 104, respectively. The driver cannot operate the touch panel 103 when the vehicle is in motion, but the passenger in the front passenger seat can operate the touch panel 103 and the aerial image 104 even when the vehicle is in motion.

[0049] A typical touch panel used as a display device for a car navigation system is controlled to prevent operation while driving in order to prevent a decrease in safety. This control is performed to prevent the driver from operating the device. However, since it is difficult to determine whether the person in the passenger seat is operating the device or the driver, touch panel operations may not be accepted regardless of who is operating the device. On the other hand, when the aerial display 10 is installed in a vehicle as a display device as shown in Figure 5, the driver cannot see the aerial image 104 and has difficulty operating the aerial image 104. Therefore, by using the aerial display 10 including the touch panel 103, the person in the passenger seat can operate the aerial image 104 while driving, but the driver cannot operate the touch panel 103. The passenger in the passenger seat can operate the device while maintaining safety while the vehicle is traveling. Therefore, various operations can be performed.

[0050] 6 is an explanatory diagram showing an installation example 2 of the aerial display 10 when used by multiple users. For example, the aerial display 10 may be installed on a table or a wall so that the surface of the optical element 101 of the aerial display 10 and the operation surface of the touch panel 103 are approximately parallel to the plane of the table or the wall.

[0051] Also, as in Figure 5, when multiple users are to operate the aerial display 10 at the same time, as shown in Figure 6, the display 102 is installed so that the operation surface of the touch panel 103 and the operation surface of the aerial image 104 are not parallel.

[0052] 6 , for example, the reception unit 111 receives an operation on the aerial image 104 by a first user and an operation on the touch panel 103 by a second user. In FIG. 6 , the first user may be a main user, and the second user may be a sub-user. Note that the first user may be a sub-user, and the second user may be the main user.

[0053] In Fig. 6, a first user views the display 102 via an aerial image 104. Meanwhile, a second user directly views the display 102. By installing the display as shown in Fig. 6, the second user will see a screen that is an inverted version of the screen seen by the first user.

[0054] 5 and 6, the aerial display 10 may be installed in a store. The aerial display 10 may be used as a display device for performing processes such as product registration, product settlement, and product search in the store.

[0055] For example, the first user who operates the aerial image 104 is a customer, and the second user who operates the touch panel 103 is a store clerk. The customer and the store clerk can view the same screen content on the aerial image 104 and the direct display 102, respectively. However, the store clerk's image is an inverted image of the aerial image 104.

[0056] For example, the reception unit 111 accepts operations by a customer on the aerial image 104 and operations by a store clerk on the touch panel 103 as different operations. For example, the operations by a customer may be operations related to product registration, operations related to payment processing, and operations related to product search. On the other hand, the operations by the store clerk may be operations to support the customer's operations or operations for performing processing exclusive to the store clerk. Here, the operations by a customer on the aerial image 104 and the operations by a store clerk on the touch panel 103 may be different in that, for example, even if the same button is pressed, the processing performed by the processing unit 112 after the button is pressed is different. Alternatively, the operations by a customer on the aerial image 104 and the operations by a store clerk on the touch panel 103 may be different in that a button is only valid on one side. For example, the screen may include a store clerk button, and the reception unit 111 may not accept touching the store clerk button on the aerial image 104 as an operation, but may accept touching the store clerk button on the touch panel 103 as an operation. Here, if the operation on the touch panel 103 by the store clerk is an operation for performing a process dedicated to the store clerk, when the reception unit 111 receives a predetermined operation on the touch panel 103 by the store clerk, the processing unit 112 may transition from the customer mode to the store clerk mode. For example, in the customer mode, the reception unit 111 may receive an operation on the aerial image 104 and an operation on the touch panel 103. On the other hand, in the store clerk mode, the reception unit 111 may not receive an operation on the aerial image 104 but may receive an operation on the touch panel 103. More specifically, as an example of transitioning to the store clerk mode, when the reception unit 111 receives a touch on the touch panel 103, the output control unit 113 displays a screen on which a store clerk code can be input. Then, the reception unit 111 may receive the input of the store clerk code as the predetermined operation. Then, when the processing unit 112 receives the input of the store clerk code, the processing unit 112 may transition to the store clerk mode.

[0057] It should be noted that the first user who operates the aerial image 104 may be a store clerk, and the second user who operates the touch panel 103 may be a customer. When the reception unit 111 receives a predetermined operation on the aerial image 104 by the store clerk, the processing unit 112 may transition from the customer mode to the store clerk mode. For example, in the customer mode, the reception unit 111 receives an operation on the touch panel 103 and receives an operation on the aerial image 104. On the other hand, in the store clerk mode, the reception unit 111 may not receive an operation on the touch panel 103, but may receive an operation on the aerial image 104.

[0058] (Correction Example) Here, the position where the user operates the touch panel 103 may differ from the position where the user actually wants to operate. Also, the position where the user operates the aerial image 104 may differ from the position where the user actually wants to operate. For this reason, the operated position may be corrected. An example of correcting the operation position will be described with reference to FIG. 7 .

[0059] FIG. 7 is an explanatory diagram showing an example in which a user is imaged for correcting the operation position. In FIG. 7 , there is a distance between the display 102 and the touch panel 103, i.e., there is a gap between the display 102 and the touch panel 103. Because the positional relationship between the display 102 and the touch panel 103 varies depending on the position of the user's viewpoint, when the touch panel 103 performs an operation on the display 102, the operation position may be shifted. In other words, the position of the touch panel 103 corresponding to the area of ​​the display 102 that the second user is gazing at changes depending on the position of the second user's eyes. The imaging device 31 is installed so as to be able to capture an image of the second user. For example, the detection unit 116 detects the position of the second user's face from the image of the second user captured by the imaging device 31. The correction unit 114 corrects the position of the received operation on the touch panel 103 based on the position of the second user's face. Specifically, for example, the correction unit 114 calculates a correction amount between the position of an operation on the touch panel 103 and the position of the display on the display 102, based on the position of the face of the second user. When the reception unit 111 receives an operation on the touch panel 103, the correction unit 114 corrects the position of the received operation using the correction amount. Then, the processing unit 112 performs processing as if the operation was performed at the corrected position.

[0060] Furthermore, the position at which the user performs an operation on the aerial image 104 may differ from the position at which the user actually wants to perform the operation. When correcting the position of the operation on the aerial image 104, the imaging device 37 detects the position of the hand of the first user. The imaging device 37 may be, for example, a 3D camera. For example, the correction unit 114 corrects the position of the operation on the aerial image 104 based on the position of the hand of the first user detected by the imaging device 37. Specifically, for example, the correction unit 114 calculates a correction amount between the position of the operation on the aerial image 104 and the position of the display on the display 102 based on the position of the hand of the first user. When the reception unit 111 receives an operation on the aerial image 104, the correction unit 114 corrects the position of the operation on the aerial image 104 based on the correction amount. The processing unit 112 processes the operation as being at the corrected position.

[0061] 8 is an explanatory diagram showing an example of a screen for correcting a standing position. In addition, in order to correct the operation position of the aerial image 104, the first user may change his / her standing position so that the desired display is visible. In FIG. 8, the first user is asked to change his / her standing position so that the number is within the circle.

[0062] Similarly, when operating the touch panel 103, the second user may change his / her position so that the desired display can be seen.

[0063] 9 is a flowchart showing an example of the operation of the control device 11 when multiple users are using the device. The output control unit 113 controls the display on the display 102 (step S101). The reception unit 111 receives an operation on the aerial image 104 by the first user (step S102). The processing unit 112 then performs processing according to the operation (step S103), and the process returns to step S101.

[0064] The accepting unit 111 accepts an operation on the touch panel 103 by the second user (step S104). The processing unit 112 performs a process according to the operation on the touch panel 103 (step S105), and the process returns to step S101.

[0065] The flowchart may be ended as appropriate.

[0066] <Operation by One User> Next, an example in which one user operates both the touch panel 103 and the aerial image 104 will be described.

[0067] 10 is an explanatory diagram showing an example of installation when one user operates both the touch panel 103 and the aerial image 104. In Fig. 10, for example, the aerial display 10 is installed on a table such as a desk so that the surface of the optical element 101 of the aerial display 10 and the operation surface of the touch panel 103 are approximately parallel to the plane of the table.

[0068] 10, the display 102 is installed so that the operation surface of the touch panel 103 and the operation surface of the aerial image 104 are not parallel. However, when one user operates the aerial display 10, the display 102 may be installed so that the operation surface of the touch panel 103 and the operation surface of the aerial image 104 are approximately parallel.

[0069] The correction is the same as that explained in the example of multiple users, and a detailed explanation will be omitted.

[0070] Here, for example, when one user operates the touch panel 103 and the aerial image 104, if the user operates the touch panel 103 from a position where the aerial image 104 can be seen as in Fig. 10, the user needs to pass through the aerial image 104 and put his / her hand toward the touch panel 103. Therefore, the determination unit 115 may determine whether the operation is directed to the aerial image 104 based on the position of the hand, for example.

[0071] FIG. 11A is an explanatory diagram showing an example in which an operation is determined to be an operation on the aerial image 104. FIG. 11B is an explanatory diagram showing an example in which an operation is determined not to be an operation on the aerial image 104. In FIGS. 11A and 11B , for example, the sensor 105 is a 3D sensor that detects the position of the hand. For example, the determination unit 115 determines whether the user's hand is located beyond the aerial image 104 and on the optical element 101 side based on the position of the user's hand detected by the sensor 105. As shown in FIG. 11A , when the hand is not located on the optical element 101 side, the reception unit 111 accepts the operation on the aerial image 104. As shown in FIG. 11B , when the hand is located on the optical element 101 side, the reception unit 111 does not accept the operation on the aerial image 104.

[0072] 11A, when a user does not have a hand on the optical element 101 side, the reception unit 111 does not receive an operation on the touch panel 103. Furthermore, as shown in Fig. 11B, when a user has a hand on the optical element 101 side, the reception unit 111 does not receive an operation on the aerial image 104. This makes it possible to prevent operations on the touch panel 103 by other users from being received while a user is performing an operation on the aerial image 104.

[0073] However, when a hand is placed on the optical element 101 side, the accepting unit 111 may be able to accept an operation on the touch panel 103 .

[0074] The method for detecting the hand position is not particularly limited and may be detection by the sensor 105 or detection from an image captured by an imaging device.

[0075] 12 is an explanatory diagram showing an example of installation of an imaging device for detecting the position of a user's hand. The sensor 105 is installed at hand, while the imaging device 38 is installed in a position where it can capture an image of the periphery of the aerial image 104. For example, the imaging device 38 may be an imaging device that includes a function capable of measuring distance information.

[0076] For example, the imaging device 38 captures an image of the user's hand. The dashed line indicates an example of the range captured by the imaging device 38. Then, for example, the detection unit 116 detects the position of the user's hand from the image of the user captured by the imaging device 38. The determination unit 115 determines whether the user's hand is located beyond the aerial image 104 on the optical element 101 side, based on the position of the user's hand. If the hand is not located on the optical element 101 side, the reception unit 111 accepts an operation on the aerial image 104. If the hand is located on the optical element 101 side, the reception unit 111 does not accept an operation on the aerial image 104.

[0077] Although an example in which the position of a hand is detected has been described, the position of a finger may be detected in more detail. For example, the determination unit 115 determines whether the user's finger is located beyond the aerial image 104 on the optical element 101 side based on the detected position of the user's finger. If the finger is not located on the optical element 101 side, the reception unit 111 accepts an operation on the aerial image 104. If the finger is located on the optical element 101 side, the reception unit 111 does not accept an operation on the aerial image 104.

[0078] (Examples of Different Operations) Here, we will explain an example in which one user performs multiple different operations when using the aerial display 10. Note that the examples described below may also be used in the case of multiple users.

[0079] For example, with an input device such as a mouse, multiple transitions are possible with one mouse cursor. As described above, it is difficult to perform various operations when operating the aerial image 104. Therefore, by providing different operations for the aerial image 104 and the touch panel 103, various operations can be performed intuitively.

[0080] For example, a left-click operation of a mouse and a right-click operation of a mouse may be realized by an operation on the aerial image 104 and an operation on the touch panel 103. The operation on the aerial image 104 may correspond to a left-click operation of a mouse. For example, the operation on the aerial image 104 is an operation corresponding to a specified position on the aerial image 104. On the other hand, the operation on the touch panel 103 may correspond to a right-click operation of a mouse. For example, the operation on the touch panel 103 may be at least one of an operation to instruct the display of a menu for performing an operation related to the position where the operation on the touch panel 103 was performed, an operation to instruct the display of a menu for making settings related to the position where the operation on the touch panel 103 was performed, and an operation to instruct the display of information related to the position where the operation on the touch panel 103 was performed.

[0081] Furthermore, for example, a general-purpose operation and a detailed operation may be realized by an operation on the aerial image 104 and an operation on the touch panel 103. The general-purpose operation may be, for example, an operation common to all users, and the detailed operation may be an operation with a different target depending on the user. For example, since an operation on the touch panel 103 has higher operability than an operation on the aerial image 104, the operation on the aerial image 104 may be a general-purpose operation, and the operation on the touch panel 103 may be a detailed operation. For example, the general-purpose operation may be a selection operation by a user touching, and the detailed operation may be an operation such as writing characters using a finger or a pen. In more detail, for example, the detailed operation may be an operation of writing characters such as signing a signature.

[0082] Furthermore, for example, a normal operation and a special operation may be realized by an operation on the aerial image 104 and an operation on the touch panel 103. For example, because an operation on the touch panel 103 is easier to operate than an operation on the aerial image 104, the operation on the aerial image 104 may be a normal operation, and the operation on the touch panel 103 may be a special operation. Taking product registration in a store as an example, for example, a normal operation is an operation to register the product, and a special operation is an operation to cancel a product that was registered when the product was registered.

[0083] Furthermore, for example, an important operation and an unimportant operation may be realized by an operation on the aerial image 104 and an operation on the touch panel 103. Because an operation on the touch panel 103 is easier to operate than an operation on the aerial image 104, the operation on the aerial image 104 may be an unimportant operation, and the operation on the touch panel 103 may be an important operation. Taking product registration and payment at a store as an example, operations such as product registration and payment selection can be easily canceled and are not considered to be important operations. On the other hand, operations that lead to payment, such as an operation to instruct payment execution, cannot be easily canceled and involve the transfer of money, so they can be considered to be important operations.

[0084] FIG. 13 is an explanatory diagram showing an example in which the aerial display 10 is used as a POS (Point of Sales) terminal. In retail stores such as convenience stores, a contactless display may be used to register and settle products. FIG. 13 shows an example in which the aerial display 10 shown in FIGS. 2A and 2B is also used as a self-service POS terminal that can be operated by customers. For example, FIG. 13 shows a barcode reader 32 and a credit card reader 33, along with the aerial display 10.

[0085] 13 , a confirmation button for confirming the payment and a back button for returning to the processing before the payment are displayed on the aerial image 104. For example, the back button can be operated by operating the aerial image 104, while the confirmation button can be operated by operating the touch panel 103.

[0086] Furthermore, for example, when used in a store, product-related operations may be operations on the touch panel 103, and operations related to shopping other than products may be operations on the aerial image 104. Product-related operations may be, for example, operations for registering products. For example, the operation for registering products may be a confirmation operation for registering a product whose product code has been read in a product purchase list, or an operation for inputting the quantity of a product whose product code has been acquired, and are not particularly limited. Furthermore, when shipping a package, an operation such as a confirmation of shipping may be an operation on the touch panel 103, and operations other than shipping may be operations on the aerial image 104. An operation other than shipping may be, for example, an operation for inputting the delivery destination or address.

[0087] 14 is a flowchart showing an example of the operation of the control device 11 when multiple users are using the device. The output control unit 113 controls the display on the display 102 (step S111). The determination unit 115 determines, based on the position of the user's hand, whether the user's hand is beyond the aerial image 104 and is on the optical element 101 side (step S112).

[0088] If the user's hand is beyond the aerial image 104 and not on the optical element 101 side (step S112: No), the accepting unit 111 accepts an operation on the aerial image 104 (step S113). Then, the processing unit 112 performs processing according to the operation on the aerial image 104 (step S114), and the process returns to step S111.

[0089] If the user's hand is beyond the aerial image 104 and on the optical element 101 side (step S112: Yes), the reception unit 111 receives an operation on the touch panel 103 (step S115). Here, if the user's hand is beyond the aerial image 104 and on the optical element 101 side, the reception unit 111 does not receive an operation on the aerial image 104. Then, the processing unit 112 performs processing according to the operation on the touch panel 103 (step S116), and returns to step S111.

[0090] The flowchart may be ended as appropriate. The process by the determination unit 115 to determine whether a user's hand is beyond the aerial image 104 and on the optical element 101 side may also be performed when multiple users perform operations. For example, the determination unit 115 may determine whether a first user's hand is beyond the aerial image 104 and on the optical element 101 side. When the first user's hand is beyond the aerial image 104 and on the optical element 101 side, the reception unit 111 does not receive an operation on the aerial image 104.

[0091] <Example of superimposed display of aerial image 104> As an application example using the aerial display 10, an example of superimposing the aerial image 104 on an image reflected in a mirror, an actual object or living thing, or another display 102 will be described.

[0092] (Example of superimposing the aerial image 104 on the image reflected in the mirror) FIG. 15 is an explanatory diagram showing an example of mirror placement when the aerial image 104 is superimposed on the image reflected in the mirror. For example, the display system 1 further includes a mirror 34. The mirror 34 is disposed on the surface opposite the aerial image 104 with respect to the optical element 101 so that the image reflected in the mirror 34 and the aerial image 104 generated by the aerial display 10 are superimposed. In other words, the mirror 34 is disposed opposite the aerial image 104 with respect to the optical element 101 included in the aerial display 10. This allows the user to view the mirror 34 in a visible extension of the aerial image 104. Therefore, the user sees their own image reflected in the mirror 34 superimposed on the aerial image 104.

[0093] 16 is an explanatory diagram showing an example of a superimposed display. For example, a user is reflected in the mirror 34. A comment such as "Slow down" is displayed on the aerial image 104. As a result, the comment is superimposed on the user reflected in the mirror 34.

[0094] In this way, the aerial display 10 can be used to superimpose graphics such as text and pictures onto the image reflected in the mirror 34, just like a smart mirror.

[0095] Fig. 17 is an explanatory diagram showing an example of installation of an imaging device to cause the display to follow the user's movements. In Fig. 17, the display system 1 may further include an imaging device 39. The imaging device 39 is installed in the optical element 101 included in the aerial display 10, and captures an image of the user or the user reflected in the mirror 34. The detection unit 116 detects the user's movements from the image captured by the imaging device 39. Then, the output control unit 113 determines the display on the display 102 of the aerial display 10 based on the user's movements.

[0096] FIG. 18 is an explanatory diagram showing an example of displaying a message that follows the user's movements. In FIG. 18, the user is reflected in the mirror 34. The comment "here" is displayed on the aerial image 104. The imaging device 39 captures an image of the user. When the user moves, the detection unit 116 detects the user's movement from the image captured by the imaging device 39. The output control unit 113 then controls the display on the display 102 of the aerial display 10 based on the user's movements so that the position of the comment "here" displayed on the aerial image 104 follows the user's movements.

[0097] Here, an example will be described in which, for example, when the aerial image 104 is superimposed on the image reflected in the mirror 34, a distinction is made between operations on the touch panel 103 and operations on the aerial image 104. For example, when a user wants to specify his or her own face reflected in the mirror 34, an operation to specify the face may be an operation on the touch panel 103, and other general operations may be operations on the aerial image 104.

[0098] Examples of uses for superimposing the aerial image 104 on the image reflected in the mirror 34 include when providing instruction or practice such as tooth brushing instruction, makeup instruction, exercise instruction, and work instruction for store clerks.

[0099] The difference between this and a smart mirror is that there is a distance between the aerial image 104 and the image reflected on the mirror 34, which is the object superimposed on the aerial image 104. Another difference from a smart mirror is that the imaging device 39 that captures the image of the user can be fixedly disposed.

[0100] (Example of superimposing the aerial image 104 on an image displayed on another display 102) Figure 19 is an explanatory diagram showing an example of the installation of another display 102 when the aerial image 104 is superimposed on an image displayed on the other display 102. The display system 1 further includes a second display 35. For example, the second display 35 is disposed on the side opposite the aerial image 104 with respect to the optical element 101 so that the display of the second display 35 and the aerial image 104 displayed by the aerial display 10 are superimposed. In other words, the second display 35 is disposed opposite the aerial image 104 with respect to the optical element 101 included in the aerial display 10. This allows the user to view the screen of the second display 35 on an extension of the line from which the aerial image 104 is visible. Therefore, the aerial image 104 is superimposed on the image displayed on the second display 35.

[0101] The output control unit 113 may, for example, control the display on the second display 35 .

[0102] (Example of superimposing aerial image 104 on actual object) Figure 20 is an explanatory diagram showing an example of object placement when superimposing aerial image 104 on the object. The display system 1 further includes an object 36. The object 36 is positioned opposite the aerial image 104 with respect to the optical element 101 included in the aerial display 10, on an extension of the aerial image 104 that is visible, so that the object 36 and the aerial image 104 created by the aerial display 10 are superimposed.

[0103] Furthermore, the display system 1 further includes an imaging device 39 capable of capturing an image of the object 36. For example, the imaging device 39 captures an image of the object 36. The detection unit 116 detects the object 36 from the captured image. The processing unit 112 may then determine the display content of the display 102 based on the detected object 36. The output control unit 113 causes the display 102 to display the determined display content.

[0104] FIG. 21 is an explanatory diagram showing an example of superimposing an aerial image 104 on an object 36. In FIG. 21 , a painting is used as an example of the object 36. An imaging device 39 captures an image of the painting. An output control unit 113 detects the painting from the captured image. A processing unit 112 may determine the display content of the display 102 based on the detected object 36. For example, the output control unit 113 causes the display 102 to display the determined display content. Specifically, for example, the output control unit 113 may cause the display 102 to display an explanation of the painting. As a result, the explanation of the painting is superimposed on the painting.

[0105] It should be noted that the object is not limited to the object 36, but may be a plant, a living thing, or the like. The installation examples in Figures 15, 19, and 20 are merely examples. For example, an image on the floor or an actual object 36 may be superimposed on the image using the aerial display 10 from the ceiling.

[0106] As described above, in the first embodiment, the display system 1 accepts an operation on the aerial image 104 and an operation on the touch panel 103 provided over a portion of the surface of the optical element 101 as different operations. This allows various operations to be performed using the aerial display 10.

[0107] For example, an operation on the aerial image 104 is an operation by a first user, and an operation on the touch panel 103 is an operation by a second user. In this way, multiple users can perform different operations on the aerial display 10 simultaneously.

[0108] Furthermore, there is a gap between the display 102 and the touch panel 103. Therefore, the display system 1 corrects the positional relationship between the display on the display 102 and the touch panel 103. This allows the operation that the user wants to perform to be recognized more accurately.

[0109] Furthermore, when the aerial display 10 is used as a display device for a car navigation system, if the driver operates the touch panel 103 and the passenger in the front passenger seat operates the aerial image 104, the display system 1 will accept the operation on the aerial image 104 while the vehicle is moving, but will not accept the operation on the touch panel 103. This allows the passenger in the front passenger seat to operate the aerial display 10 while ensuring the safety of the driver.

[0110] The aerial display 10 may also be used as a display device in a store. In such a case, the touch panel 103 may be operated by a store clerk, and the aerial image 104 may be operated by a customer. For example, the store clerk may support the customer while the customer is operating the display.

[0111] Furthermore, the display system 1 may output different sounds when an operation is performed on the aerial image 104 and when an operation is performed on the touch panel 103 .

[0112] The display system 1 outputs different vibrations when an operation is performed on the aerial image 104 and when an operation is performed on the touch panel 103 .

[0113] Furthermore, mouse operations may be realized by operations using the touch panel 103 and operations using the aerial image 104. For example, an operation on the aerial image 104 is an operation corresponding to a specified position on the aerial image 104. For example, an operation on the touch panel 103 is at least one of an operation to instruct the display of a menu for performing an operation related to the position where an operation on the touch panel 103 is performed, an operation to instruct the display of a menu for making settings related to the position where an operation on the touch panel 103 is performed, and an operation to instruct the display of information related to the position where an operation on the touch panel 103 is performed.

[0114] Furthermore, when the aerial display 10 is installed as a display device in a store, for example, operations on the touch panel 103 may be operations related to payment among the operations related to product registration and settlement, and operations on the aerial image 104 may be operations other than operations related to payment among the operations related to product registration and settlement.

[0115] Furthermore, the display system 1 determines whether the user's hand is beyond the aerial image 104 and on the optical element 101 side, based on the position of the user's hand performing an operation on the aerial image 104. The method for detecting the hand position may be detection by a sensor 105 that detects an operation on the aerial image 104, or the hand position may be detected from an image captured by an imaging device 38 that captures an image of the user's hand performing an operation on the aerial image 104. The display system 1 accepts an operation on the aerial image 104 when the hand is not on the optical element 101 side, and does not accept an operation on the aerial image 104 when the hand is on the optical element 101 side. This makes it possible to distinguish between an operation on the aerial image 104 and an operation on the touch panel 103.

[0116] Furthermore, the display system 1 detects the position of the user's face from an image captured by the imaging device 37 of the user performing an operation on the aerial image 104. The display system 1 corrects the positional relationship between the operation on the aerial image 104 and the operation on the touch panel 103 based on the detected position of the user's face.

[0117] (Second embodiment) Next, an example of basic functions of a control device according to a second embodiment will be described with reference to the drawings. FIG. 22 is a block diagram showing an example configuration of a control device according to the second embodiment. The control device 21 includes a reception unit 211, a processing unit 212, and an output control unit 213. The reception unit 211 may have the basic function of the reception unit 111 in the first embodiment. The processing unit 212 may have the basic function of the processing unit 112 in the first embodiment. The output control unit 213 may have the basic function of the output control unit 113 in the first embodiment.

[0118] The reception unit 211 receives, as different operations, an operation on an aerial image detected by a sensor included in the non-contact display and an operation on a touch panel superimposed on a portion of the surface of an optical element included in the non-contact display.

[0119] When the processing unit 212 receives an operation on the aerial image, it performs processing according to the operation on the aerial image, and when the processing unit 212 receives an operation on the touch panel, it performs processing according to the operation on the touch panel.

[0120] The output control unit 213 controls the display on the display included in the non-contact display based on the processing result by the processing unit 212.

[0121] 23 is a flowchart showing an example of an operation of the control device 21 according to the second embodiment. The output control unit 213 controls the display (step S201). The reception unit 211 receives an operation for aerial imaging (step S202). The processing unit 212 performs processing according to the operation (step S203), and the process returns to step S201. The output control unit 213 performs processing according to the operation based on the processing result by the processing unit 212.

[0122] The receiving unit 211 receives an operation on the touch panel (step S204). The processing unit 212 performs processing according to the operation on the touch panel (step S205), and the process returns to step S201. The output control unit 213 performs processing according to the operation based on the processing result by the processing unit 212.

[0123] The flowchart may be ended as appropriate.

[0124] As described above, in the second embodiment, the control device 21 accepts operations on the aerial image and operations on the touch panel as different operations, and performs processing according to each operation. This allows various operations to be performed when using a non-contact display.

[0125] In addition, in embodiment 2, the hardware configuration when the control device 21 is realized by a computer may be the same as the hardware configuration when the control device 21 described in embodiment 1 is realized by a computer, and detailed explanation will be omitted.

[0126] This concludes the description of each embodiment. Each embodiment may be modified or used in appropriate combination.

[0127] Furthermore, in each embodiment, the display system 1 may be configured to include some of the functional units and information. The control devices 11 and 21 may be configured to include some of the functional units and information. For example, the control device 11 may be configured to include a reception unit 111, a processing unit 112, an output control unit 113, a correction unit 114, and a determination unit 115. For example, the control device 11 may be configured to include the reception unit 111, the processing unit 112, the output control unit 113, and the determination unit 115. For example, the control device 11 may be configured to include the reception unit 111, the processing unit 112, the output control unit 113, the determination unit 115, and a detection unit 116.

[0128] Furthermore, the embodiments are not limited to the examples described above and can be modified in various ways. Furthermore, the configuration of the display system 1 and the configuration of the control devices 11 and 21 in the embodiments are not particularly limited. For example, the control devices 11 and 21 may be realized by different devices for different functions or data. For example, the functional units of the control devices 11 and 21 may be configured by multiple servers and realized as a control system. For example, each functional unit of the control devices 11 and 21 may be realized by a database server including each DB (Database) and a server having each functional unit.

[0129] Furthermore, each screen is merely an example and is not particularly limited. Buttons, lists, check boxes, information display fields, input fields, etc. (not shown) may be added to each screen. Furthermore, the background color of the screen may be changed.

[0130] The process of generating information to be displayed on the display included in the aerial display 10 may be performed by the output control unit. This process may also be performed by the display included in the aerial display 10.

[0131] There are also various variations in the method of realizing each device such as the control devices 11 and 21. For example, each device may be realized by any combination of a different computer and a program for each component. Furthermore, multiple components included in each device may be realized by any combination of a single computer and a program.

[0132] Furthermore, some or all of the components of each device may be realized by circuits for specific applications. Furthermore, some or all of the components of each device may be realized by general-purpose circuits including a processor such as an FPGA (Field Programmable Gate Array). Furthermore, some or all of the components of each device may be realized by a combination of circuits for specific applications and general-purpose circuits. Furthermore, these circuits may be a single integrated circuit. Alternatively, these circuits may be divided into multiple integrated circuits. The multiple integrated circuits may be connected via a bus or the like.

[0133] Furthermore, when some or all of the components of each device are realized by a plurality of computers, circuits, etc., the plurality of computers, circuits, etc. may be centrally located or distributed.

[0134] The control method described in each embodiment is realized by execution by a control device. Also, for example, the control method is realized by execution of a prepared program by a computer such as a server or a terminal device.

[0135] The programs described in each embodiment are recorded on a computer-readable recording medium such as a HDD, SSD, flexible disk, optical disk, magneto-optical disk, or USB memory. The programs are then read from the recording medium and executed by a computer. The programs may also be distributed via a communication network NT.

[0136] The components of the display system and control device in each embodiment described above may be realized by dedicated hardware, such as a computer. Alternatively, each component may be realized by software. Alternatively, each component may be realized by a combination of hardware and software.

[0137] Although the present disclosure has been described above with reference to various embodiments, the present disclosure is not limited to the above embodiments. The configuration and details of each of the present disclosures may include embodiments to which various modifications that would be apparent to those skilled in the art are applied within the scope of the present disclosure. The present disclosure may also include embodiments in which the details described herein are appropriately combined or substituted as necessary. For example, details described using a particular embodiment may also be applied to other embodiments to the extent that no contradiction occurs. For example, although multiple operations are described in sequence in the form of a flowchart, the order of the descriptions does not limit the order in which the multiple operations are performed. Therefore, when implementing each embodiment, the order of the multiple operations may be changed as long as it does not interfere with the content.

[0138] Some or all of the above-described embodiments can be described as follows: However, some or all of the above-described embodiments are not limited to the following.

[0139] (Supplementary Note 1) A control device comprising: a receiving means for receiving, as different operations, an operation on an aerial image detected by a sensor included in a contactless display, and an operation on a touch panel provided overlaying a portion of a surface of an optical element included in the contactless display; a processing means for performing processing according to the operation on the aerial image when receiving an operation on the aerial image, and performing processing according to the operation on the touch panel when receiving an operation on the touch panel; and an output control means for controlling a display on a display included in the contactless display based on a processing result by the processing means. (Supplementary Note 2) The control device according to Supplementary Note 1, wherein the operation on the aerial image is an operation by a first user, and the operation on the touch panel is an operation by a second user. (Supplementary Note 3) The control device according to Supplementary Note 2, comprising: a detecting means for detecting a position of a face of the second user from an image of the second user captured by an imaging device; and a correcting means for correcting a position of the received operation on the touch panel based on the position of the second user's face. (Supplementary Note 4) The control device according to Supplementary Note 2 or 3, wherein, in a case where the non-contact display is installed in a vehicle, the first user is a passenger in the front seat of the vehicle, the second user is a driver of the vehicle, and the acceptance means accepts operations on the aerial image but does not accept operations on the touch panel while the vehicle is traveling. (Supplementary Note 5) The control device according to Supplementary Note 2 or 3, wherein, in a case where the non-contact display is installed in a store, the first user is a customer, and the second user is a store clerk. (Supplementary Note 6) The control device according to any of Supplementary Notes 1 to 5, wherein the output control means outputs different sounds when an operation is performed on the aerial image and when an operation is performed on the touch panel. (Supplementary Note 7) The control device according to any of Supplementary Notes 1 to 6, wherein the output control means outputs different vibrations when an operation is performed on the aerial image and when an operation is performed on the touch panel.(Supplementary Note 8) The control device according to any one of Supplementary Notes 1 to 7, wherein the operation on the aerial image is an operation corresponding to a specified position on the aerial image, and the operation on the touch panel is at least one of an operation to instruct display of a menu for performing an operation related to the position where the operation on the touch panel was performed, an operation to instruct display of a menu for performing a setting related to the position where the operation on the touch panel was performed, and an operation to instruct display of information related to the position where the operation on the touch panel was performed. (Supplementary Note 9) The control device according to any one of Supplementary Notes 1 to 8, wherein, when the contactless display is installed in a store, the operation on the touch panel is a payment-related operation among operations related to product registration and settlement at the store, and the operation on the aerial image is an operation other than the payment-related operation among operations related to the product registration and the settlement. (Supplementary Note 10) The control device according to any one of Supplementary Notes 1 to 9, further comprising: a determination means for determining, based on the position of the hand of the user performing an operation on the aerial image, whether the hand of the user is located on the optical element side beyond the aerial image, wherein the acceptance means accepts the operation on the aerial image when the hand is not on the optical element side, and does not accept the operation on the aerial image when the hand is on the optical element side. (Supplementary Note 11) The control device according to Supplementary Note 10, further comprising: a detection means for detecting the hand of the user from an image captured by an imaging device of the hand of the user performing an operation on the aerial image, wherein the determination means determines, based on the detected position of the user's hand, whether the hand of the user is located on the optical element side beyond the aerial image.(Supplementary Note 13) A control device according to any one of Supplementary Notes 1 to 12, comprising: a detection means for detecting the position of a user's face from an image captured by an imaging device of the user performing an operation on the aerial image; and a correction means for correcting the position of the received operation on the aerial image based on the detected position of the user's face. (Supplementary Note 14) A display system comprising: a non-contact display; and a control device, wherein the non-contact display comprises: a touch panel; an optical element; a display; and a sensor, wherein the touch panel is arranged to overlap a portion of a surface of the optical element, the touch panel detects operations on the panel itself, and the sensor detects operations on an aerial image formed by the display and the optical element, and the control device comprises: a receiving means for receiving an operation on the aerial image and an operation on the touch panel as different operations, a processing means for, when an operation on the aerial image is received, performing processing in accordance with the operation on the aerial image, and when an operation on the touch panel is received, performing processing in accordance with the operation on the touch panel, and an output control means for controlling display on the display based on a processing result by the processing means. (Supplementary Note 15) The display system according to Supplementary Note 14, further comprising: a determination means for determining whether the user's hand is beyond the aerial image on the side of the optical element based on the position of the hand of the user performing an operation on the aerial image, wherein the acceptance means accepts the operation on the aerial image when the hand is not on the side of the optical element, and does not accept the operation on the aerial image when the hand is on the side of the optical element. (Supplementary Note 16) The display system according to Supplementary Note 14 or Supplementary Note 15, further comprising: a mirror, wherein the mirror is arranged on a surface of the optical element opposite the aerial image so that an image reflected in the mirror and the aerial image are superimposed on each other.(Supplementary Note 17) The display system according to Supplementary Note 16, further comprising an imaging device, wherein the imaging device captures an image of a user performing an operation on the aerial image, and the control device comprises: detection means for detecting a movement of the user from an image captured by the imaging device, and the output control means controls display on the display based on the movement of the user. (Supplementary Note 18) The display system according to Supplementary Note 16 or Supplementary Note 17, wherein the acceptance means accepts an operation on the touch panel as an operation on the user reflected in the mirror. (Supplementary Note 19) The display system according to Supplementary Note 14 or Supplementary Note 15, further comprising: a second display, wherein the second display is arranged on a surface of the optical element opposite the aerial image so that the display of the second display and the aerial image are superimposed. (Supplementary Note 20) The display system according to Supplementary Note 14 or Supplementary Note 15, further comprising: an object, wherein the object is arranged on a surface of the optical element opposite the aerial image so that the object and the aerial image are superimposed. (Supplementary Note 21) The display system according to Supplementary Note 20, further comprising an imaging device that images the object, wherein the control device comprises: a detection means that detects the object from an image captured by the imaging device, wherein the processing means determines what to display on the display based on the detected object, and wherein the output control means causes the determined display content to be displayed on the display.(Supplementary Note 22) A control device comprising: a reception means for receiving an operation on an aerial image detected by a sensor included in a non-contact display installed in a vehicle, and an operation on a touch panel superimposed on a portion of the surface of an optical element included in the non-contact display; a processing means for, when receiving an operation on the aerial image, performing processing according to the operation on the aerial image, and when receiving an operation on the touch panel, performing processing according to the operation on the touch panel; an output control means for controlling display on a display included in the non-contact display based on a processing result by the processing means; the operation on the aerial image is an operation by a passenger in the front seat of the vehicle, and the operation on the touch panel is an operation by the driver of the vehicle; and the reception means for receiving an operation on the aerial image while the vehicle is traveling, but not for receiving an operation on the touch panel. (Supplementary Note 23) A control method comprising: accepting an operation on an aerial image detected by a sensor included in a non-contact display and an operation on a touch panel superimposed on a portion of the surface of an optical element included in the non-contact display as different operations; when an operation on the aerial image is accepted, performing processing in accordance with the operation on the aerial image; when an operation on the touch panel is accepted, performing processing in accordance with the operation on the touch panel; and controlling display on a display included in the non-contact display based on the processing results.(Supplementary Note 24) A control method comprising: receiving an operation on an aerial image detected by a sensor included in a non-contact display installed in a vehicle and an operation on a touch panel superimposed on a portion of a surface of an optical element included in the non-contact display; when an operation on the aerial image is received, performing processing according to the operation on the aerial image; and when an operation on the touch panel is received, performing processing according to the operation on the touch panel; and controlling display on a display included in the non-contact display based on a processing result; the operation on the aerial image is an operation by a passenger in the front seat of the vehicle; and the operation on the touch panel is an operation by the driver of the vehicle; and in the receiving, while the vehicle is traveling, receiving an operation on the aerial image and not receiving an operation on the touch panel. (Supplementary Note 25) A non-transitory computer-readable recording medium having recorded thereon a program for causing a computer to execute the following processes: accept an operation on an aerial image detected by a sensor included in a non-contact display and an operation on a touch panel provided overlaid on a portion of the surface of an optical element included in the non-contact display as different operations; when an operation on the aerial image is accepted, perform a process according to the operation on the aerial image; when an operation on the touch panel is accepted, perform a process according to the operation on the touch panel; and control display on a display included in the non-contact display based on the processing results.(Supplementary Note 26) A non-transitory computer-readable recording medium having recorded thereon a program that causes a computer to execute the following processing: accept an operation on an aerial image detected by a sensor included in a non-contact display installed in a vehicle, and an operation on a touch panel superimposed on a portion of the surface of an optical element included in the non-contact display; when an operation on the aerial image is accepted, perform a process according to the operation on the aerial image; when an operation on the touch panel is accepted, perform a process according to the operation on the touch panel; and control a display on a display included in the non-contact display based on a processing result; the operation on the aerial image is an operation by a passenger in the front seat of the vehicle, and the operation on the touch panel is an operation by the driver of the vehicle, and the accepting processing accepts an operation on the aerial image but does not accept an operation on the touch panel while the vehicle is traveling. (Supplementary Note 27) A program that causes a computer to execute the following processing: accept an operation on an aerial image detected by a sensor included in a non-contact display and an operation on a touch panel superimposed on a portion of the surface of an optical element included in the non-contact display as different operations; when an operation on the aerial image is accepted, perform a process according to the operation on the aerial image; when an operation on the touch panel is accepted, perform a process according to the operation on the touch panel; and control the display on a display included in the non-contact display based on the processing results.(Supplementary Note 28) A program that causes a computer to execute processing to receive operations on an aerial image detected by a sensor included in a non-contact display installed in a vehicle and operations on a touch panel superimposed on a portion of the surface of an optical element included in the non-contact display, when an operation on the aerial image is received, perform processing in accordance with the operation on the aerial image, and when an operation on the touch panel is received, perform processing in accordance with the operation on the touch panel, and control display on a display included in the non-contact display based on a processing result, wherein the operation on the aerial image is an operation by a passenger in the front seat of the vehicle, and the operation on the touch panel is an operation by the driver of the vehicle, and the receiving processing receives operations on the aerial image but does not accept operations on the touch panel while the vehicle is traveling.

[0140] REFERENCE SIGNS LIST 1 Display system 10 Aerial display 11, 21 Control device 31 Imaging device 32 Reading device 33 Reading device 34 Mirror 35 Second display 36 Object 37 Imaging device 38 Imaging device 39 Imaging device 80 Computer 101 Optical element 102 Display 103 Touch panel 104 Aerial image formation 105 Sensor 111, 211 Reception unit 112, 212 Processing unit 113, 213 Output control unit 114 Correction unit 115 Determination unit 116 Detection unit 801 Processor 802 ROM 803 RAM 804 Storage device 805 Communication interface 806 Input / output interface 807 Bus NT Communication network

Claims

1. a receiving means for receiving, as different operations, an operation on an aerial image detected by a sensor included in the non-contact display and an operation on a touch panel provided over a portion of a surface of an optical element included in the non-contact display; a processing means for performing a process corresponding to the operation on the aerial image when an operation on the aerial image is received, and for performing a process corresponding to the operation on the touch panel when an operation on the touch panel is received; an output control means for controlling display on a display included in the non-contact display based on a processing result by the processing means; A control device comprising:

2. the operation on the aerial image is an operation by a first user, the operation on the touch panel is an operation by a second user; The control device according to claim 1 .

3. a detection means for detecting a position of a face of the second user from an image of the second user captured by an imaging device; a correction means for correcting a position of the received operation on the touch panel based on a position of the face of the second user; The control device according to claim 2 , comprising:

4. When the non-contact display is installed in a vehicle, the first user is a passenger in the front seat of the vehicle, the second user is a driver of the vehicle; the accepting unit accepts an operation on the aerial image while the vehicle is traveling, but does not accept an operation on the touch panel. The control device according to claim 2 or 3.

5. When the non-contact display is installed in a store, the first user is a customer; The second user is a store clerk. The control device according to claim 2 or 3.

6. the output control means outputs different sounds depending on whether an operation is performed on the aerial image or on the touch panel. The control device according to claim 1 .

7. the output control means outputs different vibrations when an operation is performed on the aerial image and when an operation is performed on the touch panel. The control device according to claim 1 .

8. A contactless display; a control device; Equipped with The non-contact display comprises: Touch panel and an optical element; The display and Sensor and Equipped with the touch panel is disposed over a portion of a surface of the optical element, The touch panel detects an operation on the touch panel itself, The sensor detects an operation on the display and an aerial image formed by the optical element; The control device a receiving unit that receives an operation on the aerial image and an operation on the touch panel as different operations; a processing means for performing a process corresponding to the operation on the aerial image when an operation on the aerial image is received, and for performing a process corresponding to the operation on the touch panel when an operation on the touch panel is received; an output control means for controlling display on the display based on the processing result by the processing means; Equipped with Display system.

9. an operation on an aerial image detected by a sensor included in the non-contact display and an operation on a touch panel provided over a portion of a surface of an optical element included in the non-contact display are accepted as different operations, respectively; when an operation on the aerial image is received, a process is performed in accordance with the operation on the aerial image; when an operation on the touch panel is received, a process is performed in accordance with the operation on the touch panel; Controlling display on a display included in the non-contact display based on the processing result. Control method.

10. On the computer, an operation on an aerial image detected by a sensor included in the non-contact display and an operation on a touch panel provided over a portion of a surface of an optical element included in the non-contact display are accepted as different operations, respectively; when an operation on the aerial image is received, a process is performed in accordance with the operation on the aerial image; when an operation on the touch panel is received, a process is performed in accordance with the operation on the touch panel; Controlling display on a display included in the non-contact display based on the processing result. A program that executes a process.