Focus change system, communication terminal, server, focus change method, and focus change program

Abstract

The objective is to provide a focus adjustment system that enables the automatic adjustment of the focus of a variable focus lens according to the object in the real world and the virtual image. [Solution] The focus changing system 1 comprises a communication terminal 3 having a terminal processing unit 34 that transmits eye movement information to a server 4 via a terminal communication unit 31, and a server 4 having a server storage unit 42 that stores a trained model M that has been trained to output focus information when eye movement information is input, and a server processing unit 43 that transmits the output focus information, obtained by inputting the received eye movement information into the trained model M, to the communication terminal 3 via the server communication unit 41, and the terminal processing unit 34 is characterized in that it changes the focus of the variable focus lens 22LN to either an object P1 in real space or a virtual image P2 based on the focus information.

Description

[Technical Field]

[0001] The present invention relates to a focus changing system, a communication terminal, a server, a focus changing method, and a focus changing program. [Background Art]

[0002] Conventionally, eyeglasses using a variable-focus lens whose focal length can be changed are known. For example, Patent Document 1 discloses variable-focus eyeglasses that include, for each variable-focus lens, a driving unit that changes the focal length of the variable-focus lens, a control unit that controls the driving unit, and a predetermined switch.

[0003] On the other hand, AR (Augmented Reality) technology is known as a technology that superimposes a virtual image on an actual landscape through a lens to virtually expand the world in front of the user's eyes. For example, Patent Document 2 discloses a captured image apparatus including a driving unit that adjusts the orientations of a first imaging unit and a second imaging unit according to a line of sight detected by a line-of-sight detection unit. In recent years, AR is also referred to as MR (Mixed Reality), SR (Substitutional Reality), XR (Cross reality), and the like. [Prior Art Documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Unexamined Patent Application Publication No. 2014-038302 [Patent Document 2] Japanese Unexamined Patent Application Publication No. 2019-004274 [Summary of the Invention] [Problems to be Solved by the Invention]

[0005] Conventionally, no study has been conducted on automatically changing the focus of a variable-focus lens according to an object in real space and a virtual image.

[0006] The present invention aims to provide a focus change system, a communication terminal, a server, a focus change method, and a focus change program that enable the automatic change of the focus of a variable focus lens according to an object in real space and a virtual image. [Means for solving the problem]

[0007] The present invention relates to a focus-changing system comprising eyeglasses, a server, and a communication terminal capable of communicating with the eyeglasses and the server, wherein the eyeglasses include a variable focus lens, a virtual image projection unit, a projection lens capable of projecting a virtual image output by the virtual image projection unit, and a detection unit that detects the eye movements of a user wearing the eyeglasses and generates eye movement information; the communication terminal includes a terminal communication unit that receives eye movement information and a terminal processing unit that transmits eye movement information to the server via the terminal communication unit; the server includes a server communication unit that receives eye movement information, a server storage unit that stores a trained model that has been trained to output focus information when eye movement information is input, and a server processing unit that inputs the received eye movement information into the trained model and transmits the outputted focus information to the communication terminal via the server communication unit; and the terminal processing unit is characterized in that it changes the focus of the variable focus lens to either an object in real space or a virtual image based on the focus information.

[0008] Furthermore, in the focus change system according to the present invention, the eyeglasses or communication terminal further has an operating unit that allows the user to change the focus of the variable focus lens, and the terminal processing unit generates feedback information when the focus of the variable focus lens is changed by the operating unit after the focus of the variable focus lens has been changed based on the focus information, and transmits the feedback information to the server via the terminal communication unit, and the server processing unit updates the trained model based on the feedback information received via the server communication unit.

[0009] Furthermore, in the focus change system according to the present invention, it is preferable that the variable focus lens and the projection lens are integrated.

[0010] The present invention relates to a communication terminal capable of communicating with eyeglasses having a variable focus lens, a virtual image projection unit, and a projection lens capable of projecting a virtual image output by the virtual image projection unit, and is characterized by comprising: a terminal communication unit that receives eye movement information from a detection unit that detects the eye movements of a user wearing the eyeglasses and generates eye movement information; a terminal storage unit that stores a trained model that has been trained to output focus information when eye movement information is input; and a terminal processing unit that changes the focus of the variable focus lens to either an object in the real space or a virtual image based on the focus information output by inputting the received eye movement information to the trained model.

[0011] The present invention relates to a server that can communicate with eyeglasses having a variable focus lens, a virtual image projection unit, and a projection lens capable of projecting a virtual image output by the virtual image projection unit, and transmits predetermined instructions to a processor that processes the operation of the variable focus lens, and is characterized by comprising: a server communication unit that receives eye movement information from a detection unit that detects the eye movements of a user wearing the eyeglasses and generates eye movement information; a server storage unit that stores a trained model that has been trained to output focus information when eye movement information is input; and a server processing unit that inputs the received eye movement information into the trained model and transmits the output focus information to the processor via the server communication unit.

[0012] The present invention relates to a method for changing the focus of a variable-focus lens in eyeglasses having a variable-focus lens, a virtual image projection unit, and a projection lens capable of projecting a virtual image output by the virtual image projection unit, characterized in that the method involves detecting the eye movements of a user wearing the eyeglasses and generating eye movement information, inputting the detected eye movement information into a trained model that has been trained to output focus information when eye movement information is input, and changing the focus of the variable-focus lens to either an object in real space or a virtual image based on the focus information output from the trained model.

[0013] The present invention relates to a focus changing program for eyeglasses having a variable focus lens, a virtual image projection unit, and a projection lens capable of projecting a virtual image output by the virtual image projection unit, characterized in that a processor that processes the operation of the variable focus lens receives eye movement information from a detection unit that detects the eye movements of a user wearing the eyeglasses and generates eye movement information, and when eye movement information is input, a trained model that has been trained to output focus information inputs the eye movement information detected by the detection unit, and based on the focus information output from the trained model, the focus of the variable focus lens is changed to either an object in the real space or a virtual image. [Effects of the Invention]

[0014] The present invention provides a focus-changing system, communication terminal, server, focus-changing method, and focus-changing program that enable the automatic change of focus of a variable-focus lens in accordance with an object in real space and a virtual image. [Brief explanation of the drawing]

[0015] [Figure 1] This diagram shows an overview of the focus change system according to Embodiment 1. [Figure 2] This is a functional block diagram of the focus change system according to Embodiment 1. [Figure 3] This diagram shows an overview of eye movements when focusing on an object in real space or a virtual image. [Figure 4] This figure shows an example of eye movement information input into a trained model. [Figure 5] This diagram shows the flow of the learning process. [Figure 6] This diagram shows the sequence of focus change processing. [Figure 7] This figure shows the sequence of the update process related to Modification Example 1. [Figure 8] (a) is a diagram showing an overview of the focus change system according to Embodiment 2, and (b) is a functional block diagram of the communication terminal according to Embodiment 2. [Figure 9]It is a diagram illustrating an outline of a focus changing system according to Embodiment 3. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted, however, that the technical scope of the present invention is not limited to the embodiments, and extends to the invention described in the claims and equivalents thereof.

[0017] (Embodiment 1) FIG. 1 is a diagram illustrating an outline of a focus changing system 1 according to Embodiment 1, and FIG. 2 is a functional block diagram of the focus changing system 1 according to Embodiment 1. The focus changing system 1 includes eyeglasses 2, a communication terminal 3, a server 4, and the like. The eyeglasses 2 and the communication terminal 3 can communicate with each other via wireless communication such as BLUETOOTH (registered trademark) and WiFi (registered trademark). The eyeglasses 2 and the communication terminal 3 may communicate via wired communication such as a wire harness. The communication terminal 3 and the server 4 can communicate with each other via a network N, which is the Internet or an intranet.

[0018] The eyeglasses 2 are variable-focus eyeglasses that enable changing the focal point of lenses. The eyeglasses 2 are also AR glasses that superimpose and display a virtual image on real space through the lenses. The virtual image is an image representing digital information, and includes characters and photographs.

[0019] The eyeglasses 2 include a lens unit LN, a focus control unit 21, a virtual image projection unit 22, an eyeglass communication unit 23, an eyeglass storage unit 24, an eyeglass operation unit 25, a detection unit 26, an eyeglass processing unit 27, and the like. These respective units are connected via a bus B2. These respective units may also be individually connected by a wire harness or the like. These respective units are driven by electric power supplied from a power supply unit (not shown). The power supply unit is, for example, a coin-type lithium secondary battery. Note that the electric power supplied to each unit may be supplied from a single power supply unit, or may be supplied from individual power supply units corresponding to each unit.

[0020] The lens unit LN is configured to change the focus of the lens and project a virtual image. The lens unit LN is positioned opposite the eye position of the user wearing the glasses 2. The lens unit LN includes a variable focus lens 21LN and a projection lens 22LN, etc.

[0021] The variable focus lens 21LN is a liquid lens that allows the focus of the lens to be changed by the applied voltage. Changing the focus of the lens means changing the focal length of the lens. The variable focus lens 21LN contains two types of liquids: a conductive liquid and a non-conductive (insulating) liquid. The two types of liquids form an interface without mixing. The interface is curved in a spherical shape. The degree of curvature of the interface, i.e., the curvature, determines the focal length of the liquid lens. Electrodes are arranged around the two types of liquids. The relationship between the two types of liquids and the electrodes will be described later. The variable focus lens 21LN is positioned opposite the eye position of the user wearing the glasses 2, and between the user's eye position and the position of the projection lens 22LN.

[0022] The projection lens 22LN is a lens capable of projecting the virtual image output by the virtual image projection unit 22. The projection lens 22LN is a waveguide-type AR glasses. The waveguide method is a method of guiding the virtual image output by the virtual image projection unit 22 to the user's eyes using a waveguide glass, also called a light guide plate. The projection lens 22LN is positioned opposite the eye position of the user wearing the glasses 2 and is fixed by the rim of the glasses 2.

[0023] The focus control unit 21 is configured to control the focal length of the variable focus lens 21LN. The focus control unit 21 is connected to the variable focus lens 21LN and applies a voltage to the electrodes located inside the variable focus lens 21LN. When the voltage applied to the electrodes (applied voltage) changes, the surface tension between the two types of liquids contained inside the variable focus lens 21LN changes. Here, the conductive liquid is easily affected by voltage. Therefore, the surface tension between the two types of liquids changes by increasing the voltage. This changes the shape (curvature) of the liquids. In other words, the focal length of the liquid lens changes. The focus control unit 21 controls the focal length of the variable focus lens 21LN by changing the voltage applied to the variable focus lens 21LN.

[0024] The virtual image projection unit 22 is configured to project a virtual image onto the projection lens 22LN. The virtual image projection unit 22 has a laser light source or an LED (Light Emitting Diode) light source, and emits light from these light sources to constitute the virtual image. The virtual image projection unit 22 generates a virtual image based on a signal supplied from the glasses processing unit 27 and emits light that constitutes the virtual image. The emitted light is projected onto the projection lens 22LN. This allows the virtual image projection unit 22 to allow the user wearing the glasses 2 to see a virtual image superimposed onto the real space via the projection lens 22LN. The virtual image projection unit 22 is positioned near the rim above the eye position of the user wearing the glasses 2.

[0025] The eyeglasses communication unit 23 is configured to enable the eyeglasses 2 to communicate with other devices, and includes an antenna for transmitting and receiving wireless signals, and a wireless communication circuit that conforms to a communication protocol such as wireless LAN. The eyeglasses communication unit 23 wirelessly connects with the terminal communication unit 31 in accordance with a communication standard such as wireless LAN. The eyeglasses communication unit 23 receives data from other devices and supplies it to the eyeglasses processing unit 27, and also transmits data supplied from the eyeglasses processing unit 27 to other devices.

[0026] The eyeglass memory unit 24 is configured for storing programs and data, and includes, for example, semiconductor memory. The eyeglass memory unit 24 stores programs such as operating system programs, driver programs, and application programs used for processing by the eyeglass processing unit 27. Programs are installed in the eyeglass memory unit 24 from computer-readable and non-temporary portable storage media such as CD-ROM (Compact Disc Read Only Memory) or DVD-ROM (Digital Versatile Disc Read Only Memory). Programs may also be installed in the eyeglass memory unit 24 from an external server via the eyeglass communication unit 23.

[0027] The eyeglasses control unit 25 is configured to receive user operations on the eyeglasses 2 and consists of one or more push switches. The eyeglasses control unit 25 receives operations to change the focus of the variable focus lens 21LN. The eyeglasses control unit 25 generates a signal corresponding to the user's operation and supplies it to the eyeglasses processing unit 27. The eyeglasses control unit 25 is an operation unit that allows the focus of the variable focus lens 21LN to be changed by user operation.

[0028] The detection unit 26 is an EMG (Electromyography) sensor that detects electromagnetic signals generated by the movement of the ciliary muscle associated with the user's eye adjustment and generates eye movement information. The eye movement information includes information generated by directly detecting eye movement and information generated by detecting the movement of facial muscles, including the ciliary muscle. In other words, detecting eye movement includes directly detecting eye movement and indirectly detecting eye movement by detecting the movement of facial muscles, etc. The EMG sensor has an antenna, an amplifier, and an A / D conversion circuit and is positioned near the rim of the eyeglasses 2. The antenna is a single-loop antenna and detects electromagnetic signals in a predetermined frequency band generated by the movement of the ciliary muscle. The amplifier amplifies the signal detected by the antenna to a predetermined amplitude. The A / D conversion circuit digitally converts the signal amplified by the amplifier to generate eye movement information. Note that the above-described configuration of the EMG sensor is an example, and other configurations of EMG sensors may be used, and other sensors may be used to detect electromagnetic signals generated by the movement of the ciliary muscle.

[0029] The glasses processing unit 27 is configured to comprehensively control the operation of the glasses 2 and comprises one or more processors and their peripheral circuits. The glasses processing unit 27 may include, for example, a CPU (Central Processing Unit). The glasses processing unit 27 may also include a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), etc. The glasses processing unit 27 controls the operation of each component and executes various processes so that various processes of the glasses 2 are executed in the appropriate procedure based on the program stored in the glasses storage unit 24.

[0030] The communication terminal 3 is a smartphone. The communication terminal 3 includes a terminal communication unit 31, a terminal storage unit 32, a terminal operation unit 33, and a terminal processing unit 34, etc. These components are connected via bus B3.

[0031] The terminal communication unit 31 is configured to enable the communication terminal 3 to communicate with other devices, and includes an antenna for transmitting and receiving wireless signals, and a wireless communication circuit that conforms to a communication protocol such as wireless LAN. The terminal communication unit 31 wirelessly connects with the glasses communication unit 23 and the network N in accordance with a communication standard such as wireless LAN. The terminal communication unit 31 receives data from other devices and supplies it to the terminal processing unit 34, and also transmits data supplied from the terminal processing unit 34 to other devices.

[0032] The terminal storage unit 32 is configured for storing programs and data, and includes, for example, semiconductor memory. The terminal storage unit 32 stores programs such as operating system programs, driver programs, and application programs used for processing by the terminal processing unit 34. Programs are installed into the terminal storage unit 32 from computer-readable, non-temporary, portable storage media such as CD-ROMs and DVD-ROMs. Programs may also be installed into the terminal storage unit 32 from an external server via the terminal communication unit 31.

[0033] The terminal operation unit 33 is configured to receive user operations on the communication terminal 3 and is a touch panel having a screen such as a liquid crystal display or an organic EL (Electro-Luminescence) display. In other words, the terminal operation unit 33 also functions as a display unit. The terminal operation unit 33 has an output interface circuit that outputs image data to the display and an input interface circuit that acquires signals from the touch panel. The terminal operation unit 33 generates signals in response to user operations and supplies them to the terminal processing unit 34. The terminal operation unit 33 generates and displays images based on the signals supplied from the terminal processing unit 34. The terminal operation unit 33 is an operation unit that can change the focus of the variable focus lens 21LN by user operation.

[0034] The terminal processing unit 34 is configured to comprehensively control the operation of the communication terminal 3 and comprises one or more processors and their peripheral circuits. The terminal processing unit 34 comprises, for example, a CPU. The terminal processing unit 34 may also comprise a GPU, DSP, LSI, ASIC, FPGA, etc. The terminal processing unit 34 controls the operation of each component and executes various processes so that the various processes of the communication terminal 3 are executed in appropriate procedures based on the program stored in the terminal storage unit 32.

[0035] Server 4 is an information processing device. Server 4 includes a server communication unit 41, a server storage unit 42, and a server processing unit 43, etc. Server 4 may also have an operation unit and a display unit. These components are connected via a bus (not shown).

[0036] The server communication unit 41 is configured to enable the server 4 to communicate with other devices, and includes an antenna for transmitting and receiving wireless signals, and a wireless communication circuit that conforms to a communication protocol such as wireless LAN. The server communication unit 41 wirelessly connects to the network N in accordance with a communication standard such as wireless LAN. The server communication unit 41 receives data from other devices and supplies it to the server processing unit 43, and also transmits data supplied from the server processing unit 43 to other devices.

[0037] The server storage unit 42 is configured for storing programs and data, and includes, for example, semiconductor memory. The server storage unit 42 stores programs such as operating system programs, driver programs, and application programs used for processing by the server processing unit 43. Programs are installed into the server storage unit 42 from computer-readable, non-temporary portable storage media such as CD-ROMs and DVD-ROMs. Programs may also be installed into the server storage unit 42 from an external server via the server communication unit 41.

[0038] The server memory unit 42 stores a trained model M that, when eye movement information is input, outputs focus information to determine the focal length of the variable focus lens 21LN to either an object in real space or a virtual image. The trained model M is, for example, a convolutional neural network (CNN) and is pre-trained by supervised learning. A convolutional neural network (CNN) is a deep neural network composed of convolutional layers that extract local features from input data, pooling layers that compress and aggregate information, and fully connected layers that perform the final classification. The trained model M is not limited to a convolutional neural network (CNN), but may also be a classification model using various other algorithms such as support vector machines, random forests, logistic circuits, or k-nearest neighbors.

[0039] The server processing unit 43 is configured to comprehensively control the operation of the server 4 and comprises one or more processors and their peripheral circuits. For example, the server processing unit 43 includes a CPU. The server processing unit 43 may also include a GPU, DSP, LSI, ASIC, FPGA, etc. The server processing unit 43 controls the operation of each component and executes various processes so that the various processes of the server 4 are executed in appropriate procedures based on the programs stored in the server storage unit 42.

[0040] Figure 3 shows an overview of eye movements when focusing on an object P1 in real space or a virtual image P2. The human eye focuses on an object through the movement of the ciliary muscle C. When a person tries to look at an object P1 in real space, the left and right lines of sight S1 move to coincide at the point of gaze (object P1), and the ciliary muscle C surrounding the lens L tenses and contracts. As a result, the lens L thickens as it is pressed by the ciliary muscle C, and the focal length of the lens L becomes relatively shorter compared to when looking at a virtual image P2. On the other hand, when a person tries to look at a virtual image P2, the virtual image P2 is usually projected onto the left and right projection lenses 22LN respectively, so the eyeballs move so that the left and right lines of sight S2 are parallel to each other. Therefore, the ciliary muscle C moves as if the virtual image P2 is located further away than the object P1 in real space. In this case, the ciliary muscle C surrounding the lens L relaxes and stretches. As a result, the lens L is pulled thinner by the ciliary muscle C, and the focal length of the lens L becomes relatively longer compared to when viewing an object P1 in real space. In other words, detecting the movement of the ciliary muscle C makes it possible to determine whether a person is focusing on an object P1 in real space or on a virtual image P2. Furthermore, detecting the movement of the ciliary muscle C makes it possible to estimate the focal length of the lens L.

[0041] Figure 4 shows an example of eye movement information generated by the detection unit 26. In Figure 4, the horizontal axis represents time (in milliseconds) and the vertical axis represents voltage (in volts). Curve 51 in Figure 4 is an example of eye movement information detected by the EMG sensor when the user looks at an object P1 in real space. Around point 501 on curve 51, the user is focusing on the object P1 in real space. Curve 52 in Figure 4 is an example of eye movement information detected by the EMG sensor when the user looks at a virtual image P2. Around point 502 on curve 52, the user is focusing on the virtual image P2. In the example shown in Figure 4, the position where the user focused when making eye movements is predetermined for both the object P1 in real space and the virtual image P2. From the position where the user focused when making eye movements, the user's focus information corresponding to that position is calculated. Therefore, it is possible to generate a trained model using a set of multiple eye movement information and each focus information when the eye movement information is detected as training data.

[0042] Figure 5 shows the flow of the learning process. The learning process is the process of pre-training a machine learning model to generate a trained model. The learning process is realized by the server processing unit 43 cooperating with other components based on a program stored in the server memory unit 42.

[0043] First, the server processing unit 43 acquires eye movement information (step S101). The server processing unit 43 acquires multiple eye movement information by reading multiple eye movement information stored in the server storage unit 42. The eye movement information is stored in the server storage unit 42 in advance by detecting the eye movements of the user wearing the glasses 2 using an EMG sensor. The server processing unit 43 may also acquire multiple eye movement information via the server communication unit 41.

[0044] Next, the server processing unit 43 acquires focus information corresponding to the position where the eyeball was focused during the eye movement (step S102). The server processing unit 43 acquires focus information by reading the focus information stored in the server storage unit 42. The focus information is stored in the server storage unit 42 in advance. The server processing unit 43 may also acquire focus information via the server communication unit 41.

[0045] Next, the server processing unit 43 pre-trains a machine learning model using the eye movement information and focus information datasets as training data (step S103). The server processing unit 43 trains the machine learning model so that when eye movement information is input, focus information is output. The machine learning model is the convolutional neural network (CNN) described above. In this case, eye movement information is input as input data, and the result output from the fully connected layer becomes the focus information. Note that the machine learning model may also be a classification model using various other algorithms.

[0046] Next, the server processing unit 43 stores the generated trained model in the server storage unit 42 (step S104). This completes the training process.

[0047] Alternatively, instead of using the time-varying EMG voltage information shown in Figure 4 as eye movement information, the integral, derivative, or average value of the time-varying EMG voltage, as shown in Figure 4, may be used as eye movement information.

[0048] Figure 6 shows the sequence of focus change processing. Focus change processing is mainly realized by the terminal processing unit 34 working in cooperation with other components, based on a program stored in the terminal storage unit 32.

[0049] First, the detection unit 26 of the glasses 2 detects the eye movements of the user wearing the glasses 2 (step S201). The detection unit 26 detects eye movements that focus on either an object P1 in the real space or a virtual image P2. The detection unit 26 generates eye movement information and supplies it to the glasses processing unit 27.

[0050] Next, the glasses processing unit 27 transmits the eye movement information supplied by the detection unit 26 to the communication terminal 3 via the glasses communication unit 23 (step S202). The glasses processing unit 27 transmits the eye movement information to the communication terminal 3 each time eye movement information is supplied from the detection unit 26.

[0051] Next, the terminal processing unit 34 of the communication terminal 3 receives eye movement information via the terminal communication unit 31 (step S203).

[0052] Next, the terminal processing unit 34 transmits eye movement information to the server 4 via the terminal communication unit 31 (step S204). The terminal processing unit 34 transmits eye movement information to the server 4 each time it receives glasses movement information via the terminal communication unit 31.

[0053] Next, the server processing unit 43 of server 4 receives eye movement information via the server communication unit 41 (step S205).

[0054] Next, the server processing unit 43 inputs eye movement information to the trained model M (step S206). The server processing unit 43 inputs eye movement information to the trained model M each time it receives glasses movement information via the server communication unit 41.

[0055] Next, the server processing unit 43 acquires focus information (step S207). The focus information is output by inputting the eye movement information into the trained model M.

[0056] Next, the server processing unit 43 transmits focus information to the communication terminal 3 via the server communication unit 41 (step S208). The terminal processing unit 34 of the communication terminal 3 receives the focus information via the terminal communication unit 31 (step S209).

[0057] Next, the terminal processing unit 34 changes the focus of the variable focus lens 21LN to either an object P1 in real space or a virtual image P2 based on the focus information (step S210). The terminal processing unit 34 refers to the focal length included in the focus information and generates a control instruction to change the focal length. The terminal processing unit 34 transmits the control instruction to the glasses 2 via the terminal communication unit 31.

[0058] Next, when the glasses processing unit 27 of the glasses 2 receives a control instruction via the glasses communication unit 23, it changes the focus of the variable focus lens 21LN according to the control instruction (step S211). The glasses processing unit 27 supplies the control instruction to the focus control unit 21. Based on the control instruction, the focus control unit 21 changes the voltage applied to the variable focus lens 21LN. The focus control unit 21 applies a voltage so that the focal length of the variable focus lens 21LN matches either the object P1 in the real space or the virtual image P2. As a result, the focus of the variable focus lens 21LN is changed to either the object P1 in the real space or the virtual image P2.

[0059] This concludes the focus change process.

[0060] As explained above, in the focus change system 1, the terminal processing unit 34 inputs eye movement information into a learned model M and, based on the output focus information, changes the focus of the variable focus lens 21LN to either an object P1 in the real space or a virtual image P2. The focus information is information that determines the focal length of the variable focus lens 21LN to either the object P1 in the real space or the virtual image P2. This makes it possible for the focus change system 1 to automatically change the focus of the variable focus lens 21LN to either the object P1 in the real space or the virtual image P2 in response to the user's eye movements.

[0061] Preferably, the detection unit 26 detects the eye movements of the user wearing the glasses 2 and generates eye movement information. In conventional variable-focus glasses, the focus was changed based on the measurement results of a measuring instrument. The measuring instrument irradiates the outside of the variable-focus glasses with infrared rays, light waves, ultrasound, etc., and measures the distance based on the reflection results. However, the infrared rays, etc., irradiated externally by the measuring instrument are affected by external environmental factors such as weather, temperature, humidity, and specular reflection. Therefore, when changing the focus based on the measurement results of a measuring instrument that irradiates the outside with infrared rays, etc., the appropriate focus may not be achieved.

[0062] In the focus adjustment system 1, the detection unit 26 detects the eye movements of the user wearing the glasses 2 and generates eye movement information. This makes it possible for the focus adjustment system 1 to be less affected by the external environment. This also contributes to making the focus adjustment of the glasses 2 more robust.

[0063] (Various variations and embodiments of Embodiment 1) In the following modified examples and embodiments, components similar to those in Embodiment 1 described above are denoted by the same reference numerals, and their descriptions are omitted as appropriate. Furthermore, in each modified example and embodiment, unless otherwise specified, the focus change process of Embodiment 1 (Figure 6) is performed.

[0064] (Modification 1 of Embodiment 1) The focus change system may perform an update process after executing the focus change process shown in Figure 6. The update process is the process of updating the trained model M.

[0065] Figure 7 is a sequence diagram of the update process. The update process is mainly realized by the terminal processing unit 34 working in cooperation with other components, based on a program stored in the terminal memory unit 32. The update process is executed after the focus change process S211.

[0066] First, after the focus of the variable focus lens 21LN has been changed based on the focus information, the eyeglass processing unit 27, upon receiving an operation from the eyeglass operation unit 25 to change the focus of the variable focus lens 21LN, transmits a focus change operation signal to the communication terminal 3 via the eyeglass communication unit 23 (step S301). The focus change operation signal is a signal generated in response to the user's operation to change the focus.

[0067] Next, when the terminal processing unit 34 of the communication terminal 3 receives a focus change operation signal via the terminal communication unit 31, it generates feedback information (step S302). The feedback information includes the changed focal length of the variable focus lens 21LN.

[0068] Next, the terminal processing unit 34 transmits feedback information to the server 4 via the terminal communication unit 31 (step S303).

[0069] Next, the server processing unit 43 of server 4 receives feedback information via the server communication unit 41 (step S304).

[0070] Next, the server processing unit 43 updates the trained model M by inputting feedback information into the trained model M (step S305). This update is also called retraining. The server processing unit 43 inputs feedback information into the trained model M and updates the parameters within the model that were present when the trained model M was generated.

[0071] The update process is now complete.

[0072] In Modification 1 of Embodiment 1, the terminal processing unit 34 generates feedback information if the focus of the variable focus lens 21LN is changed after the focus of the variable focus lens 21LN has been changed based on the focus information. The server processing unit 43 updates the trained model M based on the feedback information. As a result, the focus change system according to this modification makes it possible to change the focus of the variable focus lens 21LN in a way that is more in line with the user's visual acuity.

[0073] In addition, in step S305 of the feedback process shown in Figure 7, the server processing unit 43 may update some of the parameters of the trained model M based on the feedback information. For example, the server processing unit 43 fixes the initial layers of the neural network (e.g., the feature extraction layer) and retrains only the final layer (e.g., the classification layer) using the feedback information, updating its parameters. In this way, the server processing unit 43 updates the trained model M based on the feedback information.

[0074] (Modification 2 of Embodiment 1) The detection unit 26 may detect eye movements, specifically the rotation of the eyeball within the orbit (the cavity in the facial bone that houses the eyeball). The trained model M may be pre-trained using a set of the user's gaze direction and focus information as a training set.

[0075] The detection unit 26 in this modified example is a sensor having a light source and a camera. The light source is a near-infrared LED, which irradiates the user's eyeball with near-infrared light. The camera captures images of the user's eyeball reflecting the near-infrared light. A sensor having a near-infrared LED and a camera is also called an infrared distance sensor. The detection unit 26 generates dynamic image data including the pupil point and the point of reflection of near-infrared light as eye movement information and supplies it to the glasses processing unit 27. The eye movement information includes information on the user's line of sight direction.

[0076] As shown in Figure 3, a person's line of sight differs depending on the distance to the object they are looking at. When a person looks at an object P1 in real space, their eyes move so that their left and right line of sight directions S1 coincide at the object P1, compared to when they look at a virtual image P2. On the other hand, when a person looks at a virtual image P2, their eyes move so that their left and right line of sight directions S2 are parallel to each other. In other words, detecting the line of sight makes it possible to determine whether a person is trying to look at an object P1 in real space or a virtual image P2. Furthermore, by knowing the distance between the user's left and right eyes in advance, the focal length is estimated based on the angle between the line of sight directions of the user's left and right eyes (hereinafter referred to as the "line of sight angle"). In other words, detecting the line of sight makes it possible to estimate the focal length.

[0077] For example, based on the aforementioned relationship of gaze direction, the gaze angle when viewing an object P1 in real space is relatively wider than the gaze angle when viewing a virtual image P2. In other words, the height of the gaze angle makes it possible to determine whether a person is focusing on the object P1 in real space or on the virtual image P2. Furthermore, the height of the gaze angle makes it possible to estimate the focal length. Therefore, even when the user's gaze direction information is used as eye movement information, it is possible to generate a trained model using sets of multiple eye movement information and the focal information for each point when the eye movement information is detected as training data.

[0078] In step S201 of the focus change process shown in Figure 6, the detection unit 26 detects the gaze direction of the user wearing the glasses 2 as eye movement and generates eye movement information. In step S207 of the focus change process, the server processing unit 43 inputs the eye movement information, which includes the user's gaze direction, into the trained model M to obtain the output focus information.

[0079] In the modified embodiment 2 of Embodiment 1, the detection unit 26 also detects the eye movements of the user wearing the glasses 2 and generates eye movement information. This makes it possible to suppress the influence of the external environment on the focus change system according to this modified embodiment.

[0080] The detection unit 26 may not have a camera and may instead have multiple distance measuring sensors. The multiple distance measuring sensors measure the distance to multiple parts of the skin near the eye, such as the upper eyelid, lower eyelid, inner corner of the eye, and outer corner of the eye, and generate information on the time change of distance as eye movement information. In this case as well, the detection unit 26 is able to detect the direction of the gaze of the user wearing the glasses 2.

[0081] (Other modifications of Embodiment 1) The variable focus lens 21LN can be any lens other than a liquid lens, as long as its focus can be changed. For example, the variable focus lens 21LN may be a Fresnel liquid crystal lens, or a lens using polymer thin film technology and optical liquid. The variable focus lens 21LN may also be a lens that uses an optical lens and changes focus by driving an actuator.

[0082] The projection lens 22LN may be any AR glasses other than waveguide type, as long as it can project the virtual image output by the virtual image projection unit 22. The projection lens 22LN may be, for example, birdbath type AR glasses. The birdbath type is a method in which the virtual image output by the virtual image projection unit 22 is reflected by a half mirror and projected onto the eye.

[0083] The variable focus lens 21LN and the projection lens 22LN may be integrated. That is, the lens unit LN may be configured such that a virtual image P2 is superimposed on a lens that allows the focus of the lens to be changed. According to this modified example, the focus changing system can reduce the number of lenses, and the eyeglasses 2 can be made smaller. Miniaturizing the eyeglasses 2 also contributes to making the eyeglasses 2 lighter.

[0084] The virtual image projection unit 22 does not necessarily have to be located near the rim of the glasses 2; for example, it may be located near the temple of the glasses 2.

[0085] The virtual image projection unit 22 and the projection lens 22LN may be a retinal projection type display. A retinal projection type display is a display that projects an image directly onto the retina of the eye. The virtual image projection unit 22 has RGB laser light that emits RGB laser light and a MEMS mirror that reflects the RGB laser light. The projection lens 22LN is a reflective mirror that reflects the RGB laser light reflected from the MEMS mirror onto the retina.

[0086] The eyeglasses operation unit 25 may be a switch other than a push switch. The eyeglasses operation unit 25 may be, for example, a touch panel. In other words, the eyeglasses operation unit 25 may also function as a display unit. The eyeglasses operation unit 25 generates an image based on the signal supplied from the eyeglasses processing unit 27 and displays it on the touch panel.

[0087] The communication terminal 3 may be an information processing terminal other than a smartphone. For example, the communication terminal 3 may be a mobile phone other than a smartphone, a tablet device, a game console, a smartwatch, or a portable personal computer (e.g., a laptop computer).

[0088] The eyeglasses communication unit 23 and the terminal communication unit 31 may have wireless communication interface circuits conforming to communication protocols such as BLUETOOTH® and may wirelessly connect to each other in accordance with a predetermined communication standard. The terminal communication unit 31 and the server communication unit 41 may have wireless communication interface circuits conforming to communication standards such as LTE or 5G and may communicate with the network N via a base station. The terminal communication unit 31 and the server communication unit 41 may have wired communication interface circuits conforming to communication protocols such as TCP / IP and may wired connect to each other in accordance with a predetermined communication standard.

[0089] (Embodiment 2) Figure 8(a) is a diagram showing an overview of the focus change system 1' according to Embodiment 2, and (b) is a functional block diagram of the communication terminal 3' according to Embodiment 2. The focus change system 1' according to Embodiment 2 differs from the focus change system 1 according to Embodiment 1 in that it does not have a server 4, and the learned model M is stored in the terminal storage unit 32 of the communication terminal 3'.

[0090] As shown in Figure 8(a), the terminal memory unit 32 of the communication terminal 3' stores the trained model M. The trained model M is installed in the terminal memory unit 32 from an external server via the terminal communication unit 31, for example. The external server is, for example, server 4. The trained model M may also be installed in the server memory unit 42 from a computer-readable, non-temporary portable storage medium such as a CD-ROM or DVD-ROM.

[0091] In the focus change process shown in Figure 6, the terminal processing unit 34 of the communication terminal 3' executes processes S206 and S207 instead of process S204. That is, the terminal processing unit 34 inputs the eye movement information received via the terminal communication unit 31 into the learned model M stored in the terminal storage unit 32. The terminal processing unit 34 omits processes S204 and S209.

[0092] As described above, in Embodiment 2, the server 4 is omitted. The terminal processing unit 34 inputs eye movement information into a learned model M stored in the terminal memory unit 32 and changes the focus of the variable focus lens 21LN to either an object P1 in the real space or a virtual image P2 based on the output focus information. This enables the focus changing system 1' to automatically change the focus of the variable focus lens 21LN regardless of the external communication environment of the glasses 2 and the communication terminal 3'.

[0093] Furthermore, when the terminal processing unit 34 performs the feedback process shown in Figure 7, it performs the process in S305 instead of S303. That is, the terminal processing unit 34 updates the trained model M based on the feedback information. The terminal processing unit 34 omits the process in S303.

[0094] (Embodiment 3) Figure 9 shows an overview of the focus-changing system 1" according to Embodiment 3. The focus-changing system 1" according to Embodiment 3 differs from the focus-changing system 1 according to Embodiment 1 in that it does not have a communication terminal 3. Since the glasses 2 of Embodiment 3 execute the processing performed by the communication terminal 3 in Embodiment 1, the glasses 2 and the communication terminal 3 can also be said to be integrated.

[0095] In step S202 of the focus change process shown in Figure 6, the glasses processing unit 27 transmits eye movement information to the server 4 via the glasses communication unit 23. The processes S203 and S204 of the focus change process are omitted.

[0096] In the S208 of the focus change process, the server processing unit 43 transmits focus information to the glasses 2 via the server communication unit 41. That is, the server processing unit 43 inputs eye movement information into the learned model M and transmits the output focus information to the processor (glasses processing unit 27) that processes the operation of the variable focus lens 21LN via the server communication unit 41.

[0097] The eyeglasses processing unit 27 executes the processes S209 to S211 of the focus change process, excluding the processing related to communication between the eyeglasses 2 and the communication terminal 3. The eyeglasses processing unit 27 generates a control instruction to change the focal length by referring to the focal length information included in the focus information. The eyeglasses processing unit 27 supplies the control instruction to the focus control unit 21.

[0098] As described above, in Embodiment 3, the communication terminal 3 is omitted. The server processing unit 43 inputs eye movement information into the learned model M and transmits the output focus information to the glasses processing unit 27 via the server communication unit 41. This makes it possible to simplify the entire focus changing system 1". Simplifying the entire system also contributes to lowering the cost and increasing the robustness of the system.

[0099] Furthermore, if the eyeglasses operation unit 25 receives an operation to change the focus of the variable focus lens 21LN, the eyeglasses processing unit 27 may execute S302 and S303 of the feedback processing shown in Figure 7. In this case, the eyeglasses processing unit 27 omits processing S301.

[0100] Those skilled in the art will understand that various changes, substitutions, and modifications can be made without departing the scope of the present invention. For example, the embodiments and modifications described above may be combined as appropriate within the scope of the present invention. [Explanation of Symbols]

[0101] 1·1'·1” Focus change system, 2 eyeglasses, 21LN variable focus lens, 22 virtual image projection unit, 22LN projection lens, 23 eyeglasses communication unit, 26 detection unit, 3·3' communication terminal, 31 terminal communication unit, 32 terminal storage unit, 34 terminal processing unit, 4 server, 41 server communication unit, 42 server storage unit, 43 server processing unit

Claims

1. A focus adjustment system comprising eyeglasses, a server, and a communication terminal capable of communicating with the eyeglasses and the server, The aforementioned eyeglasses A variable focus lens, Virtual image projection unit, A projection lens capable of projecting the virtual image output by the virtual image projection unit, It includes a detection unit that detects the eye movements of a user wearing the aforementioned glasses and generates eye movement information, The aforementioned communication terminal is A terminal communication unit that receives the aforementioned eye movement information, The terminal processing unit includes a terminal communication unit that transmits the eye movement information to the server, The aforementioned server, A server communication unit that receives the aforementioned eye movement information, A server memory unit that stores a trained model that has been trained to output focus information when eye movement information is input, The system includes a server processing unit that inputs the received eye movement information into the trained model, outputs the focus information, and transmits it to the communication terminal via the server communication unit. The terminal processing unit changes the focus of the variable focus lens to either an object in the real space or the virtual image based on the focus information. A focus-changing system characterized by the following:

2. The aforementioned eyeglasses or communication terminal are The device further includes an operating unit that allows the user to change the focus of the variable focus lens, The aforementioned terminal processing unit is If the focus of the variable focus lens is changed by the operating unit after the focus of the variable focus lens has been changed based on the aforementioned focus information, feedback information is generated. The feedback information is transmitted to the server via the terminal communication unit. The server processing unit, Based on the feedback information received via the server communication unit, the trained model is updated. The focus changing system according to claim 1.

3. The variable focus lens and the projection lens are integrated. The focus changing system according to claim 1 or 2.

4. A communication terminal capable of communicating with eyeglasses having a variable focus lens, a virtual image projection unit, and a projection lens capable of projecting a virtual image output by the virtual image projection unit, A detection unit detects the eye movements of a user wearing the aforementioned glasses and generates eye movement information, and a terminal communication unit receives the eye movement information. A terminal memory unit that stores a trained model that has been trained to output focus information when eye movement information is input, The system includes a terminal processing unit that inputs the received eye movement information into the trained model and, based on the output focal information, changes the focus of the variable focus lens to either an object in real space or the virtual image. A communication terminal characterized by the following features.

5. A server that can communicate with eyeglasses having a variable focus lens, a virtual image projection unit, and a projection lens capable of projecting a virtual image output by the virtual image projection unit, and transmits predetermined instructions to a processor that processes the operation of the variable focus lens, A detection unit that detects the eye movements of a user wearing the aforementioned glasses and generates eye movement information, and a server communication unit that receives the eye movement information from the detection unit, A server memory unit that stores a trained model that has been trained to output focus information when eye movement information is input, The system includes a server processing unit that inputs the received eye movement information into the trained model and transmits the output focus information to the processor via the server communication unit. A server characterized by the following features.

6. A method for changing the focus of a variable-focus lens in eyeglasses having a variable-focus lens, a virtual image projection unit, and a projection lens capable of projecting a virtual image output by the virtual image projection unit, The system detects the eye movements of the user wearing the aforementioned glasses and generates eye movement information. The detected eye movement information is input to a trained model that has been trained to output focus information when eye movement information is input. Based on the focus information output from the trained model, the focus of the variable focus lens is changed to either an object in real space or the virtual image. A method for changing focus characterized by the following features.

7. A focus changing program for eyeglasses having a variable focus lens, a virtual image projection unit, and a projection lens capable of projecting a virtual image output by the virtual image projection unit, wherein the program modifies the focus of the variable focus lens, and the processor that processes the operation of the variable focus lens, The detection unit, which detects the eye movements of the user wearing the aforementioned glasses and generates eye movement information, receives the eye movement information from the detection unit. The eye movement information detected by the detection unit is input to a trained model that has been trained to output focus information when eye movement information is input. Based on the focus information output from the trained model, the focus of the variable focus lens is changed to either an object in real space or the virtual image. A focus-changing program characterized by the following:

Images