Information processing terminals and systems
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- SONY GROUP CORP
- Filing Date
- 2022-02-22
- Publication Date
- 2026-08-04
Smart Images
Figure 0007899831000001 
Figure 0007899831000002 
Figure 0007899831000003
Abstract
Description
Technical Field
[0001] The present disclosure relates to an information processing terminal, an information processing apparatus, and a system.
Background Art
[0002] In recent years, the IoT (Internet of Things) of furniture and home appliances has advanced, and many things can be operated via a network. When operating via a network, generally, an operation application pre-installed on a smartphone is called, and the operation is performed from an operation screen.
[0003] In addition, due to the development of technology in recent years, miniaturization of various devices has advanced. For example, in Patent Document 1 below, a camera capable of miniaturizing and lightening an optical system related to imaging is disclosed.
Prior Art Documents
Patent Documents
[0004]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0005] However, it was inconvenient to pick up a smartphone and operate it every time a frequently performed operation was carried out. In addition, physical input switches may cause installation costs and location restrictions.
Means for Solving the Problems
[0006] According to this disclosure, an information processing terminal is proposed comprising: an imaging unit that images an object in contact with or near a translucent member through the translucent member; a control unit that controls imaging by the imaging unit; and a communication unit that transmits information based on the image acquired by the imaging unit to an external device, wherein the imaging unit includes an image sensor and a lens provided so as to be at a variable distance from the image sensor, and the control unit controls the acquisition of a plurality of image images at different distances between the lens and the image sensor.
[0007] This disclosure proposes an information processing device that includes a control unit that transmits a control signal to a device via a communication unit, instructing it to perform one or more pre-registered functions based on a selected image from a plurality of images of an object in contact with or near an imaging unit.
[0008] The present disclosure proposes a system comprising an information processing terminal and an information processing device, wherein the information processing terminal includes an imaging unit that images an object in contact with or near a light-transmitting member through the light-transmitting member, a control unit that controls imaging by the imaging unit, and a communication unit that transmits information based on the image acquired by the imaging unit to the information processing device, the imaging unit includes an image sensor and a lens provided so as to be variable in distance from the image sensor, the control unit controls the acquisition of a plurality of image images with different distances between the lens and the image sensor, and the information processing device has a control unit that controls the transmission of a control signal to a device via the communication unit, instructing it to perform one or more pre-registered functions based on an image selected from the plurality of image images acquired by the imaging unit. [Brief explanation of the drawing]
[0009] [Figure 1] This figure illustrates an overview of an information processing system according to one embodiment of the present disclosure. [Figure 2] This block diagram shows an example of the configuration of each device included in the information processing system according to this embodiment. [Figure 3]This diagram illustrates the structure of the imaging unit according to this embodiment. [Figure 4] This diagram illustrates imaging according to this embodiment. [Figure 5] This is a schematic diagram illustrating the imaging of an object having a planar texture according to this embodiment. [Figure 6] This is a schematic diagram illustrating the imaging of an object having a three-dimensional texture according to this embodiment. [Figure 7] This flowchart shows an example of the registration process flow according to this embodiment. [Figure 8] This flowchart shows an example of the function execution process flow according to this embodiment. [Modes for carrying out the invention]
[0010] Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.
[0011] Furthermore, the explanation will be given in the following order. 1. Overview 2. Example Configuration 3. Operation Processing 3-1. Registration Process 3-2. Function Execution Process 4. Variations 5. Supplement
[0012] <<1. Overview>> Figure 1 is a diagram illustrating the outline of an information processing system according to one embodiment of the present disclosure. The information processing system according to this embodiment includes an operating terminal 10 (an example of an information processing terminal) used to operate IoT devices 30, and an information processing device 20 that controls IoT devices 30 in response to instructions from the operating terminal 10. The information processing device 20 may be a personal communication device such as a smartphone as shown in Figure 1, a tablet terminal, or a PC. The information processing device 20 may also be a dedicated terminal that communicates with a large number of IoT devices 30 and mediates communication between the operating terminal 10 and the IoT devices 30. IoT devices 30 are an example of devices that are operated by the operating terminal 10. Examples of IoT devices 30 include furniture and home appliances. The user can operate the IoT devices 30 (e.g., turn the power ON / OFF) using the operating terminal 10 worn on their finger or the like.
[0013] (Organizing the issues) As mentioned above, while the IoT integration of furniture and home appliances has progressed, allowing many items to be controlled via a network, it has been inconvenient to have to pick up a smartphone, unlock it (using fingerprint authentication or a passcode), and then launch the control application each time a frequently performed operation is needed. Assigning IoT device operations to physical input buttons is also an option, but increasing the number of input buttons with each desired function would increase installation costs. Furthermore, an increase in input buttons could potentially detract from the aesthetics of the installation location.
[0014] Therefore, in this embodiment, the operation of IoT devices is assigned to surrounding objects. Specifically, a system is provided that uses a camera to recognize objects and executes the operation of IoT devices assigned to the recognized objects. By recognizing objects around the user with a camera, the user can operate IoT devices without having to operate a smartphone or consider the location of input buttons, further enhancing the convenience of IoT devices.
[0015] The device (operation terminal) equipped with a camera is preferably a wearable device that is more compact and can be worn on the user's finger, wrist, etc. This allows the user to use the operation terminal without annoyance even for frequently occurring operations. Also, the operation terminal may be a smartphone. By performing object recognition with the camera provided in the smartphone and automatically operating the assigned IoT device, it is possible to reduce annoyance without having the user unlock the smartphone or perform screen operations.
[0016] The operation terminal according to this embodiment may be, for example, a ring-shaped operation terminal 10 worn on the user's finger as shown in FIG. 1. The user brings the operation terminal 10 provided with an imaging unit into contact with an object 40 (for example, a wall of a room) to which the ON / OFF operation of the IoT device 30 to be operated is assigned. At this time, the operation terminal 10 captures an image of the surface of the object 40 (the surface of the wall), and the captured image is transmitted to the information processing device 20. When the information processing device 20 performs object recognition based on the captured image and determines that the object 40 is the one to which the ON / OFF operation of the IoT device 30 is assigned, it transmits an instruction for the ON / OFF operation to the IoT device 30.
[0017] In this embodiment, by using an imaging unit with a short focal length, it is possible to reduce the risk of privacy infringement due to using the camera for object recognition. Also, since the texture of the object surface is diverse, the focus may be out of focus (the focus is not achieved, that is, it is blurred) when shooting with a single focus. However, if an autofocus mechanism is installed, the imaging unit will become larger and heavier. The imaging unit according to this embodiment has a structure that changes the distance between the lens and the imaging element and performs imaging multiple times when the operation terminal 10 comes into contact with the object. Thereby, it becomes possible to obtain an in-focus captured image (hereinafter, also referred to as a focused image). Details of the structure will be described later with reference to FIG. 3.
[0018] The above describes the outline of an information processing system according to one embodiment of the present disclosure. Next, the specific configurations of the operating terminal 10 and information processing device 20 included in the information processing system according to this embodiment will be described with reference to the drawings.
[0019] <<2. Example Configuration>> Figure 2 is a block diagram showing an example of the configuration of each device included in the information processing system according to this embodiment.
[0020] <2-1. Operating Terminal 10> As shown in Figure 2, the operating terminal 10 includes a communication unit 110, a control unit 120, an imaging unit 130, a vibration unit 140, and a storage unit 150.
[0021] (Communications Department 110) The communication unit 110 communicates with an external device and transmits and receives data. For example, the communication unit 110 can communicate with the information processing device 20 via wireless communication such as Wi-Fi (registered trademark) or Bluetooth (registered trademark).
[0022] (Control unit 120) The control unit 120 functions as an arithmetic processing unit and control unit, and controls the overall operation within the operating terminal 10 according to various programs. The control unit 120 is implemented by electronic circuits such as a CPU (Central Processing Unit) and a microprocessor. The control unit 120 may also include a ROM (Read Only Memory) for storing programs and calculation parameters to be used, and a RAM (Random Access Memory) for temporarily storing parameters that change as needed.
[0023] In this embodiment, the control unit 120 controls the imaging by the imaging unit 130. The control unit 120 also controls the transmission of one or more images acquired by the imaging unit 130 from the communication unit 110 to the information processing device 20.
[0024] (Imaging unit 130) The imaging unit 130 has the function of capturing images of a subject. More specifically, the imaging unit 130 according to this embodiment has a short focal length and can extract information from an extremely narrow area on the surface of an object. Furthermore, the imaging unit 130 according to this embodiment is realized by a structure that takes multiple images by changing the distance between the lens and the image sensor that constitute the imaging unit 130. Since the texture of the surface of an object is diverse and the appropriate focal length differs depending on the object, taking multiple images by changing the distance between the lens and the image sensor increases the possibility of obtaining a focused image. Here, the structure of the imaging unit 130 according to this embodiment will be specifically described with reference to Figure 3.
[0025] Figure 3 is a diagram illustrating the structure of the imaging unit 130 according to this embodiment. As shown in Figure 3, the operating terminal 10 according to this embodiment is formed by a ring-shaped main body 160 that can be worn on a finger, for example. The imaging unit 130 provided on the main body 160 includes, as shown in Figure 3, an image sensor 131, a lens 132, a light guide plate 133, a light source unit 135 (135a, 135b), a movable member 134, and a return spring 136.
[0026] The image sensor 131 images an object (subject) that is in contact with or close to the light guide plate 133 via the light guide plate 133. In this specification, proximity is also included in the description of contact. The lens 132 is provided so that the distance h from the image sensor 131 is variable. Specifically, the distance h between the image sensor 131 and the lens 132 is varied when the movable member 134 to which the lens 132 is fixed is pushed toward the image sensor 131 by contact with an object. At the tip of the movable member 134, that is, on the side opposite to the image sensor 131 relative to the lens 132, the light guide plate 133 is provided so as to cover the lens 132. When the movable member 134 comes into contact with an object such as a room wall and is pushed toward the image sensor 131, the distance h between the image sensor 131 and the lens 132 decreases. The return spring 136 has the function of pushing the movable member 134, which has been pushed toward the image sensor 131, back toward the light guide plate 133. Here, a return spring 136 is used as an example, but any component that has the function of pushing back the movable member 134 is acceptable. The light guide plate 133 is an example of a light-transmitting member. Light source units 135 (135a, 135b) are provided at both ends of the light guide plate 133. Note that the position of the light source units 135 is not limited to the example shown in Figure 3. The light source units 135 may be located around the light guide plate 133, for example, on the underside of the light guide plate 133 (the side where the image sensor 131 is located).
[0027] The control unit 120 controls the acquisition of multiple captured images with different distances h between the lens 132 and the image sensor 131. This will be explained below with reference to Figure 4.
[0028] Figure 4 illustrates imaging according to this embodiment. As shown in Figure 4, first, the user brings the operating terminal 10, which is attached to a finger or the like, close to the object 40. When the operating terminal 10 comes into contact with the object 40, the control unit 120 turns on the light source unit 135 and controls the imaging unit 130 to start imaging. Contact with the object 40 may be determined, for example, by detecting the indentation of the movable member 134. The movable member 134 may be indented by the force received when the light guide plate 133 comes into contact with the object 40. The indentation detection method may be, for example, a momentary operation method or an electrical conduction method such as a resistive film. Although not shown in Figure 4, such a detection unit may be provided in the imaging unit 130.
[0029] At the start of imaging, as shown in Figure 4, the distance h between the lens 132 and the image sensor 131 is at its longest distance (h_max). The control unit 120 continues imaging until the pushing is complete (until the movable member 134 is pushed all the way in). At this time, as shown in Figure 4, the distance h between the lens 132 and the image sensor 131 is at its shortest distance (h_min). Whether or not the movable member 134 has been pushed all the way in may be detected by providing a detection unit on the return spring 136, or by providing a detection unit around the movable member 134. The detection unit may be, for example, a contact sensor, an optical sensor, a pressure sensor, etc.
[0030] In this way, the imaging unit 130 according to this embodiment uses the force at which an object comes into contact to change the distance h between the lens 132 and the image sensor 131, thereby acquiring multiple images with different distances h. Furthermore, the imaging unit 130 according to this embodiment is miniaturized by eliminating the lens drive mechanism.
[0031] In this embodiment, for example, in the information processing device 20 described later, a focused image can be selected from multiple images with different focal lengths h, and object recognition can be performed. Here, the differences in focal lengths when imaging an object will be explained with reference to Figures 5 and 6.
[0032] Figure 5 is a schematic diagram illustrating the imaging of an object 41 having a planar texture according to this embodiment. When the image sensor 131, lens 132, and light guide plate 133 are arranged at the distance intervals shown in the left of Figure 5, an in-focus image can always be obtained if the object 41 has a planar texture, as shown in the right of Figure 5. In the arrangement shown in Figure 5, the distance a from the lens 132 to the object 41 is twice the distance from the lens 132 to the focal point Fa. Therefore, by positioning the image sensor 131 at a distance b from the lens 132 (twice the distance from the lens 132 to the focal point Fb), an image in focus can be obtained by the image sensor 131.
[0033] On the other hand, in the case of an object 42 with a three-dimensional texture, there are areas with different depths of field, so with single focus, there will be areas that are out of focus. In contrast, this embodiment makes it possible to acquire surface information of an object with depth (improving the accuracy of acquisition) by making the distance between the lens 132 and the image sensor 131 variable. The following explanation will be given with reference to Figure 6.
[0034] Figure 6 is a schematic diagram illustrating the imaging of an object 42 having a three-dimensional texture according to this embodiment. In the arrangement shown on the left of Figure 6, the light ray L1 is in focus, but the light ray L2 in the depth portion is not. In this embodiment, since the distance between the lens 132 and the image sensor 131 is variable, by changing to the arrangement shown on the right of Figure 6, for example, the light ray L2 is brought into focus, making it possible to acquire surface information of an object with depth. Thus, the imaging method according to this embodiment is particularly useful for objects having a three-dimensional texture.
[0035] In this embodiment, multiple images are acquired with different distances between the lens 132 and the image sensor 131. As a result, an image at the focal point shown on the left of Figure 6 and an image at the focal point shown on the right of Figure 6 can be acquired. In this case, for example, the information processing device 20 selects the image at the focal point shown on the right of Figure 6 to perform object recognition. In object recognition, feature quantities of the object surface are extracted as surface information of the object. The information processing device 20 then compares the extracted feature quantities with pre-registered feature quantities and executes the corresponding function (such as operating the IoT device 30).
[0036] (Vibrating part 140) The vibrating unit 140 has the function of presenting tactile stimuli according to the control of the control unit 120. The structure of the vibrating unit 140 is not particularly limited. Nor is the mounting position of the vibrating unit 140 particularly limited. The control unit 120 performs control to present vibration, for example, when the movable member 134 is pushed all the way in. This makes it possible to provide the user with operational feedback (a so-called click sensation).
[0037] (Storage unit 150) The memory unit 150 is implemented by a ROM (Read Only Memory) that stores programs and calculation parameters used in the processing of the control unit 120, and a RAM (Random Access Memory) that temporarily stores parameters that change as needed.
[0038] The configuration of the operating terminal 10 has been described in detail above. Note that the configuration of the operating terminal 10 according to this embodiment is merely an example, and this disclosure is not limited thereto. For example, the operating terminal 10 may not have a vibration unit 140. Instead of the vibration unit 140, the operating terminal 10 may have a physical structure that provides a click sensation when the movable member 134 is fully pressed in. Furthermore, the operating terminal 10 according to this embodiment is not limited to feedback by vibration, but may also provide feedback by light or sound. That is, the vibration unit 140 is just one example of a presentation unit that provides feedback, and the operating terminal 10 may have a light-emitting unit or an audio output unit. Also, the operating terminal 10 is not limited to a ring shape, but may be formed by a bangle-type or bracelet-type main body 160 worn on the user's wrist. Furthermore, the operating terminal 10 may be a pendant top.
[0039] <2-2. Information Processing Device 20> As shown in Figure 2, the information processing device 20 includes a communication unit 210, a control unit 220, an operation input unit 230, a display unit 240, and a storage unit 250.
[0040] (Communications Department 210) The communication unit 210 communicates with external devices and sends and receives data. For example, the communication unit 210 sends and receives data with the operation terminal 10 and the IoT device 30. Alternatively, the communication unit 210 may communicate with the operation terminal 10 and the IoT device 30 via a home network. The communication unit 210 can communicate with a network via, for example, wired / wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), or mobile communication networks (LTE (Long Term Evolution), 3G (third-generation mobile communication system), 4G (fourth-generation mobile communication system), 5G (fifth-generation mobile communication system)).
[0041] (Control unit 220) The control unit 220 functions as both an arithmetic processing unit and a control device, controlling the overall operation of the information processing device 20 according to various programs. The control unit 220 is implemented by electronic circuits such as a CPU (Central Processing Unit) or a microprocessor. The control unit 220 may also include a ROM (Read Only Memory) for storing programs and arithmetic parameters, and a RAM (Random Access Memory) for temporarily storing parameters that change as needed.
[0042] The control unit 220 in this embodiment can also function as a registration processing unit 221, an image selection unit 222, a matching unit 223, and a function execution unit 224. The registration processing unit 221 performs the registration process for assigning a function to an object. Specifically, the registration processing unit 221 associates the feature quantities of the object extracted from the captured image, the location name of the object (e.g., the wall next to the stairs), and the function to be assigned, and stores this as registration information in the storage unit 250. The feature quantities of the object are obtained by analyzing an image selected from multiple captured images captured by the operation terminal 10 at the time of registration. The registration information may be stored in a cloud server (not shown). The function to be assigned is, for example, the ON / OFF operation of the IoT device 30. The timing of the registration of the feature quantities of the object and the location name may differ from the timing of the registration of the function to be assigned. It is also possible to change the function to be assigned. A more specific flow of the registration process will be described later with reference to Figure 7.
[0043] The image selection unit 222 selects a focused image from multiple captured images transmitted from the operation terminal 10. The image selection unit 222 outputs the selected image to the matching unit 223.
[0044] The matching unit 223 analyzes the captured image to extract feature quantities of the object (subject) (more specifically, feature quantities of the object's surface), compares them with registered information, and calls the corresponding function. The texture of object surfaces is diverse, and by extracting information with high spatial resolution from an ultra-narrow area of the object's surface, it becomes possible to capture minute changes between objects.
[0045] The function execution unit 224 performs control to execute the function called by the matching unit 223. For example, the function execution unit 224 performs control to send a control signal to the IoT device 30 via the home network to instruct the IoT device 30 to turn ON or OFF. Upon receiving the control signal, the IoT device 30 transitions from the ON state to the OFF state, or from the OFF state to the ON state.
[0046] (Operation input unit 230) The operation input unit 230 receives operations from the user. received The input information is then output to the control unit 220. The operation input unit 230 is implemented by various input devices such as a touch panel, buttons, switches, and a keyboard.
[0047] (Display section 240) The display unit 240 has the function of displaying various screens, such as operation screens. The display unit 240 can be implemented, for example, by a liquid crystal display (LCD) device, an organic light-emitting diode (OLED) device, etc. The display unit 240 may also be a so-called optical see-through display with optical transparency, or a projector that projects images into real space.
[0048] (Storage unit 250) The memory unit 250 is implemented by a ROM (Read Only Memory) that stores programs and calculation parameters used in the processing of the control unit 220, and a RAM (Random Access Memory) that temporarily stores parameters that change as needed.
[0049] The configuration of the information processing device 20 has been described in detail above. Note that the configuration of the information processing device 20 according to this embodiment is not limited to the example shown in Figure 2. For example, the information processing device 20 may be implemented by multiple devices. Furthermore, at least some of the functions of the control unit 220 may be implemented by the operation terminal 10 or a server on the network (not shown). For example, the function of the image selection unit 222 may be implemented by the control unit 120 of the operation terminal 10. In this case, the operation terminal 10 controls the transmission of a selected, in-focus image from multiple captured images to the information processing device 20.
[0050] <<3. Operation Processing>> Next, the operation process of the information processing system according to this embodiment will be specifically explained with reference to the drawings.
[0051] <3-1. Registration Process> Figure 7 is a flowchart showing an example of the registration process according to this embodiment. The registration process shown in Figure 7 can be performed, for example, by the information processing device 20 and the operation terminal 10.
[0052] As shown in Figure 7, first, the information processing device 20 receives input from the user via the operation input unit 230 for the name of the location to be registered (step S103). The user may input a name such as "living room wall" or "entrance ornament" as the location name of the object to be registered. The name to be entered can be arbitrarily entered by the user, or the user can select from a pre-prepared list of names.
[0053] Next, the control unit 220 of the information processing device 20 switches its operating mode to registration mode ON and waits for the acquisition of the captured image (step S106).
[0054] Next, the control unit 120 of the operating terminal 10 determines whether or not the light guide plate 133 has come into contact with an object (step S109).
[0055] If contact with an object occurs (step S109 / Yes), the control unit 120 controls the image sensor 131 and the light source unit 135 to turn ON (step S112). That is, the control unit 120 controls the light source unit 135 to turn on and start imaging by the imaging unit 130.
[0056] Next, the control unit 120 uses the force of object contact to push the movable member 134 in, and continuously takes images while the distance h between the lens 132 and the image sensor 131 changes (step S115). The number of images taken may be determined by the performance of the imaging unit 130 (FPS; frames per second), or by the control unit 120 according to the contact time with the object set by the user (the length of time the operating terminal 10 is in contact with the object; a natural length is desirable for the UI) and the false authentication rate. For example, if a false authentication rate of about 5% (the percentage of times when no function is called or an incorrect function is called when authentication is performed) is considered a natural UI guideline, it may be set higher than 5% if a certain degree of malfunction of the assigned function is acceptable, or lower than 5% if the user does not want to tolerate too much malfunction of the assigned function. When the false authentication rate is low, a larger number of images is set. The more images there are, the higher the probability that an in-focus image will be included, and the higher the authentication accuracy (i.e., the accuracy of matching by the matching unit 223 when the function is executed). On the other hand, transferring and selecting the target images takes time, which can delay the execution of the function. Reducing the number of image captures can also shorten the time until the function is executed.
[0057] Next, the control unit 120 determines whether the light guide plate 133 (on which the movable member 134 is provided) has been pushed in all the way (step S118).
[0058] If the device is pushed all the way in (step S118 / Yes), the control unit 120 controls the vibration unit 140 and presents a click sensation (step S121). The user recognizes that the device has been pushed all the way in by the click sensation and releases the operating terminal 10 from the object.
[0059] Next, the control unit 120 performs control to turn off the image sensor 131 and the light source unit 135 (step S124).
[0060] Next, the control unit 120 outputs multiple captured images to the information processing device 20 (step S127).
[0061] Next, the image selection unit 222 of the information processing device 20 selects a focused image from multiple captured images (step S130). In the case of a three-dimensional texture, for example, the image with the largest focused area may be selected.
[0062] Next, the matching unit 223 performs image analysis on the focused image selected by the image selection unit 222 and extracts feature quantities from the object surface (step S133). If feature quantities exceeding a certain standard cannot be extracted, the registration processing unit 221 may determine that registration has failed. In this case, the information processing device 20 notifies the user that registration has failed. The notification may be made, for example, by the display unit 240 or by voice.
[0063] Next, if a registered image already exists (step S136 / Yes), the registration processing unit 221 updates the feature quantities (registered feature quantities) associated with such registered images (step S139). A registered image is an image captured that has already been registered as registration information in the storage unit 150 or the like. The registration processing unit 221 searches for already registered image captured images (or feature quantities) from the input name and selected focus image, and if found, registers the latest feature quantities.
[0064] On the other hand, if no registered image exists yet (step S136 / No), the registration processing unit 221 registers the feature quantities and the name (information indicating the location of the object) in association (step S142). The registration processing unit 221 may also register the selected in-focus image. In addition, the user may also register a function to be assigned to the object. Specifically, for example, the user can select any IoT device from a number of IoT devices connected to a home network and assign a predetermined operation of that IoT device as a function.
[0065] Then, the control unit 220 turns off the registration mode (step S145).
[0066] The above describes one example of the registration process. Alternatively, after registering objects in various locations within the house, the user may operate the information processing device 20 to register functions to be assigned to each object. Furthermore, the user can change the functions assigned to objects as needed, increasing the degree of customization and further improving convenience.
[0067] Furthermore, the registration process shown in Figure 7 is just one example, and this disclosure is not limited thereto. For example, the input of the name of the registration location shown in step S103 does not necessarily have to be done first, and can be done at any time before step S136, for example. Also, the control unit 220 does not have to perform the processes shown in steps S136 and S139. In addition, the image selection process shown in step S130 may be performed on the operation terminal 10. In this case, the process shown in step S127 is not performed, and after the image selection process, a process is performed to transmit the selected captured image to the information processing device 20. This reduces the time required for image transfer to the information processing device 20.
[0068] As explained above, in this embodiment, it is possible to assign functions such as operating the IoT device 30 to objects in one's surroundings. If you want to increase the number of functions to assign, you can easily increase the number of assignable functions by registering the functions to objects in your surroundings (new arbitrary locations) without having to worry about the cost of installing input buttons for operation or the design of the installation location.
[0069] <3-2. Function Execution Process> Figure 8 is a flowchart showing an example of the function execution process flow according to this embodiment. The function execution process shown in Figure 8 can be executed by, for example, the information processing device 20 and the operation terminal 10.
[0070] As shown in Figure 8, first, the control unit 120 of the operating terminal 10 determines whether or not the light guide plate 133 has come into contact with an object (step S203).
[0071] If contact with an object occurs (step S203 / Yes), the control unit 120 controls the image sensor 131 and the light source unit 135 to turn ON (step S206).
[0072] Next, the control unit 120 uses the force of object contact to push the movable member 134 in, and continues imaging while the distance h between the lens 132 and the image sensor 131 changes (step S209).
[0073] Next, when the light guide plate 133 (on the movable member 134) is pushed all the way in (step S212 / Yes), the control unit 120 controls the vibrating unit 140 to produce a click sensation (step S215). The user recognizes that it has been pushed all the way in by the click sensation and removes the operating terminal 10 from the object.
[0074] Next, the control unit 120 performs control to turn off the image sensor 131 and the light source unit 135 (step S218).
[0075] Next, the control unit 120 outputs multiple captured images to the information processing device 20 (step S221).
[0076] Next, the image selection unit 222 of the information processing device 20 selects an in-focus image from multiple captured images (step S224).
[0077] Next, the matching unit 223 performs image analysis on the focused image selected by the image selection unit 222 and extracts feature quantities from the object surface (step S227).
[0078] Next, the matching unit 223 compares the extracted features with the registered features (step S230). Registered features are features registered through the registration process described with reference to Figure 7. The matching unit 223 compares the extracted features with the registered features and searches for matching features.
[0079] Next, if there is a matching registered feature (step S233 / Yes), the function execution unit 224 executes the assigned function (step S236).
[0080] On the other hand, if no matching registered feature is found (step S233 / No), the control unit 220 feeds back a default vibration to the operation terminal 10 (step S239). The default vibration is a vibration to notify the user that the matching result was a mismatch (not registered). The control unit 220 sends a control signal to the operation terminal 10 instructing it to perform the default vibration. The default vibration may be a different pattern from the vibration used to notify the user of the end of pressing. It is also possible that the pressing speed of the operation terminal 10 may be too fast, resulting in insufficient captured images or blurring. Therefore, the user may touch the operation terminal 10 to the object again and perform the process shown in Figure 8 once more.
[0081] The above describes an example of the processing flow during function execution. Note that the function execution process shown in Figure 8 is just one example, and this disclosure is not limited thereto. For example, the image selection process shown in step S224 may be performed on the operation terminal 10. In this case, the process shown in step S221 is omitted, and after the image selection process, a process is performed to transmit the selected captured image to the information processing device 20. This reduces the time required for image transfer to the information processing device 20. Also, the feedback shown in step S239 may be performed by blinking or sound instead of vibration.
[0082] As described above, in this embodiment, it is possible to operate IoT devices 30 through natural actions such as touching the operating terminal 10 to objects around you. This eliminates the hassle of operations such as unlocking the smartphone and calling up the operation application each time an operation is performed.
[0083] Various objects can be assigned functions. In particular, by registering objects related to or near the device to be controlled (IoT device 30), the device can be operated with more natural movements. For example, by registering the bed sheets and headboard, the bedroom lights can be turned on or off by touching the sheets or headboard. Also, by registering the sofa, the TV can be turned on by placing a hand on the sofa. Furthermore, by registering the desk, attendance can be registered by touching the desk. Finally, by registering the living room wall, the living room lights can be turned on by touching the wall.
[0084] <<4. Variation>> Next, a modified example of this embodiment will be described.
[0085] In the operating terminal 10 according to this embodiment, the stiffness of the return spring 136 may be adjusted to suit the user. For example, since children and the elderly have weaker strength, the stiffness (strength) of the return spring 136 can be manually adjusted in advance to set a predetermined pressing speed (so that it can be pressed with weak force). From the viewpoint of authentication accuracy, a slower pressing speed is preferable. That is, a slower pressing speed makes it possible to acquire more captured images (reducing the false authentication rate). Also, a slower pressing speed can be expected to prevent blurring. The predetermined pressing speed may be, for example, a speed that ensures enough time to perform at least the set number of images. Furthermore, a table showing the correspondence between the pressing speed (strength of the return spring 136) and the false authentication rate may be prepared in advance and displayed on the display unit 240 of the information processing device 20.
[0086] Furthermore, feedback at the end of pressing (when the button is pressed all the way down) is not limited to vibration; it may also be provided by light or sound, for example. The operating terminal 10 may be provided with a light-emitting unit or an audio output unit in place of, or in conjunction with, the vibration unit 140.
[0087] Furthermore, the lens 132 may be a microlens array. This allows the lens 132 to be made even thinner.
[0088] Furthermore, although the above embodiment described that feedback is provided by vibration when authentication fails (see step S239), feedback may also be provided by vibration during authentication (when verification is successful by the verification unit 223). In addition, the vibration pattern during authentication on the operation terminal 10 may be set for each registered function. This allows the user to understand which function has been invoked. Note that feedback on authentication success or failure is not limited to vibration, and may also be provided by light or sound.
[0089] Furthermore, multiple functions may be assigned to a single object (location). This makes it possible to operate multiple IoT devices 30 simultaneously with a single action.
[0090] Furthermore, the information processing device 20 can recognize actions performed using the operation terminal 10 based on the captured image and execute functions corresponding to those actions. That is, the user pre-registers information indicating what functions to execute when they perform certain actions at different locations. For example, if the user touches the same object twice consecutively (within a predetermined time) using the operation terminal 10, the information processing device 20 may recognize this as a double tap and execute a different function than when the same object is touched once. Also, if the operation terminal 10 can detect that the user has moved away from the object, and the contact time is longer than a predetermined time, the information processing device 20 may recognize this as a long press and execute a function corresponding to the long press. Detection of the user moving away from the object may be performed by a detection unit provided on the return spring 136. Alternatively, the distance between the lens and the image sensor 131 may be detected by a distance measuring sensor or the like when the distance is at the shortest distance (h_min), and if this state continues for a predetermined time or longer, it may be recognized as a long press. Recognition of actions may also be performed by the operation terminal 10.
[0091] Furthermore, in this embodiment, it is also possible to recognize rotational movements and movements in the up, down, left, and right directions. This makes it possible to realize switches that require detection of twisting movements or the amount of operation, such as dimming switches, volume control switches, and temperature setting switches. Specifically, a motion sensor (accelerometer, gyro sensor, etc., or IMU; Inertial Measurement Unit) is provided on the operating terminal 10, and twisting movements and movements in the x and y axes when the operating terminal 10 is pressed can be detected. The operating terminal 10 performs control to transmit multiple captured images and motion sensor detection data to the information processing device 20.
[0092] Furthermore, the operating terminal 10 may also be controlled to acquire multiple images when it moves away from the object 40. Specifically, the control unit 120 of the operating terminal 10 controls the movable member 134 to take multiple images while the movable member 134 is pushed back by the force of the return spring 136 when the main body 160 of the operating terminal 10 moves away from the object 40. This makes it possible to acquire multiple images with different distances h between the lens 132 and the image sensor 131, not only when the operating terminal 10 is in contact, but also when it is moved away. When the pushing back of the movable member 134 begins, the control unit 120 lights up the light source unit 135 and starts imaging by the imaging unit 130. The start of the pushing back of the movable member 134 can be detected, for example, by a detection unit provided on the return spring 136 or by a distance measuring sensor that measures the distance between the lens 132 and the image sensor 131. The control unit 120 of the operating terminal 10 transmits the multiple images acquired at the time of contact and the multiple images acquired at the time of separation to the information processing device 20. The information processing device 20 can determine whether or not there was any movement while in contact by comparing a focused image selected from multiple captured images at the time of contact with a focused image selected from multiple captured images at the time of separation. Specifically, this can be determined by the shift or presence or absence of feature points extracted from the focused image. 20 If no movement occurs, it is possible to call function 1 (e.g., a decision operation), and if movement occurs, it is possible to call function 2 (e.g., a scroll operation).
[0093] Furthermore, the image sensor 131 in this embodiment is not limited to an image sensor that senses the visible light region, but may also be an image sensor that senses the non-visible light region. For example, it may be an image sensor that senses the shortwave region or the infrared region. Also, both imaging in the visible light region and imaging in the non-visible light region may be performed. If it is possible to extract features that cannot be captured with visible light from an ultra-narrow area on the surface of an object using an image sensor that senses the non-visible light region, the authentication accuracy can be further improved.
[0094] Furthermore, although the above-described embodiment explains that the image selection, matching, and function execution processes are performed by the information processing device 20, this disclosure is not limited thereto, and the image selection, matching, and function execution processes may also be performed by the control unit 120 of the operation terminal 10. In this case, the operation terminal 10 communicates with the IoT device 30 and transmits a control signal instructing the execution of the function.
[0095] Furthermore, the operating terminal 10 according to this embodiment may be implemented as a smartphone or a wearable device (smartwatch, smart band, etc.). For example, an imaging unit 130 having the structure shown in Figure 3 may be provided on a smartphone or a wearable device. The imaging unit 130 may also be used as an imaging unit for fingerprint authentication. The imaging unit 130 may be used for fingerprint authentication as its first use and for device operation as its second use.
[0096] <<5. Supplementary Information>> As described above, the operating terminal 10 according to the embodiment of this disclosure makes it possible to improve the convenience of user operation.
[0097] While preferred embodiments of the present disclosure have been described in detail above with reference to the attached drawings, the present technology is not limited to such examples. It is clear to any person with ordinary skill in the art of the present disclosure that various modifications or alterations may be conceived within the scope of the technical idea described in the claims, and these will naturally be understood to fall within the technical scope of the present disclosure.
[0098] For example, the structure of the imaging unit 130 according to this embodiment is not limited to the example shown in Figure 3. The imaging unit 130 according to this embodiment only needs to have a structure in which the distance h between the lens 132 and the image sensor 131 is variable by utilizing the force when an object is in contact.
[0099] Furthermore, although an IoT device was used as the device to be operated by the operating terminal 10, this disclosure is not limited to this, and various devices can be operated.
[0100] Furthermore, one or more computer programs can be created to enable the operation terminal 10 or information processing device 20 to perform its functions, using the CPU, ROM, RAM, and other hardware built into the aforementioned operation terminal 10 or information processing device 20. A computer-readable storage medium containing such one or more computer programs is also provided.
[0101] Furthermore, the effects described herein are merely descriptive or illustrative and not limiting. In other words, the technology relating to this disclosure may produce other effects that will be apparent to those skilled in the art from the description herein, in addition to or in lieu of the effects described herein.
[0102] Furthermore, this technology can also be configured as follows. (1) An imaging unit that images an object in contact with or near a translucent member through the translucent member, A control unit that controls imaging by the imaging unit, A communication unit that transmits information based on the captured image acquired by the imaging unit to an external device, Equipped with, The imaging unit includes an image sensor and a lens provided so as to be at a variable distance from the image sensor. The control unit performs control to acquire multiple captured images with different distances between the lens and the image sensor. Information processing terminal. (2) The information processing terminal according to (1), wherein the light-transmitting member and the lens are fixed to a movable member that is pushed toward the image sensor by contact with the object. (3) The information processing terminal according to (2), wherein the light-transmitting member is located at the tip of the movable member, on the opposite side of the lens from the image sensor, and is arranged to cover the lens. (4) The information processing terminal according to (3), wherein the light-transmitting member is a light guide plate, and a light source is provided around the light guide plate. (5) The information processing terminal according to (4), wherein the control unit controls the illumination of the light source unit and the start of imaging by the imaging unit when the pushing of the movable member is detected. (6) The information processing terminal according to (5), wherein the control unit also controls the lighting of the light source and the start of imaging by the imaging unit when the pushing back of the movable member is detected. (7) The aforementioned information processing terminal is a wearable device, as described in any one of items (1) to (6) above. (8) The information processing terminal is the information processing terminal according to (7) above, having a ring-shaped main body that can be worn on a finger. (9) The control unit performs control to transmit a plurality of captured images with different distances between the lens and the image sensor to the external device. The external device is an information processing device that performs control to execute a corresponding pre-registered function based on an image selected from the plurality of captured images. The information processing terminal described in (7) or (8) above. (10) The information processing terminal according to (7) or (8), wherein the control unit selects an image image that is in focus from a plurality of image images with different distances between the lens and the image sensor, and controls the transmission of the selected image image to the external device. (11) An information processing device comprising a control unit that performs control by transmitting a control signal to a device via a communication unit, instructing it to execute one or more pre-registered functions based on an image selected from multiple images of an object in contact with or near the imaging unit. (12) The control unit analyzes the selected captured image to obtain feature quantities and performs control to execute a function associated with registered feature quantities that match the feature quantities, as described in (11). (13) The information processing apparatus according to (11) or (12), wherein the control unit performs a process to select an image that is in focus from the plurality of captured images. (14) The control unit is an information processing device according to any one of items (11) to (13), which recognizes motion from the analysis of the selected captured image or motion sensor data and performs control to execute a function corresponding to the motion. (15) The information processing apparatus according to any one of (11) to (14), wherein the plurality of captured images are a plurality of captured images in which the distance between the lens and the image sensor included in the imaging unit is different. (16) The information processing apparatus according to (15), wherein the control unit has the imaging unit and receives the plurality of captured images from an operating terminal that controls the acquisition of a plurality of captured images with different distances between the lens and the image sensor. (17) The information processing apparatus according to (15), wherein the control unit has the imaging unit and receives an image selected from the plurality of image captured images from an operating terminal that controls the acquisition of a plurality of image captured images with different distances between the lens and the image sensor. (18) The imaging unit further comprises, The information processing apparatus according to (15), wherein the control unit performs control to acquire a plurality of captured images with different distances between the lens and the image sensor. (19) It comprises an information processing terminal and an information processing device, The aforementioned information processing terminal is An imaging unit that images an object in contact with or near a translucent member through the translucent member, A control unit that controls imaging by the imaging unit, A communication unit transmits information based on the captured image acquired by the imaging unit to the information processing device. It has, The imaging unit includes an image sensor and a lens provided so as to be at a variable distance from the image sensor. The control unit performs control to acquire multiple captured images with different distances between the lens and the image sensor. The aforementioned information processing device is The control unit has a control unit that transmits a control signal to a device via a communication unit, instructing it to execute one or more pre-registered functions based on a selected image from the plurality of images captured by the imaging unit. system. [Explanation of symbols]
[0103] 10 operations end end 110 Communications Department 120 Control Unit 130 Imaging Unit 131 Image sensor 132 lenses 133 Light guide plate 134 Movable member 135 Light source section 136 (136a, 136b) Return spring 140 Vibration section 150 Storage section 20 Information Processing Devices 210 Communications Department 220 Imaging Unit 221 Registration Processing Unit 222 Image Selection Section 223 Verification Unit 224 Function Execution Unit 230 Operation Input Section 240 Display section 250 Storage section
Claims
1. An imaging unit that images an object in contact with or near a translucent member through the translucent member, A control unit that controls imaging by the imaging unit, A communication unit that transmits information based on the captured image acquired by the imaging unit to an external device, Equipped with, The imaging unit includes an image sensor and a lens provided so as to be at a variable distance from the image sensor. The control unit performs control to acquire multiple captured images with different distances between the lens and the image sensor. The light-transmitting member and the lens are fixed to a movable member that is pushed toward the image sensor by contact with the object. Information processing terminal.
2. The information processing terminal according to claim 1, wherein the light-transmitting member is located at the tip of the movable member, on the opposite side of the lens from the image sensor, and is arranged to cover the lens.
3. The information processing terminal according to claim 2, wherein the light-transmitting member is a light guide plate, and a light source is provided around the light guide plate.
4. The information processing terminal according to claim 3, wherein the control unit controls the illumination of the light source unit and the start of imaging by the imaging unit when the pushing of the movable member is detected.
5. The information processing terminal according to claim 4, wherein the control unit also controls the lighting of the light source and the start of imaging by the imaging unit when the pushing back of the movable member is detected.
6. The information processing terminal according to claim 1, wherein the information processing terminal is a wearable device.
7. The information processing terminal according to claim 6, wherein the information processing terminal has a ring-shaped main body that can be worn on a finger.
8. The control unit performs control to transmit a plurality of captured images with different distances between the lens and the image sensor to the external device. The external device is an information processing device that performs control to execute a corresponding pre-registered function based on an image selected from the plurality of captured images. The information processing terminal according to claim 6.
9. The information processing terminal according to claim 6, wherein the control unit selects an image image that is in focus from a plurality of image images with different distances between the lens and the image sensor, and controls the transmission of the selected image image to the external device.
10. It comprises an information processing terminal and an information processing device, The aforementioned information processing terminal is An imaging unit that images an object in contact with or near a translucent member through the translucent member, A control unit that controls imaging by the imaging unit, A communication unit transmits information based on the captured image acquired by the imaging unit to the information processing device. It has, The imaging unit includes an image sensor and a lens provided so as to be at a variable distance from the image sensor. The control unit performs control to acquire multiple captured images with different distances between the lens and the image sensor. The light-transmitting member and the lens are fixed to a movable member that is pushed toward the image sensor by contact with the object. The aforementioned information processing device is The control unit has a control unit that transmits a control signal to a device via a communication unit, instructing it to execute one or more pre-registered functions based on a selected image from the plurality of images captured by the imaging unit. system.
11. The system according to claim 10, wherein the control unit of the information processing device analyzes the selected captured image to obtain feature quantities and performs control to execute a function associated with registered feature quantities that match the feature quantities.
12. The system according to claim 10, wherein the control unit of the information processing device performs a process of selecting an image that is in focus from the plurality of captured images.
13. The system according to claim 10, wherein the control unit of the information processing device recognizes an action from the analysis of the selected image or motion sensor data and performs control to execute a function corresponding to the action.
14. The system according to claim 11, wherein the plurality of captured images are a plurality of captured images in which the distance between the lens and the image sensor included in the imaging unit is different.
15. The system according to claim 14, wherein the control unit of the information processing device has the imaging unit and receives the plurality of captured images from an operating terminal that controls the acquisition of a plurality of captured images with different distances between the lens and the image sensor.
16. The system according to claim 14, wherein the control unit of the information processing device has the imaging unit and receives an image selected from the plurality of image captured images from an operating terminal that controls the acquisition of a plurality of image captured images with different distances between the lens and the image sensor.
17. The imaging unit further comprises, The system according to claim 14, wherein the control unit performs control to acquire a plurality of captured images with different distances between the lens and the image sensor.