Specific system

Abstract

This identification system for identifying a region of interest to a user comprises a wearable device and a portable apparatus. The wearable device is an HMD mounted on the head of the user. The wearable device detects the direction of the head of the user. The portable apparatus is worn on the body of the user separately from the wearable device. The portable apparatus has an imaging device, an acquisition unit, and an identification unit. The acquisition unit acquires first-direction information indicating the direction of the head of the user detected by the wearable device, and second-direction information indicating the direction of the optical axis of the imaging device. On the basis of the first-direction information and the second-direction information acquired by the acquisition unit, the identification unit identifies a region of interest to the user from a captured image captured by the imaging device.

Description

Technical Field

[0001] The present invention relates to an identification system for identifying a region of interest that a user is focusing on.

Background Art

[0002] An HMD device (Head Mounted Display) that displays additional information such as a description of an object shown in an image overlaid on a real-world view is generally known. As a prior art related to the HMD device, the technology disclosed in Patent Document 1 can be cited. In the HMD device disclosed in Patent Document 1, a display image including additional image information received from a cloud server and a landscape captured by a camera is displayed on a display panel. When this display image reaches the user's eyes, the user views the display image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, a glasses-type HMD device equipped with a lens that transmits external light and displaying an image such as additional information on this lens has been proposed. When the glasses-type HMD device becomes popular, an increase in services for identifying a region in the real space that the user is focusing on through the HMD device (hereinafter, the region of interest (eye-gaze area)) and displaying additional information corresponding to the region of interest is expected.

[0005] If an imaging device that captures an image substantially the same as the image seen by the user's eyes with sufficient image quality is mounted on the HMD device, the area of interest can be specified based on the image captured by the imaging device. However, (i) due to constraints on weight, processing power, or power consumption, the HMD device may not be equipped with an imaging device. Also, (ii) even if the HMD device is equipped with an imaging device, the image quality of the image captured by the imaging device may be insufficient for specifying the area of interest. In the cases shown in (i) or (ii), the area of interest cannot be specified, and display services such as additional information corresponding to the area of interest cannot be provided.

Means for Solving the Problem

[0006] A specific system according to a preferred aspect of the present disclosure includes a wearing device, an imaging device, an acquisition unit, and a specification unit. The wearing device is worn on the user's head. The wearing device detects the orientation of the user's head. The imaging device is worn on the user's body separately from the wearing device. The acquisition unit acquires first direction information indicating the orientation of the user's head detected by the wearing device and second direction information indicating the orientation of the optical axis of the imaging device. The specification unit specifies the area of interest that the user is looking at from the captured image captured by the imaging device based on the first direction information and the second direction information.

Effect of the Invention

[0007] According to the present disclosure, the area of interest that the user is looking at is specified using an imaging device worn on the user's body separately from the HMD device. Therefore, according to the present disclosure, even when the HMD device is not equipped with an imaging device, or when the image quality of the image captured by the imaging device mounted on the HMD device is insufficient, the area of interest of the user can be specified.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Mode for Carrying Out the Invention

[0009] (A. First Embodiment) (A-1: Overall Configuration) FIG. 1 is a block diagram showing a configuration example of a specific system 1 according to the first embodiment of the present disclosure. As shown in FIG. 1, the specific system 1 includes a mobile device 10 and a wearable device 20A. The wearable device 20A is, for example, a glasses-type HMD device. The specific system 1 is an information system that identifies a region in the real space (hereinafter referred to as the target region) that the user is looking at through the wearable device 20A, and provides a service of displaying an image corresponding to the target region, specifically, additional information, on the wearable device 20A.

[0010] The mobile device 10 is, for example, a smartphone equipped with a camera. The mobile device 10 is not limited to a smartphone, and may be, for example, a tablet or a notebook personal computer. FIG. 2 is a diagram showing an example of how to use the mobile device 10 and the wearable device 20A. The wearable device 20A is worn on the head of the user U. The mobile device 10 is worn on the body of the user U separately from the wearable device 20A. The mobile device 10 is worn at a position other than the head of the user U, that is, at a position different from the wearable device 20A. In the present embodiment, the mobile device 10 is worn on the body of the user U by hanging it from the neck using a strap or the like. The mobile device 10 may be connected to the wearable device 20A by wire. The mobile device 10 may be connected to the wearable device 20A wirelessly.

[0011] The user U in the present embodiment is, for example, a maintenance worker in a factory or the like. As shown in FIG. 2, it is assumed that the user U wearing the mobile device 10 and the wearable device 20A gazes at a two-dimensional barcode B1 attached to an instrument or the like. In this situation, when the image G1 shown in FIG. 3 is captured by the mobile device 10, the image G1 is reflected in the eyes of the user U through the wearable device 20A. Strictly speaking, due to the structure of the wearable device 20A (HMD device) and the imaging position, the image reflected in the eyes of the user U is not the same as the image G1 captured by the mobile device 10. However, the difference between the two images is small enough to be ignored, so the expression "the image G1 is reflected in the eyes of the user U" is used.

[0012] Although details will be described later, although the wearable device 20A of the present embodiment is not equipped with an imaging device (camera), the wearable device 20A detects the orientation of the user U's face, that is, the orientation of the user U's head. The wearable device 20A is an example of the wearable device in the present disclosure. The orientation of the head of the user U wearing the wearable device 20A represents the orientation of the optical axis of the wearable device 20A. The orientation of the optical axis of the wearable device 20A refers to the orientation of the optical axis of the lens through which external light passes in the wearable device 20A. In the present embodiment, based on the imaging direction of the image by the portable device 10 and the orientation of the head of the user U detected by the wearable device 20A, the attention area A1 that the user U is looking at through the wearable device 20A is specified. The portable device 10 reads the two-dimensional barcode B1 from the attention area A1. The two-dimensional barcode B1 is obtained by encoding the identification information that uniquely indicates the instrument to which the two-dimensional barcode B1 is attached. The portable device 10 acquires the identification information by decoding the two-dimensional barcode B1 read from the attention area A1. The portable device 10 transmits the acquired identification information to the management device 30 via the communication line NW.

[0013] The management device 30 is a database server device connected to the communication line NW. In the factory where the user U works, a plurality of instruments are installed. The identification information (ID) of each instrument is associated with the additional information regarding the corresponding instrument, and the additional information is stored in advance in the management device 30. Specific examples of the additional information stored in the management device 30 include the maintenance inspection history of the instrument or the cautionary notes regarding the maintenance inspection of the instrument. When the management device 30 receives the identification information via the communication line NW, it transmits the additional information corresponding to the received identification information to the wearable device 20A.

[0014] The mobile device 10 receives additional information from the management device 30 via the communication line NW. The mobile device 10 causes the mounting device 20A to display the additional information received from the management device 30. The image that the user U views through the mounting device 20A is an image in which an image representing the additional information is superimposed on a part of the image G1 captured by the mounting device 20A. The image representing the additional information may be a character string, a symbol, a table, or a figure.

[0015] (A-2: Mobile Device) FIG. 4 is a block diagram showing a configuration example of the mobile device 10. As shown in FIG. 4, the mobile device 10 includes an input device 11, an output device 12, an imaging device 13, a communication device 14, a communication device 15, a direction detection device 16, a storage device 17, a processing device 18, and a bus 19. The components of the mobile device 10 (input device 11, output device 12, imaging device 13, communication device 14, communication device 15, direction detection device 16, storage device 17, and processing device 18) are interconnected by a bus 19 for communicating information. The bus 19 may be configured using a single bus or different buses for each device.

[0016] The input device 11 includes a touch panel. The input device 11 may include a plurality of operation keys in addition to the touch panel. The input device 11 may include a plurality of operation keys without including the touch panel. The input device 11 receives operations performed by the user U. The output device 12 includes a display. The touch panel of the input device 11 is laminated on the display of the output device 12. The output device 12 displays various information.

[0017] The imaging device 13 is a camera including, for example, an image sensor. The imaging device 13 captures the field of view of the imaging device under the control of the processing device 18. The imaging device 13 outputs image (or moving image) data representing the captured image to the processing device 18. In the present embodiment, the imaging device 13 is arranged on the mobile device 10 in a posture where the imaging direction of the imaging device 13 (i.e., the direction of the optical axis of the imaging device 13) coincides with the normal direction of the display surface of the output device 12.

[0018] The communication device 14 communicates with the management device 30 via the communication line NW. The communication device 14 may communicate with the management device 30 without using the communication line NW. The communication device 15 communicates with the wearing device 20A by wire. The communication device 15 receives the first direction information transmitted from the wearing device 20A. The first direction information represents the absolute orientation of the user U's head, for example, the orientation of the user U's head in a three-dimensional coordinate system (hereinafter, the absolute coordinate system) fixed in advance with respect to the earth. The communication device 15 outputs the first direction information received from the wearing device 20A to the processing device 18. Further, the communication device 15 transmits the image data given from the processing device 18 to the wearing device 20A. Note that the communication device 15 may communicate with the wearing device 20A wirelessly.

[0019] Under the control of the processing device 18, the direction detection device 16 detects the absolute orientation of the optical axis of the imaging device 13, that is, the orientation of the optical axis of the imaging device 13 in the absolute coordinate system, every time a predetermined time elapses. The direction detection device 16 outputs second direction information indicating the detected direction to the processing device 18. The direction detection device 16 includes a three-axis geomagnetic sensor and a three-axis acceleration sensor, and generates the second direction information based on the direction of magnetic north detected by the three-axis geomagnetic sensor and the posture of the portable device 10 detected by the three-axis acceleration sensor.

[0020] The storage device 17 is a recording medium readable by the processing device 18. The storage device 17 includes, for example, a non-volatile memory and a volatile memory. The non-volatile memory is, for example, a ROM (Read Only Memory), an EPROM (Erasable Programmable Read Only Memory), and an EEPROM (Electrically Erasable Programmable Read Only Memory). The volatile memory is, for example, a RAM (Random Access Memory). The storage device 17 stores a program PR1 for causing the processing device 18 to execute the method for specifying the attention area of the present disclosure.

[0021] The processing device 18 includes one or more CPUs (Central Processing Units). The one or more CPUs are an example of one or more processors. Each of the processor and the CPU is an example of a computer. The processing device 18 reads the program PR1 from the storage device 17. By executing the program PR1, the processing device 18 functions as the acquisition unit 181, the specification unit 182, and the display control unit 183 shown in FIG. 4. That is, the acquisition unit 181, the specification unit 182, and the display control unit 183 in FIG. 4 are software modules realized by operating the processing device 18 according to software.

[0022] The acquisition unit 181 acquires the first direction information and the second direction information. More specifically, the acquisition unit 181 acquires the second direction information from the direction detection device 16. Further, the acquisition unit 181 acquires the first direction information by communicating with the wearing device 20A using the communication device 15. When the wearing device 20A transmits the first direction information to the management device 30 and the first direction information is stored in the management device 30, the acquisition unit 181 may acquire the first direction information by communicating with the management device 30 using the communication device 14.

[0023] The specification unit 182 specifies the area of interest of the user U in the captured image by the imaging device 13 based on the first direction information acquired by the acquisition unit 181 and the second direction information also acquired by the acquisition unit 181.

[0024] For example, as shown in FIG. 5, assume that the position coordinates of the portable device 10 in the absolute coordinate system are (X1, Y1, Z1), and the position coordinates of the wearing device 20A in the absolute coordinate system are (A1, B1, C1). Also, assume that the orientation of the head of the user U wearing the wearing device 20A (that is, the absolute orientation of the optical axis L1 of the wearing device 20A mounted on the head of the user U) is represented by the rotation angles (α, β, γ) around each coordinate axis in the absolute coordinate system, and the absolute orientation of the optical axis L2 of the imaging device 13 is represented by the rotation angles (r, p, y) around each coordinate axis of the absolute coordinate system.

[0025] In the example shown in FIG. 5, an image G2 obtained by magnifying an image G1 according to the magnification of the imaging lens of the imaging device 13 is used. The image G2 is an image corresponding to a position separated by the focal length of the imaging device 13 in the direction along the optical axis L2 from the position coordinates (X1, Y1, Z1) of the mobile device 10. In the absolute position coordinate system, there is an intersection point P between a straight line passing through the position coordinates (A1, B1, C1) of the mounting device 20A and along the optical axis L1 and the image G2, and the position on the image G1 corresponding to the intersection point P represents the center position (hereinafter referred to as the attention position) of the attention area A1. The image G1 is determined according to the distance from the imaging device 13 to an object such as an instrument and the angle of view of the imaging device 13. Therefore, assuming that the distance from the imaging device 13 to the object is fixed, the image G2 is uniquely determined based on the image G1, and the attention position is specified based on the position coordinates (X1, Y1, Z1), the position coordinates (A1, B1, C1), and the rotation angles (α, β, γ) and the rotation angles (r, p, y).

[0026] The first direction information in the present embodiment is information representing the rotation angles (α, β, γ), and the second direction information is information representing the rotation angles (r, p, y). In the present embodiment, the distance between the mounting device 20A and the imaging device 13 in the absolute coordinate system is about 30 cm, while the distance from the imaging device 13 to the object is generally about 3 m to 10 m. Since the distance between the mounting device 20A and the imaging device 13 is sufficiently shorter than the distance from the imaging device 13 to the object, even if the difference between the position coordinates (A1, B1, C1) of the mounting device 20A and the position coordinates (X1, Y1, Z1) of the mobile device 10 is ignored (that is, even if it is determined that X1 = A1, Y1 = B1, Z1 = C1), the attention position can be roughly specified and no particular problem occurs. For this reason, the specifying unit 182 specifies the attention position based on the differences (α - r, β - p, γ - y) in the rotation angles around the respective coordinate axes. Then, the specifying unit 182 specifies a region within a predetermined range centered on the specified attention position as the attention area A1.

[0027] The display control unit 183 causes the image (i.e., additional information) corresponding to the target area A1 specified by the specifying unit 182 to be displayed on the wearing device 20A. More specifically, the display control unit 183 reads the two-dimensional barcode B1 from the target area A1 in the image G1. Next, the display control unit 183 decodes the read two-dimensional barcode B1 and converts it into identification information. Next, the display control unit 183 transmits the identification information obtained by decoding the two-dimensional barcode B1 to the management device 30, thereby acquiring the additional information corresponding to the identification information. Then, the display control unit 183 transmits the image data representing the image in which the acquired additional information is superimposed on the target area A1 to the wearing device 20A using the communication device 15.

[0028] Further, by operating according to the program PR1, the processing device 18 executes the method for specifying the target area of the present disclosure. FIG. 6 is a flowchart showing the flow of this specifying method. As shown in FIG. 6, this specifying method includes each process from step SA110 to step SA130.

[0029] In step SA110, the processing device 18 functions as the acquisition unit 181. In step SA110, the processing device 18 acquires the second direction information from the direction detection device 16. Also, in step SA110, the processing device 18 acquires the first direction information by communicating with the wearing device 20A using the communication device 15.

[0030] In step SA120 following step SA110, the processing device 18 functions as the specifying unit 182. In step SA120, the processing device 18 specifies the target area A1 based on the first direction information and the second direction information acquired in step SA110.

[0031] In step SA130 following step SA120, the processing device 18 functions as the display control unit 183. In step SA130, the processing device 18 causes the image (additional information) corresponding to the target area A1 specified in step SA120 to be displayed on the wearing device 20A.

[0032] (A-3: Mounting Device) FIG. 7 is a block diagram showing a configuration example of the mounting device 20A. The mounting device 20A includes a display unit 2a, a communication device 2b, a direction detection device 2c, a storage device 2d, a processing device 2e, and a bus 2f. The components of the mounting device 20A (the display unit 2a, the communication device 2b, the direction detection device 2c, the storage device 2d, and the processing device 2e) are interconnected by a bus 2f for communicating information. The bus 2f may be configured using a single bus or may be configured using different buses for each element such as a device.

[0033] The display unit 2a is of the light transmissive type. The light in the visual field of the mounting device 20 (display unit 2a) passes through the display unit 2a. The display unit 2a displays an image including the attention area A1 under the control of the portable device 10. When the user U wears the mounting device 20A, the display unit 2a is positioned in front of the left and right eyes of the user U. An image representing additional information corresponding to the attention area A1 appears in the eyes of the user U wearing the mounting device 20A.

[0034] More specifically, the display unit 2a includes a lens for the left eye, a display panel for the left eye, an optical member for the left eye, a lens for the right eye, a display panel for the right eye, and an optical member for the right eye. The display panel for the left eye and the display panel for the right eye are, for example, a liquid crystal panel or an organic EL (Electro Luminescence) panel. The display panel for the left eye displays the image represented by the image data given from the portable device 10. The optical member for the left eye is an optical member that guides the light emitted from the display panel for the left eye to the lens for the left eye. Similarly, the display panel for the right eye displays the image represented by the image data given from the portable device 10. The optical member for the right eye is an optical member that guides the light emitted from the display panel for the right eye to the lens for the right eye.

[0035] Each of the left-eye lens and the right-eye lens has a half mirror. The half mirror of the left-eye lens guides the light representing the real space to the left eye of the user U by transmitting the light representing the real space. Further, the half mirror of the left-eye lens reflects the light guided by the left-eye optical member to the left eye of the user U. The half mirror of the right-eye lens guides the light representing the real space to the right eye of the user U by transmitting the light representing the real space. The half mirror of the right-eye lens reflects the light guided by the right-eye optical member to the right eye of the user U.

[0036] The communication device 2b communicates with the portable device 10 by wire. The communication device 2b may communicate with the portable device 10 wirelessly.

[0037] The direction detection device 2c detects the absolute orientation of the head of the user U every time a predetermined time elapses. The direction detection device 2c outputs first direction information indicating the detected direction to the processing device 2e. Similar to the direction detection device 16, the direction detection device 2c includes a three-axis geomagnetic sensor and a three-axis acceleration sensor. The direction detection device 2c generates the first direction information based on the direction of magnetic north detected by the three-axis geomagnetic sensor and the posture of the portable device 10 detected by the three-axis acceleration sensor.

[0038] The storage device 2d is a recording medium readable by the processing device 2e. Similar to the storage device 17, the storage device 2d includes a non-volatile memory and a volatile memory. The storage device 2d stores the program PR2.

[0039] The processing device 2e includes one or more CPUs. The processing device 2e reads the program PR2 from the storage device 2d. By executing the program PR2, the processing device 2e functions as an operation control unit 2e1.

[0040] The operation control unit 2e1 controls the operation of the mounting device 20A. The operation control unit 2e1 transmits the first direction information output from the direction detection device 2c to the mobile device 10 using the communication device 2b. By this transmission, the absolute orientation of the user U's head is transmitted to the mobile device 10. Further, the operation control unit 2e1 supplies the image data received from the mobile device 10 by the communication device 2b to the display unit 2a. The display unit 2a displays the image represented by the image data supplied from the operation control unit 2e1.

[0041] As described above, in this embodiment, the image data representing the image corresponding to the attention area A1 is transmitted from the mobile device 10 to the mounting device 20A, and the display unit 2a displays the additional information corresponding to the attention area A1. As a result, as shown in FIG. 8, an image G3 in which the image C1 is superimposed on the image G1 is projected onto the eyes of the user U. The image C1 in this embodiment is additional information corresponding to the attention area A1 and represents information such as an instrument. Alternatively, the image C1 may be an image such as a frame line surrounding the attention area A1.

[0042] As is apparent from FIG. 7, the mounting device 20A is not equipped with an imaging device. However, according to this embodiment, the attention area A1 of the user can be specified and the image corresponding to the specified attention area A1 can be displayed on the mounting device 20A by using the mounting device 20A that does not have an imaging device and the imaging device 13 of the mobile device 10 that is separately mounted on the user's body.

[0043] (B: Second Embodiment) FIG. 9 is a block diagram showing a configuration example of the specific system 1B according to the second embodiment of the present disclosure. In FIG. 9, the same components as those in FIG. 1 are given the same reference numerals. As is clear from a comparison between FIG. 9 and FIG. 1, the specific system 1B differs from the specific system 1A in that it includes a wearing device 20B and a wearing device 40 instead of the wearing device 20A. FIG. 10 is a diagram for explaining the usage methods of the wearing device 20B, the wearing device 40, and the portable device 10. As shown in FIG. 10, the portable device 10 is suspended around the neck of the user U using a strap or the like, in the same manner as in the case of the first embodiment. The portable device 10 is wired-connected to each of the wearing device 20B and the wearing device 40.

[0044] The wearing device 20B is of an earphone type. As shown in FIG. 10, the wearing device 20B is worn on the side ears of the user U. A pair of wearing devices 20B worn on both ears of the user U may be used. FIG. 11 is a block diagram showing a configuration example of the wearing device 20B. In FIG. 11, the same components as those in FIG. 7 are given the same reference numerals. As is clear from a comparison between FIG. 11 and FIG. 7, the differences between the wearing device 20B and the wearing device 20A are the following two points. The first difference is that the wearing device 20B does not have a display unit 2a. The second difference is that a program PR3 is stored in the storage device 2d of the wearing device 20B instead of the program PR2.

[0045] The processing device 2e of the wearing device 20B functions as an operation control unit 2e2 by operating according to the program PR3. The operation control unit 2e2 transmits the first direction information output from the direction detection device 2c to the portable device 10 using the communication device 2b. That is, the wearing device 20B is common with the wearing device 20A in that it detects the absolute orientation of the head of the user U. The wearing device 20B is an example of the wearing device in the present disclosure. Hereinafter, when there is no need to distinguish between the wearing device 20B and the wearing device 20A, the wearing device 20B and the wearing device 20A are referred to as the wearing device 20.

[0046] In this embodiment, the mobile device 10 identifies a target area based on the first direction information received from the wearing device 20B and the second direction information generated by the direction detection device 16. Then, the mobile device 10 transmits image data representing an image (additional information) corresponding to the identified target area to the wearing device 40.

[0047] The wearing device 40 is a glasses-type HMD device similar to the wearing device 20A. As shown in FIG. 10, the wearing device 40 is worn on the head of the user U in the same manner as the wearing device 20A. FIG. 12 is a block diagram showing a configuration example of the wearing device 40. In FIG. 12, the same reference numerals are given to the same components as those in FIG. 7. As is clear from a comparison between FIG. 12 and FIG. 7, there are two differences between the wearing device 40 and the wearing device 20A. The first difference is that the wearing device 40 does not have the direction detection device 2c. The second difference is that the program PR4 is stored in the storage device 2d of the wearing device 40 instead of the program PR2.

[0048] The processing device 2e of the wearing device 40 functions as an operation control unit 2e3 by operating according to the program PR4. The operation control unit 2e3 supplies the image data received from the mobile device 10 by the communication device 2b to the display unit 2a. By this supply, the display unit 2a displays an image (additional information) corresponding to the target area of the user U. Therefore, an image in which the image corresponding to the target area is superimposed on the real-space image is projected onto the eyes of the user U.

[0049] Also according to this embodiment, a wearing device 40 without an imaging device, an imaging device 13 mounted on a portable device 10 worn on the user's body separately from the wearing device 40, and a wearing device 20B for detecting the orientation of the user U's head are used. By using these elements, it is possible to specify the user U's attention area and display additional information (an image corresponding to the specified attention area) on the wearing device 40. Note that the wearing device 20B in this embodiment is of an earphone type, but it may be a device mounted on a helmet or a hat worn by the user U. In short, the wearing device 20B only needs to be able to detect the orientation of the user U's head. Also, if it is possible to specify the user U's attention area and the display of additional information is not required, the wearing device 40 and the display control unit 183 may be omitted.

[0050] (C: Modification Example) The present disclosure is not limited to the embodiments exemplified above. Specific modification modes are as follows. Two or more modes arbitrarily selected from the following examples may be combined.

[0051] (C-1: Modification Example 1) In each of the above embodiments, the attention area is specified by ignoring the difference between the position coordinates of the wearing device 20 and the position coordinates of the portable device 10. However, the attention area may be specified taking into account the difference between the position coordinates of the wearing device 20 and the position coordinates of the portable device 10. According to the mode of specifying the attention area taking into account the difference between the position coordinates of the wearing device 20 and the position coordinates of the portable device 10, the specification accuracy of the attention area is improved as compared with the above embodiments. Note that, in order to specify the attention area taking into account the difference between the position coordinates of the wearing device 20 and the position coordinates of the portable device 10, the wearing device 20 may have a position detection device for detecting the absolute position of the wearing device 20, and the portable device 10 may have a position detection device for detecting the absolute position of the portable device 10. Specific examples of these position detection devices include GPS (Global Positioning System, or Global Positioning Satellite) receivers.

[0052] (C-2: Modification Example 2) In the above-described embodiments, the direction detection device 16 in the imaging device 13 detected the absolute direction of the optical axis L2, and the wearing device 20 detected the absolute direction of the user U's head. However, either one of the direction detection device 16 and the direction detection device 2c included in the wearing device 20 may be a sensor that detects a relative change in the rotation angle, such as a three-axis acceleration sensor or a three-axis angular velocity sensor (hereinafter, referred to as a relative sensor). For example, when the direction detection device 2c is a relative sensor, the absolute direction of the user U's head cannot be directly detected. However, even if the direction detection device 2c is a relative sensor, by performing calibration to initially set the direction of the user U's head with respect to the optical axis L2 of the imaging device 13, it becomes possible to acquire the absolute direction of the user U's head after calibration.

[0053] A specific example of calibration is given below. The user U may be prompted to perform a predetermined operation in a state where the portable device 10 is directed toward the center of the field of view at the position in the real space corresponding to the center of the image captured by the imaging device 13. Then, the output of the direction detection device 16 at the time when the predetermined operation is performed may be stored in the portable device 10 as initial direction information. After the completion of calibration, the wearing device 20 transmits the output value of the direction detection device 2c to the portable device 10. On the other hand, the portable device 10 stores the integrated value of the output values received after calibration. Then, the portable device 10 may convert the integrated value of the output values received after calibration into the rotation angles around the respective coordinate axes, and calculate the first direction information based on the difference between these rotation angles and the rotation angle indicated by the initial direction information, and the difference between the second direction information and the initial direction information.

[0054] (C-3: Modification Example 3) In the above-described embodiments, the program PR1 was stored in the storage device 17 of the portable device 10, but the program PR1 may be manufactured or sold alone. As a method of providing the program PR1 to the purchaser when selling the program PR1, a computer-readable recording medium such as a flash ROM in which the program PR1 is written may be distributed, or the recording medium may be distributed by downloading via an electric communication line.

[0055] (C-4: Variant Example 4) In each of the above embodiments, the acquisition unit 181, the specifying unit 182, and the display control unit 183 were software modules. However, any one, a plurality, or all of the acquisition unit 181, the specifying unit 182, and the display control unit 183 may be hardware modules. Specific examples of hardware modules include a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), an FPGA (Field Programmable Gate Array), and the like.

[0056] (C-5: Variant Example 5) In each of the above embodiments, the mobile device 10 had the acquisition unit 181, the specifying unit 182, and the display control unit 183. However, the wearing device 20 or the management device 30 may have the acquisition unit 181, the specifying unit 182, and the display control unit 183. Further, although the additional information (image corresponding to the attention area) is displayed on the display unit 2a, the display unit 2a and the display control unit 183 may be omitted when it is sufficient up to the specification of the attention area. That is, the specific system of the present disclosure may include a wearing device that is worn on the user's head and detects the orientation of the user's head, an imaging device that is worn on the user's body separately from the wearing device, an acquisition unit, and a specifying unit. The acquisition unit acquires first direction information indicating the orientation of the user's head detected by the wearing device and second direction information indicating the orientation of the optical axis of the imaging device. The specifying unit specifies an attention area in the captured image by the imaging device based on the first direction information and the second direction information.

[0057] (D: Others) (1) In the above-described embodiments, ROM, RAM, etc. were exemplified as the storage device 17 and the storage device 2d. However, the storage device 17 and the storage device 2d may be a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory device (e.g., a card, a stick, a key drive), a CD-ROM (Compact Disc-ROM), a register, a removable disk, a hard disk, a floppy (registered trademark) disk, a magnetic strip, a database, a server, or other suitable storage media.

[0058] (2) In the above-described embodiments, the information, signals, etc. described may be represented using any of a variety of different technologies. For example, the data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0059] (3) In the above-described embodiments, the input and output information, etc. may be stored in a specific location (e.g., memory) or may be managed using a management table. The input and output information, etc. may be overwritten, updated, or appended. The output information, etc. may be deleted. The input information, etc. may be transmitted to other devices.

[0060] (4) In the above-described embodiments, the determination may be made based on a value represented by 1 bit (0 or 1), a truth value (Boolean: true or false), or a numerical comparison (e.g., comparison with a predetermined value).

[0061] (5) In the above-described embodiments, the processing procedures, sequences, flowcharts, etc. exemplified may be reordered as long as there is no contradiction. For example, for the methods described in the present disclosure, the elements of the various steps are presented using an exemplary order and are not limited to the specific order presented.

[0062] (6) Each function illustrated in FIGS. 4, 7, 11, and 12 is implemented by any combination of at least one of hardware and software. Also, the implementation method of each functional block is not particularly limited. That is, each functional block may be implemented using one physically or logically combined device, or two or more physically or logically separated devices may be directly or indirectly connected (for example, using wired, wireless, etc.), and these multiple devices may be used for implementation. The functional block may be implemented by combining software with the above one device or the above multiple devices.

[0063] (7) The program exemplified in the above-described embodiments, software, regardless of whether it is called software, firmware, middleware, microcode, a hardware description language, or by another name, should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc.

[0064] Also, software, instructions, information, etc. may be transmitted and received via a transmission medium. For example, when software is transmitted from a website, server, or other remote source using at least one of wired technologies (coaxial cable, optical fiber cable, twisted pair, digital subscriber line (DSL), etc.) and wireless technologies (infrared, microwave, etc.), at least one of these wired and wireless technologies is included within the definition of the transmission medium.

[0065] (8) In each of the above-described embodiments, the terms "system" and "network" are used interchangeably.

[0066] (9) The information, parameters, etc. described in this disclosure may be represented using absolute values, relative values from a predetermined value, or using corresponding other information.

[0067] (10) In the above-described embodiments, the portable device includes the case where it is a mobile station (MS: Mobile Station). A mobile station may be called by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable term. Also, in this disclosure, terms such as "mobile station", "user terminal", "user equipment (UE)", "terminal", etc. may be used interchangeably.

[0068] (11) In the above-described embodiments, the terms "connected" and "coupled", or any variations thereof, mean any direct or indirect connection or coupling between two or more elements, and can include the presence of one or more intermediate elements between two elements "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "accessed". As used in this disclosure, two elements can be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, and, as some non-limiting and non-exhaustive examples, electromagnetic energy having wavelengths in the radio frequency region, microwave region, and optical (both visible and invisible) region, etc.

[0069] In the above-described embodiments, unless otherwise specified, the description "based on" does not mean "only based on". In other words, the description "based on" means both "only based on" and "at least based on".

[0070] (13) The terms "determining" and "deciding" as used in this disclosure may encompass a wide variety of operations. "Determining" and "deciding" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up (e.g., searching in a table, database, or another data structure), ascertaining, and considering something as having "determined" or "decided" the ascertained matter. Also, "determining" and "deciding" may include receiving (e.g., receiving information), transmitting (e.g., transmitting information), inputting, outputting, accessing (e.g., accessing data in a memory), and considering something as having "determined" or "decided" the accessed matter. Further, "determining" and "deciding" may include resolving, selecting, choosing, establishing, comparing, etc., and considering something as having "determined" or "decided" the relevant matter. That is, "determining" and "deciding" may include considering something as having "determined" or "decided" some operation. Also, "determining (deciding)" may be replaced by "assuming", "expecting", "considering", etc.

[0071] (14) In the above-described embodiments, when terms such as "include", "including" and their variants are used, these terms are intended to be inclusive, similar to the term "comprising". Further, the term "or" used in the present disclosure is not intended to be an exclusive disjunction.

[0072] (15) In the present disclosure, for example, when articles are added by translation, such as a, an and the in English, the present disclosure may include that the nouns following these articles are in the plural form.

[0073] (16) In the present disclosure, the term "A and B are different" may mean that "A and B are different from each other". Note that the term may also mean that "A and B are each different from C". Terms such as "separate", "coupled", etc. may also be interpreted in the same way as "different".

[0074] (17) Each aspect / embodiment described in the present disclosure may be used alone, in combination, or switched during execution. Also, the notification of predetermined information (for example, the notification of "being X") is not limited to being explicitly performed, and may be performed implicitly (for example, without performing the notification of the predetermined information).

[0075] (E: Aspects understood from the above-described forms or variations) As described above in detail, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described in the present disclosure. The present disclosure can be implemented in modified and changed forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is for illustrative purposes and has no restrictive meaning for the present disclosure. The following aspects can be grasped from at least one of the above-described embodiments or variations.

[0076] The specific system according to the first aspect includes a wearing device 20, an imaging device 13, an acquisition unit 181, and a specification unit 182. The wearing device 20 is worn on the head of the user U. The wearing device 20 detects the orientation of the head of the user U. The imaging device 13 is worn on the body of the user U separately from the wearing device 20. The acquisition unit 181 acquires first direction information indicating the orientation of the head of the user U detected by the wearing device 20 and second direction information indicating the orientation of the optical axis L2 of the imaging device 13. The specification unit 182 specifies a target area A1 that the user U is looking at in the captured image captured by the imaging device 13 based on the first direction information and the second direction information. According to the specific system according to the first aspect, even when the imaging device is not mounted on the wearing device 20, the target area A1 in the captured image by the imaging device 13 can be specified using an imaging device 13 separate from the wearing device 20. Further, according to the specific system according to the first aspect, even when the imaging device is mounted on the wearing device 20 but the image quality of the image captured by the imaging device is insufficient for specifying the target area, the target area A1 can be specified. In the embodiment, the portable device 10 including the imaging device 13 is worn on the body of the user U, but as in the first aspect, the imaging device 13 may be worn on the body of the user U alone.

[0077] In the second aspect (an example of the first aspect), the wearing device 20 may include a display unit 2a. As a specific example of the wearing device 20 including the display unit 2a, the wearing device 20A can be mentioned. The specific system according to the second aspect may further include a display control unit 183. The display control unit 183 causes the display unit 2a to display an image including the target area A1 specified by the specification unit 182. According to the specific system according to the second aspect, using an imaging device 13 separate from the wearing device 20, the target area A1 in the captured image of the imaging device 13 can be specified and the image corresponding to this target area A1 can be displayed.

[0078] In the specific system according to the third aspect (example of the first aspect), the specific part may identify the target area A1 while ignoring the difference between the position of the imaging device 13 and the position of the wearing device 20. When the wearing device 20 and the imaging device 13 are worn on the body of the user U, the distance between the wearing device 20 and the imaging device 13 is sufficiently small compared to the distance from the imaging device 13 to the subject. Therefore, when the wearing device 20 and the imaging device 13 are worn on the body of the user U, no particular problem occurs even if the difference between the position of the wearing device 20 and the position of the imaging device 13 is ignored and the target area A1 is identified. According to the specific system according to the third aspect, the target area A1 in the captured image of the imaging device 13 can be identified without using a sensor for detecting the position of the wearing device 20 and a sensor for detecting the position of the imaging device 13.

[0079] In the specific system according to the fourth aspect (example of the first aspect), the specific part may identify the target area A1 on the premise that the distance from the imaging device 13 to the subject and the viewing angle of the imaging device 13 are fixed. According to the specific system according to the fourth aspect, the target area A1 in the captured image of the imaging device 13 can be identified without using a sensor for detecting the distance from the imaging device 13 to the subject and a sensor for detecting the viewing angle of the imaging device 13.

Explanation of Reference Numerals

[0080] 1A, 1B... Specific system, 10... Portable device, 20, 20A, 20B, 40... Wearing device, 11... Input device, 12... Output device, 13... Imaging device, 14, 15, 2b... Communication device, 16, 2c... Direction detection device, 17, 2d... Storage device, 18, 2e... Processing device, 181... Acquisition unit, 182... Specific part, 183... Display control unit, 19, 2f... Bus, 2a... Display unit, 2e1, 2e2, 2e3... Operation control unit, PR1, PR2, PR3, PR4... Program.

Claims

1. A wearing device that is worn on a user's head and detects the orientation of the user's head, An imaging device that is worn on the user's body separately from the wearing device, An acquisition unit that acquires first direction information indicating the orientation of the user's head detected by the wearing device and second direction information indicating the orientation of the optical axis of the imaging device, A specifying unit that specifies a region of interest that the user is looking at in a captured image captured by the imaging device based on the first direction information and the second direction information, A specifying system having the above.

2. Further having a display control unit, The wearing device includes a display unit, The display control unit causes the display unit to display an image corresponding to the region of interest, The specifying system according to Claim 1.

3. The specifying unit specifies the region of interest ignoring the difference between the position of the imaging device and the position of the wearing device, The specifying system according to Claim 1.

4. The specifying unit specifies the region of interest on the assumption that the distance from the imaging device to the subject and the angle of view of the imaging device are fixed, The specifying system according to Claim 1.

Images