Information processing device, information processing method, and program

Abstract

An information processing device according to the present invention comprises a position processing unit, a correction unit, and a display control unit. The position processing unit infers the head position of a user and then infers the position of a predetermined part of the body of the user on the basis of the inferred head position. The correction unit corrects, on the basis of the head position and the angle of inclination of the head of the user, the position of the predetermined part which has been inferred by the position processing unit. The display control unit performs a process in which, on the basis of the position that has been corrected by the correction unit, an image operated by the user is displayed so as to be superimposed on a space viewed by the user.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] An example of an invention for determining contact with an object in a virtual space is an information processing device disclosed in Patent Document 1. This information processing device identifies the length of the user's arm and sets the reachable range of the object's collision area based on the collision area set for the object held by the user's avatar and the range within which the user can move their hand. Because the reachable range of the object's collision area is set according to the user's arm length, the reachable range of the collision area remains the same even when the user changes. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-101468 Summary of the Invention [Problem to be solved by the invention]

[0004] In virtual space, for example, users may operate a GUI (Graphical User Interface) to display information. The user corresponding to the avatar may crouch or tilt their head in the space, and depending on the user's posture, the reach of the avatar may change, making operability difficult and potentially reducing usability.

[0005] Therefore, the present disclosure proposes an information processing device, an information processing method, and a program that can suppress a decrease in usability. [Means for solving the problem]

[0006] According to the present disclosure, there is provided an information processing device including: a position processing unit that estimates a head position of a user and estimates a position of a predetermined part of the user's body based on the estimated head position, a correction unit that corrects the position of the predetermined part estimated by the position processing unit based on the head position and an angle of tilt of the user's head, and a display control unit that performs processing to display an image operated by the user in a space viewed by the user, superimposed on the image, based on the position corrected by the correction unit. Also, according to the present disclosure, there is provided an information processing method in which information processing of the information processing device is executed by a computer, and a program that causes a computer to realize the information processing of the information processing device. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing devices constituting an information processing system according to an embodiment. [Figure 2] FIG. 2 is a diagram showing an example of a work place and a GUI that are visually recognized by a user via a terminal device. [Figure 3] FIG. 3 is a diagram for explaining a method for calculating the position of the user's waist. [Figure 4] FIG. 4 is a diagram for explaining a method for correcting the position of the user's waist. [Figure 5] FIG. 5 is a diagram showing an example of a display mode of the operation button according to the embodiment. [Figure 6] FIG. 6 is a diagram showing an example of a work place and a GUI that the user visually recognizes via a terminal device. [Figure 7] FIG. 7 is a diagram showing an example of a display mode of the operation buttons and the virtual hand according to the embodiment. [Figure 8] FIG. 8 is a block diagram showing the hardware configuration and the functional configuration of the information processing device. [Figure 9] FIG. 9 is a block diagram showing the hardware configuration and the functional configuration of the terminal device. [Figure 10] FIG. 10 is a block diagram showing the hardware of the information providing device. [Figure 11]FIG. 11 is a flowchart showing the flow of the process executed when the operation buttons are displayed. [Figure 12] FIG. 12 is a flowchart showing the flow of processing for changing the display mode of the operation buttons. [Figure 13] FIG. 13 is a flowchart showing the flow of processing for changing the display mode of the operation buttons. [Figure 14] FIG. 14 is a diagram illustrating an example of a hardware configuration of a computer that realizes the functions of the information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0008] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0009] The explanation will be given in the following order. 1. Overview of Embodiments of the Present Disclosure Introduction 1.2. Overview of the information processing system 2. Example of an information processing system 2.1. AR display in the field work area 2.2. How to identify the user's waist position 2.3. Display of operation buttons 3. Information Processing System Configuration 3.1.Configuration of information processing device 3.2. Terminal Device Configuration 3.3.Configuration of information providing device 3.4. Processing of Information Processing Systems 3.5. Processing Variations 4. Hardware configuration example 5. Summary

[0010] <<1. Overview of Embodiments of the Present Disclosure>> <1.1. Introduction> In Synecoculture (registered trademark), a technology that uses Augmented Reality (AR) technology to visualize and display various information related to the ecosystem of plants and other things in a field (for example, the amount of sunlight and the amount of moisture in the soil) to workers is attracting attention.One method of displaying various types of information is to display a GUI such as buttons for displaying information on a terminal device using AR, and then operate the displayed GUI.

[0011] If the GUI display position is fixed in three-dimensional space, the GUI will be out of the field of view when the worker moves their line of sight or moves within three-dimensional space. In this case, in order to operate the GUI, the worker must first search for the GUI that has moved out of the field of view, which takes time and effort.

[0012] Therefore, the present disclosure proposes an information processing device, an information processing method, and a program that are easy to use.

[0013] In the following description, a farm worker will be referred to as a "user" where appropriate. The user may be a user who experiences the farm worker through AR. In the embodiment, the user is an actual farm worker.

[0014] <1.2. Overview of the information processing system> An overview of an information processing system 1 according to an embodiment will be described. FIG. 1 is a diagram illustrating devices constituting the information processing system 1. As illustrated in FIG. 1, the information processing system 1 includes an information processing device 10, a terminal device 20, an information providing device 30, and a sensor group 40. For example, the information processing device 10 is connected to a communication line N via a wired connection, but may also be connected wirelessly. Various devices may be connected to the information processing device 10. For example, the terminal device 20 and the information providing device 30 are connected to the information processing device 10 via the communication line N, and information is shared between the devices. The terminal device 20, the information providing device 30, and the sensor group 40 are also connected to the communication line N via a wired or wireless connection. The wireless connection of the terminal device 20, the information providing device 30, and the sensor group 40 to the communication line N is, for example, via a wireless LAN. However, the connection is not limited to a wireless LAN and may be, for example, a connection using Bluetooth (registered trademark).

[0015] The terminal device 20 is, for example, an optical see-through head-mounted display capable of AR display, such as HoloLens (registered trademark) or HoloLens 2. The terminal device 20 may also be a terminal device such as a smartphone capable of AR display using ARCore (registered trademark) or ARKit (registered trademark). The terminal device 20 may also be a video see-through AR device or XR device, such as XR-1 by Varjo (registered trademark). The terminal device 20 may also be a VR device, such as a head-mounted display, capable of VR display. The terminal device 20 sets a target range that the user can view through the terminal device 20, and performs AR display of various information related to the target range viewed in a field, for example, based on visualization information provided from the information processing device 10. The terminal device 20 is an example of an information processing device in the present disclosure.

[0016] The sensor group 40 includes sensors that measure various information related to the field, such as a camera that photographs the field, a sensor that measures the amount of sunlight in the field, and a sensor that measures the amount of moisture in the soil of the field.

[0017] The information providing device 30 is an information processing device that provides various information related to a target area to the information processing device 10. The information providing device 30 acquires and stores various information such as the amount of sunlight, the amount of soil moisture, and images of the field from a group of sensors 40 installed in the field. The information providing device 30 provides the various stored information related to the target area to the information processing device 10 in response to a request. The information providing device 30 is realized by a PC (Personal Computer), a WS (Work Station), or the like. Note that the information providing device 30 is not limited to a PC, a WS, or the like.

[0018] The information processing device 10 is an information processing device that performs processing to provide information to be AR-displayed on the terminal device 20 to the terminal device 20. Specifically, the information processing device 10 acquires sensor information, which will be described later, from the terminal device 20. The information processing device 10 requests various information, such as the amount of sunlight and the amount of soil moisture, in a target area identified using the acquired sensor information from the information providing device 30, and acquires the various information provided in response to the request. Then, the information processing device 10 provides visualization information for displaying the amount of sunlight and the amount of soil moisture in the target area based on the information acquired from the information providing device 30. The information processing device 10 is realized by a PC, a workstation, or the like. Note that the information processing device 10 is not limited to a PC, a workstation, or the like. For example, the information processing device 10 may be an information processing device, such as a PC or a workstation, in which the functions of the information processing device 10 are implemented as an application.

[0019] <<2. Example of Information Processing System>> Next, a description will be given of an example of the information processing system 1. In the example, it is assumed that a user wears a terminal device 20 which is a see-through head-mounted display. In the example, it is assumed that a work area is provided in the field.

[0020] 2.1. AR display in farm work areas 2 is a diagram showing an example of a work location viewed by a user via terminal device 20 and an AR-displayed GUI displayed on terminal device 20 and viewed by the user. As shown in FIG. 2, the vegetation viewed by the user in the work location includes a tomato V11, a carrot V12, a chive V13, etc. The AR display viewed by the user also includes operation buttons B11 to B13, which are examples of GUIs, a virtual hand HL, which is a virtual object visualizing the left hand of the worker sensed by terminal device 20, and a virtual hand HR, which is a virtual object visualizing the right hand of the worker sensed by terminal device 20.

[0021] Operation buttons B11 to B13 are displayed based on the position of the user's waist. Operation button B11 is a button for controlling the AR display of information on the amount of sunlight at the work site. Operation button B12 is a button for controlling the AR display of the amount of moisture in the soil at the work site. Operation button B13 is a button for controlling the AR display of the complexity of vegetation at the work site.

[0022] <2.2. How to identify the user's waist position> 3 is a diagram for explaining a method for calculating the position of the user's waist. For example, in a left-handed three-dimensional Cartesian coordinate system with a predetermined position in a farm field as the origin, terminal device 20 senses the relative position from the coordinates of the origin (0,0,0), and defines the coordinates of head position P1, which is the sensed position, as the user's head coordinates (xh, yh, zh).

[0023] The terminal device 20 identifies the coordinates (xp, yp, zp) of the user's waist position P2 based on the head position P1. For example, as shown in FIG. 3(a), if the vertical height h from the ground, where the origin of the field is located, to the head position P1 is higher than a threshold H, which is a predetermined reference height, the terminal device 20 determines the user's waist position P2 to be a position that is a distance α below the head position P1 to the waist. If the coordinates of the head position P1 are (xh, yh, zh), the coordinates (xp, yp, zp) of the waist position P2 in this case are (xh, yh-α, zh). The distance α may be a distance based on a predetermined human body model, or may be a distance obtained by measuring in advance the distance from the head position to the waist when the user is standing upright.

[0024] On the other hand, when the user is in a crouched position as shown in FIG. 3(b), and the vertical height h from the ground where the origin of the field is located to head position P1 is equal to or less than a predetermined threshold H, terminal device 20 determines that user's waist position P2 is a distance β below head position P1. Distance β is shorter than distance α. If the coordinates of head position P1 are (xh, yh, zh), then the coordinates (xp, yp, zp) of waist position P2 in this case are (xh, yh-β, zh).

[0025] Having identified the waist position P2, the terminal device 20 corrects the user's waist position P2 in accordance with the tilt of the terminal device 20. FIG. 4 is a diagram for explaining a method for correcting the user's waist position P2. The terminal device 20 sets a left-handed three-dimensional Cartesian coordinate system with the head coordinate as the origin. If the tilt (pitch) angle θ around the x-axis of the left-handed three-dimensional Cartesian coordinate system with the head coordinate as the origin is not within a predetermined range, the terminal device 20 corrects the user's waist position P2.

[0026] 4(a) shows a state in which the user is in a standing position and the terminal device 20 is not rotating around the x-axis of a left-handed three-dimensional Cartesian coordinate system with the head coordinate as the origin. The terminal device 20 senses the angle θ of tilt of the terminal device 20 around the x-axis from the horizontal position shown in FIG. 4(a) and the moving distance Δh of the head position P1 in the vertical direction from the standing position.

[0027] For example, when the terminal device 20 is tilted downward beyond a predetermined range as shown in FIG. 4(b), if the coordinates of the waist position before correction are (xp, yp, zp) in a left-handed three-dimensional Cartesian coordinate system with the head coordinate as the origin, the terminal device 20 calculates the coordinates of the waist position after correction using the following equations (1) to (3). xp after correction = xp before correction (1) yp after correction = yp before correction + Δh (2) Corrected zp = uncorrected zp - Δh ÷ tanθ (3)

[0028] Furthermore, for example, when the terminal device 20 is tilted upward beyond a predetermined range as shown in FIG. 4(c), if the coordinates of the waist position before correction are (xp, yp, zp), the terminal device 20 calculates the coordinates of the waist position after correction using the following equations (4) to (6). xp after correction = xp before correction (4) Corrected yp = uncorrected yp - Δh (5) zp after correction = zp before correction + Δh ÷ tanθ (6)

[0029] The terminal device 20 displays the operation buttons B11 to B13 based on the corrected waist position P3 in a left-hand three-dimensional Cartesian coordinate system with the head position P1 as the origin. For example, the terminal device 20 displays the operation button B12 at the corrected waist position P3. Furthermore, the terminal device 20 displays the operation button B11 and the operation button B13 at positions offset by a predetermined amount in the x-axis direction and the z-axis direction from the waist position P3 in the left-hand three-dimensional Cartesian coordinate system with the head position P1 as the origin. As a result, the operation buttons B11 to B13 are AR-displayed in the user's field of view as shown in FIG. 2. The user operates the AR-displayed operation buttons B11 to B13 with the virtual hand HL or virtual hand HR. The display manner of the operation buttons B11 to B13 changes depending on the operation with the virtual hand HL or virtual hand HR.

[0030] <2.3. Display of operation buttons> 5 is a diagram showing an example of the display mode of the operation button B11. The operation button B11 has a transparent hemispherical operated portion C1, a ring-shaped indicator portion C2, and a function display portion C3 including an icon representing the function corresponding to the operation button B11.

[0031] The terminal device 20 determines whether the virtual hand HL or the virtual hand HR is in contact with the operated portion C1. If the terminal device 20 determines that the virtual hand HL or the virtual hand HR is not in contact with the operated portion C1, the terminal device 20 displays the operated portion C1 in a hemispherical shape as shown in FIG. 5(a).

[0032] When the terminal device 20 determines that the virtual hand HL or virtual hand HR has contacted the operated unit C1, it displays the height of the operated unit C1 lower than the state shown in FIG. 5(a). The terminal device 20 changes the display of the indicator unit C2 according to the contact time of the virtual hand HL or virtual hand HR with the operated unit C1. Specifically, the terminal device 20 changes the color of the indicator unit C2 clockwise according to the contact time of the virtual hand HL or virtual hand HR with the operated unit C1. As a result, the color of the indicator unit C2 changes clockwise as the contact time increases, as shown in FIGS. 5(c) and 5(d). When the contact time of the virtual hand HL or virtual hand HR with the operated unit C1 exceeds a predetermined threshold T, the terminal device 20 changes the color of the indicator unit C2 over the entire circumference, as shown in FIG. 5(e). When the virtual hand HL or virtual hand HR leaves the operated unit C1 before the contact time reaches the threshold T, the terminal device 20 returns the display of the operation button B11 to the state shown in FIG. 5(a).

[0033] When the contact time of the virtual hand HL or virtual hand HR with the operated portion C1 is equal to or longer than the threshold T, the terminal device 20 executes a process corresponding to the button with which the virtual hand HL or virtual hand HR is in contact. For example, when the contact time of the virtual hand HL or virtual hand HR with the operated portion C1 of the operation button B11 is equal to or longer than the threshold T, the terminal device 20 requests visualization information of the amount of sunlight in the target range viewed by the user from the information processing device 10. Based on the visualization information sent from the information processing device 10, the terminal device 20 AR-displays a graph GR of the amount of sunlight as shown in FIG. 6.

[0034] In addition, when the terminal device 20 determines that the virtual hand HL or the virtual hand HR has come into contact with the operated part C1, it may display the height of the operated part C1 in another display mode, rather than displaying it lower than when the virtual hand HL or the virtual hand HR is not in contact with the operated part C1.

[0035] 7 is a diagram showing an example of a display mode of the operation button B11 when a part of the virtual hand HR is located within the operation button B11. The terminal device 20 identifies an area of ​​the virtual hand HL or virtual hand HR that is located within the operated unit C1 based on the position of the virtual hand HL or virtual hand HR and the display position of the operation button B11. The terminal device 20 may display the identified area in a color different from that of an area outside the operated unit C1, as shown in FIG.

[0036] <<3. Information Processing System Configuration>> Next, the configuration of the information processing system 1 will be described.

[0037] <3.1. Configuration of information processing device> 8 is a block diagram showing the functional configuration of the information processing device 10. As shown in FIG.

[0038] The communication unit 110 has a function of communicating with an external device. For example, in communication with the external device, the communication unit 110 supplies information received from the external device to the control unit 100. Specifically, the communication unit 110 supplies information received from the information providing device 30 and information received from the terminal device 20 to the control unit 100. The communication unit 110 also transmits information supplied from the control unit 100 to the external device. Specifically, the communication unit 110 acquires, from the control unit 100, visualization information regarding the target range generated by the control unit 100 based on information supplied from the information providing device 30, and transmits the acquired visualization information to the terminal device 20. The communication unit 110 also acquires, from the control unit 100, information representing the target range generated by the control unit 100 based on sensor information supplied from the terminal device 20, and transmits the acquired information to the information providing device 30.

[0039] The storage unit 120 is realized by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 120 has a function of storing information related to processing in the information processing device 10. The storage unit 120 stores, for example, information related to the target range supplied from the information providing device 30.

[0040] The control unit 100 executes a process for controlling the operation of the information processing device 10. For example, the control unit 100 executes a process for providing visualized information to be AR-displayed in a target range on the terminal device 20, based on information supplied from the information providing device 30. To execute this process, the control unit 100 includes an acquisition unit 101, a processing unit 102, and an output unit 103, as shown in FIG.

[0041] The acquisition unit 101 has a function of acquiring information for generating visualization information. The acquisition unit 101 acquires, for example, sensor information transmitted from the terminal device 20 via the communication unit 110. For example, the acquisition unit 101 acquires sensor information for identifying the position of the terminal device 20 and a target range, such as acceleration information, direction information, depth information, angular velocity information, map information, and location information, which will be described later. The acquisition unit 101 also acquires, via the communication unit 110, information on the target range transmitted from the information providing device 30. For example, the acquisition unit 101 acquires information on the amount of sunlight and the amount of soil moisture in the target range from the information providing device 30.

[0042] The processing unit 102 has a function of generating visualization information, and includes a position specifying unit 1021 and a generating unit 1022, as shown in FIG.

[0043] The position identification unit 1021 has a function of identifying a target range. The position identification unit 1021 identifies the position of the terminal device 20 and the direction in which the user is facing, based on sensor information acquired from the terminal device 20. The position identification unit 1021 identifies a target range that can be viewed by the user through the terminal device 20 in the field, based on the identified position and direction.

[0044] The generation unit 1022 has a function of generating AR images that represent three-dimensional graphs of the amount of sunlight and the amount of soil moisture, and the complexity of vegetation in the target range identified by the position identification unit 1021. The generation unit 1022 requests information on the amount of sunlight, the amount of soil moisture, and the complexity of vegetation for the target range identified by the position identification unit 1021 from the information providing device 30 via the communication unit 110. The generation unit 1022 stores the information provided by the information providing device 30 in response to this request in the storage unit 120. The generation unit 1022 generates three-dimensional graphs of the amount of sunlight and the amount of soil moisture in the target range, and AR images that represent the complexity of vegetation, based on the information on the amount of sunlight, the amount of soil moisture, and the complexity of vegetation for the target range supplied from the information providing device 30.

[0045] The output unit 103 has a function of outputting information representing the target range. The output unit 103 outputs information representing the target range identified by the position identification unit 1021 to the communication unit 110. The information representing the target range output from the output unit 103 to the communication unit 110 is transmitted to the information providing device 30. The output unit 103 also has a function of outputting visualization information. The output unit 103 outputs visualization information generated by the generation unit 1022 to the communication unit 110. The visualization information output from the output unit 103 to the communication unit 110 is transmitted to the terminal device 20.

[0046] <3.2. Terminal Device Configuration> 9 is a block diagram showing the hardware configuration and functional configuration of the terminal device 20. The terminal device 20 has a control unit 200, a storage unit 210, a video output unit 220, an audio output unit 230, an external communication unit 240, and a sensor unit 250.

[0047] The sensor unit 250 includes a head position measurement unit 251 , a hand posture measurement unit 252 , and a voice acquisition unit 253 .

[0048] The head position measurement unit 251 has an acceleration sensor 251a, a direction sensor 251b, a depth sensor 251c, a gyro sensor 251d, a SLAM 251e, and a GPS module 251f. The acceleration sensor 251a is, for example, a three-axis acceleration sensor. The acceleration sensor 251a outputs acceleration information representing the measured acceleration. The direction sensor 251b is a sensor that measures geomagnetism and detects the direction in which the terminal device 20 is facing. The direction sensor 251b outputs direction information representing the detected direction. The depth sensor 251c is a sensor that measures the distance from the terminal device 20 to a person or object present within a target range. The depth sensor 251c outputs depth information representing the measured distance. The gyro sensor 251d is a sensor that measures the angular velocity of the terminal device 20. The gyro sensor 251d outputs angular velocity information representing the measured angular velocity. The SLAM 251e is, for example, a Lidar (Light Detection And Ranging) SLAM (Simultaneous Localization and Mapping) module equipped with a laser scanner, or a Visual SLAM (Visual SLAM) module equipped with a camera. The SLAM 251e senses a target range and outputs map information representing an environmental map of the target range. The GPS module 251f receives radio waves measured from a satellite in a satellite positioning system and measures the position of the terminal device 20. The GPS module 251f outputs position information representing the measured position.

[0049] Hand posture measurement unit 252 has depth sensor 252a and infrared camera 252b. Infrared camera 252b outputs infrared light and receives the infrared light reflected by the user's hand to capture an image of the user's hand. Depth sensor 252a measures the distance to the user's hand based on an image of the user's hand generated by infrared camera 252b. Hand posture measurement unit 252 outputs hand posture information including the measured distance to the user's hand and an image of the user's hand.

[0050] The voice acquisition unit 253 includes a microphone 253a. The microphone 253a collects sounds around the terminal device 20 and outputs voice information representing the collected sounds.

[0051] The storage unit 210 is realized by, for example, a semiconductor memory element such as a RAM or a flash memory. The storage unit 210 has a function of storing information related to processing in the terminal device 20. The storage unit 120 stores, for example, visualization information supplied from the information processing device 10. The storage unit 120 also stores application programs executed by the terminal device 20. The application programs stored in the storage unit 120 are, for example, programs that allow a user to visually recognize graphs of the amount of solar radiation and soil moisture in a field, the complexity of vegetation, etc., through AR display.

[0052] The control unit 200 is realized by executing an application program stored in the storage unit 210. The control unit 200 has a position processing unit 201, a hand posture processing unit 202, a correction unit 203, a display processing unit 204, a display control unit 205, and a communication control unit 206, as shown in FIG.

[0053] The position processing unit 201 identifies the head position P, the waist position P2 (an example of a predetermined body part), the tilt angle θ of the terminal device 20, and the like, based on sensor information output from the sensor unit 250. The hand posture processing unit 202 identifies the position and posture of the user's hands based on hand posture information output from the hand posture measurement unit 252. The correction unit 203 corrects the waist position P2 based on the head position P and the angle θ. The display processing unit 204 generates images to be displayed in AR of the operation buttons B11 to B13, etc., and generates images of the virtual hands HL and HR based on the positions and postures identified by the hand posture processing unit 202. The display processing unit 204 also generates images to be displayed in AR, such as a graph GR, based on visualization information provided by the information processing device 10. The display control unit 205 controls the video output unit 220 so that the images of the operation buttons B11 to B13, etc., generated by the display processing unit 204, are displayed in AR at the waist position P2 corrected by the correction unit 203. The display control unit 205 also controls the video output unit 220 so that images of the virtual hands HL and HR are AR-displayed at the hand positions identified by the hand posture processing unit 202. The display control unit 205 also controls the video output unit 220 so that an image generated based on the visualization information in the display processing unit 204 is AR-displayed in a symmetrical range. The communication control unit 206 controls the external communication unit 240 to transmit information to the information processing device 10 and receive information from the information processing device 10.

[0054] The video output unit 220 displays, on the half mirror, an AR image that is output from the control unit 200 and that is visually recognized by the user. The audio output unit 230 includes a speaker and outputs sound represented by an audio signal supplied from an external device. The external communication unit 240 has a function of communicating with an external device. For example, in communication with the external device, the external communication unit 240 supplies information received from the external device to the control unit 200. Specifically, the external communication unit 240 supplies visualization information received from the information processing device 10 to the control unit 200. Furthermore, in communication with the external device, the external communication unit 240 transmits information supplied from the control unit 200 to the external device. Specifically, the external communication unit 240 transmits sensor information output from the sensor unit 250 to the control unit 200 to the information processing device 10.

[0055] <3.3. Configuration of information providing device> 10 is a block diagram showing the hardware configuration of the information providing device 30. As shown in FIG.

[0056] The communication unit 320 has a function of communicating with an external device. For example, in communication with the external device, the communication unit 320 outputs information received from the external device to the control unit 300. Specifically, the communication unit 320 outputs information received from the information processing device 10 to the control unit 300. For example, the communication unit 320 outputs information indicating a target range transmitted from the information processing device 10 to the control unit 300. Furthermore, the communication unit 320 transmits information related to the target range supplied from the control unit 300 to the information processing device 10.

[0057] The control unit 300 has a function of controlling the operation of the information providing device 30. For example, the control unit 300 transmits various information related to the target range to the information processing device 10 via the communication unit 320. For example, the control unit 300 accesses the storage unit 310 and transmits to the information processing device 10 the amount of sunlight, the amount of soil moisture, and the complexity of vegetation in the target range obtained.

[0058] The storage unit 310 is realized by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 310 has a function of storing data related to processing in the information providing device 30. The storage unit 310 stores, for example, information on the amount of sunlight and the amount of moisture in the soil obtained from a group of sensors 40 installed in the field, images of the field, etc.

[0059] <3.4. Processing of information processing systems> Next, the processing performed in the information processing system 1 will be described. FIG. 11 is a flowchart showing the flow of processing executed when the operation buttons B11 to B13 are displayed on the terminal device 20. First, the terminal device 20 sets a predetermined position in the field as the origin (0,0,0) (step S101). Next, the terminal device 20 identifies the coordinates of the head position P1 when the user is in a standing position, based on the information output from the sensor unit 250 (step S102). Furthermore, the terminal device 20 identifies the angle θ of inclination of the terminal device 20 from the horizontal position and the moving distance Δh of the terminal device 20 in the vertical direction from the standing position, based on the information output from the sensor unit 250 (step S103).

[0060] The terminal device 20 determines whether the height of the head position P1 is higher than the aforementioned threshold value H based on the coordinates identified in step S102 (step S104). If the height of the head position P1 is higher than the threshold value H (Yes in step S104), the terminal device 20 determines the user's waist position P2 to be a position that is a distance α below the head position P1 (step S105). On the other hand, if the height of the head position P1 is equal to or less than the threshold value H (No in step S104), the terminal device 20 determines the user's waist position P2 to be a position that is a distance β below the head position P1 (step S106).

[0061] Next, the terminal device 20 determines whether the angle θ of tilt from the horizontal state identified in step S103 is within a predetermined range (-A≦θ≦A) (step S107). If the angle θ of tilt from the horizontal state is within the predetermined range (Yes in step S107), the terminal device 20 proceeds to step S111.

[0062] If the tilt angle θ from the horizontal state is not within a predetermined range (No in step S107), the terminal device 20 determines whether the terminal device 20 has tilted downward from the horizontal plane (step S108). If the terminal device 20 has tilted downward from the horizontal plane (Yes in step S108), the terminal device 20 corrects the coordinates of the waist position P2 upward and backward (step S109). Here, as described above, the terminal device 20 calculates the coordinates of the waist position P2 after correction using the above-mentioned equations (1) to (3). Furthermore, if the terminal device 20 has tilted upward from the horizontal plane (No in step S108), the terminal device 20 corrects the coordinates of the waist position P2 downward and forward (step S110). Here, as described above, the terminal device 20 calculates the coordinates of the waist position P2 after correction using the above-mentioned equations (4) to (6). After the process of step S109 or step S110, the terminal device 20 advances the process flow to step S111.

[0063] In step S111, the terminal device 20 determines whether the origin has moved (step S111). If the origin has moved (Yes in step S111), the terminal device 20 ends the processing shown in FIG. 11. If the origin has not moved (No in step S111), the terminal device 20 displays the operation buttons B11 to B13 at the waist position P2 (step S113). Here, if the terminal device 20 determines Yes in step S107, it displays the operation buttons B11 to B13 based on the waist position P2 set in step S105 or step S106. If the terminal device 20 determines No in step S107, it displays the operation buttons B11 to B13 based on the waist position P2 corrected in step S109 or step S110. Next, the terminal device 20 updates the coordinates of the head position P1 based on the information output from the sensor unit 250 (step S113), and returns the processing flow to step S103.

[0064] 12 is a flowchart showing the flow of processing for changing the display mode of the operation buttons B11 to B13. First, before displaying the operation buttons B11 to B13, the terminal device 20 initializes the contact time t between the virtual hand and the operation button (step S201). Next, the terminal device 20 displays the operation buttons B11 to B13 by making the operated portion C1 into a hemispherical shape based on the waist position P2 identified in the processing of FIG. 11 (step S202).

[0065] After displaying the operation buttons B11 to B13, the terminal device 20 determines whether the virtual hand HL or the virtual hand HR is in contact with the operated unit C1 (step S203). Specifically, the terminal device 20 determines whether the virtual hand HL or the virtual hand HR is in contact with the operated unit C1 based on the display positions of the operation buttons B11 to B13 and the positions of the virtual hand HL and the virtual hand HR. If the virtual hand HL or the virtual hand HR is not in contact with the operated unit C1 (No in step S203), the terminal device 20 returns the processing flow to step S201. If the virtual hand HL or the virtual hand HR is in contact with the operated unit C1 (Yes in step S203), the terminal device 20 determines whether the contact time t=0 (step S204). If the contact time t=0 (Yes in step S204), the terminal device 20 starts measuring the contact time t (step S205). If the contact time t is not 0 (No in step S204), the terminal device 20 updates the contact time t (step S206).

[0066] After step S205 or step S206, the terminal device 20 determines whether the measured contact time t is less than the aforementioned threshold value T (step S207). If the contact time t is less than the threshold value T (Yes in step S207), the terminal device 20 displays the height of the operated part C1 in contact with the virtual hand HL or virtual hand HR in a contact state display mode in which the height is lowered from the hemispherical shape as shown in FIG. 5(b) (step S208). Next, the terminal device 20 changes the color of the indicator part C2 according to the contact time t as shown in FIGS. 5(b) to 5(d) (step S209). Then, the terminal device 20 returns the process flow to step S203.

[0067] On the other hand, if the contact time t is equal to or longer than the threshold (No in step S207), the terminal device 20 displays the operated part C1 in a hemispherical non-contact state display mode (step S210), as shown in Fig. 5(a). Next, the terminal device 20 executes a process corresponding to the operation button that the virtual hand HL or virtual hand HR is in contact with (step S211), and ends the process.

[0068] For example, when the contact time t of the virtual hand HL or the virtual hand HR on the operation button B11 is equal to or longer than a threshold T, the terminal device 20 transmits sensor information to the information processing device 10. The information processing device 10 identifies the user's visible range based on the sensor information transmitted from the terminal device 20. The information processing device 10 transmits information representing the identified visible range to the information providing device 30. The information providing device 30 transmits information on the amount of sunlight in the visible range represented by the information transmitted from the information processing device 10 to the information processing device 10. The information processing device 10 generates an AR image representing a three-dimensional graph of the amount of sunlight in the target range based on the information transmitted from the information providing device 30, and transmits information about the generated AR image as visualization information to the terminal device 20. The terminal device 20 receives the visualization information transmitted from the information processing device 10 and displays a graph GR of the AR image represented by the received visualization information, for example, as shown in FIG. 6 .

[0069] <3.5. Processing Variations> Next, variations of the processing of this embodiment will be described. Note that the variations of the processing described below may be applied to this embodiment alone or in combination with each other. Furthermore, the variations of the processing may be applied in place of the configuration described in this embodiment, or may be applied in addition to the configuration described in this embodiment.

[0070] Fig. 13 is a flowchart showing the flow of processing for changing the display mode of the operation buttons B11 to B13 in accordance with the overlap of the virtual hand HR or virtual hand HL with the operated unit C1, as shown in Fig. 7. First, the terminal device 20 identifies the position and posture of the virtual hand HR or virtual hand HL based on the hand posture information output from the hand posture measurement unit 252 (step S301).

[0071] Next, the terminal device 20 determines whether at least a part of the virtual hand HR or virtual hand HL is located within the operated unit C1 based on the identified position and posture of the virtual hand HR or virtual hand HL and the display positions of the operation buttons B11 to B13 (step S302). If at least a part of the virtual hand HR or virtual hand HL is not located within the operated unit C1 (No in step S302), the terminal device 20 displays the virtual hand HR and virtual hand HL in the same color, for example, as shown in FIG. 2 (step S305), and ends the process.

[0072] On the other hand, if a part of the virtual hand HR or the virtual hand HL is located within the operated unit C1 (Yes in step S302), the terminal device 20 identifies an area of ​​the virtual hand HR or the virtual hand HL that is located within the operated unit C1 (step S303).Then, the terminal device 20 changes the color of the area identified in step S303, for example, as shown in FIG. 7 (step S304), and returns the process to step S301.

[0073] In the present invention, the display positions of the operation buttons B11 to B13 may be corrected according to the length of the user's arm. In the present invention, the display positions of the operation buttons B11 to B13 may be corrected according to the user's motion state. In the present invention, the user's posture state may be estimated from the trajectory of the user's posture identified by the acceleration sensor 251a and the gyro sensor 251d, and the waist position P2 may be corrected according to the estimated posture state. In the present invention, an object may be detected by the sensor unit 250, and if the position of the detected object overlaps with the display positions of the operation buttons B11 to B13, the display positions of the operation buttons B11 to B13 may be corrected so that they do not overlap with the detected object. In the present invention, voice acquired by the voice acquisition unit 253 may be recognized, and if the recognized voice is voice instructing the display of the operation buttons B11 to B13, the operation buttons B11 to B13 may be displayed at the position of the virtual hand HL or virtual hand HR. In the present invention, whether or not the displayed operation buttons are operable may be indicated by the color or transparency of the operated unit C1. In the present invention, the direction of the user's gaze may be detected, and if the user's gaze is directed toward one of the operation buttons B11-B13 for a certain period of time or longer, the function corresponding to the operation button in the user's gaze direction may be executed. In the present invention, the user may be notified of the passage of contact time by vibration or sound. In the present invention, the aforementioned thresholds H and T may be set to values ​​determined for each user. In the present invention, contact between the virtual hand HL and the virtual hand HR may be determined over a wider area than the displayed operated unit C1. In the present invention, the terminal device 20 displays the operation buttons B11-B13 around the user's waist. However, the display position of the operation buttons B11-B13 is not limited to the waist position and may be, for example, around the user's chest. While the above-described embodiment is configured to provide AR display compatible with symbiotic farming, the AR display displayed by the information processing system 1 is not limited to AR display compatible with symbiotic farming. For example, AR display compatible with work in a factory or at a construction site may be provided.

[0074] <<4. Hardware configuration example>> Next, an example of the hardware configuration of an information processing device according to an embodiment will be described with reference to Fig. 14. Fig. 14 is a block diagram showing an example of the hardware configuration of an example of a computer that realizes the functions of the information processing device according to the embodiment. Note that the information processing device 900 shown in Fig. 14 can realize, for example, the information processing device 10, the terminal device 20, and the information providing device 30 shown in Fig. 1. Information processing by the information processing device 10, the terminal device 20, and the information providing device 30 according to the embodiment is realized by cooperation between software and hardware described below.

[0075] 14, the information processing device 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903. The information processing device 900 also includes a host bus 904a, a bridge 904, an external bus 904b, an interface 905, an input device 906, an output device 907, a storage device 908, a drive 909, a connection port 910, and a communication device 911. Note that the hardware configuration shown here is an example, and some of the components may be omitted. The hardware configuration may also include components other than those shown here.

[0076] The CPU 901 functions as, for example, an arithmetic processing device or a control device, and controls the overall operation or part of the operation of each component based on various programs recorded in the ROM 902, the RAM 903, or the storage device 908. The ROM 902 is a means for storing programs loaded into the CPU 901 and data used for calculations. The RAM 903 temporarily or permanently stores, for example, the programs loaded into the CPU 901 and various parameters that change as appropriate when the programs are executed. These are interconnected by a host bus 904a that includes a CPU bus or the like. The CPU 901, the ROM 902, and the RAM 903 can, for example, cooperate with software to realize the functions of the control units 100, 200, and 300 described with reference to FIGS. 8 to 10 .

[0077] The CPU 901, ROM 902, and RAM 903 are interconnected via, for example, a host bus 904a capable of high-speed data transmission. On the other hand, the host bus 904a is connected to an external bus 904b, which has a relatively low data transmission speed, via, for example, a bridge 904. Furthermore, the external bus 904b is connected to various components via an interface 905.

[0078] The input device 906 is realized by a device to which information is input, such as a mouse, keyboard, touch panel, button, microphone, switch, or lever. The input device 906 may also be, for example, a remote control device that uses infrared or other radio waves, or an externally connected device such as a mobile phone or PDA that is compatible with the operation of the information processing device 900. The input device 906 may also include, for example, an input control circuit that generates an input signal based on information input using the above-described input means and outputs the signal to the CPU 901. A user of the information processing device 900 can input various data to the information processing device 900 and instruct processing operations by operating the input device 906.

[0079] Alternatively, the input device 906 may be formed by a device that detects the position of a user. For example, the input device 906 may include various sensors such as an image sensor (e.g., a camera), a depth sensor (e.g., a stereo camera), an acceleration sensor, a gyro sensor, a geomagnetic sensor, a light sensor, a sound sensor, a distance measurement sensor (e.g., a ToF (Time of Flight) sensor), and a force sensor. The input device 906 may also acquire information about the state of the information processing device 900 itself, such as the attitude and movement speed of the information processing device 900, and information about the space around the information processing device 900, such as the brightness and noise around the information processing device 900. The input device 906 may also include a GNSS module that receives GNSS signals from GNSS (Global Navigation Satellite System) satellites (e.g., GPS signals from GPS (Global Positioning System) satellites) to measure position information including the latitude, longitude, and altitude of the device. Regarding location information, the input device 906 may detect the location by sending and receiving data via Wi-Fi (registered trademark), a mobile phone, a PHS, a smartphone, or by short-range communication. The input device 906 may realize the function of the sensor unit 250 described with reference to FIG. 9, for example.

[0080] The output device 907 is formed by a device capable of visually or audibly notifying the user of acquired information. Examples of such devices include display devices such as CRT display devices, liquid crystal display devices, plasma display devices, EL display devices, laser projectors, LED projectors, and lamps, audio output devices such as speakers and headphones, and printer devices. The output device 907 outputs, for example, results obtained from various processes performed by the information processing device 900. Specifically, the display device visually displays the results obtained from various processes performed by the information processing device 900 in various formats such as text, images, tables, and graphs. On the other hand, the audio output device converts audio signals consisting of reproduced audio data, audio data, etc. into analog signals and outputs them audibly. The output device 907 can, for example, realize the functions of the video output unit 220 and the audio output unit 230 described with reference to FIG. 9.

[0081] The storage device 908 is a data storage device formed as an example of a storage unit of the information processing device 900. The storage device 908 is realized, for example, by a magnetic storage device such as an HDD, a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 908 may include a storage medium, a recording device that records data on the storage medium, a reading device that reads data from the storage medium, and a deletion device that deletes data recorded on the storage medium. The storage device 908 stores programs executed by the CPU 901, various data, and various data acquired from the outside. The storage device 908 can realize the functions of the storage units 120, 210, and 310 described with reference to FIGS. 8 to 10, for example.

[0082] The drive 909 is a reader / writer for a storage medium, and is built into or externally attached to the information processing device 900. The drive 909 reads information recorded on a removable storage medium such as an attached magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs the information to the RAM 903. The drive 909 can also write information to the removable storage medium.

[0083] The connection port 910 is a port for connecting an external device, such as a Universal Serial Bus (USB) port, an IEEE1394 port, a Small Computer System Interface (SCSI), an RS-232C port, or an optical audio terminal.

[0084] The communication device 911 is, for example, a communication interface formed by a communication device or the like for connecting to the communication network 920. The communication device 911 is, for example, a communication card for a wired or wireless LAN (Local Area Network), LTE (Long Term Evolution), Bluetooth (registered trademark), or WUSB (Wireless USB). The communication device 911 may also be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various communications. The communication device 911 can transmit and receive signals, for example, between the Internet and other communication devices in accordance with a predetermined protocol such as TCP / IP. The communication device 911 can implement, for example, the functions of the communication unit 110, the external communication unit 240, and the communication unit 320 described with reference to FIGS. 8 to 10 .

[0085] The communication network 920 is a wired or wireless transmission path for information transmitted from devices connected to the communication network 920. For example, the communication network 920 may include public network such as the Internet, a telephone network, or a satellite communication network, or various LANs (Local Area Networks) including Ethernet (registered trademark), and WANs (Wide Area Networks). The communication network 920 may also include a dedicated network such as an IP-VPN (Internet Protocol-Virtual Private Network).

[0086] The above describes an example of a hardware configuration capable of realizing the functions of the information processing device 900 according to the embodiment. Each of the above components may be realized using general-purpose components, or may be realized by hardware specialized for the function of each component. Therefore, the hardware configuration used can be changed as appropriate depending on the technical level at the time of implementing the embodiment.

[0087] <<5. Summary>> As described above, the terminal device 20 according to the embodiment performs a process of correcting the display position of the AR-displayed GUI based on the vertical position of the terminal device 20 and the tilt of the terminal device 20. As a result, the operation buttons B11 to B13 are always displayed near the user's waist. This allows the user to quickly find the operation buttons B11 to B13 even when the user changes position or posture, thereby improving usability. Furthermore, in the present embodiment, the waist position is calculated based on the user's body position, allowing the operation buttons B11 to B13 to be always operated at the same waist position. Furthermore, in the present embodiment, the operation buttons B11 to B13 are always displayed based on the user's waist. As the user becomes accustomed to operating the operation buttons, the user can learn the positional relationship between the operation buttons B11 to B13 and their own body. Therefore, the operation buttons B11 to B13 can be operated by physical sensations without having to center the operation buttons B11 to B13 in the user's field of view, and can be operated with one hand while working.

[0088] In addition, in this embodiment, the display modes of the operated portion C1 and the indicator portion C2 change in response to the operation, so the user can notice the change in the display mode and can operate the operation buttons B11 to B13 while performing other tasks. Furthermore, the display mode shown in Fig. 7 allows the user to recognize that his or her hand is in contact with the operated portion C1, so the user can easily recognize that he or she is in contact with the operated portion C1.

[0089] The present technology can also be configured as follows. (1) a position processing unit that estimates a head position of a user and estimates a position of a predetermined part of the user's body based on the estimated head position; a correction unit that corrects the position of the predetermined part estimated by the position processing unit based on the head position and an angle of inclination of the user's head; a display control unit that performs processing to superimpose and display an image operated by the user in a space visually recognized by the user based on the position corrected by the correction unit; An information processing device comprising: (2) The correction unit corrects the vertical position of the predetermined portion based on a reference height. The information processing device according to (1) above. (3) The correction unit corrects the position of the predetermined part in the front-back direction of the user when the tilt angle of the head from the horizontal state is outside a predetermined range. The information processing device according to (1) or (2). (4) The correction unit corrects the position of the predetermined part toward the rear of the user when the head is tilted downward from a horizontal position. The information processing device according to (3) above. (5) The correction unit corrects the position of the predetermined part toward the front of the user when the head is tilted upward from a horizontal position. The information processing device according to (3) or (4). (6) a display processing unit that changes a display mode of the image when the image is operated by the user; The information processing device according to (1) above. (7) The display processing unit changes the display mode of the image according to the elapsed time since the user started an operation on the image. The information processing device according to (6) above. (8) the position processing unit estimates the position of the user's hand; the display control unit displays an image of the user's hand superimposed on a space visually recognized by the user based on the position estimated by the position processing unit; The display processing unit changes a display mode of the image of the hand in accordance with an overlap with an image operated by the user. The information processing device according to (6) or (7). (9) a position processing step of estimating a head position of the user and estimating a position of a predetermined part of the user's body based on the estimated head position; a correcting step of correcting the position of the predetermined part estimated in the position processing step based on the head position and an angle of inclination of the user's head; a display control step of performing a process of superimposing and displaying an image operated by the user in a space visually recognized by the user based on the position corrected in the correction step; An information processing method comprising: (10) On the computer, a position processing step of estimating a head position of the user and estimating a position of a predetermined part of the user's body based on the estimated head position; a correcting step of correcting the position of the predetermined part estimated in the position processing step based on the head position and an angle of inclination of the user's head; a display control step of performing a process of superimposing and displaying an image operated by the user in a space visually recognized by the user based on the position corrected in the correction step; A program that executes the following. [Explanation of symbols]

[0090] 1. Information Processing Systems 10. Information processing equipment 20 Terminal equipment 30 Information provision device 40 sensors 100 control section 101 Acquisition Department 102 Processing section 1021 Location identification part 1022 Generation part 103 Output section 110 Communications Department 120 Storage section 200 control section 201 Position processing section 202 Hand posture processing unit 203 Correction Unit 204 Display processing unit 205 Display control unit 206 Communication Control Unit 210 Storage section 220 Video output section 230 Audio output section 240 External Communications Department 250 Sensor unit 251 Head position measurement section 251a Acceleration sensor 251b Orientation sensor 251c depth sensor 251d Gyro Sensor 251e SLAM 251f GPS module 252 Hand posture measurement section 252a Depth Sensor 252b Infrared camera 253 Voice Acquisition Unit 253a Mike 300 control section 310 Storage section 320 Communications Department B11~B13 Operation buttons C1 Operated part C2 indicator part C3 Function display section

Claims

1. a position processing unit that estimates a head position of a user and estimates a position of a predetermined part of the user's body based on the estimated head position; a correction unit that corrects the position of the predetermined part estimated by the position processing unit based on the head position and an angle of inclination of the user's head; a display control unit that performs processing to superimpose and display an image operated by the user in a space visually recognized by the user based on the position corrected by the correction unit; An information processing device comprising:

2. The correction unit corrects the vertical position of the predetermined portion based on a reference height. The information processing device according to claim 1 .

3. The correction unit corrects the position of the predetermined part in the front-back direction of the user when the tilt angle of the head from the horizontal state is outside a predetermined range. The information processing device according to claim 1 .

4. The correction unit corrects the position of the predetermined part toward the rear of the user when the head is tilted downward from a horizontal position. The information processing device according to claim 3 .

5. The correction unit corrects the position of the predetermined part toward the front of the user when the head is tilted upward from a horizontal position. The information processing device according to claim 3 .

6. a display processing unit that changes a display mode of the image when the image is operated by the user; The information processing device according to claim 1 .

7. The display processing unit changes the display mode of the image according to the elapsed time since the user started an operation on the image. The information processing device according to claim 6 .

8. the position processing unit estimates the position of the user's hand; the display control unit displays an image of the user's hand superimposed on a space visually recognized by the user based on the position estimated by the position processing unit; The display processing unit changes a display mode of the image of the hand in accordance with an overlap with an image operated by the user. The information processing device according to claim 6 .

9. a position processing step of estimating a head position of the user and estimating a position of a predetermined part of the user's body based on the estimated head position; a correcting step of correcting the position of the predetermined part estimated in the position processing step based on the head position and an angle of inclination of the user's head; a display control step of performing a process of superimposing and displaying an image operated by the user in a space visually recognized by the user based on the position corrected in the correction step; An information processing method comprising:

10. On the computer, a position processing step of estimating a head position of the user and estimating a position of a predetermined part of the user's body based on the estimated head position; a correcting step of correcting the position of the predetermined part estimated in the position processing step based on the head position and an angle of inclination of the user's head; a display control step of performing a process of superimposing and displaying an image operated by the user in a space visually recognized by the user based on the position corrected in the correction step; A program that executes the following.

Images