Information processing device, information processing system, support system, and information processing method

Abstract

To facilitate a user's visual recognition of information of a real world and digital information.SOLUTION: The information processing device comprises: a projection information acquisition unit which on the basis of information relating to the position of a user acquired by an information acquisition unit worn on the user, acquires projection information to be projected by a projection unit worn on the user from a storage unit; a projection object information acquisition unit which acquires projection object information relating to a projection object to project the projection information thereon; and a control unit which on the basis of the acquired projection information and projection object information, controls the projection unit to project the projection information on the projection object.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing system, a support system, and an information processing method.

Background Art

[0002] In recent years, various AR (augmented reality) glasses have been developed. AR glasses are an example of wearable devices that users wear and use. A user wearing AR glasses can view information in the real world through a transparent display and also view digital information superimposed on the transparent display.

[0003] For example, Patent Document 1 describes a wearable device in the shape of glasses that acquires information about the position of a user's estimated gaze point and displays the information on a monitor located near the user's eyes.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, with AR glasses, the user may concentrate on the digital information superimposed on the transparent display and miss the information in the real world. It would be convenient to have a technology that allows the user to easily recognize and view both the information in the real world and the digital information while solving such problems.

[0005] The present disclosure aims to make it easy for a user to recognize and view both the information in the real world and the digital information.

Means for Solving the Problems

[0006] One embodiment of the present invention includes: a projection information acquisition unit that acquires projection information projected by a projection unit worn by a user from a storage unit based on information related to the user's position acquired by an information acquisition unit worn by the user; a projection target information acquisition unit that acquires projection target information related to the projection target onto which the projection information is projected; and a control unit that controls the projection unit so that the projection information is projected onto the projection target based on the acquired projection information and projection target information. The projection target information acquisition unit acquires the projection target information determined based on the motion information related to the user's line of sight direction acquired by the motion information acquisition unit worn by the user, and further acquires the projection target information determined based on the surrounding image captured by the imaging unit worn by the user, which includes the area around the user within its imaging range. It is an information processing device. [Effects of the Invention]

[0007] According to one embodiment of the present invention, real-world information and digital information can be made easily recognizable and visually apparent to users. [Brief explanation of the drawing]

[0008] [Figure 1] This is an example diagram illustrating the outline of this embodiment. [Figure 2] This is a diagram illustrating an example of an information processing system according to this embodiment. [Figure 3] This is a hardware configuration diagram of an example of a wearable device according to this embodiment. [Figure 4] This is a hardware configuration diagram of an example of a computer according to this embodiment. [Figure 5] This is a functional configuration diagram of an example of an information processing system according to this embodiment. [Figure 6] This is a functional configuration diagram of an example of an information processing system according to this embodiment. [Figure 7] This is a functional configuration diagram of an example of an information processing system according to this embodiment. [Figure 8] This is a flowchart illustrating an example of the processing of the information processing system according to this embodiment. [Figure 9] This is a flowchart illustrating an example of the processing of the information processing system according to this embodiment. [Figure 10] This is a flowchart illustrating an example of a process for projecting projection information onto a highly visible projection target. [Figure 11] This is an example diagram illustrating the overview of a support system that assists with inspection work. [Figure 12] This is a flowchart illustrating an example of the processing steps of a support system that assists with inspection work. [Figure 13] This is an example diagram illustrating the overview of a support system for assisting construction site work. [Figure 14] This is a flowchart illustrating an example of the processing steps of a support system for assisting construction site work. [Figure 15] This is an example diagram illustrating the overview of a support system for assisting with tourist information. [Figure 16] This is a flowchart illustrating an example of the processing steps in a support system for assisting with tourist information. [Modes for carrying out the invention]

[0009] Hereinafter, each embodiment of the present invention will be described with reference to the attached drawings.

[0010] <Overview> Figure 1 is an example diagram illustrating the outline of this embodiment. User 1 is wearing one or more wearable devices. Figure 1 shows an example embodiment in which User 1 is wearing a head wearable device 2 and a torso wearable device 3. Although Figure 1 shows an example embodiment in which User 1 is wearing two wearable devices, the embodiment may also show the user wearing one or three or more wearable devices. For example, the head wearable device 2 and the torso wearable device 3 may be integrated.

[0011] The head-mounted wearable device 2 acquires information related to the user's position 1. The head-mounted wearable device 2 may be in the form of a device intended to be worn on the ear, for example. The head-mounted wearable device 2 is equipped with functions such as capturing images of the user's surroundings, measuring the user's posture and changes in posture, measuring the direction of the user's gaze with an electrooculography sensor, and GPS functionality.

[0012] The wearable device 3 for the body acquires digital information to be visually recognized by the user 1 from the storage unit based on information related to the position of the user 1. The wearable device 3 for the body may have a device form assumed to be worn on the chest, for example.

[0013] The digital information to be visually recognized by the user 1 is included in, for example, digitized real-world information called a digital twin. The digital information to be visually recognized by the user 1 is information associated with information related to the position of the user 1 in the digital twin, for example.

[0014] In addition, the wearable device 3 for the body acquires projection target information related to a projection target 4 on which the acquired digital information is projected. For example, in the digital twin, information 4' related to the projection target 4 in the real world is stored in association with information related to the position of the user 1. The wearable device 3 for the body has a projection function such as a projector, and performs projection so that projection information 5 imaged on the projection target 4 in the real world is projected.

[0015] The projection information 5 projected onto the projection target 4 in the real world by the projector included in the wearable device 3 for the body is AR information such as a mark indicating an object or a target location in the real world that the user 1 wants to visually recognize, as shown in FIG. 1 for example. Also, the projection information 5 projected onto the projection target 4 in the real world by the projector included in the wearable device 3 for the body may be AR information such as information that the user 1 wants to visually recognize.

[0016] As shown in FIG. 1, in this embodiment, by projecting the projection information 5 onto the projection target 4 in the real world by the projector included in the wearable device worn by the user 1, the display of AR information in the real world can be realized.

[0017] For example, in the display of AR information using AR glasses, unnaturalness may occur in order to superimpose the AR information on the transmissive display for visually recognizing the real-world information. Also, in the display of AR information using AR glasses, there were cases where the user 1's five senses were inhibited.

[0018] In this embodiment, compared to displaying AR information using AR glasses, the display is natural and easy for user 1 to recognize and view the AR information. Furthermore, in this embodiment, by projecting the projection information 5 onto the projection object 4 in the real world, it is possible to display AR information in the real world that can be viewed by people other than user 1 wearing the wearable device.

[0019] <System Configuration> Figure 2 is a configuration diagram of an example of an information processing system according to this embodiment. Figure 2(A) shows a configuration in which a head wearable device 2, a torso wearable device 3, and a storage unit 6 are connected in a way that enables data communication. Figure 2(B) shows a configuration in which a head wearable device 2, a torso wearable device 3, a storage unit 6, and an information processing terminal 7 are connected in a way that enables data communication.

[0020] The head-mounted wearable device 2 and the torso-mounted wearable device 3 are the same as those in Figure 1. The memory unit 6 stores information necessary for displaying AR information, such as digitized real-world information called a digital twin. The memory unit 6 may be configured to be included in either the head-mounted wearable device 2 or the torso-mounted wearable device 3. Furthermore, the memory unit 6 may be implemented on the cloud or on a PC (Personal Computer).

[0021] The information processing terminal 7 is a PC, mobile phone, smartphone, tablet, game console, or PDA (Personal Digital Assistant) owned by user 1. The memory unit 6 may be configured to be part of the information processing terminal 7. The information processing terminal 7 may also implement at least some of the functions of the head wearable device 2 and the torso wearable device 3. Furthermore, some of the functions of the head wearable device 2 and the torso wearable device 3 may be implemented on the cloud.

[0022] <Hardware Configuration> Figure 3 is a hardware configuration diagram of an example of a wearable device according to this embodiment. Figure 3 illustrates an example configuration in which the wearable device worn by user 1 is divided into a head wearable device 2 and a torso wearable device 3, but it may also be a configuration divided into three or more parts or a configuration that combines them into one.

[0023] The head-mounted wearable device 2 includes multiple peripheral cameras 400, a microphone 402, a speaker 404, an IMU (Inertial Measurement Unit) 406, and a SoC (System-on-a-Chip) 408. The torso-mounted wearable device 3 includes a Lidar 410, a wearer camera 412, a projector 414, and a wireless communication unit 416.

[0024] The peripheral camera 400 is an example of an imaging unit that captures images of the user's surroundings, and includes, for example, a lens and an image sensor. The peripheral camera 400 may be, for example, a digital camera that obtains a 360° panoramic image in a single shot. The peripheral camera 400 is used to acquire information for environmental recognition and position estimation. VSLAM (Visual Simultaneous Localization and Mapping) can be used for position estimation. VSLAM is a technology that performs self-position estimation and environmental mapping from surrounding images.

[0025] Microphone 402 picks up ambient sounds and the voice of user 1. Speaker 404 outputs voice guidance to user 1 or provides warning sounds. Microphone 402 and speaker 404 are used according to the usage scenario, for example, for voice information exchange between user 1 and a caregiver, or for notifying warning sounds.

[0026] The IMU406 is an inertial measurement device that measures the posture and changes in posture of user 1 wearing the head-mounted wearable device 2. The IMU406 also measures changes in the orientation of user 1's face. For example, the change in user 1's face orientation can be used to estimate the amount of movement by calculating the number of steps from the vertical movement of user 1's face orientation. Furthermore, the change in user 1's face orientation can be used to estimate user 1's level of concentration from the vertical and horizontal changes in user 1's face orientation.

[0027] The Lidar 410 comprises a light-emitting unit and a light-receiving unit, and measures the shape of an object or the distance to an object using remote sensing with light. The Lidar 410 measures distance information and surface information in real time in response to changes in the user's posture or changes in the positional relationship with the projection target 4. For example, changes in the user's posture or changes in the positional relationship with the projection target 4 can be used for accurate dynamic mapping of projection information 5 onto the projection target 4. For dynamic mapping of projection information 5 onto the projection target 4, real-time correction may be performed using images of the user's surroundings captured by the surrounding camera 400.

[0028] The wearer camera 412 is an example of an imaging unit that images the user 1, and includes, for example, a lens and an image sensor. For example, the image of the user 1's face captured by the wearer camera 412 can be used for estimating a state of distraction, estimating a state of concentration, and estimating gaze. Furthermore, the accuracy of gaze estimation can be improved by combining it with information measured by the IMU 406.

[0029] The projector 414 is an example of a projection unit that projects projection information 5 onto a projection target 4, and includes, for example, a lens and a light-emitting unit. The projector 414 performs projection using a laser and a projection direction shift mechanism (MEMS). Based on the surrounding image and information measured by the Lidar 410, the projector 414 can correct the positional shift of the projection position using the mirror angle of the MEMS. The wireless communication unit 416 performs data communication with the storage unit 6 and the information processing terminal 7, etc. The wireless communication unit 416 may be provided in the head-mounted wearable device 2.

[0030] The SoC408 includes a CPU, memory, peripheral IF circuitry, and an AI accelerator. The SoC408 is connected to multiple peripheral cameras 400, a microphone 402, a speaker 404, an IMU 406, a Lidar 410, a wearer camera 412, a projector 414, and a wireless communication unit 416 for data communication, and performs various processes described later to project projection information 5 onto the projection target 4.

[0031] The memory unit 6 may be implemented in a wearable device with the hardware configuration shown in Figure 3. Alternatively, the memory unit 6 may be implemented in a computer 500 with the hardware configuration shown in Figure 4, or in a cloud-based computer 500 with the hardware configuration shown in Figure 4. Figure 4 is a hardware configuration diagram of an example of a computer according to this embodiment. Note that the hardware configuration will also be as shown in Figure 4 if the information processing terminal 7 is a PC.

[0032] Computer 500 is equipped with a CPU (Central Processing Unit) 501, ROM (Read Only Memory) 502, RAM (Random Access Memory) 503, HD 504, HDD (Hard Disk Drive) controller 505, display 506, external device connection I / F (Interface) 508, network I / F 509, data bus 510, keyboard 511, pointing device 512, DVD-RW (Digital Versatile Disk Rewritable) drive 514, and media I / F 516.

[0033] The CPU 501 controls the operation of the entire computer 500 according to the program. The ROM 502 stores programs used to drive the CPU 501, such as the IPL. The RAM 503 is used as the work area for the CPU 501. The HD 504 stores various data, such as programs. The HDD controller 505 controls the reading or writing of various data to the HD 504 according to the control of the CPU 501.

[0034] The display 506 displays various information such as cursors, menus, windows, characters, or images. The external device connection interface 508 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memory. The network interface 509 is an interface for data communication using a network. The data bus 510 is an address bus and data bus for electrically connecting various components such as the CPU 501.

[0035] The keyboard 511 is a type of input means equipped with multiple keys for inputting characters, numbers, and various instructions. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting processing targets, and moving the cursor. The DVD-RW drive 514 controls the reading or writing of various data to the DVD-RW 513, which is an example of a removable recording medium. Note that it is not limited to DVD-RW, but may also be DVD-R, etc. The media interface 516 controls the reading or writing (storage) of data to the recording medium 515, such as flash memory.

[0036] Note that the hardware configuration shown in Figure 4 is just one example, and it is not necessary to include all the components shown in Figure 4, or to include components other than those shown in Figure 4. Hardware configuration diagrams for cases where the information processing terminal 7 in Figure 2 is not a PC vary widely, so they are omitted from the illustration.

[0037] <Functional Configuration> Figures 5 to 7 are functional configuration diagrams of an example of an information processing system according to this embodiment. The information processing system according to this embodiment comprises an information acquisition unit 20, an information processing unit 30, a projection unit 40, and a storage unit 50. The information acquisition unit 20, information processing unit 30, projection unit 40, and storage unit 50 in Figures 5 to 7 are realized through the cooperation of the head wearable device 2, torso wearable device 3, and storage unit 6 shown in Figure 2, and various programs. Alternatively, the information acquisition unit 20, information processing unit 30, projection unit 40, and storage unit 50 in Figures 5 to 7 may be realized through the cooperation of the head wearable device 2, torso wearable device 3, storage unit 6, and information processing terminal 7 shown in Figure 2, and various programs.

[0038] The functional configurations shown in Figures 5 to 7 will be explained in order below. The information acquisition unit 20 in Figure 5 comprises a communication unit 21 and a location information acquisition unit 22. The location information acquisition unit 22 acquires information related to the location of user 1. The location information acquisition unit 22 can acquire information related to the location of user 1 by performing environmental recognition and location estimation from information obtained from surrounding cameras 400, IMU 406, Lidar 410, wearer camera 412, GPS (Global Positioning System), etc. The communication unit 21 performs data communication with the information processing unit 30, projection unit 40, and storage unit 50.

[0039] The information processing unit 30 comprises a communication unit 31, a projection information acquisition unit 32, a projection target information acquisition unit 33, and a control unit 34. The projection information acquisition unit 32 acquires projection information corresponding to the user 1's position as projection content to be projected onto the projection unit 40, based on the information related to the user 1's position received from the information acquisition unit 20 and the information from the storage unit 50.

[0040] The projection target information acquisition unit 33 acquires projection target information related to the projection target 4 (projection position) onto which the projection information is projected, based on the information related to the user 1's position received from the information acquisition unit 20 and the information from the storage unit 50. The projection target information includes, for example, information such as the position of the projection target and the shape, material, and color of the projection surface of the projection target. The projection target information may also be stored in the storage unit 50 in association with the projection information. In that case, the projection target information may be associated with multiple projection targets for each piece of projection information. By associating a projection target with each piece of projection information, the projection target information acquisition unit 33 can acquire information on a projection target 4 that is suitable for the projection information and the user 1's position. Furthermore, if the projection target information corresponding to the projection information is not stored in the storage unit 50, the projection target information acquisition unit 33 may determine the projection target 4 from the digital twin information acquired from the storage unit 50 or from information obtained from devices worn by the user 1, such as the peripheral camera 400 or Lidar 410, and use the obtained information on the position and projection surface of the projection target 4 as projection target information. For example, it is conceivable that the projection target information stored in the memory unit 50 does not include any projection target 4 within the range of distance that the projection unit 40 can project from the user 1's position. In such cases, 3D map data stored in the memory unit 50 or data obtained from devices such as the surrounding camera 400 or Lidar 410 may be used as projection target information.

[0041] The control unit 34 controls the projection unit 40 so that the projection information 5 is projected onto the projection target 4, based on the projection information acquired by the projection information acquisition unit 32 and the projection target information acquired by the projection target information acquisition unit 33. The communication unit 31 performs data communication with the information acquisition unit 20, the projection unit 40, and the storage unit 50.

[0042] The projection unit 40 comprises a communication unit 41, a projection image creation unit 42, and a projection processing unit 43. The projection image creation unit 42 converts the projection information received from the information processing unit 30 into an image. The projection image creation unit 42 may also perform adjustments such as deformation of the image of the projection information received from the information processing unit 30 for projection onto the projection target 4. Adjustments such as deformation of the image of the projection information include, for example, trapezoidal correction.

[0043] Furthermore, the projection image creation unit 42 may adjust the image of the projection information to match the shape of the projection target 4 (shape of the projection surface) by using existing projection mapping technology. Projection mapping (PJM) is a technology that deforms the projected image according to the projection surface of the projection object, so that the projected image appears as if it is attached to the projection object. In addition, with projection mapping, a more natural projection image can be projected onto the projection object by adjusting the color and brightness of the projected image according to the color and brightness of the projection surface. Information such as the shape, color, and brightness of the projection surface of the projection target 4 may be acquired by including the information stored in the memory unit 50 into the projection target information, or it may be acquired by including it into the projection target information using the peripheral camera 400 or Lidar 410 worn by the user 1.

[0044] In this way, the imaged projection information 5, which has been adjusted using projection mapping technology, is projected by the projection processing unit 43 so as to appear as if it is attached to the projection target 4.

[0045] The projection processing unit 43 projects the projection information 5, which has been imaged by the projection image creation unit 42, onto the projection target 4 received from the information processing unit 30. The communication unit 41 performs data communication with the information acquisition unit 20, the information processing unit 30, and the storage unit 50.

[0046] The memory unit 50 stores information necessary for the information processing system according to this embodiment to realize the display of AR information in the real world. The information necessary to realize the display of AR information in the real world includes AR information that assists the actions of user 1, information about the projection target 4 onto which the AR information is projected, a 3D map of the environment surrounding user 1, and information such as the shape and material of the projection target 4.

[0047] AR information may be stored in the storage unit 50 in association with location information on a map of the surrounding environment, thereby providing user 1 with appropriate AR information according to user 1's location. Information on the projection target 4 may also be stored in the storage unit 50, including location information on a map of the surrounding environment. In addition, information on the projection target 4 may be stored in the storage unit 50 in association with AR information.

[0048] The 3D map of User 1's surrounding environment may be 3D point cloud data obtained by 3D measurement of the surrounding environment, or 3D data composed of a 3D model created with 3D CAD, etc. If the 3D map is 3D point cloud data, information indicating that it is a projection target (category or tag) may be added to the points corresponding to projection target 4 of the 3D point cloud data. If the 3D map is 3D data, attribute information indicating that it is a projection target may be added to objects (surfaces or solids) within the 3D data. By using 3D point cloud data for the 3D map, it is possible to use actual data obtained by 3D measurement of the surrounding environment, so the information processing system according to this embodiment can perform self-position estimation using VSLAM as described above using a 3D map that appropriately reflects the current state of the surrounding environment. If the 3D map is 3D data created with 3D CAD, etc., it is less susceptible to the influence of noise from 3D measurement in determining the projection target.

[0049] Note that the functional configuration diagram in Figure 5 is just an example, and each of the functional units shown in the information acquisition unit 20, information processing unit 30, and projection unit 40 may be implemented in any of the three units. Also, the storage unit 50 may be included in any of the information acquisition unit 20, information processing unit 30, and projection unit 40.

[0050] The information acquisition unit 20 in Figure 6 comprises a communication unit 21, a location information acquisition unit 22, and an action information acquisition unit 23. The communication unit 21 and the location information acquisition unit 22 are the same as those in Figure 5. The action information acquisition unit 23 acquires action information related to at least one of the user 1's gaze direction and actions, and recognizes the user 1's intentions based on the action information.

[0051] For example, the motion information acquisition unit 23 can recognize the user's intention, such as an instruction to start projecting AR information or an instruction to project AR information onto target 4, from motion information that recognizes the user's gestures obtained from the surrounding camera 400, IMU 406, electrooculography sensor, etc.

[0052] The functional configuration of the information processing unit 30 is the same as in Figure 5. The projection information acquisition unit 32 acquires projection information to be projected onto the projection unit 40 based on the information of the user 1's intentions received from the information acquisition unit 20, information related to the user 1's location, and information from the storage unit 50.

[0053] The projection target information acquisition unit 33 acquires projection target information related to the projection target 4 onto which the projection information will be projected, based on the information about the user 1's intention and the information about the user 1's location received from the information acquisition unit 20, and the information from the storage unit 50. The control unit 34 controls the projection unit 40 so that the projection information 5 is projected onto the projection target 4, based on the projection information acquired by the projection information acquisition unit 32 and the projection target information acquired by the projection target information acquisition unit 33. The functional configuration of the projection unit 40 is the same as in Figure 5.

[0054] Note that the functional configuration diagram in Figure 6 is just an example, and each of the functional units shown in the information acquisition unit 20, information processing unit 30, and projection unit 40 may be implemented in any of the three units. Also, the storage unit 50 may be included in any of the three units. The information processing system with the functional configuration shown in Figure 6 can recognize the intent of user 1 and project AR information onto the projection target 4 specified by user 1.

[0055] The information acquisition unit 20 in Figure 7 comprises a communication unit 21 and an imaging unit 24. The imaging unit 24 acquires an image of the user's surroundings. The imaging unit 24 can acquire an image of the user's surroundings, which is an example of information related to the user's position, from a surrounding camera 400 or the like. The communication unit 21 performs data communication with the information processing unit 30, the projection unit 40, and the storage unit 50.

[0056] The functional configuration of the information processing unit 30 is the same as in Figure 5. The projection information acquisition unit 32 estimates the position of user 1 based on the surrounding image of user 1 received from the information acquisition unit 20 and the information from the storage unit 50. The projection information acquisition unit 32 acquires projection information corresponding to the position of user 1 as the projection content to be projected onto the projection unit 40. The projection target information acquisition unit 33 acquires projection target information related to the projection target 4 onto which the projection information will be projected, based on the surrounding image of user 1 received from the information acquisition unit 20, as described below.

[0057] The control unit 34 controls the projection unit 40 so that the projection information 5 is projected onto the projection target 4, based on the projection information acquired by the projection information acquisition unit 32 and the projection target information acquired by the projection target information acquisition unit 33. The communication unit 31 performs data communication with the information acquisition unit 20, the projection unit 40, and the storage unit 50.

[0058] The functional configuration of the projection unit 40 is the same as in Figure 5. The projection image creation unit 42 converts the projection information received from the information processing unit 30 into an image. The projection processing unit 43 projects the projection information 5, which has been imaged by the projection image creation unit 42, onto the projection target 4 received from the information processing unit 30. The communication unit 41 performs data communication with the information acquisition unit 20, the information processing unit 30, and the storage unit 50.

[0059] Note that the functional configuration diagram in Figure 7 is just an example, and each of the functional units shown in the information acquisition unit 20, information processing unit 30, and projection unit 40 may be implemented in any of the three units. Also, the storage unit 50 may be included in any of the information acquisition unit 20, information processing unit 30, and projection unit 40.

[0060] The information processing system with the functional configuration shown in Figure 7 acquires an ambient image containing AR information projected onto the projection target 4 by the projection unit 40. Therefore, the projection target information acquisition unit 33 of the information processing unit 30 evaluates the image of the AR information contained in the ambient image and can project the AR information to a location that is easily visible to the user 1.

[0061] The functional configurations shown in Figures 5 to 7 are just examples; combinations of the functional configurations in Figures 5 and 7, or combinations of the functional configurations in Figures 6 and 7, are also acceptable.

[0062] <Processing> Figure 8 is a flowchart of an example of the processing of the information processing system according to this embodiment. Here, the functional configuration of Figure 5 will be explained as an example. The flowchart in Figure 8 is an example in which the projection target 4 of the AR information is specified in advance. For example, the flowchart in Figure 8 corresponds to a case where the projection target 4 is to be an area on an object such as equipment or furnishings around the user 1, or an area at the user's feet that displays the direction the user 1 is moving, in which case the area to be projected (AR information) is predetermined.

[0063] In step S10, the information acquisition unit 20 acquires information related to the user 1's location and transmits it to the information processing unit 30. The information related to the user 1's location includes information obtained from the surrounding camera 400, IMU 406, Lidar 410, wearer camera 412, or GPS, and information obtained by performing environmental recognition and position estimation from the acquired information. The information acquisition unit 20 may store the information necessary for environmental recognition and position estimation, or it may acquire it from, for example, the storage unit 50. For example, the information acquisition unit 20 may acquire point cloud data that forms a 3D map of the surrounding environment from the storage unit 50, and perform environmental recognition and position estimation by matching the point cloud data with the surrounding image of the user 1.

[0064] In step S12, the information processing unit 30 acquires AR information associated with the location of user 1 based on the received information relating to the location of user 1 and the AR information associated with the location of the 3D map of the surrounding environment recorded in the storage unit 50.

[0065] In step S14, the information processing unit 30 detects the area to be projected with AR information by performing detection processing based on the surrounding image from the surrounding camera 400 and the information from the digital twin, and determines the projection target 4.

[0066] In step S16, the information processing unit 30 controls the projection unit 40 so that the projection information 5 is projected onto the projection target 4, based on the projection information acquired in step S12 and the projection target 4 determined in step S14. The processing in steps S10 to S16 is repeated until user 1 performs the operation to end the projection.

[0067] Figure 9 is a flowchart of an example of the processing of the information processing system according to this embodiment. Here, a functional configuration combining Figures 6 and 7 will be explained as an example. The flowchart in Figure 9 is an example where the projection target 4 of the AR information is not specified in advance. For example, the flowchart in Figure 9 corresponds to cases where the area on which the projection information 5 (AR information) is projected is not determined in advance, such as when the user 1 is asked to specify the projection target 4, or when the user 1 wants to choose a location that is easily visible to them as the projection target 4. Alternatively, the flowchart in Figure 9 corresponds to cases where information related to the projection target associated with the projection information (projection target information) is not stored in the storage unit 50, or when the projection targets within the range that the projection unit 40 can project are not stored in the storage unit 50.

[0068] The processing in steps S20 to S22 is the same as the processing in steps S10 to S12 in Figure 8, so the explanation is omitted. In step S24, the information acquisition unit 20 acquires motion information relating to at least one of the user 1's gaze direction and movement. In step S26, the information acquisition unit 20 performs gesture recognition based on the acquired motion information and determines whether or not user 1 has made a gesture to project projection information 5. If user 1 has not made a gesture to project projection information 5, the information acquisition unit 20 returns to the processing in step S20.

[0069] The gesture used to project the projection information 5 is merely one example of how user 1 can specify the projection of the projection information 5, and may also be specified by user 1's line of sight, by a physical button, by voice, etc.

[0070] If user 1 performs a gesture to project projection information 5, the information acquisition unit 20 determines whether the gesture performed by user 1 is a gesture to specify projection target 4. If the gesture performed by user 1 is a gesture to specify projection target 4, the information acquisition unit 20 transmits the information of projection target 4 specified by user 1 to the information processing unit 30.

[0071] The gesture used to specify projection target 4 is merely one example of how user 1 can specify projection target 4; it may also be specified by the direction of user 1's gaze, by a physical button, or by voice. For example, a gesture to specify projection target 4 may include holding up the palm of the hand to specify the palm as projection target 4, or pointing with a finger to specify the area at the end of the finger as projection target 4. Furthermore, a method for specifying projection target 4 may include specifying the area that user 1 is fixating on as projection target 4.

[0072] Upon receiving information about the projection target 4 specified by user 1, the information processing unit 30 proceeds to step S32. The information processing unit 30 controls the projection unit 40 so that the projection information 5 is projected onto the specified projection target 4. The information acquisition unit 20 also proceeds to step S30 if the gesture made by user 1 is not a gesture to specify the projection target 4. The information acquisition unit 20 transmits information to the information processing unit 30 indicating that user 1 has not specified the projection target 4. Upon receiving the information indicating that user 1 has not specified the projection target 4, the information processing unit 30 controls the projection unit 40 so that the projection information 5 is projected onto the highly visible projection target 4. The processes in steps S20 to S34 are repeated until user 1 performs the operation to end the projection.

[0073] The highly visible projection target 4 in step S30 may be estimated, for example, from the surrounding image of user 1 using AI or the like. Alternatively, the highly visible projection target 4 in step S30 may be estimated, for example, as shown in Figure 10.

[0074] Figure 10 is a flowchart of an example of a process for projecting projection information onto a highly visible projection target. In step S40, the information acquisition unit 20 determines a candidate projection target onto which to project the sample image by detection processing based on the surrounding image and digital twin information. If no suitable projection target candidate is stored in the digital twin information, the information acquisition unit 20 may detect a projection target candidate based on the surrounding image and 3D information detected by the Lidar 410. The information acquisition unit 20 transmits the information of the determined projection target candidate to the information processing unit 30. Upon receiving the information of the projection target candidate, the information processing unit 30 proceeds to the process in step S42. The information processing unit 30 controls the projection unit 40 so that the sample image is projected onto the projection target candidate.

[0075] In step S44, the projection unit 40 projects a sample image onto a candidate projection target. The information acquisition unit 20 captures the sample image projected onto the candidate projection target. In step S46, the information acquisition unit 20 determines the candidate projection target with high visibility of the projected sample image as the projection target 4 based on the captured image.

[0076] In step S48, the information processing unit 30 controls the projection unit 40 so that the projection information 5 is projected onto the projection target 4 with high visibility, which was determined in step S46. Therefore, the projection unit 40 can project the projection information 5 onto the projection target 4 with high visibility. According to the flowchart in Figure 10, by feeding back the state when a sample image is actually projected onto the projection target 4, the most suitable projection target 4 with high visibility can be selected from among multiple projection targets 4.

[0077] For example, in this embodiment, a highly visible projection surface is not always directly in front of user 1. Depending on user 1's state and surrounding conditions, it is necessary to project AR information onto a projection target 4 desired by user 1 or a highly visible projection target 4. In construction sites or urban areas with many objects and complex shapes, if the projection target 4 in front of user 1 has properties unsuitable for projection, user 1 may not be able to see the projected information 5. For example, bright surfaces, patterned surfaces, uneven surfaces, specularly reflective surfaces, surfaces that do not diffusely reflect light well, surfaces not directly facing user 1, and surfaces that are not white or black are examples of projection surfaces with properties unsuitable for projection.

[0078] Furthermore, dark surfaces, uniform surfaces without patterns, flat surfaces, surfaces that do not reflect specularly, surfaces that easily reflect diffusely, surfaces directly facing user 1, and surfaces with background colors such as white or black are examples of projection surfaces that have properties suitable for projection and are easy to see. The properties of the projection surface may be determined by referring to information registered in the digital twin, for example, or by estimating them from the surrounding images captured by the surrounding camera 400 using image processing or AI.

[0079] Furthermore, the information processing system according to this embodiment may, for example, recognize the user's intention from action information relating to at least one of the user's gaze direction and actions, and project a telephone function UI onto the palm of the hand, or project information about a ticket held in the hand. In this case, the user's hand or a ticket held in the user's hand are examples of projection targets 4. When projection targets 4 such as the user's hand or a ticket held in the user's hand are to be used as projection targets, the information processing unit 30 may determine them as projection targets based on data acquired by the surrounding camera 400 or Lidar 410. In this case, the information related to the projection targets is information based on detection data from the surrounding camera 400 or Lidar 410.

[0080] Thus, the information processing system according to this embodiment can project projection information onto the user 1's hand or belongings without using projection target information contained in the digital twin information stored in the memory unit 50. As a result, the information processing system according to this embodiment can provide the user 1 with easily visible and appropriate information even when it is not possible to utilize the digital twin information (which does not include information on moving objects such as people or vehicles) when there are many moving obstacles such as people or vehicles around the user 1.

[0081] Note that the flowcharts in Figures 8 and 9 are just examples. For example, it may be possible to determine whether or not the projection target 4 onto which the AR information will be projected has been specified in advance. If the projection target 4 has been specified in advance, the process shown in the flowchart in Figure 8 will be performed, and if the projection target 4 has not been specified in advance, the process shown in the flowchart in Figure 9 will be performed.

[0082] <Example 1> The information processing system of this embodiment can be used, for example, as a support system to assist inspection work. Figure 11 is a diagram illustrating an example of a support system for assisting inspection work. User 1, who performs inspection work, wears a head-mounted wearable device 2 and a torso-mounted wearable device 3 and tours the inspection area.

[0083] The head-mounted wearable device 2 acquires information related to the user 1's position. The torso-mounted wearable device 3 projects a patrol route to the inspection location as projected information 5b onto the user 1's feet or other location, based on the information related to the user 1's position. In this way, the support system of this embodiment allows the user 1 to be navigated using projected information 5b such as arrows along the patrol route to the inspection location, enabling efficient inspection.

[0084] Furthermore, the torso wearable device 3 projects projection information 5a indicating inspection points onto the projection target 4 based on information related to the user 1's position. The torso wearable device 3 can also indicate the procedure of the inspection work by showing the order in which the projection information 5a indicating the inspection points is projected onto the projection target 4. In this way, the support system of this embodiment allows the user 1 to perform the inspection work efficiently by indicating the inspection points with projection information 5a such as arrows and highlights.

[0085] For example, when displaying AR information using AR glasses, AR information is superimposed on a transparent display that allows the user to view information from the real world. As a result, when displaying AR information using AR glasses, user 1 may concentrate excessively on the AR display in front of them, and may not notice abnormalities (such as strange noises or smells) that they would normally notice. In the support system of this embodiment, projection information 5 is projected onto a projection object 4 in the real world, thereby realizing the display of AR information in the real world. This allows AR information to be displayed without unnecessarily taking away user 1's attention and gaze.

[0086] Figure 12 is a flowchart of an example of the processing of a support system that assists with inspection work. The flowchart in Figure 12 is an example where the projection target 4 of the AR information is specified in advance.

[0087] In step S60, the information acquisition unit 20 acquires information related to the location of user 1 and transmits it to the information processing unit 30. In step S62, the information processing unit 30 acquires inspection information associated with the location of user 1 based on the received information related to the location of user 1 and the inspection information associated with the location of the 3D map of the surrounding environment recorded in the storage unit 50. The inspection information acquired here includes point cloud data that forms a 3D map, location information of the inspection site, location information of the inspection points, a checklist for the inspection work, and cautionary information for the inspection work.

[0088] In step S64, the information processing unit 30 determines, based on the received information relating to the location of user 1 and the inspection information, whether the distance to the inspection target, such as equipment or fixtures, is short and whether projection onto the inspection target is possible.

[0089] If projection onto the target object is possible, the process proceeds to step S66, where the information processing unit 30 detects the inspection target and inspection location to which projection information 5a will be projected by detection processing based on the surrounding image from the surrounding camera 400 and inspection information. The information processing unit 30 controls the projection unit 40 so that projection information 5a and other information are projected onto the detected inspection target.

[0090] On the other hand, if the distance to the inspection target, such as equipment or fixtures, is too far and projection onto the target is not possible, the process proceeds to step S68. The information processing unit 30 then performs detection processing based on the surrounding image from the surrounding camera 400 and the inspection information to select, for example, projection information 5b to be projected onto the projection unit 40. The information processing unit 30 then controls the projection unit 40 so that navigation information such as the selected projection information 5b is projected. The processes in steps S60 to S70 are repeated until the user 1 performs the operation to end the projection.

[0091] Although this embodiment has been described as a system to support inspection work, it can also be applied to support systems that assist with predetermined tasks. For example, this embodiment can be applied to support cargo sorting work in a logistics warehouse or to support product assembly work in a manufacturing plant.

[0092] <Example 2> The information processing system of this embodiment can be used, for example, as a support system to assist with construction site work. Figure 13 is a diagram illustrating an example of a support system for assisting with construction site work. User 1, who performs work at the construction site, wears a head-mounted wearable device 2 and a torso-mounted wearable device 3 while working at the construction site.

[0093] The head-mounted wearable device 2 acquires information related to the user 1's position. Based on the information related to the user 1's position, the torso-mounted wearable device 3 projects dangerous areas, restricted areas, secured areas, and work targets at the construction site as projection information 5 so that the user 1 can easily notice them. In this way, the support system of this embodiment can support work at a construction site.

[0094] For example, when displaying AR information using AR glasses, AR information is superimposed on a transparent display that allows the user to view information from the real world. As a result, when displaying AR information using AR glasses, user 1 may concentrate excessively on the AR display in front of them, and may not notice dangers that they would normally notice (such as steps at their feet or obstacles overhead). In this embodiment, the support system displays AR information in the real world by projecting projection information 5 onto a projection object 4 in the real world, so that AR information can be displayed without unnecessarily taking away user 1's attention and gaze.

[0095] Figure 14 is a flowchart of an example of the processing of a support system that assists with construction site work. In step S80, the information acquisition unit 20 acquires information related to the location of user 1 and transmits it to the information processing unit 30.

[0096] In step S82, the information processing unit 30 acquires construction site information associated with the location of user 1 based on the received information relating to the location of user 1 and the construction site information associated with the location of the 3D map of the surrounding environment recorded in the storage unit 50. The construction site information acquired here includes point cloud data that forms a 3D map, location information of places that are dangerous to touch, location information of places with steps, location information of places with obstacles overhead, location information of places where entry is prohibited, location information of places that have been reserved as space, location information of the work object, information on the work content, and information on precautions to take during work.

[0097] In step S84, the information processing unit 30 determines whether there is any construction site information within the projection range based on the received information relating to the location of user 1 and the construction site information. If there is any construction site information within the projection range, the process proceeds to step S86, where the information processing unit 30 detects a projection target 4 onto which to project the projection information 5 of the construction site information, based on the surrounding image from the surrounding camera 400 and the construction site information. For example, in the example in Figure 13, there is a hazardous object near user 1, and that hazardous object is detected as the projection target 4. Alternatively, the detected projection target 4 may be the location of the hazardous object. The information processing unit 30 controls the projection unit 40 so that projection information 5, such as a color cone (registered trademark) or a highlight display, is projected onto the detected projection target 4 as construction site information.

[0098] On the other hand, if there is no construction site information within the projectable range, the information processing unit 30 skips the process in step S86. The processes in steps S80 to S88 are repeated until user 1 performs the operation to end the projection.

[0099] <Example 3> The information processing system of this embodiment can be used, for example, as a support system for assisting with tourist information. Figure 15 is a diagram illustrating an example of a support system for assisting with tourist information. User 1, who uses the tourist information system, wears a head-mounted wearable device 2 and a torso-mounted wearable device 3 while sightseeing. The places where sightseeing takes place include tourist spots, art museums, or museums.

[0100] The head-mounted wearable device 2 acquires information related to the user 1's location. Based on the information related to the user 1's location, the torso-mounted wearable device 3 projects information such as projected information 5a indicating store information, projected information 5b indicating the route to the destination, and projected information 5c indicating information to notify the user 1, in a way that is easily noticeable to the user 1. Projected information 5a is information about tourist attractions (buildings, local products, etc.) projected onto the object itself or a projection target in the vicinity of the object. In this way, the support system of this embodiment can assist in tourist guidance.

[0101] For example, when displaying AR information using AR glasses, only users wearing the AR glasses can see the projected information 5a-5c. Therefore, when displaying AR information using AR glasses, users not wearing the AR glasses cannot see the projected information 5a-5c.

[0102] In this embodiment, the support system projects projection information 5 onto a projection target 4 in the real world, thereby enabling the display of AR information in the real world. Therefore, if one person in the group wears the support system of this embodiment, everyone in the group can see the displayed AR information.

[0103] Figure 16 is a flowchart illustrating an example of the processing steps of a support system for assisting with tourist information.

[0104] In step S100, the information acquisition unit 20 acquires information related to the location of user 1 and transmits it to the information processing unit 30. In step S102, the information processing unit 30 acquires tourist information associated with the location of user 1 based on the received information related to the location of user 1 and tourist information associated with the location of the 3D map of the surrounding environment recorded in the storage unit 50. The tourist information acquired here includes point cloud data that forms the 3D map, information on the places where sightseeing takes place, etc.

[0105] In step S104, the information processing unit 30 determines whether there is any tourist information within the projection range based on the received information relating to the location of user 1 and the tourist information. If there is tourist information within the projection range, the process proceeds to step S106, where the information processing unit 30 detects the projection target 4 onto which the projection information 5a to 5c of the sightseeing location will be projected, based on the surrounding image from the surrounding camera 400 and the tourist information.

[0106] On the other hand, if there is no tourist information within the projectable range, the information processing unit 30 skips the process in step S106. The processes in steps S100 to S108 are repeated until user 1 performs the operation to end the projection.

[0107] The support system of this embodiment can be used in educational settings, for example, where multiple people want to share AR information. For example, when conducting a lesson in an educational setting where AR information is shared using AR glasses, one pair of AR glasses is required for each participant. With the support system of this embodiment, for example, one teacher can wear a wearable device, allowing everyone participating in the lesson to view the AR information.

[0108] Each of the functions of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each of the functions described above.

[0109] The apparatus described in the examples represents only one of several computing environments for carrying out the embodiments disclosed herein. The present invention is not limited by these embodiments, and the components in these embodiments include those readily conceivable to those skilled in the art, those substantially identical, and those within the scope of so-called equivalents. Furthermore, various omissions, substitutions, modifications, and combinations of components can be made without departing from the spirit of these embodiments. [Explanation of symbols]

[0110] 2. Wearable devices for the head 3. Wearable devices for the torso 4. Projection target 5 Projection information 6 Memory section 7. Information Processing Terminal 20 Information acquisition department 22 Location information acquisition unit 23 Operation information acquisition section 30 Information Processing Department 32 Projection information acquisition section 33 Projection Target Information Acquisition Unit 34 Control Unit 40 Projection section 50 Storage section [Prior art documents] [Patent Documents]

[0111] [Patent Document 1] Patent No. 6306985

Claims

1. A projection information acquisition unit acquires projection information projected by a projection unit worn by the user from a storage unit, based on information related to the user's position acquired by an information acquisition unit worn by the user. A projection target information acquisition unit that acquires projection target information relating to the projection target onto which the aforementioned projection information is projected, A control unit controls the projection unit so that the projection information is projected onto the projection target, based on the acquired projection information and projection target information. Equipped with, The projection target information acquisition unit acquires the projection target information determined based on the motion information related to the user's line of sight acquired by the motion information acquisition unit worn by the user, and further, The projection target information acquisition unit acquires the projection target information determined based on a surrounding image that includes the area around the user, captured by the imaging unit worn by the user. Information processing device.

2. The projection target information acquisition unit acquires the projection target information based on the information relating to the user's position. The information processing apparatus according to claim 1.

3. The projection target information acquisition unit acquires the projection target information stored in the storage unit in association with the projection information from the storage unit. The information processing apparatus according to claim 1 or 2.

4. The control unit controls the projection unit to change at least one of the projection information and the projection target projected by the projection unit, based on the image captured by the imaging unit of the projection target onto which the projection information is projected. The information processing apparatus according to claim 1.

5. The control unit adjusts the projection information to be projected onto the projection unit based on the shape of the projection surface of the projection target included in the projection target information and information relating to the user's position. The information processing apparatus according to any one of claims 1 to 4.

6. An information processing system comprising: an information acquisition unit that acquires information regarding the location of a user wearing a wearable device; a projection unit that projects projection information from the wearable device worn by the user onto a projection target; and an information processing unit that controls the projection unit, A projection information acquisition unit acquires projection information from a storage unit based on the information related to the user's location acquired by the information acquisition unit, A projection target information acquisition unit that acquires projection target information relating to the projection target onto which the aforementioned projection information is projected, A control unit controls the projection unit so that the projection information is projected onto the projection target, based on the acquired projection information and projection target information. Equipped with, The projection target information acquisition unit acquires the projection target information determined based on the operation information related to the user's line of sight acquired by the operation information acquisition unit of the wearable device worn by the user, and further, The projection target information acquisition unit acquires the projection target information determined based on a surrounding image that includes the area around the user, captured by the imaging unit of the wearable device worn by the user. Information processing system.

7. The projection target information acquisition unit acquires the projection target information based on the information relating to the user's position. The information processing system according to claim 6.

8. The projection target information acquisition unit acquires the projection target information stored in the storage unit in association with the projection information from the storage unit. The information processing system according to claim 6 or 7.

9. The control unit controls the projection unit to change at least one of the projection information and the projection target projected by the projection unit, based on the image captured by the imaging unit of the projection target onto which the projection information is projected. The information processing system according to claim 6.

10. The control unit adjusts the projection information to be projected onto the projection unit based on the shape of the projection surface of the projection target included in the projection target information and information relating to the user's position. The information processing system according to any one of claims 6 to 9.

11. Using the information processing system according to any one of claims 6 to 10, support information for assisting the user, obtained from the storage unit, is projected onto the projection target. Support system.

12. Information processing device, A projection information acquisition step involves acquiring projection information projected by a projection unit worn by the user from a storage unit, based on information related to the user's position acquired by an information acquisition unit worn by the user. A step of acquiring projection target information related to the projection target onto which the aforementioned projection information is projected, A control step that controls the projection unit so that the projection information is projected onto the projection target based on the acquired projection information and projection target information, Execute, The aforementioned projection target information acquisition step acquires the projection target information determined based on the motion information related to the user's line of sight direction, acquired by the motion information acquisition unit worn by the user, and further, The aforementioned step of acquiring projection target information involves acquiring projection target information determined based on a surrounding image captured by an imaging unit worn by the user, which includes the area around the user within its imaging range. Information processing methods.

Images