Information processing apparatus, operating system, information processing method, and program

Abstract

An information processing device (1) according to the present disclosure comprises: a detected information reception unit (14) that receives five senses information recognizable by at least one of the five senses of human and enhanced five senses information which is difficult to recognize by the five senses of human; a visual data conversion unit (51), an audio data conversion unit (52), a tactile data conversion unit (53), an olfactory data conversion unit (54) and a gustatory data conversion unit (55) that convert the enhanced five senses information into converted five senses data representing information recognizable by at least one of the five senses of human; a visual data synthesis unit (61), an audio data synthesis unit (62), a tactile data synthesis unit (63), an olfactory data synthesis unit (64) and a gustatory data synthesis unit (65) that synthesize the converted five senses data with the corresponding five senses information; and an expression information transmission unit (16) that transmits the synthesized data as the result of the synthesis to an expression device that conveys the synthesized data to a user in a user-recognizable manner.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus in an operation system in which a device is controlled according to a user's operation, an operation system including the information processing apparatus, an information processing method, and a program.

Background Art

[0002] There is known an operation system that enables a user located at a location different from a device such as a robot to operate the device. For example, in Patent Document 1, sensors such as a camera and a microphone are provided at the installation position of a robot, and based on the information received from the sensors, the information is reproduced so that the user can recognize the information visually or aurally, thereby transmitting the information to the user. Thus, with the technique described in Patent Document 1, a user located at a location away from the robot can remotely operate the robot while grasping the situation around the robot.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] With the technique described in Patent Document 1, information equivalent to the information that can be recognized visually or aurally when a human user is at the position of the robot is reproduced. In order to improve work efficiency or presence, reproduction of only the information that can be recognized by the five human senses may be insufficient.

[0005] The present disclosure has been made in view of the above, and an object thereof is to obtain an information processing apparatus capable of allowing a user to recognize information that cannot be recognized by the five human senses.

Means for Solving the Problems

[0006] In order to solve the above-described problems and achieve the object, an information processing apparatus according to the present disclosure includes a detection information receiving unit that receives at least one piece of five-sense information recognizable by at least one of the five senses of a person in a device controlled in response to an operation from a user existing at a location remote from the device, and extended five-sense information that is information difficult to recognize by the five senses of a person in the device, and a conversion unit that converts the extended five-sense information into conversion five-sense data indicating information that can be recognized by at least one of the five senses of a person. The information processing apparatus further includes a synthesizing unit that synthesizes the conversion five-sense data and the corresponding five-sense information, and a presentation information transmitting unit that transmits the synthesized data, which is the result of the synthesis by the synthesizing unit, to a presentation device that transmits the synthesized data to the user so that the user can recognize the synthesized data by at least one of the five senses. A filter unit that filters synthetic data, comprises and the expression information transmission unit transmits the filtered synthetic data to the expression device it.

Advantages of the Invention

[0007] The information processing apparatus according to the present disclosure has an effect that it can cause a user to recognize information that cannot be recognized by the five senses of a person.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Embodiments for Carrying Out the Invention

[0009] Hereinafter, an information processing apparatus, an operating system, an information processing method, and a program according to the embodiments will be described in detail with reference to the drawings.

[0010] Embodiment 1. FIG. 1 is a diagram showing a configuration example of the operating system according to Embodiment 1. The operating system 5 of the present embodiment includes an information processing apparatus 1, a device 2, a motion information detection device 3, and a presentation device 4. The device 2 has a movable part and operates based on control information transmitted from the information processing apparatus 1. The device 2 is, for example, a robot, a manipulator, a working machine, or the like. In FIG. 1, a humanoid robot is illustrated as an example of the device 2, but the device 2 is not limited to a humanoid robot.

[0011] The operating system 5 of the present embodiment can be applied, for example, to rescue activities during disasters, operations in places where there may be harmful substances to humans such as explosive demolition work, operations in places where it is difficult for humans to work such as high places or narrow places, and the like. By installing the device 2 at these workplaces and allowing the user to remotely operate the device from a location away from the device 2, the user can perform the work safely. In addition, the operating system 5 can be used for a user located at a location away from the device 2 to operate the device 2, such as in remote surgery, remote cooking, and the like. It can also be used for travel experiences, natural experiences, education using the environment and objects in remote areas, and the like. The applications of the operating system 5 of the present embodiment described above are examples and are not limited to the examples described above.

[0012] The motion information detection device 3 detects motion information, which is information regarding the motion of a user who is at a location separated from the device 2, and transmits the detected motion information to the information processing device 1. The motion information is, for example, information indicating the position, direction, and motion of the user's eyes, ears, nose, hands, and mouth. The motion information is used to operate the device 2 according to the motion of the user. Note that it is not essential to acquire motion information regarding all of the eyes, ears, nose, hands, and mouth, and depending on the configuration of the device 2, information that is not necessary may not be acquired. For example, if the device 2 is not provided with a mechanism for simulating the motion of the mouth, motion information regarding the mouth may not be detected. Further, as will be described later, the motion information detection device 3 is composed of one or more devices.

[0013] The device 2 is controlled according to an operation from a user who is at a location separated from the device 2. The device 2 includes a control information reception unit 21, a drive control unit 22, a drive unit 23, a detection information acquisition unit 24, and a detection information transmission unit 25. The control information reception unit 21 receives control information from the information processing device 1 and outputs the received control information to the drive control unit 22. The drive control unit 22 controls the drive unit 23 using the control information received from the control information reception unit 21. The detection information acquisition unit 24 detects five-sense information, which is information that can be recognized by a person's five senses, and outputs the detected five-sense information to the detection information transmission unit 25. The five-sense information acquired by the detection information acquisition unit 24 is information that can be recognized by at least one of the five senses when the user is present at the position of the device 2. That is, the five-sense information is information that can be recognized by at least one of a person's five senses in the device 2. The detection information acquisition unit 24 further detects extended five-sense information, which is information that is difficult for a person to recognize with the five senses in the device 2, and outputs the detected extended five-sense information to the detection information transmission unit 25. Further, as will be described later, the extended five-sense information may be information regarding each of the five-sense information, that is, information obtained by extending each of the five senses. The detection information transmission unit 25 transmits the five-sense information and the extended five-sense information to the information processing device 1. Details of the drive control unit 22, the drive unit 23, and the detection information acquisition unit 24 will be described later.

[0014] The extended five-sense information is information regarding physical quantities that are the same as or similar to the information that a person can recognize through the five senses, and is information outside the range that a person can recognize through the five senses. The extended five-sense information is information indicating the detection result of light outside the visible wavelength range, information indicating the detection result of sound with a frequency outside the audible range of a person, information at a level smaller than the minimum value that a person can recognize, information at a level larger than the maximum value that a person can recognize (such as sound that can break the eardrum, strong light that can damage the retina, etc.), information that a person cannot separately detect (such as the type of gas), information outside the visual field range of a person, and the like. For example, regarding vision, visible light can be recognized by the five senses, so the detection result of visible light is information that can be recognized by the five senses, but light other than visible light, such as infrared light, ultraviolet light, etc., is extended five-sense information that is difficult to recognize by the five senses. The extended five-sense information can also be called information of more than five senses or information outside the five senses. Details of the extended five-sense information will be described later.

[0015] Based on the expression information received from the information processing device 1, the expression device 4 presents the information detected by the detection information acquisition unit 24 of the device 2 to the user by performing an expression that the user can recognize as at least one of the five senses. That is, the expression device 4 transmits the received expression information to the user so that the user can recognize it with at least one of the five senses. Thereby, the user can recognize each piece of information at the position of the device 2 through the five senses and operate based on this information. This operation of the user is detected by the movement information detection device 3 described above. Note that the expression device 4 only needs to be able to perform an expression corresponding to at least one of the five senses of vision, hearing, touch, smell, and taste. Details of the expression device 4 will be described later.

[0016] The information processing apparatus 1 generates control information for controlling the device 2 by using the motion information received from the motion information detection device 3, and transmits the generated control information to the device 2. Further, the information processing apparatus 1 converts the extended five-sense information received from the device 2 into five-sense data recognizable by human five senses, synthesizes the five-sense information received from the device 2 and the five-sense data obtained by the conversion, and uses the synthesized five-sense information to generate expression information for causing the expression device 4 to perform an expression that can be recognized as at least one of the five senses, and transmits the generated expression information to the expression device 4.

[0017] As described above, the information processing apparatus 1 of the present embodiment can cause the user to recognize information that cannot be recognized by human five senses by converting the extended five-sense information into five-sense data and synthesizing it with the acquired five-sense information for presentation to the user.

[0018] The information processing apparatus 1 includes a control information transmission unit 11, a device control unit 12, a motion information reception unit 13, a detection information reception unit 14, an information conversion unit 15, and an expression information transmission unit 16. The control information transmission unit 11, the device control unit 12, and the motion information reception unit 13 perform remote operation processing, which is control for causing the device 2 to perform an operation according to the user's motion. The detection information reception unit 14, the information conversion unit 15, and the expression information transmission unit 16 perform information transmission processing for transmitting the five-sense information and the extended five-sense information acquired by the detection information acquisition unit 24 of the device 2 to the user. Hereinafter, it will be described separately as remote operation processing and information transmission processing.

[0019] FIG. 2 is a diagram showing a detailed configuration example related to the remote control process of the present embodiment. The motion information detection device 3 includes an eye information detection unit 31, an ear information detection unit 32, a hand information detection unit 33, a nose information detection unit 34, and a mouth information detection unit 35. The eye information detection unit 31 detects the position, orientation, and movement of the user's eyes. The ear information detection unit 32 detects the position, orientation, and movement of the user's ears. The hand information detection unit 33 detects the position, orientation, and movement of the user's hands and fingers, and also detects the pressure at each part of the hands and fingers. The nose information detection unit 34 detects the position, orientation, and movement of the user's nose. The mouth information detection unit 35 detects the position, orientation, and movement of the user's mouth and tongue. The eye information detection unit 31, the ear information detection unit 32, the hand information detection unit 33, the nose information detection unit 34, and the mouth information detection unit 35 include an imaging device such as a camera, for example, and the hand information detection unit 33 includes a pressure sensor. In addition, for the detection of the orientation and position of the eyes, ears, nose, and mouth in the eye information detection unit 31, the ear information detection unit 32, the nose information detection unit 34, and the mouth information detection unit 35, an imaging device that detects the orientation and position of the user's body and the orientation and position of the user's face may be used. The eye information detection unit 31, the ear information detection unit 32, the hand information detection unit 33, the nose information detection unit 34, and the mouth information detection unit 35 may be worn by the user, or may be a camera that photographs the user from the outside. Further, the motion information detection device 3 may include a positioning sensor attached to a device worn by the user.

[0020] Also, as shown in FIG. 2, the drive unit 23 of the device 2 includes a vision-related device drive unit 231, an auditory-related device drive unit 232, a tactile-related device drive unit 233, an olfactory-related device drive unit 234, and a gustatory-related device drive unit 235, which are examples of a plurality of device drive units. When the device 2 is a humanoid robot, for example, the vision-related device drive unit 231, the auditory-related device drive unit 232, the tactile-related device drive unit 233, the olfactory-related device drive unit 234, and the gustatory-related device drive unit 235 are parts corresponding to the eyes, ears, hands, nose, and mouth of the robot, respectively.

[0021] FIG. 3 is a flowchart showing an example of a remote operation processing procedure in the information processing apparatus 1 of the present embodiment. The motion information receiving unit 13 of the information processing apparatus 1 acquires motion information (information indicating the position, orientation, motion, etc. of the user's eyes, ears, hands, nose, and mouth) (step S1). Specifically, the motion information receiving unit 13 acquires the motion information by receiving the detection results from the eye information detection unit 31, ear information detection unit 32, hand information detection unit 33, nose information detection unit 34, and mouth information detection unit 35, respectively, and outputs the acquired motion information to the device control unit 12.

[0022] The device control unit 12 generates control information for reflecting the motion information in the motion of the device (step S2). Specifically, the device control unit 12 uses the motion information received from the eye information detection unit 31, ear information detection unit 32, hand information detection unit 33, nose information detection unit 34, and mouth information detection unit 35 via the motion information receiving unit 13 to generate control information for controlling the corresponding visual-related device driving unit 231, auditory-related device driving unit 232, tactile-related device driving unit 233, olfactory-related device driving unit 234, and gustatory-related device driving unit 235, respectively. That is, the device control unit 12 generates control information corresponding to each of vision, hearing, touch, smell, and taste so that the visual-related device driving unit 231, auditory-related device driving unit 232, tactile-related device driving unit 233, olfactory-related device driving unit 234, and gustatory-related device driving unit 235 can simulate the motions of the user's eyes, ears, hands, nose, and mouth, respectively. The device control unit 12 outputs the generated control information to the control information transmitting unit 11.

[0023] The control information transmitting unit 11 transmits the control information (step S3). Specifically, the control information transmitting unit 11 transmits the control information corresponding to each of vision, hearing, touch, smell, and taste received from the device control unit 12 to the device 2. The information processing apparatus 1 performs the above-described steps S1 to S3 for each remote operation control cycle. The remote operation control cycle may be the same as or different from the acquisition cycle of the motion information.

[0024] In this way, the motion information receiving unit 13 receives motion information from the motion information detection device 3 and outputs the received motion information to the device control unit 12. That is, the motion information receiving unit 13 acquires motion information indicating the positions and movements of each part of the user corresponding to a plurality of device driving units in the device. The device control unit 12 uses the motion information received from the motion information detection device 3 to generate control information for causing the device 2 to perform an operation corresponding to the motion information, and outputs the generated control information to the control information transmission unit 11. Specifically, the device control unit 12 generates control information for controlling the operations of a plurality of device driving units in the device 2 using the motion information. The control information transmission unit 11 transmits the control information received from the device control unit 12 to the device 2.

[0025] The control information receiving unit 21 of the device 2 outputs control information for vision, hearing, touch, smell, and taste to the drive control unit 22. The drive control unit 22 uses the control information for vision, hearing, touch, smell, and taste to control the corresponding vision-related device drive unit 231, hearing-related device drive unit 232, touch-related device drive unit 233, smell-related device drive unit 234, and taste-related device drive unit 235, respectively. Thereby, the device 2 can simulate the movements of the user's eyes, ears, hands, nose, and mouth. That is, the movements of the user's eyes, ears, hands, nose, and mouth become operation information indicating the operation content that determines the operation of the device 2, and remote operation of the device 2 is realized.

[0026] Note that, as described above, the device 2 is not limited to a humanoid robot, and the device 2 does not have to simulate all of the eyes, ears, hands, nose, and mouth, and motion information that is not necessary may not be acquired according to the configuration of the device 2. Also, here, an example in which the user's movement itself is the operation content has been described, but it is not limited to this. For example, operation means for operating the device 2 such as a lever, a touch pad, or a button may be provided separately, and the device 2 may be remotely operated by the user operating the operation means.

[0027] FIG. 4 is a diagram showing a detailed configuration example related to the information transmission process of the present embodiment. As shown in FIG. 4, the detection information acquisition unit 24 of the device 2 includes a visual information detection unit 241, an extended visual information detection unit 242, an auditory information detection unit 243, an extended auditory information detection unit 244, a tactile information detection unit 245, an extended tactile information detection unit 246, an olfactory information detection unit 247, an extended olfactory information detection unit 248, a gustatory information detection unit 249, and an extended gustatory information detection unit 250.

[0028] The visual information detection unit 241 detects visual information, which is information that can be recognized by a person visually. As the visual information detection unit 241, for example, a visible camera or the like can be used. The extended visual information detection unit 242 detects extended visual information, which is information related to vision and is difficult for a person to recognize by the five senses. The extended visual information is, for example, an image outside the range of a person's visual field or a detection result of light with a wavelength other than visible light. The extended visual information detection unit 242 may be, for example, an infrared camera, a temperature sensor, an ultraviolet camera, etc. that acquire light with a wavelength different from visible light as an image, or a rear camera, a high-sensitivity camera, etc. that image a direction where a person cannot visually recognize, such as the rear direction or a distant place.

[0029] The auditory information detection unit 243 detects auditory information, which is information that can be recognized by a person auditorily. As the auditory information detection unit 243, for example, a microphone (mic) or the like can be used. The extended auditory information detection unit 244 detects extended auditory information, which is information related to audition and is difficult for a person to recognize by the five senses. The extended auditory information is, for example, information related to sounds that are difficult for a person to recognize. The extended auditory information detection unit 244 detects sounds outside the frequency range of human audible range or small-amplitude sounds that a person cannot detect. For example, a high-sensitivity microphone, a broadband microphone, etc.

[0030] The tactile information detection unit 245 detects tactile information, which is information that can be recognized by a person through the sense of touch. As the tactile information detection unit 245, for example, a pressure sensor or the like can be used. The extended tactile information detection unit 246 detects extended tactile information, which is information related to touch and is difficult for a person to recognize through the five senses. The extended tactile information is, for example, information indicating at least one of vibration, pressure, air flow, and inclination that is difficult for a person to detect. The extended tactile information detection unit 246 detects, for example, minute vibrations or minute air pressure fluctuations (air currents) that a person cannot recognize. The extended tactile information detection unit 246 is, for example, a high-sensitivity vibration sensor or the like.

[0031] The olfactory information detection unit 247 detects olfactory information, which is information that can be recognized by a person through the sense of smell. As the olfactory information detection unit 247, for example, a smell sensor or the like can be used. The extended olfactory information detection unit 248 detects extended olfactory information, which is information related to smell and is difficult for a person to recognize through the five senses. The extended olfactory information is, for example, information related to a smell that is difficult for a person to recognize. As the extended olfactory information detection unit 248, a high-sensitivity smell sensor, a gas sensor, or the like can be used.

[0032] The taste information detection unit 249 detects taste information, which is information that can be recognized by a person through the sense of taste. As the taste information detection unit 249, for example, a taste sensor or the like can be used. The extended taste information detection unit 250 detects extended taste information, which is information related to taste and is difficult for a person to recognize through the five senses. The extended taste information is, for example, information related to a taste that is difficult for a person to recognize. As the extended taste information detection unit 250, a high-sensitivity taste sensor or the like can be used.

[0033] In the example shown in FIG. 4, the detection information acquisition unit 24 includes a visual information detection unit 241, an extended visual information detection unit 242, an auditory information detection unit 243, an extended auditory information detection unit 244, a tactile information detection unit 245, an extended tactile information detection unit 246, an olfactory information detection unit 247, an extended olfactory information detection unit 248, a gustatory information detection unit 249, and an extended gustatory information detection unit 250. However, depending on the application, it is not necessary to include all of these. The detection information acquisition unit 24 includes at least one of the visual information detection unit 241, the auditory information detection unit 243, the tactile information detection unit 245, the olfactory information detection unit 247, and the gustatory information detection unit 249, and it is sufficient to include at least one of the extended visual information detection unit 242, the extended auditory information detection unit 244, the extended tactile information detection unit 246, the extended olfactory information detection unit 248, and the extended gustatory information detection unit 250. That is, the detection information acquisition unit 24 only needs to detect at least one of the five senses information, which is at least one of the visual information, auditory information, tactile information, olfactory information, and gustatory information described above, and detect at least one of the extended five senses information, which is at least one of the extended visual information, extended auditory information, extended tactile information, extended olfactory information, and extended gustatory information. In other words, the detection information acquisition unit 24 includes a five senses information acquisition unit that acquires five senses information and an extended five senses information acquisition unit that acquires extended five senses information. In the example shown in FIG. 4, the five senses information acquisition unit includes the visual information detection unit 241, the auditory information detection unit 243, the tactile information detection unit 245, the olfactory information detection unit 247, and the gustatory information detection unit 249, and the extended five senses information acquisition unit includes the extended visual information detection unit 242, the extended auditory information detection unit 244, the extended tactile information detection unit 246, the extended olfactory information detection unit 248, and the extended gustatory information detection unit 250.

[0034] Also, as shown in FIG. 4, the expression device 4 includes a visual expression device 41, an auditory expression device 42, a tactile expression device 43, an olfactory expression device 44, and a gustatory expression device 45.

[0035] The visual presentation device 41 is a display device such as a display capable of displaying images, for example, a 3D display, VR (Virtual Reality) goggles, a VR head-mounted display, a glasses-type display, etc., but is not limited thereto. The auditory presentation device 42 is a device that emits sound, for example, a speaker, and may be a 360-degree speaker or the like. The tactile presentation device 43 is a device that generates pressure, for example, a tactile glove, a force-sensing glove, etc. The olfactory presentation device 44 is a device that generates an odor, for example, an odor-generating device. The gustatory presentation device 45 is a device that generates a taste, for example, a taste-generating device worn on the inside of the mouth, the jaw, the hand, etc.

[0036] In the example shown in FIG. 4, the presentation device 4 includes a visual presentation device 41, an auditory presentation device 42, a tactile presentation device 43, an olfactory presentation device 44, and a gustatory presentation device 45. However, depending on the application, it is not necessary to include all of these. The presentation device 4 only needs to include at least one of the visual presentation device 41, the auditory presentation device 42, the tactile presentation device 43, the olfactory presentation device 44, and the gustatory presentation device 45.

[0037] FIG. 5 is a diagram showing a configuration example of the information conversion unit 15 in the information processing device 1 of the present embodiment. As shown in FIG. 5, the information conversion unit 15 includes a visual data conversion unit 51, an auditory data conversion unit 52, a tactile data conversion unit 53, an olfactory data conversion unit 54, a gustatory data conversion unit 55, a visual data synthesis unit 61, an auditory data synthesis unit 62, a tactile data synthesis unit 63, an olfactory data synthesis unit 64, a gustatory data synthesis unit 65, a visual data filter unit 71, an auditory data filter unit 72, a tactile data filter unit 73, an olfactory data filter unit 74, and a gustatory data filter unit 75.

[0038] Each of the visual data conversion unit 51, the auditory data conversion unit 52, the tactile data conversion unit 53, the olfactory data conversion unit 54, and the gustatory data conversion unit 55 is an example of a conversion unit that converts extended five-sense information into conversion five-sense data (five-sense data) indicating information that can be recognized by at least one of the five senses of a human. Each of the visual data synthesis unit 61, the auditory data synthesis unit 62, the tactile data synthesis unit 63, the olfactory data synthesis unit 64, and the gustatory data synthesis unit 65 is an example of a synthesis unit that synthesizes five-sense data and corresponding five-sense information. Each of the visual data filter unit 71, the auditory data filter unit 72, the tactile data filter unit 73, the olfactory data filter unit 74, and the gustatory data filter unit 75 is an example of a filter unit that filters the synthesized data.

[0039] Note that in FIG. 5, an example of performing processing corresponding to all five senses is shown. However, depending on the configuration of the detection information acquisition unit 24 and the configuration of the expression device 4, some of the units in the information conversion unit 15 shown in FIG. 5 may not be provided. For example, when the gustatory expression device 45 is not provided, the gustatory data conversion unit 55, the gustatory data synthesis unit 65, and the gustatory data filter unit 75 may not be provided.

[0040] FIG. 6 is a flowchart showing an example of an information transmission processing procedure in the information processing apparatus 1 of the present embodiment. As shown in FIG. 6, the information processing apparatus 1 acquires five-sense information (step S11). Specifically, the detection information reception unit 14 receives visual information, auditory information, tactile information, olfactory information, and gustatory information that can be recognized by a human through the five senses from the device 2, and outputs the received visual information, auditory information, tactile information, olfactory information, and gustatory information to the information conversion unit 15. Further, the information processing apparatus 1 acquires extended five-sense information (step S12). Specifically, the detection information reception unit 14 receives extended visual information, extended auditory information, extended tactile information, extended olfactory information, and extended gustatory information that are difficult for a human to recognize through the five senses from the device 2, and outputs the received extended visual information, extended auditory information, extended tactile information, extended olfactory information, and extended gustatory information to the information conversion unit 15. Note that the order of step S11 and step S12 is not limited to that shown in FIG. 6, and they may be in the reverse order or performed simultaneously.

[0041] Next, the information conversion unit 15 converts the extended five-sense information into five-sense data (step S13). The five-sense data is at least one of visual data, auditory data, tactile data, olfactory data, and gustatory data that can be recognized by a person through the five senses.

[0042] Specifically, the visual data conversion unit 51 converts at least one of the extended visual information, extended auditory information, extended tactile information, extended olfactory information, and extended gustatory information into visual data that can be recognized by a person visually, and outputs the converted visual data to the visual data synthesis unit 61. The auditory data conversion unit 52 converts at least one of the extended visual information, extended auditory information, extended tactile information, extended olfactory information, and extended gustatory information into auditory data that can be recognized by a person aurally, and outputs the converted auditory data to the auditory data synthesis unit 62. The tactile data conversion unit 53 converts at least one of the extended visual information, extended auditory information, extended tactile information, extended olfactory information, and extended gustatory information into tactile data that can be recognized by a person tactilely, and outputs the converted tactile data to the tactile data synthesis unit 63. The olfactory data conversion unit 54 converts at least one of the extended visual information, extended auditory information, extended tactile information, extended olfactory information, and extended gustatory information into olfactory data that can be recognized by a person olfactorily, and outputs the converted olfactory data to the olfactory data synthesis unit 64. The gustatory data conversion unit 55 converts at least one of the extended visual information, extended auditory information, extended tactile information, extended olfactory information, and extended gustatory information into gustatory data that can be recognized by a person gustatorily, and outputs the converted gustatory data to the gustatory data synthesis unit 65.

[0043] The conversion from the extended five-sense information to the five-sense data may be performed by converting it into data of the same type of five senses, for example, the visual data conversion unit 51 converts the extended visual information into visual data, or may be converted into data of different types of five senses, such as converting the extended taste information into visual data. Also, the visual data conversion unit 51 may generate the five-sense data using a plurality of pieces of information, such as converting a plurality of pieces of information among the extended visual information, the extended auditory information, the extended tactile information, the extended olfactory information, and the extended taste information into visual data. Also, one piece of extended five-sense information may be converted into a plurality of types of five-sense data, such as the extended auditory information being converted into visual data and auditory data.

[0044] Next, the information conversion unit 15 performs a synthesis process (step S14). Specifically, the visual data synthesis unit 61 synthesizes the visual data obtained by the conversion and the visual information acquired in step S11, and outputs the synthesized information (synthesized data) to the visual data filter unit 71. The auditory data synthesis unit 62 synthesizes the auditory data obtained by the conversion and the auditory information acquired in step S11, and outputs the synthesized information to the auditory data filter unit 72. The tactile data synthesis unit 63 synthesizes the tactile data obtained by the conversion and the tactile information acquired in step S11, and outputs the synthesized information to the tactile data filter unit 73. The olfactory data synthesis unit 64 synthesizes the olfactory data obtained by the conversion and the olfactory information acquired in step S11, and outputs the synthesized information to the olfactory data filter unit 74. The taste data synthesis unit 65 synthesizes the taste data obtained by the conversion and the taste information acquired in step S11, and outputs the synthesized information to the taste data filter unit 75.

[0045] Next, the information conversion unit 15 filters the synthesized information (step S15). Specifically, the visual data filter unit 71, the auditory data filter unit 72, the tactile data filter unit 73, the olfactory data filter unit 74, and the gustatory data filter unit 75 perform filtering according to the value of the synthesized data (synthesized information), which is the result of synthesis by the corresponding synthesis unit. For example, if the value of the synthesized information is too large, that is, if the intensity expressed for presentation to the user is too strong, it may have an adverse effect on the user. However, by performing filtering, it is possible to prevent an adverse effect on the user.

[0046] Examples of filtering include, for example, when the input value, which is the value of the synthesized information, exceeds a predetermined constant value, a process of converting the input value to a smaller value and outputting it, or when the input value exceeds a constant value, a process of outputting a predetermined value (including 0) below the constant value. Also, as filtering, when the input value is small, a process of multiplying the input value by a value greater than 1 to output a value greater than the input value may be performed. For example, filtering may include a process of multiplying by a gain according to the input value, that is, the value of the synthesized data. In this case, for example, when the input value is less than or equal to the first value, the gain may be a value greater than 1, and when the input value exceeds the second value greater than the first value, the gain may be a value less than 1. That is, for example, as filtering, when the input value is less than or equal to the first value, it is multiplied by G1 (G1 > 1), when the input value is greater than the first value and less than or equal to the second value (the second value > the first value), the input value is output as it is, and when the input value is greater than the second value, it is multiplied by G2 (G2 < 1).

[0047] Also, the range of the input value may be classified into four or more ranges, and a value (gain) to be multiplied may be determined for each range. Also, the output value may be formulated as a non-linear function of the input value, and a process of outputting a function value corresponding to the function may be performed as filtering. The specific content of filtering is not limited to this example.

[0048] Next, the information processing apparatus 1 transmits the filtered information to the presentation apparatus 4 (step S16). Specifically, the visual data filter unit 71, the auditory data filter unit 72, the tactile data filter unit 73, the olfactory data filter unit 74, and the gustatory data filter unit 75 output the filtered information to the presentation information transmission unit 16 as presentation information (visual presentation information, auditory presentation information, tactile presentation information, olfactory presentation information, and gustatory presentation information), respectively. The presentation information transmission unit 16 transmits the visual presentation information, the auditory presentation information, the tactile presentation information, the olfactory presentation information, and the gustatory presentation information to the corresponding visual presentation apparatus 41, auditory presentation apparatus 42, tactile presentation apparatus 43, olfactory presentation apparatus 44, and gustatory presentation apparatus 45, respectively.

[0049] The information processing apparatus 1 performs the above-described information transmission process for each information transmission control cycle. The information transmission control cycle may be the same as or different from the acquisition cycle of the five-sense information and the extended five-sense information. Also, the information transmission control cycle may be the same as or different from the remote operation control cycle.

[0050] The visual presentation apparatus 41, the auditory presentation apparatus 42, the tactile presentation apparatus 43, the olfactory presentation apparatus 44, and the gustatory presentation apparatus 45 can cause the user to recognize this information as the five senses by performing presentations based on the visual presentation information, the auditory presentation information, the tactile presentation information, the olfactory presentation information, and the gustatory presentation information, respectively. As described above, in the present embodiment, since the extended five-sense information that cannot be recognized by a person is synthesized with the five-sense information that can be recognized by a person, the user can grasp the state of the device 2 and the surrounding situation in more detail than when only the five-sense information is transmitted. As a result, the sense of strangeness of the user's remote operation can be reduced, the user can perform intuitive operations, and the work efficiency can be improved. Also, since the user can grasp the state of the device 2 and the surrounding situation in more detail, more appropriate operations can be realized, and the work efficiency can be improved. Further, by performing filtering, remote operations can be safely performed without harming the user.

[0051] Examples of extended five-sense information and five-sense data are given below. Note that the content of the extended five-sense information and five-sense data is not limited to the following examples.

[0052] Example 1: As extended auditory information, detect minute sounds that humans cannot hear, such as gas leakage sounds, and convert the extended auditory information into visual data so that the user can recognize gas leakage. For example, it may be indicated by characters showing an alert such as "Gas leakage occurred" as visual data, or gas leakage may be indicated by an image or video. Note that the specific characters for alert display are not limited to this example. When gas leakage is indicated by an image or video, for example, the direction of the gas sound (the gas generation source) may be indicated by color, an arrow, etc. Also, depending on the type of minute sound, conduct tests in advance to learn through machine learning what visual data humans can use to imagine the phenomenon corresponding to that sound, and the information conversion unit 15 may determine the visual data using the learned model. Also, the correspondence of conversion may be determined based on rules rather than machine learning.

[0053] Example 2: As extended auditory information, detect minute sounds that humans cannot hear, such as the voices of people buried under rubble, and convert the extended auditory information into auditory data corresponding to loud sounds that humans can recognize so that the user can recognize the minute sounds.

[0054] Example 3: As extended olfactory information, detect minute odors that humans cannot recognize, such as almost odorless toxic gases, and convert the extended olfactory information into auditory data of an alert sound that humans can recognize so that the user can recognize the odorless toxic gas.

[0055] Example 4: As extended olfactory information, detect minute odors that humans cannot recognize, such as almost odorless toxic gases, and convert the extended olfactory information into visual data that humans can recognize, such as a concentration distribution map, so that the user can estimate the source of the odorless toxic gas.

[0056] Example 5: As extended tactile information or extended visual information, detect a temperature (ultra-high temperature) that cannot be recognized by humans, and convert it into visual data that can be recognized by humans, such as making the ultra-high temperature part red in the extended tactile information or extended visual information, so that the user can recognize the ultra-high temperature location.

[0057] Example 6: As extended visual information, obtain an image in a direction that cannot be recognized by humans, such as an image of the rear, and convert the extended visual information into visual data that can be recognized by humans, so that the user can recognize the state of the rear.

[0058] Example 7: As extended visual information, obtain an image of a dark environment that cannot be recognized by humans, and convert the extended visual information into visual data with enhanced sensitivity that can be recognized by humans, so that the user can recognize the state in the dark environment.

[0059] Example 8: As extended tactile information, detect the generation of a slight force, such as a minute vibration that cannot be recognized by humans (minute vibrations of the ground or a building (air) due to explosives or aging), minute air pressure fluctuations (flow of air currents), etc. When the generation of a slight force is detected in the extended tactile information, convert the extended tactile information into auditory data indicating an alert sound or the like, so that the user can recognize the generation of a slight force.

[0060] Example 9: As extended taste information, detect a delicate taste that cannot be recognized by humans, and convert the extended taste information into auditory data such as voice indicating the components constituting the taste, so that the user can recognize the delicate taste.

[0061] Example 10: As extended tactile information, detect minute inclinations that are imperceptible to humans, such as deviations from the gravitational or horizontal directions, and convert the extended tactile information into visual data as numerical values indicating the inclinations, so that the user can recognize the inclinations. For example, when causing the robot, which is Device 2, to perform rescue activities, if the condition of the feet is poor, phenomena such as slipping or being buried may occur, which may impede the work. Therefore, as the extended tactile information detection unit 246 or the extended visual information detection unit 242, a visible camera, an infrared camera, a sound wave sensor, LiDAR (Light Detection And Ranging), a pressure sensor, a gyro, etc. may be provided on the sole of Device 2 so as to be able to detect the state of objects (inclination, fragility, slippery when wet) and the presence of people around Device 2. Thereby, when there is a person under the rubble near the feet of Device 2, it is possible to prevent getting on that part.

[0062] Note that the information processing apparatus 1 may determine which of the five-sense data to convert the extended five-sense information into according to the value and characteristics (such as time change, frequency, etc.) of the extended five-sense information, either using machine learning or rule-based. For example, according to the characteristics of a minute sound indicated by the extended five-sense information, etc., what kind of sound it is is learned in advance by machine learning to generate a learned model, and the information processing apparatus 1 obtains an inference result of what kind of sound it is by inputting the extended five-sense information into the learned model. The correspondence between the type of sound (what kind of sound it is) and which of the five-sense data (any of visual data, auditory data, tactile data, olfactory data, gustatory data) to convert is determined in advance in a rule-based manner, and the information processing apparatus 1 determines the type of five-sense data to be converted according to the inferred type of sound. Alternatively, the correspondence between the type of sound and the five-sense data to be converted may also be determined by machine learning. For example, for each type of sound, which of the five-sense data (any of visual data, auditory data, tactile data, olfactory data, gustatory data) is easy for a person to recognize the state corresponding to the sound may be learned by machine learning using the results of tests, etc. Also, the correspondence between the value and characteristics of the extended five-sense information and the type of five-sense data to be converted may be directly determined in a rule-based manner. Also, a learned model for obtaining the five-sense data to be converted from the value and characteristics of the extended five-sense information may be created.

[0063] Next, the effect of filtering is exemplified. Note that the content of filtering is not limited to the following examples.

[0064] Example 1 of filtering (tactile): When an explosion occurs during disaster relief and a strong impact hits the device 2, by suppressing or blocking the intensity through filtering, it is possible to prevent the impact from being directly given to the user.

[0065] Example 2 of filtering (auditory): When an explosion occurs during disaster relief and a loud explosion sound is detected, by suppressing or blocking the intensity through filtering, it is possible to prevent a loud explosion sound that could burst the eardrum from being transmitted to the user.

[0066] Example 3 of Filtering (Vision): When an explosion occurs during disaster relief and high-intensity light reaches device 2, filtering can be used to suppress or block the light intensity, so as not to transmit strong light that has an adverse effect on vision to the user.

[0067] Example 4 of Filtering (Smell): When a strong strange smell reaches device 2 due to gas leakage or the like during disaster relief, filtering can be used to suppress or block the smell, so as not to transmit a strange smell that harms the sense of smell to the user.

[0068] Example 5 of Filtering (Taste): When the robot, which is device 2, tastes during cooking and the taste has an impact on the sense of taste, filtering can be used to suppress or block the intensity of the taste, so as not to transmit a taste that harms the sense of taste to the user.

[0069] In addition, in FIG. 1, an example where the communication between device 2 and information processing device 1, between information processing device 1 and motion information detection device 3, and between information processing device 1 and expression device 4 is wireless communication is shown. However, it is not limited to this. The communication between each device in this embodiment may be wired communication, or both wired communication and wireless communication may be used. Also, for the communication between each device in this embodiment, 5G (5th Generation: the 5th generation mobile communication system) or Beyond 5G may be used. By using a low-latency communication method such as Beyond 5G, information can be transmitted to the user without a sense of discomfort. For example, by using a communication method with a shorter delay time than the time resolution of information changes that can be recognized by a person (for example, about 10 to 30 msec), the sense of discomfort of the user can be reduced, and device 2 can be operated efficiently. Also, by using a communication method with a shorter delay time, the sense of presence can be enhanced.

[0070] Next, the hardware configuration of the information processing apparatus 1 according to the present embodiment will be described. The information processing apparatus 1 according to the present embodiment functions as the information processing apparatus 1 when a program, which is a computer program describing the processing in the information processing apparatus 1, is executed on a computer system. FIG. 7 is a diagram showing a configuration example of a computer system that realizes the information processing apparatus 1 according to the present embodiment. As shown in FIG. 7, this computer system includes a control unit 101, an input unit 102, a storage unit 103, a display unit 104, a communication unit 105, and an output unit 106, which are connected via a system bus 107. The control unit 101 and the storage unit 103 constitute a processing circuit.

[0071] In FIG. 7, the control unit 101 is a processor such as a CPU (Central Processing Unit), for example, and executes a program describing the processing in the information processing apparatus 1 according to the present embodiment. Note that a part of the control unit 101 may be realized by dedicated hardware such as a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array). The input unit 102 is composed of, for example, a keyboard, a mouse, etc., and is used by the user of the computer system to input various types of information. The storage unit 103 includes various memories such as a RAM (Random Access Memory) and a ROM (Read Only Memory), and storage devices such as a hard disk, and stores a program to be executed by the control unit 101, necessary data obtained during the processing, and the like. The storage unit 103 is also used as a temporary storage area for programs. The display unit 104 is composed of a display, an LCD (Liquid Crystal Display Panel), etc., and displays various screens to the user of the computer system. The communication unit 105 is a receiver and a transmitter that perform communication processing. The output unit 106 is a printer, a speaker, etc. Note that FIG. 7 is an example, and the configuration of the computer system is not limited to the example of FIG. 7.

[0072] Here, an operation example of the computer system until the program of the present embodiment becomes executable will be described. In a computer system having the above-described configuration, for example, a computer program is installed in the storage unit 103 from a CD-ROM or a DVD-ROM set in a CD (Compact Disc)-ROM drive or a DVD (Digital Versatile Disc)-ROM drive (not shown). Then, when the program is executed, the program read from the storage unit 103 is stored in the main storage area of the storage unit 103. In this state, the control unit 101 executes the processing as the information processing apparatus 1 of the present embodiment according to the program stored in the storage unit 103.

[0073] In the above description, a program describing the processing in the information processing apparatus 1 is provided using a CD-ROM or a DVD-ROM as a recording medium. However, the present invention is not limited to this, and depending on the configuration of the computer system, the capacity of the program to be provided, etc., for example, a program provided by a transmission medium such as the Internet via the communication unit 105 may be used.

[0074] The program of the present embodiment causes the computer system to execute, for example, steps of receiving five-sense information and extended five-sense information, converting the extended five-sense information into conversion five-sense data indicating information that can be recognized by at least one of the five senses of a person, generating composite data by synthesizing the conversion five-sense data and the corresponding five-sense information, and transmitting the composite data to a presentation device 4 that transmits the composite data to the user so that the user can recognize it with at least one of the five senses.

[0075] The device control unit 12 and the information conversion unit 15 shown in FIG. 1 are realized by executing a computer program stored in the storage unit 103 shown in FIG. 7 by the control unit 101 shown in FIG. 7. The storage unit 103 shown in FIG. 7 is also used to realize the device control unit 12 and the information conversion unit 15 shown in FIG. 1. The control information transmission unit 11, the motion information reception unit 13, the detection information reception unit 14, and the expression information transmission unit 16 shown in FIG. 1 are realized by the communication unit 105 shown in FIG. 7. Further, the information processing apparatus 1 may be realized by a plurality of computer systems. For example, the information processing apparatus 1 may be realized by a cloud computer system.

[0076] Also, in the example shown in FIG. 1, an example in which the information processing apparatus 1 performs both remote operation control and information transmission control has been described. However, a remote operation device that performs remote operation control and an information transmission device that performs information transmission control may be provided separately. In this case, the remote operation device includes a control information transmission unit 11, a device control unit 12, and a motion information reception unit 13, and the information transmission device includes a detection information reception unit 14, an information conversion unit 15, and an expression information transmission unit 16.

[0077] As described above, in the present embodiment, in the information processing apparatus 1 in the operation system 5 in which the user operates the device 2 from a remote location, in addition to the five-sense information that can be recognized by a person, extended five-sense information that is difficult for a person to recognize is acquired, the extended five-sense information is converted into five-sense data that can be recognized by a person, and the five-sense data is synthesized with the five-sense information. Then, the information processing apparatus 1 transmits the synthesized information to the expression device 4 that transmits the information to the user as five-sense information, thereby transmitting the synthesized information to the user. Therefore, information that cannot be recognized by a person's five senses can be recognized by the user.

[0078] Embodiment 2. FIG. 8 is a diagram showing a configuration example related to remote operation control of the operation system according to Embodiment 2. The operation system of the present embodiment is the same as the operation system 5 of Embodiment 1 except that the information processing apparatus 1a is provided instead of the information processing apparatus 1. Components having the same functions as those in Embodiment 1 are denoted by the same reference numerals as those in Embodiment 1, and redundant descriptions are omitted. Hereinafter, differences from Embodiment 1 will be mainly described.

[0079] As shown in FIG. 8, the information processing apparatus 1a is the same as the information processing apparatus 1 of Embodiment 1 except that the first delay adjustment unit 17 is added and the information conversion unit 15a is provided instead of the information conversion unit 15. The first delay adjustment unit 17 is a remote control delay adjustment unit that adjusts a delay so that operations based on control information generated using movement information acquired at the same time are simultaneously executed in a plurality of device drive units. Specifically, the first delay adjustment unit 17 acquires movement information from each of the eye information detection unit 31, the ear information detection unit 32, the hand information detection unit 33, the nose information detection unit 34, and the mouth information detection unit 35 via the movement information reception unit 13, and calculates an adjustment amount of the timing so that the timings at which the respective device drive units in the device 2 corresponding to the movement information detected at the same time are driven are the same, and instructs the calculated adjustment amount to the device control unit 12.

[0080] The device control unit 12 adjusts the timing for outputting control information corresponding to each of the visual-related device drive unit 231, the auditory-related device drive unit 232, the tactile-related device drive unit 233, the olfactory-related device drive unit 234, and the gustatory-related device drive unit 235 based on the adjustment amount. Thereby, the first delay adjustment unit 17 can adjust and synchronize the delays of the respective drive units corresponding to the respective movement information detected at the same time. Further, when a plurality of movement information is used for generating the control information, the first delay adjustment unit 17 synchronizes the plurality of movement information corresponding to the same time and outputs it to the device control unit 12.

[0081] For example, it is assumed that time information (timestamp) indicating the detected time is added to the movement information in the eye information detection unit 31, ear information detection unit 32, hand information detection unit 33, nose information detection unit 34, and mouth information detection unit 35. The first delay adjustment unit 17 discriminates the movement information corresponding to the same time using the time information of each movement information. Here, it is assumed that the case where the time difference is equal to or less than the threshold value is the same time. The first delay adjustment unit 17 sets the latest time among the times when each movement information corresponding to the same time is received as the reference time, and calculates the difference between the time when the other movement information is received and the reference time as the adjustment amount. By the first delay adjustment unit 17 outputting this adjustment amount to the device control unit 12, the device control unit 12 uses the adjustment amount to control the visual-related device drive unit 231, auditory-related device drive unit 232, tactile-related device drive unit 233, olfactory-related device drive unit 234, and taste-related device drive unit 235, respectively. The timing for outputting the control information to the control information transmission unit 11 is adjusted. Thereby, the first delay adjustment unit 17 can adjust the delay (reception delay) from the acquisition by the eye information detection unit 31, ear information detection unit 32, hand information detection unit 33, nose information detection unit 34, and mouth information detection unit 35 until the information processing device 1a receives each movement information. Here, the latest time among the times when each movement information corresponding to the same time is received is set as the reference time, but the method for determining the reference time is not limited to this.

[0082] Note that, for example, the acquisition periods of information in the eye information detection unit 31, ear information detection unit 32, hand information detection unit 33, nose information detection unit 34, and mouth information detection unit 35 are the same, and the acquisition period and the remote operation control period are the same, but it is not limited to this. When the acquisition period is shorter than the remote operation control period, the device control unit 12 generates control information using the information of one or more acquisition periods received within one remote operation control period. For example, the control information may be generated using the average value of the information of a plurality of acquisition periods, or the control information may be generated using the median value of the information of the acquisition periods. In this case, the first delay adjustment unit 17 may perform delay adjustment using, for example, the average value of the time information of the information corresponding to a plurality of acquisition periods. Also, when the acquisition period is longer than the remote operation control period, the device control unit 12 may generate control information using the most recent information, or may use an extrapolated value using the most recent information and the information before that.

[0083] Also, when there is a difference in the processing time for the device control unit 12 to generate each of the visual-related device drive unit 231, auditory-related device drive unit 232, tactile-related device drive unit 233, olfactory-related device drive unit 234, and gustatory-related device drive unit 235, the difference between the longest processing time among these processing times and each processing time is obtained as the processing delay amount, and the first delay adjustment unit 17 calculates the adjustment amount by adding the adjustment amount for the above-described reception delay and the processing delay amount. Further, when there is a difference in the time from when the device 2 receives the control information until the visual-related device drive unit 231, auditory-related device drive unit 232, tactile-related device drive unit 233, olfactory-related device drive unit 234, and gustatory-related device drive unit 235 each start operating, similarly, these differences may be considered as the adjustment amount.

[0084] In the example shown in FIG. 8, the first delay adjustment unit 17 is provided in the front stage of the device control unit 12. However, the present invention is not limited to this, and the first delay adjustment unit 17 may be provided in the rear stage of the device control unit 12. In this case, time information indicating the time when the motion information used for generating the control information is detected is added to each control information generated by the device control unit 12, and the first delay adjustment unit 17 controls based on the time information so that the control information corresponding to the motion information detected simultaneously is output simultaneously.

[0085] FIG. 9 is a diagram showing a configuration example related to information transmission control of the operating system according to Embodiment 2. As shown in FIG. 9, the information conversion unit 15a of the information processing apparatus 1a in the present embodiment is the same as the information conversion unit 15 in Embodiment 1 except that the second delay adjustment unit 81 is added.

[0086] The second delay adjustment unit 81 is an information transmission delay adjustment unit that adjusts the delay so that a plurality of composite data corresponding to the five-sense information and the extended five-sense information acquired at the same time are transmitted to the user simultaneously by a plurality of expression devices. Specifically, the second delay adjustment unit 81 acquires the five-sense information and the extended five-sense information via the detection information reception unit 14, calculates an adjustment amount of the delay so that the five-sense information and the five-sense data generated from the extended five-sense information detected at the same time as the five-sense information are combined, and instructs the visual data synthesis unit 61, the auditory data synthesis unit 62, the tactile data synthesis unit 63, the olfactory data synthesis unit 64, and the gustatory data synthesis unit 65 with the calculated adjustment amount as input information, respectively. Further, the second delay adjustment unit 81 calculates an adjustment amount (output adjustment amount) so that the expressions in the expression device 4 corresponding to the five-sense information and the extended five-sense information detected at the same time are executed simultaneously, and instructs the corresponding visual data filter unit 71, auditory data filter unit 72, tactile data filter unit 73, olfactory data filter unit 74, and gustatory data filter unit 75, respectively. The visual data filter unit 71, the auditory data filter unit 72, the tactile data filter unit 73, the olfactory data filter unit 74, and the gustatory data filter unit 75 control the output timing of the expression information so that the expression information generated from the extended five-sense information detected at the same time is output simultaneously based on the output adjustment amount, respectively.

[0087] Further, the second delay adjustment unit 81 may further consider, as an adjustment amount, the difference in processing time from when the visual data filter unit 71, the auditory data filter unit 72, the tactile data filter unit 73, the olfactory data filter unit 74, and the gustatory data filter unit 75 output expression information until the visual expression device 41, the auditory expression device 42, the tactile expression device 43, the olfactory expression device 44, and the gustatory expression device 45 in the expression device 4 execute the expression. Also, a delay adjustment unit may be provided between the detection information receiving unit 14 and the visual data conversion unit 51, the auditory data conversion unit 52, the tactile data conversion unit 53, the olfactory data conversion unit 54, and the gustatory data conversion unit 55 to perform delay adjustment of the input. That is, the delay adjustment unit obtains a delay adjustment amount (input adjustment amount) so that each conversion unit and each synthesis unit perform processing using the five-sense information and the extended five-sense information acquired at the same time, and instructs the delay adjustment amount to each conversion unit and each synthesis unit. Thereby, each conversion unit and each synthesis unit can perform processing using the information acquired at the same time. Note that by performing delay adjustment on either the input or the output, an effect of reducing the user's sense of discomfort can be obtained, but by performing delay adjustment on both the input and the output, delay adjustment can be performed more effectively. Also, regarding remote operation control, similarly, delay adjustment may be performed on either the input or the output or on both the input and the output.

[0088] Note that, for example, the detection information acquisition unit 24 of the device 2 has the same acquisition period for the information in the visual information detection unit 241, the extended visual information detection unit 242, the auditory information detection unit 243, the extended auditory information detection unit 244, the tactile information detection unit 245, the extended tactile information detection unit 246, the olfactory information detection unit 247, the extended olfactory information detection unit 248, the gustatory information detection unit 249, and the extended gustatory information detection unit 250, and the acquisition period and the information transmission control period are the same, but it is not limited thereto. When the acquisition period is shorter than the information transmission control period, each conversion unit and each synthesis unit perform conversion and synthesis respectively using the information of one or a plurality of acquisition periods received within one information transmission control period. For example, control information may be generated using the average value of the information of a plurality of acquisition periods, or conversion and synthesis may be performed using the median value of the information of the acquisition period. In this case, the second delay adjustment unit 81 may perform delay adjustment using, for example, the average value of the time information of the information corresponding to a plurality of acquisition periods. Also, when the acquisition period is longer than the information transmission control period, conversion and synthesis may be performed using the latest information, or a value extrapolated using the latest information and the information before that may be used.

[0089] Also, when a large delay occurs in a part of the information received by the information processing device 1, as described above, when delay adjustment is performed by the first delay adjustment unit 17 or the second delay adjustment unit 81, the delay of the entire operation system 5 becomes large. In particular, in wireless communication, the delay fluctuation is larger than that in wired communication, and it is assumed that the transmission delay of some information may become large when wireless communication is used. Therefore, information with low importance for realizing an intuitive operation may be excluded from the target of delay adjustment. In this case, for each piece of information, it may be determined in advance whether or not it is the target of delay adjustment, and information that is not the target of delay adjustment may be excluded regardless of the delay amount, or may be excluded when the delay (the difference between the time information and the reception time) of the information that is not the target of delay adjustment is a certain value. That is, the first delay adjustment unit 17 may exclude the control information corresponding to the drive unit with low importance among the plurality of device drive units from the target of delay adjustment, and the second delay adjustment unit 81 may exclude the synthesized data corresponding to the sense with low importance among the plurality of senses from the target of delay adjustment.

[0090] For example, since olfactory information and the like can be remotely operated without discomfort even with some delay, olfactory information may be excluded from the target of delay adjustment. Also, in disaster relief and the like, since taste information has a low importance level, taste information may be excluded from the target of delay adjustment. Further, in rescue activities at the disaster site and sports in the snow mountains as a leisure (travel experience), olfactory information and taste information also have a low importance level, and auditory information in remote cooking is considered to have a low importance level, so these may be excluded from the target of delay adjustment. Since the importance level of each piece of information depends on how the operation system 5 is used, it is not limited to the examples described above, and the information to be excluded from the target of delay adjustment may be determined according to the importance level in the application. Also, since the information emphasized by the user may vary depending on the user, the information to be excluded from the target of delay adjustment may be selectable according to the user's settings.

[0091] Note that, in the examples described above, an example in which the first delay adjustment unit 17 and the second delay adjustment unit 81 are added to the information processing apparatus 1 of the first embodiment has been described. However, only one of the first delay adjustment unit 17 and the second delay adjustment unit 81 may be added to the information processing apparatus 1 of the first embodiment, and the other may not be added.

[0092] Also, in the examples described above, an example in which the first delay adjustment unit 17 and the second delay adjustment unit 81 are added to the information processing apparatus 1 of the first embodiment has been described. However, the first delay adjustment unit 17 and the second delay adjustment unit 81 may be applied to an operation system that does not use extended five-sense information. For example, the first delay adjustment unit 17 and the second delay adjustment unit 81 may be applied to an information processing apparatus that does not include the visual data conversion unit 51, the auditory data conversion unit 52, the tactile data conversion unit 53, the olfactory data conversion unit 54, the taste data conversion unit 55, the visual data synthesis unit 61, the auditory data synthesis unit 62, the tactile data synthesis unit 63, the olfactory data synthesis unit 64, and the taste data synthesis unit 65 shown in FIG. 9. Further, this information processing apparatus may not include the visual data filter unit 71, the auditory data filter unit 72, the tactile data filter unit 73, the olfactory data filter unit 74, and the taste data filter unit 75.

[0093] The information processing apparatus 1a of this embodiment is realized by a computer system in the same manner as the information processing apparatus 1 of Embodiment 1.

[0094] As described above, in this embodiment, by adjusting the delay in remote operation control, it is possible to synchronize the operations of the device 2 based on the information detected at the same time. Thereby, it is possible to prevent a deviation in the operation of the device 2. Further, by adjusting the delay in information transmission control, it is possible to synchronize the effectiveness of the expression in the presentation device 4 based on the information detected at the same time. Thereby, the user can recognize the information detected at the same time without a sense of discomfort.

[0095] The configurations shown in the above embodiments are examples, and it is possible to combine them with other known technologies, combine the embodiments with each other, or omit or change a part of the configuration without departing from the gist.

Explanation of Reference Numerals

[0096] 1, 1a Information processing apparatus, 2 Device, 3 Motion information detection device, 4 Expression device, 5 Operating system, 11 Control information transmission unit, 12 Device control unit, 13 Motion information reception unit, 14 Detection information reception unit, 15, 15a Information conversion unit, 16 Expression information transmission unit, 17 First delay adjustment unit, 21 Control information reception unit, 22 Drive control unit, 23 Drive unit, 24 Detection information acquisition unit, 25 Detection information transmission unit, 31 Eye information detection unit, 32 Ear information detection unit, 33 Hand information detection unit, 34 Nose information detection unit, 35 Mouth information detection unit, 41 Visual expression device, 42 Auditory expression device, 43 Tactile expression device, 44 Olfactory expression device, 45 Gustatory expression device, 51 Visual data conversion unit, 52 Auditory data conversion unit, 53 Tactile data conversion unit, 54 Olfactory data conversion unit, 55 Gustatory data conversion unit, 61 Visual data synthesis unit, 62 Auditory data synthesis unit, 63 Tactile data synthesis unit, 64 Olfactory data synthesis unit, 65 Gustatory data synthesis unit, 71 Visual data filter unit, 72 Auditory data filter unit, 73 Tactile data filter unit, 74 Olfactory data filter unit, 75 Gustatory data filter unit, 81 Second delay adjustment unit, 231 Visual-related device drive unit, 232 Auditory-related device drive unit, 233 Tactile-related device drive unit, 234 Olfactory-related device drive unit, 235 Gustatory-related device drive unit, 241 Visual information detection unit, 242 Extended visual information detection unit, 243 Auditory information detection unit, 244 Extended auditory information detection unit, 245 Tactile information detection unit, 246 Extended tactile information detection unit, 247 Olfactory information detection unit, 248 Extended olfactory information detection unit, 249 Gustatory information detection unit, 250 Extended gustatory information detection unit.

Claims

1. A detection information receiving unit that receives at least one piece of five-sense information recognizable by at least one of the five senses of a person in the device, which is controlled according to an operation from a user existing at a location away from the device, and extended five-sense information that is information difficult to recognize by the five senses of a person in the device; A conversion unit that converts the extended five-sense information into conversion five-sense data indicating information that can be recognized by at least one of the five senses of a person; A synthesis unit that synthesizes the conversion five-sense data and the corresponding five-sense information; An expression information transmission unit that transmits the synthesized data, which is the result of synthesis by the synthesis unit, to an expression device that transmits the synthesized data to the user so that the user can recognize it by at least one of the five senses; A filter unit that filters the synthesized data; comprising; The expression information transmission unit is an information processing device characterized by transmitting the synthesized data after filtering to the expression device.

2. The information processing device according to claim 1, wherein the extended five-sense information is information related to sounds that are difficult for a person to recognize.

3. The information processing device according to claim 1, wherein the extended five-sense information is an image outside the range of a person's field of vision or a detection result of light having a wavelength other than visible light.

4. The information processing device according to claim 1, wherein the extended five-sense information is information indicating at least one of vibrations, pressures, airflows, and inclinations that are difficult for a person to detect.

5. The information processing device according to claim 1, wherein the extended five-sense information is information related to odors that are difficult for a person to recognize.

6. The information processing device according to claim 1, wherein the extended five-sense information is information related to tastes that are difficult for a person to recognize.

7. The information processing apparatus according to any one of claims 1 to 6, wherein the filtering includes a process of multiplying a gain according to the value of the synthetic data.

8. The information processing apparatus according to claim 7, wherein the gain is a value greater than 1 when the value of the synthetic data is equal to or less than a first value, and a value less than 1 when the value of the synthetic data exceeds a second value greater than the first value.

9. The synthetic data is a plurality of data respectively corresponding to a plurality of sensations among the five senses, The information processing apparatus, An information transmission delay adjustment unit that adjusts a delay so that a plurality of the synthetic data corresponding to the five-sense information and the extended five-sense information acquired at the same time are transmitted to the user simultaneously by a plurality of the expression devices, The information processing apparatus according to any one of claims 1 to 8, comprising:

10. The information processing apparatus according to claim 9, wherein the information transmission delay adjustment unit excludes the synthetic data corresponding to a sensation with a low importance among the plurality of sensations from being an object of delay adjustment.

11. A motion information receiving unit that acquires motion information indicating the position and motion of each part of the user, corresponding to a plurality of device driving units in the device, A device control unit that generates control information for controlling the operations of the plurality of device driving units in the device, using the motion information, A control information transmission unit that transmits the control information to the device, A remote control delay adjustment unit that adjusts a delay so that operations based on the control information generated using the motion information acquired at the same time are executed simultaneously in the plurality of device driving units, The information processing apparatus according to any one of claims 1 to 10, comprising:

12. The remote control delay adjustment unit excludes, from the target of delay adjustment, the control information corresponding to the device drive unit with a low importance among the plurality of device drive units, in the information processing apparatus according to claim 11.

13. A detection information receiving unit that receives at least one piece of five-sense information recognizable by at least one of the five senses of a person in the device controlled in response to an operation from a user existing at a location remote from the device, and extended five-sense information that is information difficult to recognize by the five senses of a person in the device, a conversion unit that converts the extended five-sense information into conversion five-sense data indicating information recognizable by at least one of the five senses of a person, a synthesis unit that synthesizes the conversion five-sense data and the corresponding five-sense information, a presentation information transmission unit that transmits, to a presentation device that transmits the synthesized data, which is the result of synthesis by the synthesis unit, to the user so that the user can recognize the synthesized data by at least one of the five senses, The synthesized data is a plurality of data respectively corresponding to a plurality of senses among the five senses, and an information transmission delay adjustment unit that adjusts a delay so that a plurality of the synthesized data corresponding to the five-sense information and the extended five-sense information acquired at the same time are transmitted to the user simultaneously by the plurality of presentation devices, An information processing apparatus characterized by comprising:

14. A detection information receiving unit that receives at least one piece of five-sense information recognizable by at least one of the five senses of a person in the device controlled in response to an operation from a user existing at a location remote from the device, and extended five-sense information that is information difficult to recognize by the five senses of a person in the device, a conversion unit that converts the extended five-sense information into conversion five-sense data indicating information recognizable by at least one of the five senses of a person, a synthesis unit that synthesizes the conversion five-sense data and the corresponding five-sense information, a presentation information transmission unit that transmits, to a presentation device that transmits the synthesized data, which is the result of synthesis by the synthesis unit, to the user so that the user can recognize the synthesized data by at least one of the five senses, A motion information receiving unit that acquires motion information indicating the positions and motions of respective parts of the user, corresponding to a plurality of device driving parts in the device; A device control unit that generates control information for controlling the operations of the plurality of device driving parts in the device, using the motion information; A control information transmitting unit that transmits the control information to the device; A remote control delay adjustment unit that adjusts a delay so that operations based on the control information generated using the motion information acquired at the same time are simultaneously executed in the plurality of device driving parts; An information processing apparatus, characterized by comprising the above.

15. A device; An information processing apparatus that transmits control information for controlling the device to the device in response to an operation from a user existing at a location away from the device; An expression device that transmits information recognizable by the user's five senses to the user; Comprising: The device includes: A detection information acquisition unit that detects at least one piece of five-sense information recognizable by at least one of the five senses of a person in the device and extended five-sense information that is information difficult to recognize by the five senses of a person in the device; The information processing apparatus includes: A detection information receiving unit that receives the five-sense information and the extended five-sense information from the device; A conversion unit that converts the extended five-sense information into conversion five-sense data indicating information recognizable by at least one of the five senses of a person; A synthesis unit that synthesizes the conversion five-sense data and the corresponding five-sense information; An expression information transmitting unit that transmits the synthesis data, which is the result of synthesis by the synthesis unit, to the expression device; A filter unit that filters the synthesis data; Comprising: The expression information transmitting unit transmits the filtered synthesis data to the expression device, which is a feature of the operating system.

16. A machine, an information processing apparatus that transmits control information for controlling the machine to the machine in response to an operation from a user existing at a location remote from the machine, a presentation device that transmits information recognizable by the user's five senses to the user, comprising: the machine includes: a detection information acquisition unit that detects at least one piece of five-sense information recognizable by at least one of the five senses of a person in the machine and extended five-sense information that is information difficult to recognize by the five senses of a person in the machine; the information processing apparatus includes: a detection information reception unit that receives the five-sense information and the extended five-sense information from the machine; a conversion unit that converts the extended five-sense information into conversion five-sense data indicating information recognizable by at least one of the five senses of a person; a synthesis unit that synthesizes the conversion five-sense data and the corresponding five-sense information; a presentation information transmission unit that transmits the synthesis data, which is the result of synthesis by the synthesis unit, to the presentation device; the synthesis data is a plurality of data respectively corresponding to a plurality of senses among the five senses, and an information transmission delay adjustment unit that adjusts a delay so that a plurality of the synthesis data corresponding to the five-sense information and the extended five-sense information acquired at the same time are transmitted to the user simultaneously by a plurality of the presentation devices; An operating system, characterized by comprising the above.

17. A machine, an information processing apparatus that transmits control information for controlling the machine to the machine in response to an operation from a user existing at a location remote from the machine, a presentation device that transmits information recognizable by the user's five senses to the user, comprising: the machine includes: A detection information acquisition unit that detects at least one piece of five-sense information recognizable by at least one of the five senses of a person in the device and extended five-sense information that is information difficult to recognize by the five senses of a person in the device. The information processing apparatus A detection information receiving unit that receives the five-sense information and the extended five-sense information from the device, A conversion unit that converts the extended five-sense information into conversion five-sense data indicating information recognizable by at least one of the five senses of a person, A synthesis unit that synthesizes the conversion five-sense data and the corresponding five-sense information, A presentation information transmission unit that transmits the synthesized data, which is the result of synthesis by the synthesis unit, to the presentation device, A motion information receiving unit that acquires motion information indicating the position and motion of each part of the user corresponding to a plurality of device driving parts in the device, A device control unit that generates control information for controlling the operations of the plurality of device driving parts in the device using the motion information, A control information transmission unit that transmits the control information to the device, A remote control delay adjustment unit that adjusts a delay so that operations based on the control information generated using the motion information acquired at the same time are simultaneously executed in the plurality of device driving parts. An operating system characterized by comprising the above.

18. An information processing method in an information processing apparatus, A step of receiving at least one piece of five-sense information recognizable by at least one of the five senses of a person in the device controlled in response to an operation from a user existing at a location away from the device and extended five-sense information that is information difficult to recognize by the five senses of a person in the device, A step of converting the extended five-sense information into conversion five-sense data indicating information recognizable by at least one of the five senses of a person, A step of generating synthesized data by synthesizing the conversion five-sense data and the corresponding five-sense information, The step of filtering the synthetic data; The step of transmitting the synthetic data after filtering to a presentation device that transmits the synthetic data to the user so that the user can recognize it with at least one of the five senses; An information processing method characterized by including the above.

19. An information processing method in an information processing apparatus, The step of receiving at least one piece of five-sense information recognizable by at least one of the five senses of a person in the device controlled in response to an operation from a user existing at a location away from the device, and extended five-sense information that is information difficult to recognize by the five senses of a person in the device; The step of converting the extended five-sense information into conversion five-sense data indicating information that can be recognized by at least one of the five senses of a person; The step of generating synthetic data by synthesizing the conversion five-sense data and the corresponding five-sense information; The step of transmitting the synthetic data to a presentation device that transmits the synthetic data to the user so that the user can recognize it with at least one of the five senses; The synthetic data is a plurality of data respectively corresponding to a plurality of sensations among the five senses, and the step of adjusting the delay so that a plurality of the synthetic data corresponding to the five-sense information and the extended five-sense information acquired at the same time are transmitted to the user simultaneously by a plurality of the presentation devices; An information processing method characterized by including the above.

20. The step of generating control information for controlling the device; The step of transmitting the control information to the device; The information processing method according to claim 18 or 19, characterized by including the above.

21. An information processing method in an information processing apparatus, Receiving at least one piece of five-sense information recognizable by at least one of the five senses of a person in the device controlled according to an operation from a user existing at a location remote from the device, and extended five-sense information that is information difficult to recognize by the five senses of a person in the device. Converting the extended five-sense information into conversion five-sense data indicating information that can be recognized by at least one of the five senses of a person. Generating synthesized data by synthesizing the conversion five-sense data and the corresponding five-sense information. Transmitting the synthesized data to a presentation device that transmits the synthesized data to the user so that the user can recognize it with at least one of the five senses. Obtaining movement information indicating the position and movement of each part of the user corresponding to a plurality of device driving parts in the device. Generating control information for controlling the operations of the plurality of device driving parts in the device using the movement information. Transmitting the control information to the device. Adjusting a delay so that operations based on the control information generated using the movement information obtained at the same time are executed simultaneously in the plurality of device driving parts. An information processing method characterized by including the above.

22. In a computer system, Receiving at least one piece of five-sense information recognizable by at least one of the five senses of a person in the device controlled according to an operation from a user existing at a location remote from the device, and extended five-sense information that is information difficult to recognize by the five senses of a person in the device. Converting the extended five-sense information into conversion five-sense data indicating information that can be recognized by at least one of the five senses of a person. Generating synthesized data by synthesizing the conversion five-sense data and the corresponding five-sense information. Filtering the synthesized data. Sending the synthesized data after filtering to a presentation device that transmits the synthesized data to the user so that the user can recognize it with at least one of the five senses. A program characterized by causing the above to be executed.

23. In a computer system, Receiving at least one piece of five-sense information recognizable by at least one of the five senses of a person in the device controlled in response to an operation from a user existing at a location remote from the device, and extended five-sense information that is information difficult to recognize with the five senses of a person in the device. Converting the extended five-sense information into conversion five-sense data indicating information recognizable by at least one of the five senses of a person. Generating synthesized data by synthesizing the conversion five-sense data and the corresponding five-sense information. Sending the synthesized data to a presentation device that transmits the synthesized data to the user so that the user can recognize it with at least one of the five senses. The synthesized data is a plurality of data respectively corresponding to a plurality of senses among the five senses, and adjusting the delay so that a plurality of the synthesized data corresponding to the five-sense information and the extended five-sense information acquired at the same time are transmitted to the user simultaneously by a plurality of the presentation devices. A program characterized by causing the above to be executed.

24. In a computer system, Receiving at least one piece of five-sense information recognizable by at least one of the five senses of a person in the device controlled in response to an operation from a user existing at a location remote from the device, and extended five-sense information that is information difficult to recognize with the five senses of a person in the device. Converting the extended five-sense information into conversion five-sense data indicating information recognizable by at least one of the five senses of a person. Generating synthesized data by synthesizing the conversion five-sense data and the corresponding five-sense information. Transmitting the synthetic data to a presentation device that transmits the synthetic data to the user so that the user can recognize it with at least one of the five senses; Obtaining movement information indicating the position and movement of each part of the user corresponding to a plurality of device drive units in the device; Generating control information for controlling the operations of the plurality of device drive units in the device using the movement information; Transmitting the control information to the device; Adjusting a delay so that operations based on the control information generated using the movement information obtained at the same time are executed simultaneously in the plurality of device drive units; A program characterized by causing the above to be executed.

Images