Information processing device, operating system, information processing method, and program

Through the synthesis and transformation technology of the information processing device, users can identify the five sense information that cannot be used through the five senses, solving the problem of insufficient information perception during long-distance operation in the prior art, and improving work efficiency and sense of presence.

CN118922801APending Publication Date: 2024-11-08MITSUBISHI ELECTRIC CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202280093972.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2022-03-28
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

In the prior art, users cannot identify information that cannot be identified by people through five senses when operating robots at a distance, resulting in low operational efficiency and poor sense of presence.

Method used

An information processing device is designed to detect user's movement information and the five sense information in the device, and synthesize and transform it into the five sense data of available people, and communicate it to the user, so that the user can identify the inability through vision, hearing, touch, smell and taste. Use the five senses of people.

Benefits of technology

It improves the user's perceived refinement of equipment and surrounding environment, reduces the sense of dissonance in remote operations, and improves the operation efficiency and operation accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118922801A_ABST
    Figure CN118922801A_ABST
Patent Text Reader

Abstract

An information processing device (1) according to the present disclosure is provided with: a detection information reception unit (14) that receives five-sense information that can be recognized by at least one of five senses of a person, and extended five-sense information that is information that is difficult to recognize by the five senses of the person; a visual data conversion unit (51), an auditory data conversion unit (52), a tactile data conversion unit (53), an olfactory data conversion unit (54), and a taste data conversion unit (55) that convert the expanded five-sense information into converted five-sense data indicating information recognizable by at least one of the five senses of a person; a visual data synthesis unit (61), an auditory data synthesis unit (62), a tactile data synthesis unit (63), an olfactory data synthesis unit (64), and a taste data synthesis unit (65) which synthesize the converted five-sense data and the corresponding five-sense information; and a performance information transmission unit (16) that transmits the synthesized data, which is the result of the synthesis, to a performance device that transmits the synthesized data to the user so as to be recognizable by the user.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to an information processing device in an operating system that controls a device according to a user's operation, an operating system including the information processing device, an information processing method, and a program. Background Art

[0002] It is known that an operating system that enables a user who is in a different place from a device such as a robot to operate the device. For example, in Patent Document 1, sensors such as a camera and a microphone are installed at the installation location of the robot, and information received from the sensor is reproduced in a manner that enables the user to visually and auditorily recognize the information, thereby conveying the information to the user. As a result, in the technology described in Patent Document 1, a user who is in a place far away from the robot can remotely operate the robot while understanding the surrounding conditions of the robot.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2021-49642 Summary of the invention

[0006] The technology described in Patent Document 1 reproduces information equivalent to that which can be recognized by vision and hearing when a human user is in the position of the robot. In order to improve work efficiency or enhance the sense of presence, it is sometimes not sufficient to only reproduce information that can be recognized by the five senses of a person.

[0007] The present disclosure has been made in view of the above, and an object of the present disclosure is to obtain an information processing device that enables a user to recognize information that cannot be recognized by the five senses of a person.

[0008] In order to solve the above-mentioned problems and achieve the purpose, the information processing device involved in the present disclosure includes: a detection information receiving unit that receives five sense information that can be recognized by at least one of the five senses of a person in a device that is controlled according to an operation from a user who is located in a place far away from the device and extended five sense information in the device that is information that is difficult to recognize by the five senses of a person; and a conversion unit that converts the extended five sense information into converted five sense data that represents information that can be recognized by at least one of the five senses of a person. The information processing device also includes: a synthesis unit that synthesizes the converted five sense data and the corresponding five sense information; and a presentation information sending unit that sends the synthesized data that is the result of synthesis by the synthesis unit to a presentation device, and the presentation device conveys the synthesized data to the user so that the user can recognize it with at least one of the five senses.

[0009] The information processing device according to the present disclosure has an effect of enabling a user to recognize information that cannot be recognized by the five senses of a person. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a diagram showing a configuration example of an operating system according to the first embodiment.

[0011] Figure 2 This is a diagram showing a detailed configuration example related to the remote operation process according to the first embodiment.

[0012] Figure 3 This is a flowchart showing an example of a remote operation processing procedure in the information processing device according to the first embodiment.

[0013] Figure 4 This is a diagram showing a detailed configuration example related to information transmission processing according to the first embodiment.

[0014] Figure 5 This is a diagram showing a configuration example of an information conversion unit in the information processing device according to the first embodiment.

[0015] Figure 6 This is a flowchart showing an example of an information transmission processing procedure in the information processing device according to the first embodiment.

[0016] Figure 7 This is a diagram showing a configuration example of a computer system that realizes the information processing device according to the first embodiment.

[0017] Figure 8 This is a diagram showing a configuration example related to remote operation control of an operating system according to the second embodiment.

[0018] Fig. 9 This is a diagram showing a configuration example related to information transmission control of an operating system according to the second embodiment. DETAILED DESCRIPTION

[0019] Hereinafter, an information processing device, an operating system, an information processing method, and a program according to the embodiments will be described in detail with reference to the accompanying drawings.

[0020] Implementation method 1.

[0021] Figure 1 1 is a diagram showing a configuration example of an operating system according to Embodiment 1. The operating system 5 of this embodiment includes an information processing device 1, a device 2, a motion information detection device 3, and a display device 4. The device 2 has a movable part and operates according to control information sent from the information processing device 1. The device 2 is, for example, a robot, a manipulator, a work machine, etc. Figure 1 In FIG. 1 , a humanoid robot is illustrated as an example of the device 2 , but the device 2 is not limited to a humanoid robot.

[0022] The operating system 5 of the present embodiment can be applied to, for example, rescue activities in the event of a disaster, operations in places where harmful substances to humans may exist, such as operations in places where it is difficult for humans to work, such as high places or narrow places, etc. By setting up the device 2 in these work places, the user can remotely operate the device from a place far away from the device 2, so that the user can work safely. In addition, the operating system 5 can be used for users who are in a place far away from the device 2 to operate the device 2, such as remote surgery, remote cooking, etc. In addition, it can be used for travel experience, nature experience, education using remote environments or objects, etc. The use of the operating system 5 of the present embodiment described above is an example and is not limited to the above example.

[0023] The motion information detection device 3 detects motion information as information related to the motion of a user who is in a place far away from the device 2, and sends the detected motion information to the information processing device 1. The motion information is, for example, information indicating the position, direction, and motion of the eyes, ears, nose, hands, and mouth of the user. The motion information is used to operate the device 2 according to the user's motion. In addition, it is not necessary to obtain motion information about all of the eyes, ears, nose, hands, and mouth, and unnecessary information may not be obtained depending on the structure of the device 2. For example, in the case where the device 2 is not provided with a mechanism for simulating the movement of the mouth, the motion information related to the mouth may not be detected. In addition, as described later, the motion information detection device 3 is composed of one or more devices.

[0024] The device 2 is controlled according to an operation from a user who is located at a place far from the device 2. The device 2 includes a control information receiving unit 21, a drive control unit 22, a drive unit 23, a detection information acquisition unit 24, and a detection information sending unit 25. The control information receiving unit 21 receives control information from the information processing device 1, and outputs the received control information to the drive control unit 22. The drive control unit 22 controls the drive unit 23 using the control information received from the control information receiving unit 21. The detection information acquisition unit 24 detects five sense information that is information that can be recognized by the five senses of a person, and outputs the detected five sense information to the detection information sending unit 25. The five sense information acquired by the detection information acquisition unit 24 is information that can be recognized by at least one of the five senses when the user is located at the location of the device 2. That is, the five sense information is information in the device 2 that can be recognized by at least one of the five senses of a person. The detection information acquisition unit 24 also detects extended five sense information that is information that is difficult for a person to recognize by the five senses in the device 2, and outputs the detected extended five sense information to the detection information sending unit 25. In addition, as described later, the expanded five senses information may also be information related to each of the five senses information, that is, information expanded from each of the five senses. The detection information sending unit 25 sends the five senses information and the expanded five senses information to the information processing device 1. The details of the drive control unit 22, the drive unit 23, and the detection information acquisition unit 24 will be described later.

[0025] Extended five senses information is, for example, information related to physical quantities that are the same or similar to information that people can identify through their five senses and is outside the range that people can identify with their five senses. Extended five senses information is information that indicates the detection results of light outside the visible wavelength range, information that indicates the detection results of sounds with frequencies outside the human audible range, information at a level smaller than the minimum value that people can identify, information at a level larger than the maximum value that people can identify (such as the sound of a damaged eardrum, such as strong light that damages the retina, etc.), information that people cannot detect separately (types of gas, etc.), information outside the human field of vision, etc. For example, with regard to vision, since visible light can be identified with the five senses, the detection results of visible light are information that can be identified with the five senses, but light other than visible light, such as infrared light, ultraviolet light, etc., is extended five senses information that is difficult to identify with the five senses. Extended five senses information can also be referred to as information above the five senses and information outside the five senses. Details about extended five senses information will be described later.

[0026] The presentation device 4 performs a presentation that can be identified by at least one of the five senses of the user based on the presentation information received from the information processing device 1, thereby prompting the user with the information detected by the detection information acquisition unit 24 of the device 2. That is, the presentation device 4 conveys the received presentation information to the user in a manner that enables the user to identify it with at least one of the five senses. As a result, the user can identify each information at the position of the device 2 through the five senses and act according to the information. The movement of the user is detected by the above-mentioned motion information detection device 3. In addition, the presentation device 4 can perform a presentation corresponding to at least one of the five senses of vision, hearing, touch, smell and taste. The details of the presentation device 4 will be described later.

[0027] The information processing device 1 generates control information for controlling the device 2 using the motion information received from the motion information detection device 3, and sends the generated control information to the device 2. In addition, the information processing device 1 transforms the extended five senses information received from the device 2 into five senses data that can be recognized by the five senses of a person, synthesizes the five senses information received from the device 2 and the five senses data obtained by the transformation, uses the synthesized five senses information, generates performance information for causing the performance device 4 to perform a performance that can be recognized by at least one of the five senses, and sends the generated performance information to the performance device 4.

[0028] The information processing device 1 of this embodiment converts the expanded five senses information into five senses data, combines it with the acquired five senses information, and presents it to the user, so that the user can recognize information that cannot be recognized by the five senses of humans.

[0029] The information processing device 1 includes a control information sending unit 11, a device control unit 12, a motion information receiving unit 13, a detection information receiving unit 14, an information conversion unit 15, and a performance information sending unit 16. The control information sending unit 11, the device control unit 12, and the motion information receiving unit 13 perform a remote operation process for controlling the device 2 to perform an action corresponding to the user's motion. The detection information receiving unit 14, the information conversion unit 15, and the performance information sending unit 16 perform an information communication process for communicating the five sense information and the extended five sense information acquired by the detection information acquisition unit 24 of the device 2 to the user. The remote operation process and the information communication process are described below.

[0030] Figure 2 : is a diagram showing a detailed structural example related to the remote operation processing of the present embodiment. The motion information detection device 3 includes an eye information detection unit 31, an ear information detection unit 32, a hand information detection unit 33, a nose information detection unit 34, and a mouth information detection unit 35. The eye information detection unit 31 detects the position, direction, and movement of the user's eyes. The ear information detection unit 32 detects the position, direction, and movement of the user's ears. The hand information detection unit 33 detects the position, direction, and movement of the user's hands and fingers, and detects the pressure of each part of the hands and fingers. The nose information detection unit 34 detects the position, direction, and movement of the user's nose. The mouth information detection unit 35 detects the position, direction, and movement of the user's mouth and tongue. The eye information detection unit 31, the ear information detection unit 32, the hand information detection unit 33, the nose information detection unit 34, and the mouth information detection unit 35 include, for example, a camera or other imaging device, and the hand information detection unit 33 includes a pressure sensor. In addition, in the detection of the orientation and position of the eyes, ears, nose and mouth in the eye information detection unit 31, the ear information detection unit 32, the nose information detection unit 34 and the mouth information detection unit 35, a camera device that detects the orientation and position of the user's body, the orientation or position of the user's face may be used. The eye information detection unit 31, the ear information detection unit 32, the hand information detection unit 33, the nose information detection unit 34 and the mouth information detection unit 35 may be a component worn by the user, or a camera that photographs the user from the outside. In addition, as the motion information detection device 3, a positioning sensor mounted on a device worn by the user may also be included.

[0031] In addition, if Figure 2As shown, the driver 23 of the device 2 includes a vision-related device driver 231, an hearing-related device driver 232, a touch-related device driver 233, an olfactory-related device driver 234, and a taste-related device driver 235 as an example of a plurality of device drivers. When the device 2 is a humanoid robot, for example, the vision-related device driver 231, the hearing-related device driver 232, the touch-related device driver 233, the olfactory-related device driver 234, and the taste-related device driver 235 are parts corresponding to the eyes, ears, hands, nose, and mouth of the robot, respectively.

[0032] Figure 3 1 is a flowchart showing an example of a remote operation processing procedure in the information processing device 1 of the present embodiment. The motion information receiving unit 13 of the information processing device 1 obtains motion information (information indicating the position, orientation, motion, etc. of the user's eyes, ears, hands, nose, and mouth) (step S1). Specifically, the motion information receiving unit 13 obtains motion information by receiving detection results from the eye information detection unit 31, the ear information detection unit 32, the hand information detection unit 33, the nose information detection unit 34, and the mouth information detection unit 35, respectively, and outputs the obtained motion information to the device control unit 12.

[0033] The device control unit 12 generates control information for reflecting the motion information to the motion of the device (step S2). Specifically, the device control unit 12 uses the motion information received from the eye information detection unit 31, the ear information detection unit 32, the hand information detection unit 33, the nose information detection unit 34, and the mouth information detection unit 35 via the motion information receiving unit 13 to generate control information for controlling the corresponding vision-related device driver 231, the hearing-related device driver 232, the touch-related device driver 233, the smell-related device driver 234, and the taste-related device driver 235, respectively. That is, the device control unit 12 generates control information corresponding to vision, hearing, touch, smell, and taste, respectively, in such a way that the vision-related device driver 231, the hearing-related device driver 232, the touch-related device driver 233, the smell-related device driver 234, and the taste-related device driver 235 can simulate the movements of the user's eyes, ears, hands, nose, and mouth, respectively. The device control unit 12 outputs the generated control information to the control information transmission unit 11 .

[0034] The control information sending unit 11 sends the control information (step S3). Specifically, the control information sending unit 11 sends the control information corresponding to the vision, hearing, touch, smell and taste respectively received from the device control unit 12 to the device 2. The information processing device 1 implements the above-described steps S1 to S3 every remote operation control cycle. The remote operation control cycle may be the same as or different from the acquisition cycle of the motion information.

[0035] In this way, the motion information receiving unit 13 receives the motion information from the motion information detection device 3, and outputs the received motion information to the device control unit 12. That is, the motion information receiving unit 13 obtains motion information representing the position and action of each part of the user, which corresponds to each of the multiple device drivers in the device. The device control unit 12 uses the motion information received from the motion information detection device 3 to generate control information for causing the device 2 to perform actions corresponding to the motion information, and outputs the generated control information to the control information sending unit 11. In detail, the device control unit 12 uses the motion information to generate control information for controlling the actions of the multiple device drivers in the device 2 respectively. The control information sending unit 11 sends the control information received from the device control unit 12 to the device 2.

[0036] The control information receiving unit 21 of the device 2 outputs the control information of vision, hearing, touch, smell and taste to the drive control unit 22. The drive control unit 22 uses the control information of vision, hearing, touch, smell and taste to control the corresponding vision-related device driver 231, hearing-related device driver 232, touch-related device driver 233, smell-related device driver 234 and taste-related device driver 235. Thus, the device 2 can simulate the actions of the user's eyes, ears, hands, nose and mouth. That is, the movements of the user's eyes, ears, hands, nose and mouth become the operation information indicating the operation content that determines the action of the device 2, and the remote operation of the device 2 is realized.

[0037] Furthermore, as described above, the device 2 is not limited to a humanoid robot, and the device 2 may not simulate all of the eyes, ears, hands, nose, and mouth, and may not obtain unnecessary motion information according to the structure of the device 2. In addition, an example in which the user's motion itself becomes the operation content is described here, but it is not limited to this. For example, an operation unit such as a joystick, a touch pad, or a button for operating the movement of the device 2 may be separately provided, and the user may remotely operate the device 2 by operating the operation unit.

[0038] Figure 4 1 is a diagram showing a detailed configuration example related to information transmission processing according to this embodiment. Figure 4 As shown, the detection information acquisition unit 24 of the device 2 includes a visual information detection unit 241, an extended visual information detection unit 242, an auditory information detection unit 243, an extended auditory information detection unit 244, a tactile information detection unit 245, an extended tactile information detection unit 246, an olfactory information detection unit 247, an extended olfactory information detection unit 248, a taste information detection unit 249 and an extended taste information detection unit 250.

[0039] The visual information detection unit 241 detects visual information that is information that can be recognized by a person through vision. As the visual information detection unit 241, for example, a visual camera or the like can be used. The extended visual information detection unit 242 detects extended visual information that is information related to vision and is difficult for a person to recognize through the five senses. For example, extended visual information is an image outside the range of a person's field of vision, or a detection result of light of a wavelength other than visible light. The extended visual information detection unit 242 can be, for example, an infrared camera, a temperature sensor, an ultraviolet camera, etc. that obtains light of a wavelength different from visible light as an image, etc., or a rear camera, a high-sensitivity camera, etc. that captures images in the rear direction, a direction far away, and a direction that cannot be visually recognized by a person.

[0040] The auditory information detection unit 243 detects auditory information that is information that a person can recognize through hearing. As the auditory information detection unit 243, for example, a microphone (microphone) or the like can be used. The extended auditory information detection unit 244 detects extended auditory information that is information related to hearing and is difficult for a person to recognize through the five senses. For example, the extended auditory information is information related to sounds that are difficult for a person to recognize. The extended auditory information detection unit 244 detects sounds outside the frequency range of the human audible range and sounds of small amplitude that cannot be detected by a person. For example, a high-sensitivity microphone, a wide-band microphone, etc.

[0041] The tactile information detection unit 245 detects tactile information that is information that can be recognized by a person through the sense of touch. As the tactile information detection unit 245, for example, a pressure sensor or the like can be used. The extended tactile information detection unit 246 detects extended tactile information that is information related to the sense of touch and is difficult for a person to recognize through the five senses. The extended tactile information is, for example, information indicating at least one of vibration, pressure, air flow, and tilt that are difficult for a person to detect. The extended tactile information detection unit 246 detects, for example, tiny vibrations that cannot be recognized by a person, tiny changes in air pressure (airflow), etc. The extended tactile information detection unit 246 is, for example, a high-sensitivity vibration sensor, etc.

[0042] The olfactory information detection unit 247 detects olfactory information that is information that a person can recognize through the sense of smell. As the olfactory information detection unit 247, for example, an odor sensor or the like can be used. The extended olfactory information detection unit 248 detects extended olfactory information that is information related to the sense of smell and is difficult for a person to recognize through the five senses. For example, the extended olfactory information is information related to odors that are difficult for a person to recognize. As the extended olfactory information detection unit 248, a highly sensitive odor sensor, a gas sensor, or the like can be used.

[0043] The taste information detection unit 249 detects taste information that is information that can be recognized by a person through the sense of taste. As the taste information detection unit 249, for example, a taste sensor or the like can be used. The extended taste information detection unit 250 detects extended taste information that is information related to taste and is difficult for a person to recognize through the five senses. For example, the extended taste information is information related to a taste that is difficult for a person to recognize. As the extended taste information detection unit 250, a high-sensitivity taste sensor or the like can be used.

[0044] In addition, Figure 4 In the example shown, the detection information acquisition unit 24 includes a visual information detection unit 241, an extended visual information detection unit 242, an auditory information detection unit 243, an extended auditory information detection unit 244, a tactile information detection unit 245, an extended tactile information detection unit 246, an olfactory information detection unit 247, an extended olfactory information detection unit 248, a taste information detection unit 249, and an extended taste information detection unit 250, but it is not necessary to include all of them depending on the purpose. The detection information acquisition unit 24 may include at least one of the visual information detection unit 241, the auditory information detection unit 243, the tactile information detection unit 245, the olfactory information detection unit 247, and the taste information detection unit 249, and may include at least one of the extended visual information detection unit 242, the extended auditory information detection unit 244, the extended tactile information detection unit 246, the extended olfactory information detection unit 248, and the extended taste information detection unit 250. That is, the detection information acquisition unit 24 detects at least one of the five sense information of the visual information, auditory information, tactile information, olfactory information, and taste information, and detects at least one of the extended five sense information of the extended visual information, extended auditory information, extended tactile information, extended olfactory information, and extended taste information. In other words, the detection information acquisition unit 24 includes a five sense information acquisition unit for acquiring five sense information and an extended five sense information acquisition unit for acquiring extended five sense information. Figure 4 In the example shown, the five sense information acquisition unit includes a visual information detection unit 241, an auditory information detection unit 243, a tactile information detection unit 245, an olfactory information detection unit 247 and a taste information detection unit 249, and the extended five sense information acquisition unit includes an extended visual information detection unit 242, an extended auditory information detection unit 244, an extended tactile information detection unit 246, an extended olfactory information detection unit 248 and an extended taste information detection unit 250.

[0045] In addition, if Figure 4 As shown, the representation device 4 includes a visual representation device 41 , an auditory representation device 42 , a tactile representation device 43 , an olfactory representation device 44 , and a taste representation device 45 .

[0046] The visual display device 41 is a display device such as a display capable of displaying images, such as a three-dimensional display, VR (Virtual Reality) goggles, a VR head-mounted display, and a glasses-type display, but is not limited to these. The auditory display device 42 may also be a device that emits sound, such as a speaker, a 360-degree speaker, etc. The tactile display device 43 is a device that generates pressure, such as a tactile glove, a force glove, etc. The olfactory display device 44 is a device that generates an odor, such as an odor generating device. The taste display device 45 is a device that generates a taste, such as a taste generating device worn in the mouth, jaw, hand, etc.

[0047] In addition, Figure 4 In the example shown, the display device 4 includes a visual display device 41, an auditory display device 42, a tactile display device 43, an olfactory display device 44, and a taste display device 45, but it is not necessary to include all of them depending on the application. The display device 4 only needs to include at least one of the visual display device 41, the auditory display device 42, the tactile display device 43, the olfactory display device 44, and the taste display device 45.

[0048] Figure 5 1 is a diagram showing a configuration example of the information conversion unit 15 in the information processing device 1 of the present embodiment. Figure 5 As shown, it includes a visual data conversion unit 51, an auditory data conversion unit 52, a tactile data conversion unit 53, an olfactory data conversion unit 54, a taste data conversion unit 55, a visual data synthesis unit 61, an auditory data synthesis unit 62, a tactile data synthesis unit 63, an olfactory data synthesis unit 64, a taste data synthesis unit 65, a visual data filter unit 71, an auditory data filter unit 72, a tactile data filter unit 73, an olfactory data filter unit 74 and a taste data filter unit 75.

[0049] The visual data conversion unit 51, the auditory data conversion unit 52, the tactile data conversion unit 53, the olfactory data conversion unit 54, and the taste data conversion unit 55 are each an example of a conversion unit that converts the expanded five senses information into converted five senses data (five senses data) that represents information that can be recognized by at least one of the five senses of a person. The visual data synthesis unit 61, the auditory data synthesis unit 62, the tactile data synthesis unit 63, the olfactory data synthesis unit 64, and the taste data synthesis unit 65 are each an example of a synthesis unit that synthesizes the five senses data and the corresponding five senses information. The visual data filter unit 71, the auditory data filter unit 72, the tactile data filter unit 73, the olfactory data filter unit 74, and the taste data filter unit 75 are each an example of a filter unit that filters the synthesized data.

[0050] In addition, Figure 5An example of performing processing corresponding to all five senses is shown in FIG. 1 , but depending on the structure of the detection information acquisition unit 24 and the structure of the display device 4, it is also possible not to set Figure 5 The information conversion unit 15 shown is a part of each unit. For example, when the taste representation device 45 is not provided, the taste data conversion unit 55, the taste data synthesis unit 65 and the taste data filtering unit 75 may not be provided.

[0051] Figure 6 1 is a flowchart showing an example of the information transmission processing procedure in the information processing device 1 of this embodiment. Figure 6 As shown, the information processing device 1 obtains five sense information (step S11). In detail, the detection information receiving unit 14 receives each of the visual information, auditory information, tactile information, olfactory information and taste information that can be recognized by a person through the five senses from the device 2, and outputs the received visual information, auditory information, tactile information, olfactory information and taste information to the information conversion unit 15. In addition, the information processing device 1 obtains extended five sense information (step S12). In detail, the detection information receiving unit 14 receives each of the extended visual information, auditory information, tactile information, olfactory information and taste information that are difficult for a person to recognize through the five senses from the device 2, and outputs the received extended visual information, auditory information, tactile information, olfactory information and taste information to the information conversion unit 15. In addition, the order of step S11 and step S12 is not limited to. Figure 6 , or it can be in the reverse order, or it can be done simultaneously.

[0052] Next, the information conversion unit 15 converts the expanded five senses information into five senses data (step S13). The five senses data is at least one of visual data, auditory data, tactile data, olfactory data, and taste data that can be recognized by a person through the five senses.

[0053] Specifically, the visual data conversion unit 51 converts at least one of the extended visual information, the extended auditory information, the extended tactile information, the extended olfactory information, and the extended taste information into visual data that can be visually recognized by a person, and outputs the converted visual data to the visual data synthesis unit 61. The auditory data conversion unit 52 converts at least one of the extended visual information, the extended auditory information, the extended tactile information, the extended olfactory information, and the extended taste information into auditory data that can be auditory recognized by a person, and outputs the converted auditory data to the auditory data synthesis unit 62. The tactile data conversion unit 53 converts at least one of the extended visual information, the extended auditory information, the extended tactile information, the extended olfactory information, and the extended taste information into tactile data that can be tactilely recognized by a person, and outputs the converted tactile data to the tactile data synthesis unit 63. The olfactory data conversion unit 54 converts at least one of the extended visual information, the extended auditory information, the extended tactile information, the extended olfactory information, and the extended taste information into olfactory data that can be recognized by a human through the sense of smell, and outputs the converted olfactory data to the olfactory data synthesis unit 64. The taste data conversion unit 55 converts at least one of the extended visual information, the extended auditory information, the extended tactile information, the extended olfactory information, and the extended taste information into taste data that can be recognized by a human through the sense of taste, and outputs the converted taste data to the taste data synthesis unit 65.

[0054] Regarding the conversion of the expanded five senses information into the five senses data, for example, the visual data conversion unit 51 may convert the expanded visual information into the same kind of five senses data as in the case of converting the expanded visual information into visual data, or may convert the expanded taste information into the data of different five senses as in the case of converting the expanded taste information into visual data. In addition, the visual data conversion unit 51 may generate the five senses data using a plurality of information as in the case of converting a plurality of information among the expanded visual information, the expanded auditory information, the expanded tactile information, the expanded olfactory information, and the expanded taste information into visual data. In addition, one expanded five senses information may be converted into a plurality of types of five senses data as in the case of converting the expanded auditory information into visual data and auditory data.

[0055] Next, the information conversion unit 15 performs synthesis processing (step S14). Specifically, the visual data synthesis unit 61 synthesizes the visual data obtained by the transformation and the visual information obtained in step S11, and outputs the synthesized information (synthesized data) to the visual data filtering unit 71. The auditory data synthesis unit 62 synthesizes the auditory data obtained by the transformation and the auditory information obtained in step S11, and outputs the synthesized information to the auditory data filtering unit 72. The tactile data synthesis unit 63 synthesizes the tactile data obtained by the transformation and the tactile information obtained in step S11, and outputs the synthesized information to the tactile data filtering unit 73. The olfactory data synthesis unit 64 synthesizes the olfactory data obtained by the transformation and the olfactory information obtained in step S11, and outputs the synthesized information to the olfactory data filtering unit 74. The taste data synthesis unit 65 synthesizes the taste data obtained by the transformation and the taste information obtained in step S11, and outputs the synthesized information to the taste data filtering unit 75.

[0056] Next, the information conversion unit 15 filters the synthesized information (step S15). Specifically, the visual data filter unit 71, the auditory data filter unit 72, the tactile data filter unit 73, the olfactory data filter unit 74, and the taste data filter unit 75 perform filtering corresponding to the value of the synthesized data (synthesized information) synthesized by the corresponding synthesizer. For example, if the value of the synthesized information is too large, that is, the intensity of the information displayed to the user is too strong, it may have a bad effect on the user, but filtering can prevent the bad effect on the user.

[0057] As filtering, for example, when the input value as the value of the synthesized information exceeds a predetermined certain value, the input value is converted into a small value and output, and when the input value exceeds a certain value, a predetermined value (including 0) below the certain value is output. In addition, as filtering, when the input value is small, a process of outputting a value larger than the input value by multiplying the input value by a value larger than 1 can also be performed. For example, filtering can also include a process of multiplying the input value, i.e., the value of the synthesized data. In this case, for example, when the input value is less than the first value, the gain can be a value larger than 1, and when the input value exceeds the second value larger than the first value, the gain can be a value smaller than 1. That is, for example, as filtering, the following process can also be performed, that is, when the input value is less than the first value, the input value is multiplied by G1 (G1>1), when the input value is greater than the first value and less than the second value (second value>first value), the input value is output as it is, and when the input value is greater than the second value, the input value is multiplied by G2 (G2<1).

[0058] In addition, the range of input values ​​may be classified into four or more ranges, and the multiplication value (gain) may be determined for each range. In addition, the output value may be formulated as a nonlinear function of the input value, and the function value corresponding to the function may be output as a filter. The specific content of the filter is not limited to this example.

[0059] Next, the information processing device 1 sends the filtered information to the presentation device 4 (step S16). Specifically, the visual data filter 71, the auditory data filter 72, the tactile data filter 73, the olfactory data filter 74, and the taste data filter 75 output the filtered information as presentation information (visual presentation information, auditory presentation information, tactile presentation information, olfactory presentation information, and taste presentation information) to the presentation information sending unit 16. The presentation information sending unit 16 sends the visual presentation information, the auditory presentation information, the tactile presentation information, the olfactory presentation information, and the taste presentation information to the corresponding visual presentation device 41, the auditory presentation device 42, the tactile presentation device 43, the olfactory presentation device 44, and the taste presentation device 45, respectively.

[0060] The information processing device 1 performs the above-described information communication processing every information communication control cycle. The information communication control cycle may be the same as or different from the acquisition cycle of the five senses information and the extended five senses information. In addition, the information communication control cycle may be the same as or different from the remote operation control cycle.

[0061] The visual representation device 41, the auditory representation device 42, the tactile representation device 43, the olfactory representation device 44 and the taste representation device 45 respectively perform representations based on the visual representation information, the auditory representation information, the tactile representation information, the olfactory representation information and the taste representation information, thereby enabling the user to recognize these information as the five senses. As described above, in the present embodiment, the five sense information that can be recognized by humans is further synthesized into extended five sense information that cannot be recognized by humans, so compared with the case where only the five sense information is transmitted, the user can grasp the state of the device 2 and the surrounding conditions in more detail. As a result, the user's sense of disharmony in remote operation can be reduced, the user can operate intuitively, and the work efficiency can be improved. In addition, by grasping the state of the device 2 and the surrounding conditions in more detail, the user can achieve more appropriate operations and improve work efficiency. In addition, by filtering, remote operation can be performed safely without endangering the user.

[0062] The following is a specific example of the expanded five senses information and five senses data. In addition, the contents of the expanded five senses information and five senses data are not limited to the following examples.

[0063] Example 1: As extended auditory information, tiny sounds that are inaudible to humans, such as gas leakage sounds, are detected, and the extended auditory information is converted into visual data so that the user can recognize gas leakage. For example, as visual data, it can be represented by text indicating a warning such as "gas leakage has occurred", or it can be represented by an image or video. In addition, the specific text displayed in the warning is not limited to this example. In the case of representing gas leakage by an image or video, for example, the direction of the sound of the gas (the source of the gas) can also be represented by color, arrows, etc. In addition, according to the type of tiny sound, experiments can be conducted in advance, and machine learning can be used to learn under what kind of visual data a person can visualize the phenomenon corresponding to the sound, and the information conversion unit 15 uses the learned model to determine the visual data. In addition, the corresponding transformation can also be determined based on rules that are not machine learning.

[0064] Example 2: As extended auditory information, a tiny sound that is inaudible to humans, such as the cry for help from a person buried in rubble, is detected and converted into auditory data corresponding to a loud sound that is recognizable to humans, thereby enabling the user to recognize the tiny sound.

[0065] Example 3: As extended olfactory information, minute odors that are not recognizable to humans, such as almost odorless toxic gases, are detected and converted into auditory data of warning sounds that are recognizable to humans, thereby enabling users to recognize odorless toxic gases.

[0066] Example 4: As extended olfactory information, tiny odors that are not recognizable to humans, such as almost odorless toxic gases, are detected and the extended olfactory information is converted into visual data that humans can recognize, such as concentration distribution maps, so that users can infer the source of the odorless toxic gases.

[0067] Example 5: As extended tactile information or extended visual information, a temperature that is not recognizable to humans (ultra-high temperature) is detected, and the ultra-high temperature part in the extended tactile information or extended visual information is turned red to transform it into visual data that can be recognized by humans, so that the user can recognize the ultra-high temperature part.

[0068] Example 6: As extended visual information, an image in a direction that is not recognizable to humans, such as an image behind, is obtained, and the extended visual information is converted into visual data that can be recognized by humans, thereby allowing the user to recognize the state behind.

[0069] Example 7: As extended visual information, an image of a dark environment that cannot be recognized by humans is obtained, and the extended visual information is converted into visual data that can be recognized by humans with increased sensitivity, so that the user can recognize the state in the dark environment.

[0070] Example 8: As extended tactile information, the occurrence of tiny forces such as tiny vibrations that are imperceptible to humans (tiny vibrations of the ground and buildings (air) caused by explosives and aging), tiny air pressure changes (flow of air), etc. are detected. When the occurrence of tiny forces is detected in the extended tactile information, the extended tactile information is converted into auditory data such as a warning sound, thereby enabling the user to recognize the occurrence of tiny forces.

[0071] Example 9: As extended taste information, a delicate taste that cannot be recognized by humans is detected, and the extended taste information is converted into auditory data such as sound representing components constituting the taste, thereby enabling the user to recognize the delicate taste.

[0072] Example 10: As extended tactile information, a slight tilt that is not perceptible to humans due to displacement from the direction of gravity or the horizontal direction is detected, and the extended tactile information is converted into visual data as a numerical value representing the tilt, so that the user can recognize the tilt. For example, when a robot as device 2 is used for rescue activities, when the state of the feet is poor, there is a possibility that the operation will be hindered due to slipping or burial. Therefore, a visual camera, an infrared camera, an acoustic wave sensor, a LiDAR (Light Detection And Ranging), a pressure-sensitive sensor, a gyroscope, etc. may be provided on the instep of device 2 as an extended tactile information detection unit 246 or an extended visual information detection unit 242, so as to detect the state of objects around device 2 (easy to fall, fragile, slippery) or the presence of humans. As a result, when a person is under rubble near the feet of device 2, it is possible to prevent the person from stepping on the part.

[0073] In addition, the information processing device 1 may determine which five sense data the extended five sense information is to be converted into based on the value and characteristics (time change, frequency, etc.) of the extended five sense information by using machine learning or based on rules. For example, based on the characteristics of the tiny sound represented by the extended five sense information, the information processing device 1 may learn in advance what the sound is by machine learning, and generate a learned model. The information processing device 1 obtains an inference result of what the sound is by inputting the extended five sense information into the learned model. The correspondence between the type of sound (what sound it is) and which five sense data (visual data, auditory data, tactile data, olfactory data, taste data) it is to be converted into is predetermined based on the rule, and the information processing device 1 determines the type of five sense data to be converted based on the inferred type of sound. Alternatively, the correspondence between the type of sound and the five sense data to be converted may also be determined by machine learning. For example, for each type of sound, using the results of experiments, etc., it is also possible to learn by machine learning which five sense data (visual data, auditory data, tactile data, olfactory data, taste data) is the five sense data that humans can easily recognize as corresponding to the sound. Alternatively, the correspondence between the values ​​and characteristics of the extended five senses information and the types of five senses data to be converted may be determined directly based on the rules. Alternatively, a learned model may be created for obtaining the five senses data to be converted based on the values ​​and characteristics of the extended five senses information.

[0074] Next, the effects of filtering are illustrated. In addition, the content of filtering is not limited to the following examples.

[0075] Example 1 of filtering (tactile sensation): When an explosion occurs during disaster relief and causes a strong impact on the device 2, filtering is used to suppress the intensity or cut off the intensity, thereby preventing the user from being directly impacted.

[0076] Example 2 of filtering (hearing): When an explosion occurs during disaster relief and a loud explosion sound is detected, filtering is used to suppress or cut off the intensity, thereby preventing the loud explosion sound, such as a ruptured eardrum, from being transmitted to the user.

[0077] Example 3 of filtering (vision): When high-brightness light reaches device 2 due to an explosion during disaster relief, filtering is used to suppress or cut off the intensity of the light, thereby preventing strong light that has a negative impact on vision from reaching the user.

[0078] Example 4 of filtering (smell): When a strong odor due to gas leakage or the like reaches the device 2 during disaster relief, filtering is used to suppress or cut off the odor, thereby preventing the odor that damages the sense of smell from reaching the user.

[0079] Example 5 of filtering (taste): When the robot as device 2 tastes the food during cooking and finds that the taste is detrimental to the taste, filtering is used to suppress the intensity of the taste or cut off the intensity of the taste, thereby preventing the taste that is detrimental to the taste from being transmitted to the user.

[0080] In addition, Figure 1 In the figure, the communication between the device 2 and the information processing device 1, between the information processing device 1 and the motion information detection device 3, and between the information processing device 1 and the presentation device 4 is shown as an example of wireless communication, but it is not limited to this. The communication between the devices in this embodiment may also be wired communication, and both wired communication and wireless communication may be used. In addition, in the communication between the devices in this embodiment, 5G (5th Generation: 5th generation mobile communication system) or Beyond 5G may also be used. By using a low-latency communication method such as Beyond 5G, information can be conveyed to the user without disharmony. For example, by using a communication method with a short delay time compared to the time resolution of changes in information that can be recognized by humans (for example, 10 to 30 msec), the user's sense of disharmony can be reduced and the device 2 can be operated efficiently. In addition, by using a communication method with a short delay time, the sense of presence can be improved.

[0081] Next, the hardware configuration of the information processing device 1 of this embodiment will be described. The information processing device 1 of this embodiment executes a program as a computer program describing the processes in the information processing device 1 on the computer system, and the computer system functions as the information processing device 1 . Figure 7 1 is a diagram showing a configuration example of a computer system for realizing the information processing device 1 according to the present embodiment. Figure 7 As shown, the computer system includes a control unit 101, an input unit 102, a storage unit 103, a display unit 104, a communication unit 105, and an output unit 106, which are connected via a system bus 107. The control unit 101 and the storage unit 103 constitute a processing circuit.

[0082] exist Figure 7In the embodiment, the control unit 101 is a processor such as a CPU (Central Processing Unit), which executes a program describing the processing in the information processing device 1 of this embodiment. In addition, a part of the control unit 101 can also be implemented using dedicated hardware such as a GPU (Graphics Processing Unit) and an FPGA (Field-Programmable Gate Array). The input unit 102 is composed of, for example, a keyboard, a mouse, etc., and is used for the user of the computer system to input various information. The storage unit 103 includes various memories such as RAM (Random Access Memory), ROM (Read Only Memory), and storage devices such as hard disks, which store the program to be executed by the above-mentioned control unit 101, the necessary data obtained during the processing, etc. In addition, the storage unit 103 is also used as a temporary storage area for the program. The display unit 104 is composed of a display, an LCD (liquid crystal display panel), etc., and displays various screens for the user of the computer system. The communication unit 105 is a receiver and a transmitter that implements communication processing. The output unit 106 is a printer, a speaker, etc. In addition, Figure 7 is an example, the structure of the computer system is not limited to Figure 7 Example.

[0083] Here, an example of the operation of the computer system until the program of the present embodiment is executable is described. In the computer system adopting the above structure, for example, the computer program is installed from a CD-ROM or DVD-ROM set in a CD (Compact Disc)-ROM drive or DVD (Digital Versatile Disc)-ROM drive (not shown) to the storage unit 103. And when the program is executed, the program read from the storage unit 103 is stored in the main storage area of ​​the storage unit 103. In this state, the control unit 101 executes the processing of the information processing device 1 of the present embodiment according to the program stored in the storage unit 103.

[0084] In addition, in the above description, a CD-ROM or DVD-ROM is used as a recording medium to provide a program that records the processing in the information processing device 1, but this is not limited to this. Depending on the structure of the computer system, the capacity of the program provided, etc., for example, a program provided via a transmission medium such as the Internet via the communication unit 105 can also be used.

[0085] The program of this embodiment, for example, causes the computer system to execute: the step of receiving five sense information and extended five sense information; the step of transforming the extended five sense information into transformed five sense data representing information that can be identified by at least one of the five senses of a person; the step of generating synthesized data by synthesizing the transformed five sense data and the corresponding five sense information; and the step of sending the synthesized data to the presentation device 4, which conveys the synthesized data to the user so that the user can identify it with at least one of the five senses.

[0086] Figure 1 The device control unit 12 and the information conversion unit 15 shown in the figure are operated by using Figure 7 The control unit 101 shown executes the Figure 7 The computer program of the storage unit 103 shown in FIG. Figure 1 In the implementation of the device control unit 12 and the information conversion unit 15 shown in the figure, the Figure 7 The storage unit 103 is shown. Figure 1 The control information transmitting unit 11, the motion information receiving unit 13, the detection information receiving unit 14 and the performance information transmitting unit 16 are shown using Figure 7 The information processing device 1 may be implemented by the communication unit 105 shown. In addition, the information processing device 1 may also be implemented by using a plurality of computer systems. For example, the information processing device 1 may also be implemented by using a cloud computer system.

[0087] In addition, Figure 1 In the example shown, an example is described in which the information processing device 1 performs both remote operation control and information transmission control, but a remote operation device that performs remote operation control and an information transmission device that performs information transmission control may be independently provided. In this case, the remote operation device includes a control information transmission unit 11, a device control unit 12, and a motion information receiving unit 13, and the information transmission device includes a detection information receiving unit 14, an information conversion unit 15, and a performance information transmission unit 16.

[0088] As described above, in this embodiment, the information processing device 1 in the operating system 5 operated by the user from a place far away from the device 2 obtains not only the five sense information that can be recognized by humans, but also the extended five sense information that is difficult for humans to recognize, converts the extended five sense information into five sense data that can be recognized by humans, and synthesizes the five sense data with the five sense information. In addition, the information processing device 1 transmits the synthesized information to the presentation device 4 that is transmitted to the user as information of the five senses, thereby transmitting the synthesized information to the user. Therefore, the user can recognize information that cannot be recognized by the five senses of humans.

[0089] Implementation method 2.

[0090] Figure 81 is a diagram showing a configuration example of an operating system according to Embodiment 2 related to remote operation control. The operating system according to this embodiment is the same as the operating system 5 according to Embodiment 1, except that the information processing device 1 is replaced with the information processing device 1a. The same reference numerals as those in Embodiment 1 are attached to the components having the same functions as those in Embodiment 1, and repeated descriptions are omitted. The following mainly describes the differences from Embodiment 1.

[0091] like Figure 8 As shown, the information processing device 1a is the same as the information processing device 1 of the first embodiment except that a first delay adjustment unit 17 is added and an information conversion unit 15a is provided instead of the information conversion unit 15. The first delay adjustment unit 17 is a remote control delay adjustment unit that adjusts the delay so that the action based on the control information generated using the motion information obtained at the same time is simultaneously executed in multiple device drivers. In detail, the first delay adjustment unit 17 obtains motion information from the eye information detection unit 31, the ear information detection unit 32, the hand information detection unit 33, the nose information detection unit 34, and the mouth information detection unit 35 via the motion information receiving unit 13, respectively, so that the timing of each device driver in the device 2 corresponding to the motion information detected at the same time is driven at the same time, calculates the adjustment amount of the timing, and indicates the calculated adjustment amount to the device control unit 12.

[0092] The device control unit 12 adjusts the timing of outputting the control information corresponding to the vision-related device driver 231, the hearing-related device driver 232, the touch-related device driver 233, the smell-related device driver 234, and the taste-related device driver 235, respectively, according to the adjustment amount. Thus, the first delay adjustment unit 17 can adjust the delay of each driver corresponding to each motion information detected at the same time to achieve synchronization. In addition, when the first delay adjustment unit 17 uses a plurality of motion information in generating the control information, the plurality of motion information corresponding to the same time is synchronized and output to the device control unit 12.

[0093] For example, it is assumed that the motion information in the eye information detection unit 31, the ear information detection unit 32, the hand information detection unit 33, the nose information detection unit 34, and the mouth information detection unit 35 is added with time information (time stamp) indicating the time of detection. The first delay adjustment unit 17 uses the time information of each motion information to determine the motion information corresponding to the same time. In addition, here, the case where the difference in time is less than the threshold is set as the same time. The first delay adjustment unit 17 uses the latest time of the time when each motion information corresponding to the same time is received as the reference time, and calculates the difference between the time when the other motion information is received and the reference time as the adjustment amount. The first delay adjustment unit 17 outputs the adjustment amount to the device control unit 12, and the device control unit 12 uses the adjustment amount to adjust the timing of outputting the control information to the control information sending unit 11, and the control information is used to control the vision-related device driver 231, the hearing-related device driver 232, the touch-related device driver 233, the olfactory-related device driver 234, and the taste-related device driver 235 respectively. Thus, the first delay adjustment unit 17 can adjust the delay (delay in reception) from the acquisition of the eye information detection unit 31, the ear information detection unit 32, the hand information detection unit 33, the nose information detection unit 34, and the mouth information detection unit 35 to the reception of each motion information by the information processing device 1a. In addition, here, the latest time among the times when each motion information corresponding to the same time is received is used as the reference time, but the method for determining the reference time is not limited to this.

[0094] In addition, for example, the acquisition cycle of the information in the eye information detection unit 31, the ear information detection unit 32, the hand information detection unit 33, the nose information detection unit 34, and the mouth information detection unit 35 is the same, and the acquisition cycle is the same as the remote operation control cycle, but it is not limited to this. In the case where the acquisition cycle is shorter than the remote operation control cycle, the device control unit 12 uses the information of one or more acquisition cycles received within one remote operation control cycle to generate control information. For example, the control information can be generated using the average value of the information of multiple acquisition cycles, or the central value of the information of the acquisition cycle, etc. In this case, the first delay adjustment unit 17 can also adjust the delay using, for example, the average value of the time information of the information corresponding to the multiple acquisition cycles. In addition, when the acquisition cycle of the device control unit 12 is longer than the remote operation control cycle, the control information can be generated using the most recent information or a value interpolated using the most recent information and the previous information.

[0095] In addition, when there is a difference in the processing time generated by the device control unit 12 for the vision-related device driver 231, the hearing-related device driver 232, the tactile-related device driver 233, the olfactory-related device driver 234, and the taste-related device driver 235, the difference between the longest processing time of these processing times and each processing time is calculated as the processing delay amount, and the first delay adjustment unit 17 calculates the adjustment amount by adding the adjustment amount of the delay amount received and the processing delay amount. In addition, further, when there is a difference in the time from receiving the control information from the device 2 to the time when the vision-related device driver 231, the hearing-related device driver 232, the tactile-related device driver 233, the olfactory-related device driver 234, and the taste-related device driver 235 start to operate, these differences can also be considered as the adjustment amount.

[0096] In addition, Figure 8 In the example shown, the first delay adjustment unit 17 is provided at the preceding stage of the device control unit 12, but the present invention is not limited thereto, and the first delay adjustment unit 17 may be provided at the succeeding stage of the device control unit 12. In this case, the time information at which the motion information used in the generation of the control information is detected is added to each control information generated by the device control unit 12, and the first delay adjustment unit 17 performs control based on the time information so that the control information corresponding to the motion information detected at the same time is output at the same time.

[0097] Fig. 9 2 is a diagram showing a configuration example of an operating system according to Embodiment 2 related to information transmission control. Fig. 9 As shown, the information conversion unit 15a of the information processing device 1a of this embodiment is the same as the information conversion unit 15 of the first embodiment except that a second delay adjustment unit 81 is added.

[0098] The second delay adjustment unit 81 is an information transmission delay adjustment unit that adjusts the delay so that a plurality of synthesized data corresponding to the five sense information and the extended five sense information acquired at the same time are simultaneously transmitted to the user through a plurality of presentation devices. Specifically, the second delay adjustment unit 81 acquires the five sense information and the extended five sense information via the detection information receiving unit 14, calculates the delay adjustment amount in such a manner that the five sense information and the five sense data generated based on the extended five sense information detected at the same time as the five sense information are synthesized, and instructs the calculated adjustment amount as input information to the visual data synthesis unit 61, the auditory data synthesis unit 62, the tactile data synthesis unit 63, the olfactory data synthesis unit 64, and the taste data synthesis unit 65, respectively. In addition, the second delay adjustment unit 81 calculates the adjustment amount (outputs the adjustment amount) in such a manner that the presentation in the presentation device 4 corresponding to the five sense information and the extended five sense information detected at the same time is simultaneously executed, and instructs the corresponding visual data filter 71, the auditory data filter 72, the tactile data filter 73, the olfactory data filter 74, and the taste data filter 75, respectively. The visual data filter unit 71, the auditory data filter unit 72, the tactile data filter unit 73, the olfactory data filter unit 74 and the taste data filter unit 75 respectively control the output timing of the expression information according to the output adjustment amount so that the expression information generated according to the extended five senses information detected at the same time is output at the same time.

[0099] In addition, the second delay adjustment unit 81 may further consider the difference in processing time from when the visual data filter unit 71, the auditory data filter unit 72, the tactile data filter unit 73, the olfactory data filter unit 74, and the taste data filter unit 75 output the performance information to the visual performance device 41, the auditory performance device 42, the tactile performance device 43, the olfactory performance device 44, and the taste performance device 45 to perform the performance as the adjustment amount. In addition, a delay adjustment unit may also be provided between the detection information receiving unit 14 and the visual data conversion unit 51, the auditory data conversion unit 52, the tactile data conversion unit 53, the olfactory data conversion unit 54, and the taste data conversion unit 55 to perform input delay adjustment. That is, the delay adjustment unit calculates the delay adjustment amount (input adjustment amount) in such a manner that each conversion unit and each synthesis unit performs the processing using the five sense information and the expanded five sense information obtained at the same time, and instructs each conversion unit and each synthesis unit to perform the delay adjustment amount. As a result, each conversion unit and each synthesis unit can perform the processing using the information obtained at the same time. Furthermore, by adjusting the delay on one of the input and output, the user's sense of disharmony is reduced, but by adjusting the delay on both the input and output, the delay adjustment can be performed more effectively. In addition, regarding remote operation control, similarly, the delay adjustment can be performed on one of the input and output or on both the input and output.

[0100] In addition, for example, in the detection information acquisition unit 24 of the device 2, the acquisition cycle of information in the visual information detection unit 241, the extended visual information detection unit 242, the auditory information detection unit 243, the extended auditory information detection unit 244, the tactile information detection unit 245, the extended tactile information detection unit 246, the olfactory information detection unit 247, the extended olfactory information detection unit 248, the taste information detection unit 249, and the extended taste information detection unit 250 is the same, and the acquisition cycle is the same as the information transmission control cycle, but it is not limited to this. In the case where the acquisition cycle is shorter than the information transmission control cycle, each conversion unit and each synthesis unit uses the information of one or more acquisition cycles received within one information transmission control cycle to perform conversion and synthesis respectively. For example, the average value of the information of multiple acquisition cycles can be used to generate control information, and the central value of the information of the acquisition cycle can be used for conversion and synthesis. In this case, the second delay adjustment unit 81 can also adjust the delay by, for example, using the average value of the time information of the information corresponding to the multiple acquisition cycles. In addition, when the acquisition cycle is longer than the information transmission control cycle, the latest information may be used for conversion and synthesis, or a value obtained by extrapolation using the latest information and previous information may be used.

[0101] In addition, when a large delay occurs in a part of the information received by the information processing device 1, as described above, if the delay adjustment is performed by the first delay adjustment unit 17 and the second delay adjustment unit 81, the delay of the operating system 5 as a whole becomes larger. In particular, it is assumed that the fluctuation of the delay in wireless communication is larger than that in wired communication, and when wireless communication is used, there is a case where the delay of the transmission of a part of the information becomes larger. Therefore, the information with low importance in terms of realizing intuitive operation is removed from the objects of delay adjustment. In this case, it is also possible to predetermine whether to be the object of delay adjustment for each information, and remove the information outside the object of delay adjustment regardless of the delay amount, and remove the information outside the object of delay adjustment when the delay (difference between time information and reception time) of the information outside the object of delay adjustment is a certain value. That is, the first delay adjustment unit 17 may remove the control information corresponding to the driver with low importance among the plurality of device drivers from the objects of delay adjustment, and the second delay adjustment unit 81 may remove the synthetic data corresponding to the sense with low importance among the plurality of senses from the objects of delay adjustment.

[0102] For example, since olfactory information and the like can be remotely operated without incoordination even if there is a slight delay, olfactory information can also be excluded from the object of delay adjustment. In addition, in disaster relief, etc., the importance of taste information is low, so taste information can also be excluded from the object of delay adjustment. In addition, it is believed that the importance of olfactory information and taste information in rescue activities at the disaster site and sports on the snowy mountains as leisure activities (travel experience) are low, and the importance of auditory information in remote cooking is low, so these can also be excluded from the object of delay adjustment. The importance of each information also depends on the purpose for which the operating system 5 is used, so it is not limited to the above example, and the information set as the object of delay adjustment can be determined according to the importance in the purpose. In addition, depending on the user, there is also a situation where different information is valued, so it is also possible to allow the user's setting to select the information set as the object of delay adjustment.

[0103] In addition, in the example described above, an example of adding the first delay adjustment unit 17 and the second delay adjustment unit 81 to the information processing device 1 of embodiment 1 is described, but one of the first delay adjustment unit 17 and the second delay adjustment unit 81 may be added to the information processing device 1 of embodiment 1 without adding the other.

[0104] In the above-described example, the first delay adjustment unit 17 and the second delay adjustment unit 81 are added to the information processing device 1 of the first embodiment. However, the first delay adjustment unit 17 and the second delay adjustment unit 81 may be applied to an operating system that does not use the extended five senses information. Fig. 9 The first delay adjustment unit 17 and the second delay adjustment unit 81 are applied to the information processing device including the visual data conversion unit 51, the auditory data conversion unit 52, the tactile data conversion unit 53, the olfactory data conversion unit 54, the taste data conversion unit 55, the visual data synthesis unit 61, the auditory data synthesis unit 62, the tactile data synthesis unit 63, the olfactory data synthesis unit 64, and the taste data synthesis unit 65. Furthermore, the information processing device may not include the visual data filter unit 71, the auditory data filter unit 72, the tactile data filter unit 73, the olfactory data filter unit 74, and the taste data filter unit 75.

[0105] The information processing device 1 a of the present embodiment is realized by a computer system similarly to the information processing device 1 of the first embodiment.

[0106] As described above, in this embodiment, by adjusting the delay in remote operation control, the actions of the device 2 based on the information detected at the same time can be synchronized. Thus, it is possible to prevent the deviation of the actions of the device 2. In addition, by adjusting the delay in information transmission control, the effectiveness of the presentation in the presentation device 4 based on the information detected at the same time can be synchronized. Thus, the user can recognize the information detected at the same time without any disharmony.

[0107] The configuration shown in the above embodiment is merely an example, and may be combined with other known technologies, the embodiments may be combined with each other, and part of the configuration may be omitted or changed within a scope not departing from the gist.

[0108] (Explanation of symbols)

[0109] 1, 1a: information processing device; 2: device; 3: motion information detection device; 4: display device; 5: operating system; 11: control information sending unit; 12: device control unit; 13: motion information receiving unit; 14: detection information receiving unit; 15, 15a: information conversion unit; 16: display information sending unit; 17: first delay adjustment unit; 21: control information receiving unit; 22: drive control unit; 23: drive unit; 24: detection information acquisition unit; 25: detection unit Detection information sending unit; 31: eye information detection unit; 32: ear information detection unit; 33: hand information detection unit; 34: nose information detection unit; 35: mouth information detection unit; 41: visual representation device; 42: auditory representation device; 43: tactile representation device; 44: olfactory representation device; 45: taste representation device; 51: visual data conversion unit; 52: auditory data conversion unit; 53: tactile data conversion unit; 54: olfactory data conversion unit; 55: taste data conversion unit ; 61: visual data synthesis unit; 62: auditory data synthesis unit; 63: tactile data synthesis unit; 64: olfactory data synthesis unit; 65: taste data synthesis unit; 71: visual data filtering unit; 72: auditory data filtering unit; 73: tactile data filtering unit; 74: olfactory data filtering unit; 75: taste data filtering unit; 81: second delay adjustment unit; 231: visual related device driver; 232: auditory related device driver; 233: tactile related device driver driving unit; 234: olfactory related device driving unit; 235: taste related device driving unit; 241: visual information detection unit; 242: extended visual information detection unit; 243: auditory information detection unit; 244: extended auditory information detection unit; 245: tactile information detection unit; 246: extended tactile information detection unit; 247: olfactory information detection unit; 248: extended olfactory information detection unit; 249: taste information detection unit; 250: extended taste information detection unit.

Claims

1. An information processing device, It is characterized in that have: a detection information receiving unit for receiving five sense information that can be recognized by at least one of the five senses of a person in the device controlled according to an operation from a user who is located at a place far from the device, and extended five sense information that is information that is difficult to be recognized by the five senses of a person in the device; a conversion unit that converts the expanded five senses information into converted five senses data representing information that can be recognized by at least one of the five senses of a person; a synthesis unit that synthesizes the transformed five senses data and the corresponding five senses information; and The presentation information transmitting unit transmits the synthesized data as a result of the synthesis by the synthesizing unit to the presentation device, and the presentation device communicates the synthesized data to the user so that the user can recognize it with at least one of the five senses.

2. The information processing device according to claim 1, It is characterized in that The extended five senses information is information related to sounds that are difficult for humans to recognize.

3. The information processing device according to claim 1, It is characterized in that The extended five senses information is the detection result of images outside the human field of vision or light of wavelengths other than visible light.

4. The information processing device according to claim 1, It is characterized in that The extended five senses information is information indicating at least one of vibration, pressure, air flow, and tilt that are difficult for humans to detect.

5. The information processing device according to claim 1, It is characterized in that The extended five senses information is information related to smells that are difficult for humans to recognize.

6. The information processing device according to claim 1, It is characterized in that The extended five senses information is information related to tastes that are difficult for humans to distinguish.

7. The information processing device according to any one of claims 1 to 6, It is characterized in that The information processing device includes a filtering unit for filtering the synthetic data. The presentation information transmitting unit transmits the filtered synthetic data to the presentation device.

8. The information processing device according to claim 7, It is characterized in that The filtering includes a process of multiplying a gain corresponding to a value of the synthesized data.

9. The information processing device according to claim 8, It is characterized in that When the value of the synthesized data is equal to or less than a first value, the gain is a value greater than 1, and when the value of the synthesized data exceeds a second value greater than the first value, the gain is a value less than 1.

10. The information processing device according to any one of claims 1 to 9, It is characterized in that The synthetic data is a plurality of data corresponding to a plurality of senses among the five senses, respectively. The information processing device includes an information transmission delay adjustment unit that adjusts the delay so that the plurality of synthetic data corresponding to the five senses information and the extended five senses information acquired at the same time are simultaneously transmitted to the user through the plurality of presentation devices.

11. The information processing device according to claim 10, It is characterized in that The information transmission delay adjustment unit removes the synthesized data corresponding to a sense having a low degree of importance among the plurality of senses from targets for delay adjustment.

12. The information processing device according to any one of claims 1 to 11, It is characterized in that have: a motion information receiving unit for acquiring motion information indicating positions and movements of various parts of the user, respectively corresponding to a plurality of device driving units in the device; a device control unit that uses the motion information to generate control information for respectively controlling the actions of the plurality of device driving units in the device; A control information sending unit, which sends the control information to the device; as well as The remote control delay adjustment unit adjusts the delay so that operations based on the control information generated using the motion information acquired at the same time are simultaneously executed in the plurality of device drivers.

13. The information processing device according to claim 12, It is characterized in that The remote control delay adjustment unit removes the control information corresponding to the device driver having a low importance among the plurality of device drivers from targets for delay adjustment.

14. An operating system, It is characterized in that have: equipment; an information processing device that transmits control information for controlling the device according to an operation from a user who is located at a place far from the device to the device; and a display device for conveying information to the user that the user can recognize with the five senses, The device includes a detection information acquisition unit that detects five sense information that can be recognized by at least one of the five senses of a person in the device and extended five sense information that is information that is difficult to recognize by the five senses of a person in the device. The information processing device comprises: a detection information receiving unit, receiving the five senses information and the extended five senses information from the device; a conversion unit that converts the expanded five senses information into converted five senses data representing information that can be recognized by at least one of the five senses of a person; a synthesis unit that synthesizes the transformed five senses data and the corresponding five senses information; and The presentation information transmitting unit transmits the synthesized data which is the result of the synthesis by the synthesizing unit to the presentation device.

15. An information processing method in an information processing device, It is characterized in that include: A step of receiving five sense information that can be recognized by at least one of the five senses of a person in the device controlled by an operation from a user who is located at a place far from the device and extended five sense information that is information that is difficult to recognize by the five senses of a person in the device; a step of converting the expanded five senses information into converted five senses data representing information that can be recognized by at least one of the five senses of a person; A step of generating synthesized data by synthesizing the transformed five sense data and the corresponding five sense information; as well as The step of sending the synthesized data to a presentation device, wherein the presentation device communicates the synthesized data to the user so that the user can recognize it using at least one of the five senses.

16. The information processing method according to claim 15, It is characterized in that include: a step of generating control information for controlling the device; as well as The step of sending the control information to the device.

17. A procedure, It is characterized in that Causes the computer system to: A step of receiving five sense information that can be recognized by at least one of the five senses of a person in the device controlled by an operation from a user who is located at a place far from the device and extended five sense information that is information that is difficult to recognize by the five senses of a person in the device; a step of converting the expanded five senses information into converted five senses data representing information that can be recognized by at least one of the five senses of a person; A step of generating synthesized data by synthesizing the transformed five sense data and the corresponding five sense information; as well as The step of sending the synthesized data to a presentation device, wherein the presentation device communicates the synthesized data to the user so that the user can recognize it using at least one of the five senses.