Information processing device, information processing system, information processing method and program

Abstract

To provide technology that guides people with disabilities and other users, enabling them to walk and move safely and with peace of mind. [Solution] The information processing device includes an image data acquisition means for acquiring image data obtained by an imaging device, an acoustic data acquisition means for acquiring acoustic data obtained by a sound collection device, a generation means for generating control data to be supplied to one or more tactile devices by inputting input data including the image data and the acoustic data into a machine learning model, the control data being for guiding a user of the tactile device through tactile stimulation by the tactile device, and a supply means for supplying the control data to the one or more tactile devices.

Description

[Technical Field]

[0001] This disclosure relates to an information processing device, an information processing system, an information processing method, and a program. [Background technology]

[0002] Assistive technology to support the daily lives of the visually impaired is becoming an increasingly important social issue from the perspective of the SDGs (Sustainable Development Goals). As a technology that can help with this, for example, Patent Document 1 discloses a device and method that is worn by the visually impaired to assist them in walking safely. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2006-305214 Summary of the Invention [Problem to be solved by the invention]

[0004] The walking support device disclosed in Patent Document 1 acquires three-dimensional data of a predetermined space on a walking path of a visually impaired person, detects obstacles on the path in three dimensions based on the acquired three-dimensional data, generates vibration frequencies that indicate information about the direction, height, size, and distance to the obstacle, and transmits the information to the visually impaired person via transmission means attached in a dispersed manner to multiple parts where the visually impaired person can feel the vibrations.

[0005] However, although this walking assistance device can three-dimensionally grasp the presence and size of obstacles on the path, there are problems in terms of providing more appropriate guidance regarding the obstacles.

[0006] The present disclosure has been made in consideration of the above points, and one exemplary purpose thereof is to provide technology that can appropriately guide users who need guidance, such as people with disabilities. [Means for solving the problem]

[0007] An information processing device according to an exemplary aspect of the present disclosure includes an image data acquisition means for acquiring image data obtained by an imaging device, an acoustic data acquisition means for acquiring acoustic data obtained by a sound collection device, a generation means for generating control data to be supplied to one or more tactile devices by inputting input data including the image data and the acoustic data into a machine learning model, the control data being for guiding a user of the tactile device through tactile stimulation by the tactile device, and a supply means for supplying the control data to the one or more tactile devices.

[0008] An information processing system relating to an illustrative aspect of this disclosure comprises the information processing device, the imaging device, and the one or more tactile devices.

[0009] An information processing method according to an exemplary aspect of the present disclosure acquires image data obtained by an imaging device, acquires acoustic data obtained by a sound collection device, and inputs input data including the image data and the acoustic data into a machine learning model to generate control data to be supplied to one or more tactile devices, the control data being for guiding a user of the tactile device through tactile stimulation by the tactile device, and supplies the control data to the one or more tactile devices.

[0010] A program according to an exemplary aspect of the present disclosure is a program that causes a computer to function as an information processing device, and causes the computer to execute an image data acquisition process that acquires image data obtained by an imaging device, an acoustic data acquisition process that acquires acoustic data obtained by a sound collection device, a generation process that generates control data to be supplied to one or more tactile devices by inputting input data including the image data and the acoustic data into a machine learning model, the control data being for guiding a user of the tactile device through tactile stimulation by the tactile device, and a supply process that supplies the control data to the one or more tactile devices. [Effects of the Invention]

[0011] According to one exemplary aspect of the present disclosure, one exemplary effect is achieved in that it is possible to appropriately guide people with disabilities and other users who require guidance. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram illustrating a configuration of an information processing device according to the present disclosure. [Figure 2] FIG. 1 is a flow diagram showing the flow of an information processing method according to the present disclosure. [Figure 3] This is a block diagram showing the configuration of the information processing system related to this disclosure. [Figure 4] FIG. 1 is a block diagram illustrating an application example of an information processing system according to the present disclosure. [Figure 5] FIG. 10 is a block diagram illustrating another application example of the information processing system according to the present disclosure. [Figure 6] 1 is a schematic diagram illustrating an example of the appearance of an information processing system according to the present disclosure and a camera used therein. [Figure 7] 1A to 1C are schematic diagrams illustrating examples of haptic devices used in an information processing system according to the present disclosure. [Figure 8] FIG. 1 is a block diagram illustrating an example of data processing in an information processing system according to the present disclosure. [Figure 9]FIG. 1 is a block diagram for explaining a processing example based on a generation AI in an information processing system according to the present disclosure. [Figure 10] This is a closed-loop block diagram illustrating an example of processing based on generated AI in the information processing system related to this disclosure. [Figure 11] This is a block diagram showing the configuration of a computer that functions as an information processing device related to this disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0013] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, the effects mentioned in the exemplary embodiments shown below are examples of effects expected in the exemplary embodiments, and do not define the scope of the present invention. In other words, embodiments that do not exhibit the effects mentioned in the exemplary embodiments shown below may also be included in the scope of the present invention.

[0014] [First Exemplary Embodiment] A first exemplary embodiment, which is one example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is the basic form of each exemplary embodiment described later. Note that the scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure to the extent that no particular technical obstacles arise. Furthermore, each technical means shown in the drawings referred to in describing this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure to the extent that no particular technical obstacles arise.

[0015] (Configuration of information processing device 1) The configuration of an information processing device 10 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the information processing device 10. As shown in Fig. 1, the information processing device 10 includes an acquisition unit 11, a generation unit 12, and a supply unit 13.

[0016] (Acquisition part 11) The acquisition unit 11 functions as an image data acquisition means and acquires image data obtained by an imaging device. The imaging device, for example, is at least one camera worn by a user, capturing an image of at least a portion of the user's surroundings. Wearing the camera on the user in this manner is preferable because it can dynamically capture images of the user's surroundings in accordance with the user's movements. While the image data obtained by the imaging device is basically video data, it may also be still image data as appropriate depending on the situation. The acquisition unit 11 may also be configured to further acquire acoustic data obtained by a sound collection device and supply the acquired acoustic data to the generation unit 12. However, these examples do not limit the present exemplary embodiment.

[0017] (Generation unit 12) The generation unit 12 generates control data by inputting input data including image data from the acquisition unit 11 into a trained model (also referred to as a machine learning model or a machine learning model) described below. The control data is data to be supplied to one or more haptic devices described below, and is intended to guide a user of the haptic device through tactile stimulation by the haptic device. The control data may include control data for providing at least one of surrounding information and guidance information for the user to support the user in making decisions.

[0018] (Supply section 13) The supply unit 13 supplies the control data from the generation unit 12 to the haptic device. Then, the haptic device provides a haptic stimulus to the user in accordance with the supplied control data. Note that the haptic stimulus here can be, for example, Pressing into the user's skin with a noticeable level of pressure on the skin - Release of the pressure Repeated pressing and releasing The haptic device provides such haptic stimuli to the user based on the control data from the generation unit 12. Note that the above examples do not limit the present exemplary embodiment.

[0019] (Effects of the information processing device 10) As described above, the information processing device 10 acquires image data captured by an imaging device, inputs input data including the image data into a trained model, generates control data to be supplied to one or more haptic devices, the control data being used to guide a user of the haptic device through tactile stimulation by the haptic device, and supplies the control data to the haptic device. Therefore, the information processing device 10 can appropriately guide users who require guidance, such as people with disabilities. More specifically, the information processing device 10 provides tactile stimulation from the haptic device in accordance with control data generated based on a trained model that uses image data as input to users who require guidance, thereby providing guidance that allows the user to walk and move safely and with peace of mind.

[0020] (Flow of information processing method S1) Next, the flow of an information processing method S10 according to this exemplary embodiment will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of the information processing method S10. As shown in Fig. 2, the information processing method S10 includes an acquisition process (acquisition step) S11, a generation process (generation step) S12, and a supply process (supply step) S13. The information processing method S10 is executed, as an example, by the information processing device 10 described above.

[0021] (Step S11) In step S11, the acquisition unit 11 of the information processing device 10 acquires image data acquired by an imaging device. Specific examples of image data have been described above, so redundant description will be omitted here. The acquisition unit 11 may also be configured to acquire acoustic data acquired by a sound collection device and supply the acquired acoustic data to the generation unit 12.

[0022] (Step S12) In step S12, the generation unit 12 of the information processing device 10 inputs the input data, including the image data acquired in step S11, into a trained model (also called a machine learning model or a machine learning model) to generate control data to be supplied to one or more haptic devices, and to guide a user of the haptic device through tactile stimulation by the haptic device. The specific processing by the generation unit 12 has been described above, so a duplicated description will be omitted here.

[0023] (Step S13) In step S13, the supply unit 13 of the information processing device 10 supplies the control data generated in step S12 to one or more haptic devices. Then, the haptic devices provide haptic stimulation to the user in accordance with the supplied control data. Note that the haptic stimulation here can be, for example, Pressing into the user's skin with a noticeable level of pressure on the skin - Release of the pressure Repeated pressing and releasing The haptic device provides such haptic stimuli to the user based on the control data from the generation unit 12. Note that the above examples do not limit the present exemplary embodiment.

[0024] After step S13, the process proceeds to step S11. This repeats the process from steps S11 to S13 described above.

[0025] (Effect of information processing method S10) As described above, the information processing method S10 employs a process in which image data obtained by an imaging device is acquired, and input data including the image data is input into a trained model to generate control data for supplying to one or more tactile devices, which is used to guide the user of the tactile device through tactile stimulation from the tactile device, and the control data is supplied to one or more tactile devices. Therefore, according to the information processing method S10, it is possible to appropriately guide disabled persons and other users who require guidance. More specifically, by providing tactile stimulation from a tactile device to disabled persons and other users who require guidance, in accordance with the control data generated based on a trained model using image data as input, it is possible to provide guidance that enables the user to walk and move safely and with peace of mind.

[0026] (Information processing system 100) The configuration of the information processing system 100 to which the information processing device 10 according to this exemplary embodiment is applied will be described with reference to Figure 3. Figure 3 is a block diagram showing the configuration of the information processing system 100. As shown in Figure 3, the information processing system 100 includes an information processing device 10 equipped with an acquisition unit 11, a generation unit 12, and a supply unit 13, and a tactile device 21 and an imaging device 22 connected to the information processing device 10. Since the specific processing performed by the information processing device 10 has been described above, a redundant explanation will be omitted here.

[0027] The imaging device 22 can be realized by a camera that captures an image of the user's surroundings, and the camera can generate the image data. The haptic device 21 serves as one or more of the haptic devices described above. The imaging device 22 may be, for example, at least one camera worn by the user that captures an image of at least a portion of the user's surroundings. The haptic device 21 is worn on or in contact with a specific part of the user's body, and has a configuration that can guide the user by tactile stimulation other than the user's hearing.

[0028] (Effects of the information processing system 100) As described above, the information processing system 100 is configured to acquire image data captured by the imaging device 22, input input data including the image data into a trained model, generate control data to be supplied to one or more haptic devices 21, and provide the control data to guide the user of the haptic device through tactile stimulation by the haptic device. Therefore, the information processing system 100 can appropriately guide users who require guidance, such as people with disabilities. More specifically, the information processing system 100 provides tactile stimulation from the haptic device according to control data generated based on a trained model that uses image data as input to users who require guidance, thereby providing guidance that allows the user to walk and move safely and with peace of mind.

[0029] Second Exemplary Embodiment A second exemplary embodiment, which is one example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be assigned the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technical means shown in each drawing referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0030] (Conventional issues) Assistive technologies are crucial for supporting the daily activities and decision-making of people with visual impairments. To find objects and locations, people with such disabilities use traditional devices that help them identify objects and routes. These traditional devices can be navigational devices such as canes or animals such as guide dogs. In addition, visually impaired people need to be familiar with other reading technologies, such as Braille codes, to read documents and other print media. These devices come at a significant cost and require a significant learning curve.

[0031] People with visual impairments rely heavily on hearing and surrounding sounds to understand situations and dangerous events. To further support their audio perception, visually impaired people use long canes to sense their surroundings in public places such as train stations. Sometimes they carry additional devices such as radar or ultrasound devices to further understand the situation of objects and public routes. However, because the number of devices available to users is limited, their detection of their surroundings is limited and not very detailed. This can lead to dangerous situations not only for the user but also for other participants around them, such as falling or being hit by the cane.

[0032] Current assistive technologies can only provide limited support for the daily lives of people with visual impairments. Furthermore, these technologies are difficult to learn, very expensive, and difficult to use. Daily tasks are limited by the functions and features they can provide. This also limits how these people can participate in the job market and workforce. Their social participation is severely limited, and they experience high levels of stress and discomfort. Reducing these high levels of stress and discomfort is important from the perspective of healthcare for people with visual impairments.

[0033] (Summary of the present exemplary embodiment) This embodiment includes a system that guides visually impaired individuals and their daily living activities with the help of AI-based functionality (a machine learning model according to this embodiment) and a new output function that includes tactile signals. More specifically, as will be described later, the system is equipped with one or more cameras that record objects and the user's surroundings in real time and uses an AI-based algorithm to detect and classify objects of interest. This includes detecting directions and paths, and detecting dangerous objects or locations, which can be notified to the user in real time. To minimize the impact on the hearing of visually impaired individuals, the system uses a tactile device worn on the user's body to notify them of the AI-based detection results. This system does not interfere with the hearing of audio signals that are important to such users in their daily lives, thus causing minimal inconvenience to the user. Instead, specific body parts, such as the wrist, are used to convey tactile signal information to the user. The wrist is very convenient for visually impaired individuals to wear the device and blends easily with other user devices, so it does not obstruct the user's overall view.

[0034] This embodiment supports the use of AI-based functions such as object detection, path planning, and behavior detection, and provides basic guidance support to the user. Such guidance is provided by AI via image recognition from a camera device. The camera records objects and their locations in real time and transmits the image signals to an AI module that runs machine learning algorithms for object detection, path planning, behavior detection, and face recognition. Furthermore, generative AI is used to add additional information for the user. The generative AI uses the input data to generate high-level information such as reasoning and situational awareness. Such reasoning further improves support for the user by providing additional information based on context and situation. This allows the user to reduce risky behavior and more easily participate in daily life. In this embodiment, generative AI is used to generate new, context-based haptic signals in real time that indicate a specific situation. This embodiment can teach the user about complex environments and situations through generative haptic signals.

[0035] One example of this exemplary embodiment is a haptic device that is easy to wear and blends seamlessly with the user. Such a haptic device is embedded in a wristband or body vest. The wristband can provide basic information such as "left," "right," "up," and "down" via a vibrating haptic motor. This is beneficial to the user because it eliminates the need to hold additional devices such as a cane or other route-finding devices. Both hands are free and can be used for other daily tasks. Furthermore, the configuration includes a body vest composed of multiple vibrating haptic motors. This has multiple vibrating haptic motors arranged in a matrix on the front and back of the vest. Such a vest helps visually impaired individuals perceive their surroundings in detail and how to avoid dangerous situations by indicating their body position through feedback from the vibrating haptic sensors. Generative AI is used to generate haptic signals transmitted to the vibrating haptic motors, informing the user of the situation and surroundings in real time.

[0036] (Configuration of information processing system 100A) Next, the configuration of the information processing system 100A according to this exemplary embodiment will be described with reference to Figures 4 to 7. Figure 4 is a block diagram showing the configuration of the information processing system 100A. As shown in Figure 4, the information processing system 100A includes, as an example, an information processing device 1A, a group of devices 20, a server device 30, and a communication network N.

[0037] The information processing device 1A includes a control unit 10A having the same basic configuration as the information processing device 10 shown in Figure 3. Therefore, the control unit 10A includes an acquisition unit 11, a generation unit 12, and a supply unit 13, which are equivalent to those of the information processing device 10. However, the acquisition unit 11 here not only acquires the image data described above, but also acts as an acoustic data acquisition means for acquiring acoustic data obtained by a sound collection device. Thus, the acquisition unit 11 here has the functions of both an image data acquisition means and an acoustic data acquisition means. Accordingly, the input data to the generation unit 12 includes acoustic data from the sound collection device in addition to image data. This input data, including the acoustic data and image data, is input to a trained model, and control data is generated to guide the user of the haptic device through tactile stimulation from the haptic device.

[0038] The device group 20 includes a haptic device 21, a camera 22, a microphone 23, and the like, and each of these devices is connected to the information processing device 1A via a communication network N. A server device 30 is also connected to the network N, and the server device 30 and the information processing device 1A can exchange data via the network N.

[0039] (Configuration of information processing device 1A) 4, the information processing device 1A includes a control unit 10A, a storage unit 15A, and a communication unit 16A. The information processing device 1A also includes an internal device group 17A, which will be described later.

[0040] (Communication unit 16A) The communication unit 16A communicates with devices outside the information processing device 1A. For example, the communication unit 16A communicates with the haptic device 21, camera 22, microphone 23, server device 30, etc. The communication unit 16A transmits data supplied from the control unit 10A to the haptic device 21 and server device 30, and supplies data received from any of the devices in the device group 20 to the control unit 10A.

[0041] (Storage section 15A) The storage unit 15A stores various data referenced by the control unit 10A, as well as various data generated by the control unit 10A. For example, the storage unit 15A stores: • Image data ID referenced by the control unit 10A • Acoustic data SD referenced by the control unit 10A Control data CD, an example of the processing result (generated output) by the control unit 10A. • Learned model LM used in processing by the control unit 10A • Library data LIB referenced by the control unit 10A It is stored there.

[0042] (Control Unit 10A) As partially mentioned above, as shown in Figure 4, the control unit 10A includes an acquisition unit 11, a generation unit 12, and a supply unit 13.

[0043] (Acquisition part 11) As in the first exemplary embodiment, the acquisition unit 11 functions as an image data acquisition means and acquires image data ID obtained by an imaging device (camera 22). Here, the imaging device (camera 22) is, for example, at least one camera worn by a user, which captures an image of at least a portion of the user's surroundings. The imaging device (camera 22) may also be configured to include multiple cameras. For example, the imaging device (camera 22) may be configured to include one or more front cameras that capture images in front of the user and one or more rear cameras that capture images behind the user. However, this does not limit the present exemplary embodiment, and the imaging device (camera 22) may also be configured with other camera arrays.

[0044] The above-described configuration in which the camera is worn by the user is preferable because it allows dynamic capture of images of the surroundings in response to the user's movements. Furthermore, while the image data obtained by the imaging device is generally video data, it may also be still image data as appropriate depending on the situation. However, these examples do not limit this exemplary embodiment. As an example, the acquisition unit 11 may be configured to further acquire acoustic data obtained by a sound collection device and supply the acquired acoustic data to the generation unit 12. With this configuration, the generation unit 12 generates control data CD by referring to the acoustic data collected from around the user, thereby enabling more optimal guidance to the user.

[0045] (Generation unit 12) The generation unit 12 generates control data CD by inputting input data including image data from the acquisition unit 11 into a learned model LM (also called a machine learning model or a machine learning model). The control data CD is data to be supplied to one or more haptic devices 21, and is intended to guide a user of the haptic device 21 through tactile stimulation by the haptic device 21. The control data CD may include control data for providing the user with at least one of surrounding information and guidance information for the user to support the user's decision-making.

[0046] (Supply section 13) The supply unit 13 supplies the control data CD from the generation unit 12 to the haptic device 21. Here, the haptic device 21 is configured to give a haptic stimulus to the user in accordance with the supplied control data. Note that the haptic stimulus here can be, for example, • Pressing into the user's skin with a perceptible level of pressure. - Release of the pressure Repeated pressing and releasing This refers to things like the above. The tactile device 21 provides such tactile stimuli to the user based on the control data CD from the generation unit 12.

[0047] More specifically, the haptic device 21 may guide the user by providing appropriate stimuli to enable the user to perform safe actions in accordance with the surrounding environment, using one of the following patterns or a combination of these patterns. The stimulation position on the user's body, which corresponds to the control data CD, and the stimulation position of the tactile stimulation -Tactile stimulation strength according to control data CD - Frequency of tactile stimulation according to control data CD · Stimulation pressure waveform of tactile stimulation according to control data CD.

[0048] Thus, the haptic device 21 is attached to or in contact with a specific part of the user and has a configuration that allows it to guide the user with tactile stimuli other than hearing (e.g., vibration).

[0049] (Application example 1 of information processing system 100A) Next, with reference to Figure 5, Application Example 1 of the information processing system 100A will be described. Figure 5 is a diagram illustrating an application example of the information processing system 100A. As shown in Figure 5, the information processing system 100A includes a group of user devices 20, including a smartphone 24, headphones 25, a haptic device 21, a camera 22, and a microphone 23.

[0050] The smartphone 24 is equipped with a control unit that controls the functions of the smartphone, and this control unit runs an application that receives messages and alarm messages via a graphical user interface (GUI). The smartphone 24 also functions as a device for visually impaired persons who can visually operate the smartphone, but it may also be implemented as a specially prepared smartphone that generates Braille or other tactile output. Furthermore, if the user of the information processing system 100A is a biker (motorcycle operator), the smartphone 24 may be used as a dashcam for that user. In such cases, the smartphone 24 may also be used as a device that presents guide information such as alarm messages to that user.

[0051] The headphones 25 output audio signals such as alarm messages and other detection signals. These headphones may be configured as specific bone conduction headphones to provide the user with tactile signals as guiding information. Thus, because the headphones 25 are configured as bone conduction headphones, the guiding information can be provided to the user as bone conduction vibrations without interfering with the user's ability to hear ambient sounds. Such headphones 25 can be considered an example of a tactile device 21.

[0052] The tactile device 21 transmits tactile stimuli indicated by tactile signals to the user's body. This notifies and warns the user of the detection results of their surroundings through tactile stimuli. The tactile signals here are generated by the control data CD generated by the generation unit 12 of the control unit 10A, and then adapted by the supply unit 13 of the control unit 10A to the tactile device.

[0053] Camera 22 is responsible for the imaging device 22 described above, receiving image signals from the user's surroundings and transmitting them to the system. It can be configured as a combination of one front camera and one back camera, or as other camera arrays, further improving the detection of the surroundings. Camera 22 can be a standard USB-compliant 4K camera with MPEG coding, or a high-performance camera including an 8K camera or surround camera.

[0054] The microphone 23 functions as a sound collection device as described in FIG. 4 , and the acoustic data obtained thereby is supplied to the control unit 10A, sent to the acquisition unit 11 therein as part of the input data, and then passed to the generation unit 12. The microphone 23 records audio signals from the user's surroundings for voice recognition and noise detection, and is used to detect the surrounding situation and important audio signals related to the situation. The microphone 23 is also used for voice input to instruct the information processing system 100A to perform specific tasks. The user can request the system 100A to perform specific tasks, such as finding an object or a route, via the microphone 23. The user can interact with the system through voice input via the microphone 23.

[0055] The control unit 10A, which may be implemented as a CPU or the like, controls the flow of various input and output signals. The control unit 10A also executes the machine learning tasks necessary to activate the machine learning algorithms executed by the learned model LM (machine learning model) and other AI-based functions, and stores related data such as neural network weights and parameters in local memory. The control unit 10A is preferably power-efficient and supports extended use of more than one hour.

[0056] An internal device group 17A consisting of multiple internal devices is connected to this system 100A, and the internal device group 17A is configured with an (external) battery 171, an AI accelerator 172, a model / generative model LM, and a haptic library LIB. In addition to the battery 171, an external battery (not shown) may be provided to accommodate even longer use, and a fast-charging lithium battery may be used.

[0057] The AI ​​accelerator 172 supports the processing of the learned model LM (machine learning model) used by the generation unit 12 included in the control unit 10A. As an example, the AI ​​accelerator 172 is an additional hardware component that supports the calculation of the AI-based machine learning neural network (trained model LM) by the generation unit 12 of the control unit 10A. The AI ​​accelerator 172 provides sufficient computing power, such as 26 TOPS (Tera Operations Per Second), to perform real-time processing of camera images, noise removal, and other image features. By offloading such AI-based computations from the control unit 10A to the AI ​​accelerator 172, high computing performance and minimal power consumption (efficiency) can be achieved.

[0058] The model / generative model LM is composed of software functions that support AI-based machine learning, such as object detection, path planning, and behavior detection. The model / generative model LM is an example of the trained model LM (machine learning model) described above. The model / generative model LM also has the function of estimating the situation around the user (such as the occurrence of an accident or emergency) by referencing acoustic data indicating the environmental sounds around the user collected by the microphone 23 (sound collection device). The model / generative model LM also has the function of analyzing the user's voice data collected by the microphone 23 (sound collection device) and identifying the content of the user's speech.

[0059] Additionally, the Model / Generative Model LM consists of generative AI that can infer input data for context recognition. This extends object detection processing to help users understand the state of an object, such as a dangerous temperature, to mitigate accidents. The Model / Generative Model LM is important for further freeing users from everyday obstacles and enabling people with disabilities to better understand their surroundings and their dynamics in that location.

[0060] The haptic library LIB is composed of a haptic signal library containing a set of pre-recorded haptic signals and patterns. The haptic library LIB contains a set of specific tasks, such as drinking, pushing, eating, cutting, and swallowing, and a haptic signal associated with each of these specific tasks. These haptic signals from the haptic library LIB are selected by the control unit 10A and played by a haptic device 21 provided in a haptic wristband or a haptic body vest, which will be described later. Therefore, the generation unit 12 of the control unit 10A has a haptic library LIB containing a set of haptic signal patterns associated with various tasks performed by the user, and generates the control data by referring to this haptic library.

[0061] The external device group 30 is composed of a cloud access AC and an external haptic library LIB_EXT. For example, the external device group 30 is realized by the server device 30 described above via a network N (cloud access AC). The cloud access AC is used to support local machine learning within the control unit 10A on demand. The external haptic library LIB_EXT can be downloaded on demand when needed for specific tasks or daily operations. Since the primary use is local AI-based machine learning for guiding people with disabilities, cloud access is not required for the system. The cloud access provides additional event recording and map search services, which can be used by the system 100A on demand.

[0062] (Application example 2 of information processing system 100A) Next, with reference to Figure 6, we will explain Application Example 2 of the information processing system 100A. Figure 6 is a diagram showing Application Example 2 of the information processing system 100A.

[0063] The tactile guidance system developed by this system is a closed-loop system that calculates and executes AI-based detection in real time without requiring constant cloud access.

[0064] As shown in Fig. 6, system 100A includes components indicated by reference numerals 203 to 205, 207, and 208, and system 100A is shown to be directly attached to the body of user 201. To improve the comfort of using the tactile guide system (system 100A), system main body extension 204 is attached to the front 202 of the body over the shoulder, i.e., suspended from the shoulder by a shoulder strap, and is connected to the tip of strap front 203 in Fig. 6.

[0065] The system main body extension 204 includes a front camera 205. The front camera 205 serves as the imaging device and can be a single or multiple cameras attached to the user 201, who is the user, and captures an image of at least a portion of the surroundings of the user 201. Here, the front camera 205 is used to record an image in front of the user 201. The front camera 205 corresponds to the camera 22 described in FIGS. 4 and 5.

[0066] The main part of system 100A described in Fig. 5 is located on the back, i.e., back surface 206, of user 201, and is suspended from the shoulder by the shoulder strap for increased comfort, and is connected to the tip of rear strap portion 207 in Fig. 6. Main body 208 of system 100A shown in Fig. 5 is placed on the back of user 201, and houses main parts such as a CPU that runs control unit 10A, battery 171, and AI accelerator 172.

[0067] Fig. 7 shows the configuration of the haptic device of the haptic guide system included in the system 100A. In Fig. 7, the haptic device is attached directly to the user's body 301. An array of multiple vibrotactile motors 303 included in the haptic device 21 is attached to the chest, i.e., the front, 302, of the user's body.

[0068] The vibration tactile motor 303, which is responsible for the tactile device, is connected to a controller 304 that is connected to the guide system (control unit 10A) described in Figures 5 and 6 via wireless communication (Wi-Fi, Bluetooth, etc.). The controller 304 is also an example of the configuration of the tactile device 21. The array of vibration tactile motors 303 can be built into the body vest, i.e., the tactile vest. The tactile vest may consist of a front array and a rear array of vibration tactile motors 303.

[0069] In this way, the haptic device 21 is configured as a body vest worn on the user's chest. The body vest is also configured to have multiple vibration source motors that provide the user with vibration patterns indicating either ambient information or guidance information based on control data from the supply unit 13. This makes it possible to provide the user with appropriate haptic guidance without restricting the movement of the user's hands and fingers, that is, without being restricted in many everyday activities.

[0070] Another form of tactile device is an array of vibratory tactile motors 306, which is mounted on a wristband attached to or worn on the wrist of the arm of the body 301.

[0071] In this example, the wrist, where the wristband is worn, is a highly sensitive area of ​​the human body to all kinds of pressure and other tactile sensations. Therefore, it is beneficial to configure the vibration tactile feedback system via the user's wrist. Furthermore, the array of vibration tactile motors 306 is connected to the controller 307. Here, the vibration tactile motors 306 and the controller 307 are examples of the configuration of the tactile device 21.

[0072] In this example, the controller 307 is connected to the guide system (controller 10A) by wireless communication (Wi-Fi, Bluetooth, etc.) as described in Figures 5 and 6. The controller 307 receives an input haptic signal (i.e., a signal corresponding to the control data CD from the supply unit 13 of the control unit 10A) and transmits the signal to the vibrotactile motor 306. A guidance pattern indicating a direction relative to the user, such as "left," "right," "up," or "down," is generated by the operation of the vibrotactile motor 306 and is recognized by the user.

[0073] Thus, the wristband has a configuration that includes a vibration source motor for providing the user with vibrations that indicate a direction relative to the user based on control data. This has the advantage of effectively guiding the user through the user's relatively sensitive wrist.

[0074] As another form of haptic device, the headphones 25 can be bone conduction headphones equipped with a bone conduction actuator, as described above. Headphones 25 configured in this manner can receive haptic signals carrying control data from the supply unit 13 of the system internal control unit 10A, and drive the actuator in response to the received signals to generate vibrations and generate haptic stimuli for the user. This allows the headphones 25 to function as a guide for the system in addition to allowing the user to hear audio.

[0075] (Processing example 1 by information processing system 100A) Figure 8 shows an example of processing by the information processing system 100A. More specifically, Figure 8 shows the data flow and model selection performed in the information processing system 100A.

[0076] (Step S101) In step S101 of Figure 8, data is input to system 100A. The input data to the system consists, for example, of audio and video signals recorded via the user's camera 22 and microphone 23. This input data is acquired, for example, by the acquisition unit 11 described above.

[0077] (Step S102) Next, in step S102, one or more machine learning models LM are selected from the multiple machine learning models LM. Here, the model selection process is executed by, for example, the acquisition unit 11 or the generation unit 12. The acquisition unit 11 or the generation unit 12 selects a machine learning model depending on the input data and the user's usage status of the system, and supplies the input data to the selected machine learning model. Then, the generation unit 12 acquires control data CD output by the selected machine learning model, which is used to guide the user of the haptic device through tactile stimulation by the haptic device 21. For example, the acquisition unit 11 or the generation unit 12 You may select an object detection model for the simple object detection process S1032 based on camera input, You may choose a model that performs the complex path planning process S1034, which requires more power consumption. You may also select a model that performs the distance estimation process S1033. The selection of such a model can be performed adaptively depending on the input data and the user's usage status of the system.

[0078] Furthermore, for example, when a dangerous situation occurs, it is appropriate to execute additional processing using a machine learning model to help the user understand the surrounding situation. As an example, communication information regarding the surrounding situation of the user may be generated by a generation process S1031 using a generation AI, which is an example of a machine learning model.

[0079] In this way, the generation unit 12 can execute a combination of multiple machine learning algorithms, such as the generation process S1031, the object detection process S1032, the distance estimation process S1033, and the route planning process S1034. In particular, the transmission information generation process using the generation AI is suitable for providing the user with the context of the surrounding situation in a dynamic situation where a large number of people, vehicles, and objects are present. Furthermore, the basic object detection process S1032 is useful for supporting the user in daily tasks in the home, kitchen, living room, etc. For outdoor detection, the distance estimation process S1033 and the route planning process S1034 are preferably executed in unknown locations.

[0080] Thus, in this step, the generation unit 12 inputs input data into the machine learning model LM to generate control data CD to be supplied to one or more tactile devices 21, and to guide the user of the tactile device 21 through tactile stimulation by the tactile device 21.

[0081] (Step S104) Subsequently, in step S104, the supply unit 13 supplies the control data CD generated by the generation unit 12 in step S103 to an output device. Here, such an output device includes a haptic device 21, headphones 25 as an audio device, a smartphone 24 as a visual indicator, and a controller 26 responsible for motor control.

[0082] As shown in FIG. 7, the haptic device 21 may be configured with a haptic vest and a haptic wristband, for example. The headphones 25 serving as the audio device may be headphones for audio output via a bone conduction hearing device. The smartphone 24 serving as the visual indicator is a device that emits light, such as an LED, to indicate a dangerous situation in the surrounding area. This is useful, for example, for a biker (motorcycle rider) when the guidance system is used as a dashcam. The light can indicate a dangerous situation in a car behind the biker.

[0083] The controller 26, which is responsible for motor control, transmits control signals to the attached wheelchair used for mobility, or to other electric vehicles, bicycles, or scooters. When the user is moving and using the guidance system, it is important to accelerate or brake to avoid dangerous situations. The guidance system can also be applied as a useful unit to improve the safety and security of the user's vehicle, such as e-bikes, motorcycles, and bicycles.

[0084] Figure 9 shows a generative AI (an example of a machine learning model LM) used in the haptic guidance system. Input signals are supplied by one or more cameras 22. In addition, a single microphone 23 or a microphone array 23 (hereinafter simply referred to as microphone 23) is installed to help the system 100A understand its surroundings. Acoustic data obtained by microphone 23 is also supplied as an input signal.

[0085] The generation network 12, which functions as the generation unit 12, generates additional information using video and audio input from the camera 22 and microphone 23. For this purpose, a pre-trained encoder network 121 is used. Such a network encoder can be a conditional generative adversarial network (GAN) capable of generating vibration patterns for haptic devices.

[0086] The generator 122 generates data consisting of object data and context data. This is done by creating a spectral representation of the input data and converting it into a waveform for tactile signals. Context data is discovered by the generator 122, enabling the user to clearly understand their surroundings. Furthermore, such context information is used to generate tactile signals CD in real time based on the surrounding dynamics.

[0087] Thus, the dynamics or dynamic situation is converted into a complex haptic signal CD that is perceived by the user via the haptic device 21. An example would be that a "cup" has been detected by the object detection algorithm. Furthermore, the generative AI adds information about what has been put in the cup and whether there is a dangerous situation such as hot liquid.

[0088] This information is conveyed to the user through novel haptic signals that the user can understand. For example, the haptic vest described above can mimic the haptic patterns of drinking and swallowing, thereby providing situational awareness of what may happen next and how it may affect the user. The combination of generative AI and haptic vibration patterns can generate haptic patterns related to the user's current situation, thereby highly enhancing the user's situational awareness.

[0089] 10 is a diagram showing an example of processing when the information processing system 100A performs closed-loop feedback. More specifically, FIG. 10 shows the closed loop of the generation AI tactile guide system.

[0090] The closed loop is configured, for example, by a machine learning / detection process S210, a context generation process S220, a route planning process S230, a signal generation process S240, and a user action S250. The machine learning / detection process S210 is executed by the generation unit 12 with reference to input data. In the machine learning / detection process S210, for example, the generation unit 12 references the input data (image data, acoustic data) and performs the following: Identifying (recognizing, detecting) that the user's current location is a coffee shop (I211) Identifying (recognizing, detecting) that the target object for the user is a cup (I212) Identifying (recognizing, detecting) the sound of spilling liquid occurring in the user's vicinity (I213) Identifying (recognizing, detecting) the behavior of the user or a person around the user as a waiter (I214) · Tracking the hands of the user or people around the user (I215) • Recognition processing of faces of people around the user (I216) Each process is executed.

[0091] The location detection process I211 provides basic information such as where the user is located and what the surrounding classes are like, such as coffee shops, shopping malls, train stations, airports, etc., for example.

[0092] The object detection process I212 further decomposes the situation into what objects the user is facing, such as a cup, a spoon, or a table, as an example.

[0093] The voice detection process I213 assists the classifier in verifying the results and increasing the reliability of detection, such as detecting spills or car brake sounds.

[0094] The action detection process I214, as an example, helps to classify not only a person or a person, but also what action the person is performing, such as a waiter, staff member, or conductor.

[0095] The hand tracking process I215 is important, for example, to support this system by determining the location of the user's hands.

[0096] For example, face recognition I216 is useful for understanding the emotions and feedback of people around the user. Such signals are transferred to execute a context generation process S220. For example, the generation unit 12 can create a message or context information I221 such as "Coffee was poured into the cup two minutes ago" as context information about the user's surroundings. Furthermore, the generation unit 12 can refer to the context information I221 to generate guide information (tactile signals) for the user, including a message I222 such as "Be careful of hot objects. You may get burned." The context thus generated is transferred to a path planning process S230. The function of this process S230 includes at least one of a danger area estimation process, a user's body position estimation process, a body movement estimation process, and a hand movement estimation process.

[0097] The result of the path planning process S230 is transferred to the signal generation process S240, which generates tactile waveforms and signals in real time that indicate tactile alarm signals, position (left, right, up, down), and other new tactile vibration signals (swallow, drink, cut, etc.).

[0098] The user action S250 is executed in response to the output (tactile stimulation from the haptic device) of the signal generation process S240, and the machine learning / detection process S210 is further executed through closed-loop feedback S260. In this way, the information processing system 100A executes a closed-loop system consisting of classification / detection, guidance / route planning, feedback, and user action.

[0099] [Software implementation example] Some or all of the functions of the information processing device 10 or 1A or the control unit 10A (hereinafter also referred to as "each of the above devices") may be implemented by hardware such as an integrated circuit (IC chip) or by software.

[0100] In the latter case, each of the above devices is implemented, for example, by a computer that executes instructions for a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 11. Figure 11 is a block diagram showing the hardware configuration of computer C, which functions as each of the above devices.

[0101] Computer C comprises at least one processor C1 and at least one memory C2. Memory C2 stores a program P that causes computer C to operate as each of the above-mentioned devices. In computer C, processor C1 reads program P from memory C2 and executes it, thereby realizing each of the above-mentioned devices.

[0102] For processor C1, for example, a CPU (Central Processing Unit), GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Point Number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination thereof can be used. For memory C2, for example, flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination thereof can be used.

[0103] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, mouse, display, and printer.

[0104] Furthermore, the program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0105] [Appendix A] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0106] (Appendix A1) Image data acquisition means for acquiring image data obtained by an imaging device, a generation means for generating control data to be supplied to one or more haptic devices by inputting input data including the image data into a machine learning model, the control data being for guiding a user of the haptic device through tactile stimulation by the haptic device; A supply means for supplying the control data to the one or more tactile devices, An information processing device comprising:

[0107] (Appendix A2) The information processing device according to Appendix A1, further comprising an acoustic data acquisition means for acquiring acoustic data obtained by a sound collection device, wherein the input data includes the acoustic data.

[0108] (Appendix A3) The control data includes: The control data includes control data for providing the user with at least one of surrounding information and guidance information for the user to support the user in making decisions. An information processing device according to appendix A1 or A2.

[0109] (Appendix A4) The imaging device is at least one camera worn by the user, and is a camera that captures an image of at least a part of the user's surroundings. An information processing device according to any one of appendices A1 to A3.

[0110] (Appendix A5) The tactile device is a device that is attached to or in contact with a specific part of the user and guides the user by providing tactile stimulation other than hearing. An information processing device according to any one of appendices A1 to A4.

[0111] (Note A6) An information processing device according to any one of appendices A1 to A5; The imaging device and, the one or more haptic devices; Equipped with Information processing system.

[0112] (Note A7) The information processing system described in Appendix A6, wherein the tactile device is configured as a wristband worn on the user's wrist.

[0113] (Appendix A8) The information processing system described in Appendix A7, wherein the wristband has a vibration source motor for applying vibrations to the user that indicate a direction relative to the user based on the control data.

[0114] (Note A9) The tactile device is configured as a body vest worn on the user's chest, as described in Appendix A6 of the information processing system.

[0115] (Note A10) The information processing system described in Appendix A8, wherein the body vest has a plurality of vibration source motors for providing the user with a vibration pattern indicating either surrounding information or guidance information about the user based on the control data.

[0116] (Note A11) The tactile device is a bone conduction headphone, as described in Appendix A6 of the information processing system.

[0117] (Note A12) The information processing system described in Appendix A6, wherein the generating means has a haptic library including a set of haptic signal patterns associated with various tasks performed by the user, and generates the control data while referring to this haptic library.

[0118] [Additional Notes B] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0119] (Note B1) At least one processor performs image data acquisition processing to acquire image data obtained by the imaging device, a generation process in which the at least one processor inputs input data including the image data into a machine learning model to generate control data to be supplied to one or more haptic devices, the control data being for guiding a user of the haptic device through tactile stimulation by the haptic device; a providing process in which the at least one processor provides the control data to the one or more haptic devices; An information processing method comprising:

[0120] (Note B2) The information processing method according to Appendix B1, further comprising an acoustic data acquisition process for acquiring acoustic data obtained by a sound collection device, wherein the input data includes the acoustic data.

[0121] (Note B3) The control data includes: The control data includes control data for providing the user with at least one of surrounding information and guidance information for the user to support the user in making decisions. 1. An information processing method according to Appendix B1 or B2.

[0122] (Note B4) The imaging device is at least one camera worn by the user, and is a camera that captures an image of at least a part of the user's surroundings. 1. An information processing method according to any one of appendices B1 to B3.

[0123] (Note B5) The tactile device is a device that is attached to or in contact with a specific part of the user and guides the user by providing tactile stimulation other than hearing. 1. An information processing method according to any one of appendices B1 to B4.

[0124] (Note B6) An information processing device according to any one of appendices B1 to B5; The imaging device and, the one or more haptic devices; Includes Information processing methods.

[0125] (Note B7) The information processing method according to Appendix B6, wherein the tactile device is configured as a wristband worn on the wrist of the user.

[0126] (Note B8) The information processing method according to Appendix B7, wherein the wristband has a vibration source motor for applying vibrations to the user that indicate a direction relative to the user based on the control data.

[0127] (Note B9) The information processing method according to Appendix B6, wherein the tactile device is configured as a body vest worn on the chest of the user.

[0128] (Note B10) The information processing method described in Appendix B8, wherein the body vest has a plurality of vibration source motors for providing the user with a vibration pattern indicating either surrounding information or guidance information about the user based on the control data.

[0129] (Note B11) The information processing method according to Appendix B6, wherein the tactile device is a bone conduction headphone.

[0130] (Note B12) An information processing method as described in Appendix B6, wherein in the generation process, the at least one processor has a haptic library containing a set of haptic signal patterns associated with various tasks performed by the user, and generates the control data while referring to this haptic library.

[0131] [Additional Note C] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0132] (Appendix C1) A program that makes a computer function as an information processing device. The computer Image data acquisition means for acquiring image data obtained by an imaging device, a generation means for generating control data to be supplied to one or more haptic devices by inputting input data including the image data into a machine learning model, the control data being for guiding a user of the haptic device through tactile stimulation by the haptic device; A supply means for supplying the control data to the one or more tactile devices, An information processing program that functions as such.

[0133] (Appendix C2) The information processing program described in Appendix C1 further comprises an acoustic data acquisition means for acquiring acoustic data obtained by a sound collection device, wherein the input data includes the acoustic data.

[0134] (Appendix C3) The control data includes: The control data includes control data for providing the user with at least one of surrounding information and guidance information for the user to support the user in making decisions. The information processing program described in Appendix C1 or C2.

[0135] (Appendix C4) The imaging device is at least one camera worn by the user, and is a camera that captures an image of at least a part of the user's surroundings. An information processing program described in any one of the appendices C1 to C3.

[0136] (Appendix C5) The tactile device is a device that is attached to or in contact with a specific part of the user and guides the user by providing tactile stimulation other than hearing. An information processing program according to any one of appendices C1 to C4.

[0137] (Appendix C6) An information processing device according to any one of appendices C1 to C5; The imaging device and, the one or more haptic devices; Equipped with Information processing program.

[0138] (Appendix C7) The information processing program according to Appendix C6, wherein the tactile device is configured as a wristband worn on the wrist of the user.

[0139] (Appendix C8) The wristband has a vibration source motor for providing the user with vibrations that indicate a direction relative to the user based on the control data, according to the information processing program described in Appendix C7.

[0140] (Appendix C9) The information processing program according to Appendix C6, wherein the tactile device is configured as a body vest worn on the chest of the user.

[0141] (Appendix C10) The body vest has a plurality of vibration source motors for providing the user with a vibration pattern that indicates either the user's surrounding information or guidance information based on the control data, according to the information processing program described in Appendix C8.

[0142] (Appendix C11) The information processing program according to Appendix C6, wherein the tactile device is a bone conduction headphone.

[0143] (Appendix C12) The generation means includes a tactile library containing a set of tactile signal patterns associated with various tasks performed by the user, and the information processing program described in Appendix C6 generates the control data while referring to this tactile library.

[0144] [Appendix D] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0145] (Appendix D1) at least one processor, an image data acquisition process for acquiring image data obtained by an imaging device; a generation process for generating control data to be supplied to one or more haptic devices by inputting input data including the image data into a machine learning model, the control data being used to guide a user of the haptic device through tactile stimulation by the haptic device; a supplying process for supplying the control data to the one or more haptic devices; An information processing device that executes the above.

[0146] The information processing device may further include a memory, and the memory may store a program for causing the at least one processor to execute each of the processes.

[0147] (Appendix D2) The information processing device according to Appendix D1, further comprising an acoustic data acquisition process for acquiring acoustic data obtained by a sound collection device, wherein the input data includes the acoustic data.

[0148] (Appendix D3) The control data includes: The control data includes control data for providing the user with at least one of surrounding information and guidance information for the user to support the user in making decisions. An information processing device according to appendix D1 or D2.

[0149] (Appendix D4) The imaging device is at least one camera worn by the user, and is a camera that captures an image of at least a part of the user's surroundings. An information processing device according to any one of appendices D1 to D3.

[0150] (Appendix D5) The tactile device is a device that is attached to or in contact with a specific part of the user and guides the user by providing tactile stimulation other than hearing. An information processing device according to any one of appendices D1 to D4.

[0151] (Appendix D6) An information processing device according to any one of appendices D1 to D5; The imaging device and, the one or more haptic devices; Equipped with Information processing system.

[0152] (Note D7) The information processing system described in Appendix D6, wherein the tactile device is configured as a wristband worn on the user's wrist.

[0153] (Appendix D8) The information processing system according to Appendix D7, wherein the wristband has a vibration source motor for providing the user with vibrations that indicate a direction relative to the user based on the control data.

[0154] (Appendix D9) The information processing system described in Appendix D6, wherein the tactile device is configured as a body vest worn on the user's chest.

[0155] (Appendix D10) The information processing system according to Appendix D8, wherein the body vest has a plurality of vibration source motors for providing the user with a vibration pattern that indicates either the user's surrounding information or guidance information based on the control data.

[0156] (Appendix D11) The information processing system of Appendix D6, wherein the tactile device is a bone conduction headphone.

[0157] (Appendix D12) The information processing system according to Appendix D6, wherein in the generation process, the at least one processor has a haptic library including a set of haptic signal patterns associated with various tasks performed by the user, and generates the control data while referring to this haptic library.

[0158] [Appendix E] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims. (Appendix E1) A program that makes a computer function as an information processing device. The computer, an image data acquisition process for acquiring image data obtained by an imaging device; a generation process for generating control data to be supplied to one or more haptic devices by inputting input data including the image data into a machine learning model, the control data being used to guide a user of the haptic device through tactile stimulation by the haptic device; a supplying process for supplying the control data to the one or more haptic devices; A non-transitory recording medium on which an information processing program for executing the above is recorded. [Explanation of symbols]

[0159] 10, 1A, 2A ···Information processing device 100,100A ···Information Processing System 10A Control unit 11...Acquisition part 12...Generation section 13...Supply section 21 Haptic Devices 22 Camera 23. Mike

Claims

1. image data acquisition means for acquiring image data obtained by an imaging device; an acoustic data acquisition means for acquiring acoustic data obtained by the sound collection device; a generation means for generating control data to be supplied to one or more haptic devices by inputting input data including the image data and the acoustic data into a machine learning model, the control data being for guiding a user of the haptic device through tactile stimulation by the haptic device; supplying means for supplying the control data to the one or more haptic devices; An information processing device comprising:

2. The control data includes: The control data includes control data for providing the user with at least one of surrounding information and guidance information for the user to support the user in making decisions. The information processing device according to claim 1 .

3. The information processing apparatus according to claim 2 , wherein the imaging device is at least one camera worn by the user, and captures an image of at least a portion of the user's surroundings.

4. The information processing device according to claim 3 , wherein the tactile device is a device that is attached to or comes into contact with a specific part of the user and guides the user by providing a tactile stimulus other than an auditory stimulus to the user.

5. An information processing device according to any one of claims 1 to 4; the imaging device; the one or more haptic devices; Equipped with Information processing system.

6. The information processing system according to claim 5 , wherein the haptic device is configured as a wristband worn on the wrist of the user.

7. 7. The information processing system according to claim 6, wherein the wristband has a vibration source motor for applying vibrations to the user that indicate a direction relative to the user based on the control data.

8. The information processing system according to claim 5 , wherein the haptic device is configured as a body vest worn on the chest of the user.

9. 9. The information processing system according to claim 8, wherein the body vest has a plurality of vibration source motors for providing the user with a vibration pattern indicating either surrounding information or guidance information for the user based on the control data.

10. The information processing system according to claim 5 , wherein the haptic device is a bone conduction headphone.

11. 6. The information processing system according to claim 5, wherein the generating means has a haptic library containing sets of haptic signal patterns associated with various tasks performed by the user, and generates the control data while referring to this haptic library.

12. Acquire image data obtained by an imaging device; Acquire acoustic data obtained by a sound collection device, inputting input data including the image data and the acoustic data into a machine learning model to generate control data for supplying to one or more haptic devices, the control data being for guiding a user of the haptic device through tactile stimulation by the haptic device; providing the control data to the one or more haptic devices; Information processing methods.

13. A program that causes a computer to function as an information processing device, The computer, an image data acquisition process for acquiring image data obtained by an imaging device; an acoustic data acquisition process for acquiring acoustic data obtained by a sound collection device; a generation process for generating control data to be supplied to one or more haptic devices by inputting input data including the image data and the acoustic data into a machine learning model, the control data being used to guide a user of the haptic device through tactile stimulation by the haptic device; a supplying process for supplying the control data to the one or more haptic devices; Execute program.

Images