Sensory substitution devices and methods

Abstract

Provided are a sensory substitution device and method for combining two or more pieces of multisensory sensory information to convert them into different multisensory sensory information, wherein the received multiple pieces of information are combined to suit the sensory characteristics of a user's body and are effectively transmitted to different sensory organs. The sensory substitution method according to the present invention includes: receiving input sensory information; classifying the input sensory information; calculating the importance of each individual sensory information; determining a target sensation, wherein some or all of the individual sensory information for which the importance is calculated is to be converted into a new sensory signal that is inconsistent with the corresponding individual sensation; and converting the input sensory information into target sensory information to be transmitted to the determined target sensation as an organ.

Description

[0001] This application claims priority to and the benefit of Korean Patent Application No. 10-2019-0132848, filed on October 24, 2019, and Korean Patent Application No. 10-2020-0114967, filed on September 8, 2020, the disclosures of which are hereby incorporated by reference in their entirety. Technical Field

[0002] The present invention relates to a technology for converting sensory information into other sensory information, and more particularly, to a technology for combining two or more pieces of sensory information and converting the two or more pieces of sensory information into two or more other sensory signals. Background Art

[0003] With the increasing number of elderly or disabled people whose senses, such as vision and hearing, are impaired or degenerated, sensory substitution is gaining attention as a technological alternative that can improve impaired senses and perception to ensure continued economic activities and a higher quality of life.

[0004] Generally, sensory substitution involves changing the form of a damaged or degraded sensation into another form of sensation, and then transmitting the altered form of sensation to the corresponding sensory organ. Such sensory substitution techniques are known to be based on brain plasticity, where the human brain structurally and functionally changes and reorganizes to adapt to new environments. When sensory abilities are reduced or lost, the areas of the cerebral cortex responsible for different sensory information are rearranged. As this rearrangement progresses, when a sensory organ in the human body loses its function, it is functionally replaced by another sensory organ and adapts to the loss.

[0005] Existing sensory substitution technologies are limited to converting a single sensory signal of a human sense into another sensory signal and transmitting the converted sensory signal. In addition, in some technologies, multiple sensory organs are used as transmission paths to transmit a single sensory signal.

[0006] However, sensory organs, which serve as sensory receptors, have different limitations on the amount of sensory information they can receive (bandwidth), and users have different levels of stimulation of their sensory organs that they prefer or respond to. Therefore, existing sensory substitution technologies have difficulty in effectively transmitting information. Summary of the Invention

[0007] The present invention provides a sensory substitution technology capable of converting multisensory sensory information into different sensory information and efficiently transmitting the different sensory information to a sensory organ corresponding to the different sensory information.

[0008] The technical objects of the present invention are not limited to the above-mentioned contents, and other objects may become apparent to those skilled in the art based on the following description.

[0009] According to one aspect of the present invention, a sensory substitution device and method are provided for combining two or more pieces of multisensory sensory information to convert them into different multisensory sensory information, wherein the received multiple pieces of multisensory sensory information are combined to suit the sensory characteristics of the user's body and efficiently transmitted to different sensory organs. Here, the term "sensory information" refers to specific information used to combine multiple pieces of input sensory information to generate a new form of information (e.g., the position, distance, color, etc. of an object in the case of vision, and similar specific information in the case of other senses). Furthermore, the term "sensory signal" used in this specification refers to sensory information in terms of simple signals of received external information.

[0010] Specifically, a sensory substitution device according to one aspect of the present invention includes: a sensory information inputter configured to receive a plurality of different sensory information (multisensory sensory information); a sensory information combiner configured to combine the received plurality of sensory information to generate a signal in a new form, and convert the signal in the new form into different multisensory sensory information; and a sensory information outputter configured to output another sensory information to be sent to a sensory organ corresponding to the different multisensory sensory information.

[0011] Here, the sensory information combiner includes: an individual sensory classifier, configured to receive multisensory sensory information having two or more of visual information, auditory information, tactile information, olfactory information and taste information as an input sensory information, and classify the input multisensory sensory information according to individual sensations; an importance calculator, configured to calculate the importance of the classified individual sensory information; and a sensory information converter, configured to determine a target sensation and convert the input multisensory sensory information into target sensory information to be sent to the determined target sensation, wherein, for the target sensation, part or all of the individual sensory information for which the importance is calculated will be converted into a new sensory signal that is inconsistent with the corresponding individual sensation.

[0012] In addition, the sensory substitution method according to another aspect of the present invention includes: receiving two or more of auditory information, tactile information, olfactory information and taste information as one piece of input sensory information; classifying the one piece of input sensory information according to individual sensation for effective combination; calculating the importance of each piece of individual sensory information for the classified individual sensation; and determining the target sensation using the calculated importance or necessary bandwidth information about the bandwidth of the sensory organ possessed by the user, wherein, for the target sensation, part or all of the individual sensory information for which the importance is calculated will be converted into a new sensory signal that is inconsistent with the corresponding individual sensation; and converting the input sensory information into target sensory information to be sent to the determined target sensation (organ).

[0013] Here, the method may optionally further include: in order to determine the necessary bandwidth information, adjusting the bandwidth of the target sensation according to the importance calculated for each individual sensation.

[0014] The configuration and effects of the present invention will become apparent when considering the following detailed description of embodiments in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram showing an embodiment for implementing the sensory substitution device proposed by the present invention.

[0016] Figure 2 is a flowchart illustrating a processing procedure of the sensory substitution method according to the present invention.

[0017] Figures 3A to 3F An example of intermediate data related to the action of the sensory substitution device and method according to the present invention is shown. DETAILED DESCRIPTION

[0018] Hereinafter, when considered in conjunction with the accompanying drawings, with reference to the description of the following detailed embodiments, the advantages and features of the present invention and the ways to achieve them will become apparent. However, the scope of the present invention is not limited to such embodiments, and the present invention can be implemented in various forms. The embodiments to be described below are embodiments provided only to complete the disclosure of the present invention and to help those skilled in the art fully understand the scope of the present invention. The present invention is limited only by the scope of the appended claims. In addition, the terms used herein are used to help explain and understand the present invention, rather than to limit the scope and spirit of the present invention. It should be understood that, unless the context clearly provides otherwise, the singular also includes the plural. When the terms "comprise", "comprising", "including" and / or "comprising" are used in this article, they specify the existence of the group of stated features, integers, steps, operations, elements, components and / or features, integers, steps, operations, elements and components, and do not exclude the existence or addition of one or more other features, integers, steps, operations, elements, components and / or features, integers, steps, operations, elements and components.

[0019] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. With regard to assigning reference numerals to elements in the accompanying drawings, the same reference numerals will be used throughout the specification to indicate the same elements, and in the description of the embodiments, detailed descriptions of related known functions or configurations will be omitted to avoid obscuring the subject matter of the present invention.

[0020] Figure 1 is a block diagram showing an embodiment for implementing the sensory substitution device proposed by the present invention.

[0021] The sensory substitution device according to an embodiment includes: a sensory information inputter 100 configured to collect various sensory information including visual information, auditory information, tactile information, olfactory information, and taste information; a sensory information combiner 200 configured to combine the collected multiple pieces of sensory information (input sensory information) 10 into a new form of sensory signal to generate target sensory information 20; and a sensory information outputter 300 configured to output the converted target sensory information 20 to be transmitted to another sensory organ. Furthermore, the sensory substitution device may further include a storage device 400 configured to store data used when the sensory information combiner 200 combines the input multiple pieces of multisensory sensory information 10 and converts the combined result into the target sensory information 20.

[0022] Figure 2 A flowchart showing a processing procedure of a sensory substitution method according to the present invention, in which input multisensory sensory information is combined to convert and generate target sensory information, and the converted multisensory sensory information is output. Figure 2 The process shown in Figure 1 The sensory substitution device shown in FIG may be performed by a sensory substitution device having the same Figure 1 First, in order to help understand the present invention, we will first refer to Figure 2 The flowchart depicts the overall processing of the sensory substitution method according to the present invention.

[0023] Operation S10: Two or more of visual information, auditory information, tactile information, olfactory information, and taste information are input as one piece of input sensory information (ie, one piece of multi-sensory sensory information).

[0024] Operation S20: Classifying a piece of input sensory information by individual senses for effective combination. In other words, operation S20 is a process that analyzes the types of individual information that make up the multiple pieces of sensory information collected around the user and synchronizes the analyzed multiple pieces of individual sensory information by their appearance time so that the input sensory information is classified.

[0025] Operation S30: Calculating the importance of each piece of individual sensory information for the classified individual senses. In this case, as will be described below, priorities and weights may be used.

[0026] Operation S40: Determine a target sensation, wherein part or all of the individual sensory information for which the importance is calculated is converted into a new sensory signal that is inconsistent with the corresponding individual sensation. In this case, the importance calculated in operation S30 is used, but the target sensation may be determined based on necessary bandwidth information regarding the bandwidth of the user's sensory organs.

[0027] Operation S50: As an option, in order to determine the "necessary bandwidth information" described in operation S40, the bandwidth of the target feeling may be adjusted according to the importance of each individual feeling.

[0028] Operation S60: Finally, the input multisensory sensory information is converted into target sensory information to be sent to the determined target sense (sensory organ).

[0029] Operation S70: Outputting the target sensory information generated by the conversion to be sent to a corresponding sensory organ.

[0030] In the following, reference will be made to Figure 1 and Figure 2 A sensory substitution device according to an embodiment of the present invention is described in detail.

[0031] The sensory information inputter 100 receives two or more of visual information, auditory information, tactile information, olfactory information, and taste information as one piece of input sensory information 10, and transmits the input sensory information 10 to the sensory information combiner 200. Furthermore, the sensory information outputter 300 outputs target sensory information 20 as one piece of different sensory information, wherein the target sensory information 20 is a result of the input sensory information 10 being combined by the sensory information combiner 200 and converted into two or more of visual information, auditory information, tactile information, olfactory information, and taste information. Here, the target sensory information 20 is sensory information that is partially or entirely different from the input sensory information 10.

[0032] The sensory information combiner 200 specifically includes: an individual sensory classifier 210, configured to classify multiple input sensory information 10 according to individual sensations for effective combination; an importance calculator 220, configured to calculate the importance of each individual sensory information based on the priority and weight of the classified individual sensations (to be described in detail below); and a sensory information converter 230, configured to convert part or all of the individual sensory information for which the importance has been calculated into sensory information different from the corresponding individual sensations.

[0033] The various sensory information perceived by humans each has a varying degree of importance depending on the situation. Generally speaking, humans rely heavily on vision, so visual information is important. However, in situations where visual dependence is reduced, such as in a dark environment, auditory or olfactory information becomes more important than visual information. Therefore, in order to effectively convert input sensory information 10 into target sensory information 20, it is important to identify the individual sensory information that forms the input multisensory sensory information 10, which is a mixture of various pieces of information.

[0034] Therefore, the individual sensory classifier 210 performs the following tasks: analyzing what types of individual information form the multiple pieces of sensory information collected around the user through the sensory information inputter 100, and synchronizing the analyzed multiple pieces of individual sensory information by appearance time so that the multiple pieces of individual sensory information are classified. Figure 3AAn example of grouped data is shown, in which multiple pieces of sensory information input in the form of visual information V, auditory information A, tactile information T, olfactory information S, and taste information E are separated into n items, where the multiple pieces of sensory information need to be transmitted based on each individual sense. For example, sensory information specific to vision can include information such as color, shape, average brightness of an image, histogram, and extended information (such as distance to an object, location of an object, and type of object) that can be obtained through additional image processing. This separation of sensory information can be represented by dividing the information that needs to be transmitted based on each individual sense (such as vision, hearing, touch, smell, taste, etc.) into n items. Therefore, the value of n, which indicates the type (number) of the multiple pieces of sensory information to be transmitted, can vary for each sense.

[0035] In addition, the individual sensory classifier 210 can analyze or generate additional information for the input visual, auditory, tactile, olfactory, and gustatory information based on the individual sensory information and use this additional information. To generate additional information based on the individual sensory information, machine learning methods commonly used in data processing can be used. For example, in the case of visual information, in addition to the input image, video, or streaming video, detailed information such as the objects displayed in the image or video, the type of objects, the color of the objects, the distance to the objects, the distance between objects, and the location of the objects can also be analyzed and used. In addition, when the input sensory information is auditory, in addition to sound information, volume, pitch, tone, and other feature values processed in the frequency domain can be generated and used as additional information. When the input sensory information is tactile, information that can convey sensations through the skin (such as pressure, temperature, and vibration intensity) can be used as additional information. In the case of smell or taste, information such as the type, intensity, and duration of the smell or taste can be used as additional information.

[0036] Next, the importance calculator 220 will be described in detail.

[0037] The sensory information combiner 200 calculates the importance of each piece of sensory information classified by the individual sensory classifier 210 to efficiently combine two or more pieces of multi-sensory sensory information. It also uses the importance to combine multiple pieces of currently input sensory information 10 to generate new information. To this end, the importance calculator 220 considers both the importance of the multiple pieces of input sensory information 10 and the importance of additional information generated for the individual senses. To help understand the present invention, the meanings of importance, priority, and weight are described below.

[0038] Importance: The importance level in this invention is used to determine the sensory organ to which the combined result is ultimately transmitted. The importance level is calculated for each individual sense (vision: V, hearing: A, touch: T, smell: S, taste: E). The weights considered for calculating the importance level include: D, which accounts for the variability of the input senses; W, which accounts for the priority of the input senses; and U, which accounts for the continuity of the input sense information. Therefore, the importance level of each individual sense information can be calculated as follows.

[0039] -Importance of visual information: P v =D v +W v +U v

[0040] - Importance of auditory information: P a =D a +W a +U a

[0041] - Importance of tactile information: P t =D t +W t +U t

[0042] - Importance of olfactory information: P s =D s +W s +U s

[0043] - Importance of taste information: P e =D e +W e +U e

[0044] D, W, and V can represent the sensory information to be sent for each individual sense as n items, and can be represented as the sum of the priorities of multiple pieces of information expected to be sent according to individual senses at any point in time. Figures 3B to 3D ), the sensory information generated at any point in time is information generated by combining multiple pieces of sensory information that are ranked high in order of priority while taking into account the importance of each individual sensory information.

[0045] [Priority]: Priority is an indicator used to calculate the preferred combination of multiple pieces of information among n information groups that represent an individual's feelings at any given moment. For example, even when the same person is in the same space and takes the same action (sitting on a sofa), the information they expect to receive may differ. That is, even when the user remains seated in the same place, the shape of objects outside the window may be important, or the position of the TV remote control in front of the user may be used as important information.

[0046] [Weight]: Weight is a value D assigned to the change of sensation, a value W assigned to the priority of individual sensation, and a value U assigned to the continuity of sensation in order to calculate the most important information from n information groups representing individual sensations at any point in time. For example, weights can be assigned to three levels, where 1 represents high, 0.5 represents medium, and 0.2 represents low, so that a large weight is assigned to important sensory information. On the other hand, weights can be assigned to five levels at even intervals, where 1 represents very high, 0.8 represents high, 0.6 represents medium, 0.4 represents low, and 0.2 represents very low.

[0047] In summary, to determine which sensory information from multiple incoming pieces of sensory information will be combined for generation, priority is first used to distinguish the most important information at a specific point in time, either per unit time or discrete time. Priority is related to assigning weights to sensory information. Weights are differentiated values used to determine the importance of information over time and are assigned based on the priority of individual sensations. Taking into account the variability D and continuity U of the sensations, weights are assigned as separate values to calculate the importance of each sensation. In short, the target sensory information generated at a specific point in time is generated by combining multiple pieces of sensory information belonging to individual sensations with high importance.

[0048] Like this, the importance calculator 220 assigns weights to each piece of sensory information to calculate importance according to the priorities set to the individual sensory information classified by the individual sensory classifier 210 and the priorities set to the plurality of pieces of additional information, and calculates importance by summing the assigned weights.

[0049] The following are examples of prioritizing additional information. When the individual sense is vision, the additional information may include at least one of the distance to an object in the image, the object's color, the object's position, and the object's shape. When the individual sense is hearing, the additional information may include at least one of volume, pitch, and tone. When the individual sense is touch, the additional information may include at least one of pressure and temperature. Similarly, when the individual sense is smell or taste, the additional information may include at least one of the type, intensity, and duration of the taste or smell. The steps for prioritizing additional information generated from individual sensory information may vary depending on the situation. For example, in the case of additional information related to vision, priority may be set in the order of distance, color, position, and shape, but depending on the specific situation, priority may be set in the order of shape, distance, position, and color. In the case of hearing, priority may be set in the order of volume, pitch, and tone, but this priority setting may also vary depending on the situation. Additional information related to touch, smell, and taste may also be prioritized in a similar manner as described above, depending on the situation.

[0050] The importance P for generating a new signal by combining two or more pieces of sensory information may be calculated in consideration of the user's behavior or environment. To this end, the importance calculator 220 assigns weights having values varying between the following three conditions.

[0051] (1) Considering the amount of change D between sensory information input before a preset time and sensory information input at the current time, a higher weight may be assigned to sensory information having a larger amount of change.

[0052] (2) Based on the priority set according to the user's situation, different weights W can be assigned to the input sensory information.

[0053] (3) Considering the continuity U of the transmitted information, weights may be assigned to the sensory information based on the weights assigned to the previous sensory information input previously. That is, the higher the weight assigned to the previous sensory information, the higher the weight assigned to the sensory information corresponding to the previous sensory information.

[0054] In the case of the above condition (1), when the change amount of the currently input sensory information is greater than the change amount of the previous signal (previous sensory information), the currently input sensory information is determined to be important information (see Figure 3B ). Figure 3B: is a diagram showing a representation of data in which a plurality of detailed information to be transmitted based on each individual feeling is divided into n groups for determining a weight D for the amount of change in detailed information about the individual feeling at any point in time. For example, the weight Dv for the amount of change in visual information is expressed by dividing a plurality of detailed information {v1, v2, ..., v n}, which reflects the weight D of the change in the detailed information about vision at any point in time. Here, v represents all visual information including the detailed information about vision indicated as color information v1, the shape of the object v2, the distance information v3, and the position of the object v4. Similarly, information that needs to be transmitted based on other individual senses (such as hearing (a), touch (t), smell (s), and taste (e)) can also be specified in the same manner as above.

[0055] In the case of condition (2), the importance of each sensory information is considered to vary depending on the user's situation or behavior (see Figure 3C ). Figure 3C is a diagram showing a representation of data in which a plurality of detailed information to be transmitted based on each individual feeling is divided into n groups for determining a weight W for the priority of detailed information about individual feelings at any point in time. For example, the weight W for the priority of visual information v Indicates that a set of multiple detailed information {v1, v2, ..., v n} is obtained by reflecting the weight W of the priority of important detailed information about vision at any time point.

[0056] In the case of the above condition (3), it is considered whether to use the sensory information input at the previous time point based on the continuity of the information sent (refer to Figure 3D ). Figure 3D is a diagram showing a representation of data in which a plurality of detailed information to be transmitted based on each individual feeling is divided into n groups for determining a weight U for the continuity of detailed information about the individual feeling at any point in time. For example, the weight U for the continuity of visual information v Indicates that a set of multiple detailed information {v1, v2, ..., v n} is the information obtained by reflecting the weight U for the continuity of visual information at each specific time.

[0057] In this way, the importance for generating a new signal by combining two or more pieces of sensory information can be obtained by combining all three pieces of information (P=D+W+U) or combining at least two pieces of information based on the user's current situation.

[0058] Finally, sensory information converter 230 will be described in detail. As described above, sensory information converter 230 determines the target sensory organ to which target sensory information is to be transmitted based on the importance of the user's current situation, and converts the sensory information into a signal form suitable for the determined target sensory organ, ultimately generating target sensory information. More specifically, sensory information converter 230 includes a target sensory information determiner configured to determine a target sensory information based on at least one of bandwidth information of the user's sensory organ, environmental information regarding the user's location, and behavioral information related to the user's actions, and converts the plurality of sensory information determined in this manner into a signal form. The sensory information is then converted for the target sensory information.

[0059] In order to display the target sensory information generated, it is necessary to determine the total amount of information (i.e., bandwidth, B target ) are sent respectively. According to the calculated importance of the target sensory information and the bandwidth B of the information that can be sent by the individual sensory organs v 、B a 、B t 、B s and B e The target sensory organ is determined. The available transmittable bandwidth of the sensory organ is first allocated to the target sensory information with the highest importance. Therefore, the target sensory information determiner determines the target sensory information for which the multiple sensory information will be converted based on the required bandwidth information regarding the bandwidth of the user's sensory organ. In other words, the sensory information to be transmitted is determined based on its importance and the available transmittable bandwidth, and the sensory information to be transmitted is selected in order of highest importance within the available bandwidth.

[0060] exist Figure 3E The necessary bandwidth of the converted sensory information for each sensory information is shown in FIG. The target sensory organ is determined by sequentially arranging the sensory organs starting from the sense with the highest importance among the five senses mentioned above, wherein the target sensory information generated into a new form by combining multiple sensory information is to be sent to the target sensory organ, and the total amount of information B to be sent is determined by target The total number of sensory organs that send information is determined by allocation, so that the total amount of information B target is allocated to the amount of information (bandwidth) that can be sent by each sensory organ, B v 、B a 、B t 、B s and B eIn this case, the generated information is subdivided into m pieces of information so that the generated information is transmitted at a predetermined target time, and the bandwidth felt by the individual can be expressed as the sum of the amount of information (bandwidth) that can be transmitted at each subdivided time. Figure 3E is a diagram showing the bandwidth of individual perception.

[0061] In this specification, "target sense" and "target sensory organ" are used with the same meaning, and refer to a transmission path (channel) determined by a target sensory determiner for transmitting information generated by combining input sensory information. In addition, "transmitted sensory information" refers to information converted into information for the sensory organ serving as the determined transmission path. For example, when a signal generated by combining auditory information and olfactory information is transmitted through vision and touch, the visual signal serving as the transmitted sensory signal may be detailed information such as graphics or colors of an image or video, and the tactile signal serving as the transmitted sensory signal may be provided in the form of a vibration pattern or stimulation transmitted to the skin. In this specification, the term "transmitted sensory signal" is also used together with the term "specific sensory signal conversion information."

[0062] In addition, the sensory information converter 230 may further include a bandwidth adjuster, wherein the bandwidth adjuster adjusts the bandwidth of the target sense according to the importance set for each sense. The above description states that "the target sense determiner determines the target sense for which the sensory information is to be converted based on the necessary bandwidth information about the bandwidth of the sensory organ possessed by the user", wherein the bandwidth adjuster is required in order to determine the "necessary bandwidth information". The bandwidth adjuster adjusts the bandwidth of the target sense by reflecting the environmental variable (θ) for each individual sense. 1-5 ) to convert the information to be transmitted based on each individual sense, so that at least one of environmental information about the environment of the place where the user is located and behavioral information about the behavior performed by the user is reflected in the conversion of information for the determined target sense. Basically, there is a bandwidth (represented as B above) that can be transmitted for each individual sensory organ. v 、B a 、B t 、B s and B e ), but the sense of interest changes depending on the user's environment or behavior as the user's intentions change. In other words, the information received by the same sensory organ can change depending on the situation. Therefore, to determine which sense is more important in a given situation, the transmission amount is determined by adjusting the environmental variable (θ) for each individual sense.

[0063] Used to determine the transmittable bandwidth B for sending sensory information target as follows.

[0064]

[0065] Here, θ affects the amount of sensory channel transmission and varies depending on the user's surroundings, their behavior, their individual sensory characteristics, and the interplay of multiple senses. For example, when ambient sound levels are high, the value of the environmental variable θ2 is less than 1. During exercise, because vision suppresses auditory information, the values of the environmental variables θ1 for vision and θ3 for touch increase in a manner that varies depending on the situation.

[0066] Return to Figure 1 The sensory information outputter 300 synchronizes the signal, wherein the signal is converted for the target sensory organ determined by the sensory information converter 230 and used to correspond to multiple target sensory information according to the bandwidth of the target sensory organ, and the sensory information outputter 300 transmits the signal to the individual sensory organ in the form of multi-sensory sensory information. The transmitted multi-sensory sensory information is as follows: Figure 3F shown.

[0067] The function or processing of each element of the present invention described above may be implemented in a hardware combination including at least one of the following items: a digital signal processor (DSP), a processor, a controller, an application-specific IC (ASIC), a programmable logic device (e.g., a field programmable gate array (FPGA)), etc.), other electronic devices or a combination of a digital signal processor (DSP), a processor, a controller, an application-specific IC (ASIC), a programmable logic device (e.g., a field programmable gate array (FPGA)), etc.) and other electronic devices, or the function or processing of each element of the present invention described above may be implemented in software alone or in combination with hardware components, wherein the software may be stored in a recording medium.

[0068] As is apparent from the above, two or more pieces of multisensory sensory information are received and converted into a plurality of different target sensory information to be transmitted to the user. In addition, according to the present invention, various conditions are taken into account in the process of converting the input multisensory sensory information into the target sensory information, thereby providing the user with a customized sensory substitution function with enhanced efficiency.

[0069] Although the present invention has been described with reference to the embodiments, it will be understood by those skilled in the art that various modifications, equivalents and other embodiments are possible without departing from the scope and spirit of the invention. Therefore, the embodiments disclosed above should be interpreted as exemplary rather than limiting the present invention. The scope of protection of the present invention is not limited by the above-described embodiments, but is defined by the claims appended hereto, and the present invention will encompass all modifications, equivalents and substitutes that fall within the spirit and scope of the present invention.

Claims

1. A sensory substitution device comprising: an individual sensory classifier configured to receive multisensory sensory information having two or more sensory information among a plurality of sensory information and classify the received multisensory sensory information according to the individual senses, wherein the plurality of sensory information includes visual information, auditory information, tactile information, olfactory information, and taste information; an importance calculator configured to calculate the importance of each individual sensory information classified according to the individual senses; and a sensory information converter configured to determine a target sensation and convert the input multisensory sensory information into target sensory information to be sent to a target organ corresponding to the determined target sensation, wherein part or all of the individual sensory information for which importance is calculated is converted into a new sensory signal inconsistent with the corresponding individual sensation for the target sensation, The importance calculator is configured to calculate the importance of each individual sensory information by considering at least two of the following weights: a weight for the amount of change in the input sensory information, a weight for the sensory information taking priority into consideration, and a weight for the continuity of the input sensory information.

2. The sensory substitution device according to claim 1, wherein The sensory information converter is further configured to additionally use a transmission bandwidth of a sensory organ possessed by the user when determining the target sensation.

3. The sensory substitution device according to claim 1, wherein The individual sensory classifier is further configured to separate the multisensory sensory information input in the form of visual information, auditory information, tactile information, olfactory information, and taste information into groups that need to be transmitted based on each individual sense.

4. The sensory substitution device according to claim 1, wherein To calculate the importance, the importance calculator is configured to assign weights to the individual sensory information in consideration of a change amount between sensory information input before a preset time and sensory information input at a current time.

5. The sensory substitution device according to claim 1, wherein To calculate the importance, the importance calculator is configured to assign weights to individual sensory information according to a degree of priority set by the user's situation.

6. The sensory substitution device according to claim 1, wherein To calculate the importance, the importance calculator is configured to assign weights to individual sensory information according to weights assigned to previously input past sensory information so that continuity of the input sensory information is taken into account.

7. The sensory substitution device of claim 1, wherein: The sensory information converter includes: a target feeling determiner, which is configured to determine the target feeling based on at least one of bandwidth information of a sensory organ possessed by a user, environmental information about the environment of a place where the user is located, and behavioral information about a behavior performed by the user, and convert the determined target feeling into the form of a signal.

8. The sensory substitution device of claim 1, wherein: The sensory information converter includes a bandwidth adjuster for adjusting a transmittable bandwidth of a target organ corresponding to the target sensation according to the importance set for each individual sensation.

9. A sensory substitution method comprising: receiving multisensory sensory information having an input of two or more pieces of sensory information among a plurality of pieces of sensory information, wherein the plurality of pieces of sensory information include visual information, auditory information, tactile information, olfactory information, and taste information; classifying the piece of input multisensory sensory information according to individual sensations for effective combination; Calculate the importance of each piece of individual sensory information for the classified individual sensations; determining a target sensation for which part or all of the individual sensory information for which the importance is calculated is to be converted into a new sensory signal that is inconsistent with the corresponding individual sensation; and converting the input multisensory sensory information into target sensory information to be sent to the target organ corresponding to the determined target sensation, Among them, the step of calculating the importance of each individual sensory information for the classified individual sensation includes: calculating the importance of each individual sensory information by considering at least two of the following weights: the weight for the change amount of the input sensory information, the weight for the sensory information considering the priority, and the weight for the continuity of the input sensory information.

10. The sensory substitution method according to claim 9, wherein: When determining the target sensation, the transmittable bandwidth of the sensory organs possessed by the user is additionally used.

11. The sensory substitution method according to claim 9, wherein: When classifying the one piece of input multisensory sensory information according to individual senses, the one piece of input multisensory sensory information input in the form of visual information, auditory information, tactile information, olfactory information, and taste information that needs to be transmitted based on each individual sense is separated into groups.

12. The sensory substitution method according to claim 9, wherein: To calculate the importance, weights are assigned to individual sensory information in consideration of a change amount between sensory information input before a preset time and sensory information input at a current time.

13. The sensory substitution method according to claim 9, wherein: To calculate the importance, weights are assigned to individual sensory information according to the degree of priority set by the user's situation.

14. The sensory substitution method according to claim 9, wherein: To calculate the importance, weights are assigned to individual sensory information according to weights assigned to previously input past sensory information so that continuity of the input sensory information is taken into account.

15. The sensory substitution method according to claim 9, wherein: The step of converting the input multisensory sensory information into target sensory information includes: determining the target sensation based on at least one of bandwidth information of a sensory organ possessed by the user, environmental information about the environment of the place where the user is located, and behavioral information about the behavior performed by the user, and converting the determined target sensation into the form of a signal.

16. The sensory substitution method according to claim 9, wherein: The step of converting the input multisensory sensory information into target sensory information includes adjusting a transmittable bandwidth of a target organ corresponding to the target sense according to the importance set for each individual sense.

Images