An information interaction system and method based on sensory enhancement

Abstract

The application discloses a kind of information interaction system and method based on sensory enhancement.In the system, image display and feedback module obtains force signal characteristics by interactive touch with user, and then force signal characteristics, image feature signal quantity is transferred to image content identification and coding module, and reference signal quantity is transferred to sensory evoked module;Image content identification and coding module are used to: based on image feature signal, utilize classification model to obtain image content information;Based on force signal characteristics, utilize deep learning model, obtain the contact pressure information and contact position information that user touches image;Further, generate sensory coding information according to image content information, contact pressure information and contact position information;Sensory evoked module is used to generate electrical stimulation signal that can induce user brain to produce real feeling according to sensory coding information.The application can improve the efficiency and accuracy of user in information interaction process in the manner of sensory enhancement.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction, and more particularly to an information interaction system and method based on sensory enhancement. BACKGROUND

[0002] Traditional information interaction methods are generally realized through vision and hearing. However, for the blind, the visually impaired, the deaf and mute, children and adolescents, and other groups, when they cannot obtain complete audio-visual interaction information, they usually interact information through touch. In information interaction based on touch, the relevant information is usually reflected through the three-dimensional structure or roughness of the surface of the medium such as Braille and images, or through the vibration feedback of the medium. However, the three-dimensional structure, roughness of the medium surface, or the vibration effect of the medium feedback cannot enable users to efficiently and accurately interact information, especially users with imperfect audio-visual functions cannot accurately obtain or feedback the information displayed by the medium.

[0003] With the development of cloud computing, big data, artificial intelligence and other technologies, touch interaction based on virtual reality and augmented reality technologies can provide users with a better information interaction experience. For example, patent application CN201910395324.2 provides an information sending method, a storage medium and an electronic device. The method comprises: when a first operation signal is obtained, determining a target touch electrode according to the first operation signal; sequentially outputting at least two kinds of preset electric stimulation signals through the target touch electrode, each kind of preset electric stimulation signal corresponding to a user; when a second operation signal is obtained, determining the current preset electric stimulation signal as a target preset electric stimulation signal according to the second operation signal; obtaining a target user according to the target preset electric stimulation signal, and sending information to the target user; by binding different electric stimulation signals to different users, information can be quickly sent to the corresponding user, simplifying the information sharing process. However, the touch electrode and electric stimulation signal in this scheme are used to feedback the tactile signal, only through different electric stimulation signals to make the user feel different touch feeling.

[0004] Patent application CN201910400250.7 provides a control method and device of a touch screen, a storage medium and an electronic device. In this scheme, when a touch instruction located in a first touch area is received, the touch electrode of the first touch area is controlled to output a corresponding electric stimulation signal, so that the user can feel a corresponding electric stimulation touch feeling, increasing the diversity of touch sensing. However, this scheme mainly enhances the user's touch feeling of the touch screen through electric stimulation, and enriches the instructions of human-computer interaction through touch instructions, and in the structure of the touch screen component, only a row of touch electrodes is added to the touch surface of the touch screen, which cannot obtain the real interactive feeling of image content information through touch.

[0005] Patent application CN202010301023.1 discloses a system and method for converting ordinary walls or ground into a touch screen using a laser radar, comprising a host computer, a power supply, a laser radar device and a projector, the power supply output end of the host computer, the projector and the laser radar device is electrically connected through the wire, the connection end of the laser radar device is connected with the connection end of the host computer through the network cable, the output end of the host computer is connected with the input end of the projector through the data line, and the emission end of the laser radar device emits a virtual touch screen. However, this scheme only detects gesture position, gesture action and other information through laser radar to feedback the interactive action of touch pattern content, and cannot make the operator feel the real feeling feedback by the image.

[0006] Patent application CN201310302106.2 proposes a multi-sensory interactive multi-mode control functional electrical stimulation system which combines functional electrical stimulation technology, motion aid, Internet of Things technology, multi-sensory interactive game and human vital sign parameters. It is composed of functional electrical stimulation, vital sign parameter acquisition, motion aid information acquisition, information fusion, multi-sensory interactive game competition based on Internet of Things technology. However, this scheme only controls the functional electrical stimulation and the multi-sensory interactive game competition module through the human vital sign parameters and the motion aid parameters, and not only has no connection between the electrical stimulation and the multi-sensory interactive game, but also the electrical stimulation does not involve sensory feedback.

[0007] Patent application US12881376 discloses a touch panel module that can provide electrical stimulation sensory feedback, which is composed of a touch panel, a sensory feedback panel, a control unit and a stimulation signal generation circuit; the sensory feedback panel is installed on the touch panel and has a plurality of stimulation circuits arranged in a matrix configuration or a non-overlapping configuration; the stimulation signal generation circuit is connected with the stimulation circuit and the control unit; when the control unit receives the touch signal generated by the touch panel, the control unit will control the stimulation signal generation circuit to deliver stimulation current to the stimulation circuit; when the user touches the stimulation circuit, the stimulation current can flow through the user's finger to electrically stimulate the nerve, thereby achieving the effect of sensory feedback, so that the function touched by the user of the touch panel is activated. Then this scheme only integrates the stimulation circuit in the touch panel, and feedbacks the touch state by electrically stimulating the skin of the finger, and does not have the information interaction function of feedbacking the real feeling with the content in the touch panel.

[0008] In summary, the current information interaction based on touch mode is usually through the stereoscopic structure or roughness of the medium surface such as Braille and image to reflect the relevant information, or through the vibration feedback of the medium to express the relevant information, and the user with limited visual and auditory function cannot accurately obtain or feedback the information displayed by the medium. For example, when touching the sharp object in the picture medium, the user cannot truly feel the pain caused by the sharp object; when touching the flame picture in the video medium, the user cannot truly feel the burning caused by the flame. SUMMARY

[0009] The purpose of the present application is to overcome the defects of the prior art, and to provide an information interaction system and method based on sensory enhancement.

[0010] According to a first aspect of the present application, an information interaction system based on sensory enhancement is provided. The system comprises an image display and feedback module, an image content recognition and coding module, and a sensory induction module, wherein:

[0011] The image display and feedback module is used to display images and obtain force signal characteristics through interactive touch with the user, and then transmit the force signal characteristics and image feature signal quantities to the image content recognition and coding module, and transmit reference signal quantities to the sensory induction module;

[0012] The image content recognition and coding module is used to: based on the image feature signal, obtain image content information using a classification model; based on the force signal characteristics, obtain contact pressure information and contact position information when the user touches the image using a deep learning model; and then generate sensory coding information according to the image content information, the contact pressure information and the contact position information;

[0013] The sensory induction module is used to generate an electric stimulation signal capable of inducing the user's brain to produce a real feeling according to the sensory coding information, and act on the nerve bundle and nerve ending of the user's stimulation site;

[0014] The image display and feedback module is provided with an image display layer, a force detection layer and a reference electrode layer, the image display layer is used to display image content and output the image feature signal to the image content recognition and coding module; the force detection layer is used to detect the force when the user touches the image display layer content, and output the force signal characteristics to the image content recognition and coding module; the reference electrode layer is used to transmit the reference signal quantities to the sensory induction module.

[0015] According to a second aspect of the present application, an information interaction method based on sensory enhancement is provided. The method comprises the following steps:

[0016] The image is displayed to the user, and the force signal features are obtained through interaction with the user, and then the force signal features, image feature signal quantity are transmitted to the image content recognition and coding module, and the reference signal quantity is transmitted to the sensation induction module;

[0017] Based on the image feature signal, the image content information is obtained by using a classification model; based on the force signal feature, the contact pressure information and the contact position information of the user when touching the image are obtained by using a deep learning model; and then the sensation coding information is generated according to the image content information, the contact pressure information and the contact position information.

[0018] According to the sensation coding information, the electric stimulation signal capable of inducing the user's brain to produce a real feeling is generated, and acts on the nerve bundle and nerve ending of the stimulation site of the user.

[0019] Compared with the prior art, the information interaction system and method based on sensation enhancement provided by the application can enhance the acquisition and feedback effect of users including blind people, weak-sighted people, deaf-mutes, children and adolescents, and people with normal visual and auditory functions on information when they touch the medium such as image, through the real feeling of vibration, burning, tingling and the like of the medium feedback, so as to improve the efficiency and accuracy of the user in the information interaction process in the form of sensation enhancement.

[0020] Other features and advantages of the present application will become apparent from the following detailed description of exemplary embodiments thereof, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings incorporated in and forming a part of the specification, illustrate embodiments of the application and, together with the description, serve to explain the principles of the application.

[0022] Figure 1 is a schematic diagram of an information interaction system based on sensation enhancement according to an embodiment of the application;

[0023] Figure 2 is a signal output schematic diagram of an image display and feedback module according to an embodiment of the application;

[0024] Figure 3 is a signal processing flow chart of an image content recognition and coding module according to an embodiment of the application;

[0025] Figure 4 is a schematic diagram of the composition and signal relationship of a sensation induction module according to an embodiment of the application. DETAILED DESCRIPTION

[0026] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. Note that the relative arrangement of the components and steps illustrated in these embodiments, numerical expressions, and numerical values are merely examples, and do not limit the scope of the present application unless otherwise specifically stated.

[0027] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the scope of the application its application or uses.

[0028] Techniques, methods, and apparatus known to those of ordinary skill in the relevant art can not be discussed in detail herein, but should be considered as part of the specification, where appropriate.

[0029] In all of the examples shown and discussed herein, any specific values should be interpreted as merely illustrative, and not as a limitation. Thus, other examples of the exemplary embodiments can have different values.

[0030] Note that like reference numerals and letters indicate like items in the following drawings and that, once an item is defined in one drawing, it need not be discussed further in subsequent drawings.

[0031] The present application proposes a sensory enhancement-based information interaction scheme, which generates an electric stimulation signal corresponding to the encoded information according to the recognition and encoding of the image content touched by the user, so as to induce the user's brain to generate real sensations such as vibration, tingling, and burning corresponding to the image content.

[0032] Referring to Figure 1 As shown, the provided sensory enhancement-based information interaction system includes an image display and feedback module, an image content recognition and encoding module, and a sensory induction module. The image display and feedback module is used for the display of images and the touch interaction of the user, the image content recognition and encoding module is used for recognizing the image content displayed in the image display and feedback module and generating sensory induction encoding information, and the sensory induction module generates an electric stimulation signal capable of inducing the user's brain to generate real sensations such as vibration, tingling, and burning according to the encoding information of the image content recognition and encoding module, and acts on the stimulation site such as the user's wrist.

[0033] In one embodiment, the image display and feedback module is provided with an image display layer, a force detection layer, a reference electrode layer, and the like. The image display layer and the force detection layer are connected to the image content recognition and encoding module, and the reference electrode layer is connected to the sensory induction module. It should be noted that the images in the information interaction system of the present application include but are not limited to image content such as pictures and videos.

[0034] The image display layer is used to display image content, including but not limited to LED, LCD, etc. The force detection layer is used to detect the force of the user touching the image display layer, including but not limited to rigid or flexible force sensors of resistance, capacitance, piezoelectricity, etc. The reference electrode layer is used to provide a reference signal for the sensory induction module when the user touches the image content, including but not limited to using transparent indium tin oxide as the conductive material of the reference electrode layer.

[0035] Figure 2 is a signal output schematic diagram of the image display and feedback module. In the image display and feedback module, the image content in the image display layer is processed, including but not limited to greying, binarization, denoising, inclination correction, character cutting, normalization, etc., and then the image feature signal is output. The image feature signal includes but is not limited to color feature C O , shape feature S H , texture feature T E , spatial relationship feature S R , etc. For example, the amount of image feature signal output by the image display layer is f(C O , S H , T E , S R ); the force detection layer outputs the force signal when the user touches the module, including but not limited to force peak F P , force change ΔF, finger touch position (Δx, Δy), for example, the amount of force signal output by the force detection layer is f(F P , ΔF, Δx, Δy); the reference signal amount Rs output by the reference electrode layer is used to provide a signal reference for the sensory induction module. For example, the reference signal amount Rs can make the information interaction system and the human body have the same reference potential, so that the sensory induction module forms an input-output loop when working and plays an effective electric stimulation role, and also can reduce the interference of human body static electricity on the information interaction system.

[0036] Figure 3 is a signal processing flowchart of the image content recognition and coding module. The image content recognition and coding module is used to identify the image content displayed in the image display and feedback module and generate sensory induction coding information. First, the image content recognition and coding module uses image recognition processing methods (or classification models) including but not limited to convolutional neural network, depth-first search, breadth-first search, Dijkstra algorithm, etc. to classify and identify the image content displayed in the image display and feedback module according to the image feature signal amount f(C O , S H , T E , S R ) transmitted from the image display and feedback module, so as to obtain the content information C ISimultaneously, the image content recognition and encoding module will use the force signal f(F) transmitted from the image display and feedback module. P Using ΔF, Δx, Δy) and deep learning models, the contact pressure information F of the user touching the image is obtained. T and contact location information P T Finally, based on the image content information C I Contact pressure information F T and contact location information P T After processing by the sensory information encoding algorithm, sensory encoded information S is generated. C .

[0037] For example, the process of classifying and recognizing image content using a convolutional neural network model involves: pre-training the convolutional neural network model, where each sample in the training set reflects the correspondence between image feature signals and image categories; and then inputting the real-time acquired image feature signals into the pre-trained convolutional neural network model to obtain the corresponding image category. Another example is using a deep learning model to obtain the contact pressure information F when a user touches an image. T and contact location information P T At that time, the training set for pre-training the deep learning model reflects the correspondence between the sample force signal features and the sample contact pressure information and sample contact position information.

[0038] In one embodiment, sensory encoding information S is generated. C The process is a multimodal signal fusion process, which generates sensory coding information S by fusing multiple modal information such as image and force. C Specifically, image content information C I It includes spatial information of the image displayed in the image display layer, and image object attributes. For example, if the image display layer displays a cone-shaped object, the image content information C... I This will include parameters such as the spatial location of the pointed object, the contour of its tip, and the hardness of its physical material; contact pressure information F. T and contact location information P T These are all interactive attribute parameters fed back by the force detection layer when a user touches an image in the image display layer. Examples include the magnitude of the force applied by the user when touching the image, and the positional information of the user's contact with the image display layer during the touch. There is a correspondence between the spatial positional information of the image content fed back by the image display layer and the spatial positional information of the contact force fed back by the force detection layer. Furthermore, the image content is correlated with its spatial position within the image display layer. Therefore, based on the logic and correlation of touching the image, the image content information C... I Contact pressure information F T and contact location information P TThe sensory encoding information S can be obtained after multi-modal fusion of the above information C .

[0039] Figure 4 is a composition and signal relationship diagram of the sensory induction module. The sensory induction module generates an electric stimulation signal capable of inducing the user's brain to generate a real feeling of vibration, tingling, burning, etc. according to the encoding information of the image content recognition and encoding module, and acts on the wrist and other stimulation sites of the user.

[0040] In one embodiment, the sensory induction module includes an encoding information conversion unit, a stimulation current generation unit, and an electric stimulation output unit, etc. The encoding information conversion unit is used to convert the sensory encoding information S C in the image content recognition and encoding module into an electric stimulation control signal U S ; the stimulation current generation unit is used to generate a stimulation current signal I S according to the electric stimulation control signal U S ; and the electric stimulation output unit combines the signal reference amount R S in the image display and feedback module to act the stimulation current signal I S on the wrist and other related stimulation sites through signal electrodes, wearable devices, etc. In the present application, the user feels the real feeling of vibration, stimulation, burning, etc. corresponding to the image content, which is not through the current stimulation contact site to make the skin feel vibration, tingling, burning, etc. but through the stimulation of the nerve bundle and nerve ending of the wrist and other related sites to induce the user's brain to generate a real feeling of vibration, tingling, burning, etc. on the non-stimulation site such as the finger touching the image content. Therefore, the electric stimulation control signal U S generated by the encoding information conversion unit in the sensory induction module contains but is not limited to stimulation characteristic parameters such as waveform, amplitude, width, interval, etc. and the combination of different stimulation characteristic parameters corresponds to the stimulation current signal I S .

[0041] It should be noted that the sensory induction in the present application refers to that the receptors in the skin transmit the received electric stimulation to the nerve fibers connected with the receptors, thereby activating the nerve fibers to generate electric activity, and then the electric activity is transmitted to the brain along the upward sensory nerve transmission path to induce the brain to generate a real feeling of vibration, tingling, etc. which is the proprioceptive sensation. However, the existing ordinary electric stimulation signal cannot induce the brain to generate a natural intuitive feeling. For example, the different stimulation current signals I SWhen the user's brain induces the real feeling of the non-stimulated part, the user's middle finger, ring finger and the corresponding palm position below the ring finger will feel the real feeling of knocking when the current with pulse interval 200 μs, pulse width 200 μs, current amplitude 1 mA and frequency 50 Hz is applied to the median nerve position of the user's wrist; the user's thumb and thenar eminence position will feel the real feeling of vibration when the current with pulse interval 200 μs, pulse width 200 μs, current amplitude 0.75 mA and frequency 200 Hz is applied to the user's wrist ulnar nerve position; the user's little finger will feel the real feeling of tingling when the current with pulse interval 200 μs, pulse width 350 μs, current amplitude 3 mA and frequency 200 Hz is applied to the user's wrist flexor nerve position. When the above different stimulation current signals I S When the above different stimulation current signals I

[0042] After demonstration and pre-experiment, the information interaction system based on sensory enhancement provided by the present application can be applied to information exchange, cognitive training, teaching guidance, game entertainment, augmented reality and the like.

[0043] Embodiment 1

[0044] The information interaction system provided by the application can enable the user with impaired vision, such as the blind and the weak-sighted, to read Braille or browse pictures. For example, after the user with impaired vision wears the relevant signal electrode on the wrist, the user can feel the real feeling corresponding to the Braille or picture content when the user touches the Braille or picture content displayed on the image display and feedback module of the information interaction system. First, the information interaction system identifies and encodes the Braille or picture content. Then, when the user touches the Braille or picture, the information interaction system generates the electric stimulation signal corresponding to the touched Braille or picture content according to the encoding information of the content at the touched position, and applies the electric stimulation signal to the wrist of the user through the electric stimulation output unit of the sensory evoking module, so that the user can feel the real feeling, such as vibration, tingling, burning, etc., corresponding to the touched Braille or picture content, thereby enhancing the reading ability and interactive experience of the user with impaired vision in obtaining text and picture information. For example, when the user with impaired vision touches the Braille representing the meaning of "fire", the user can feel the real feeling, such as burning and tingling, when touching the flame, so that the user can understand and recognize the meaning and characteristics of "fire" more quickly.

[0045] Embodiment 2

[0046] The information interaction system provided by the application can enable the user with impaired hearing, such as the deaf and mute, and the user with incomplete auditory and visual functions, such as children and adolescents, to learn fire-fighting knowledge, such as fire alarm first aid. For example, after the user with impaired hearing, such as the deaf and mute, and the user with incomplete auditory and visual functions, such as children and adolescents, wears the relevant signal electrode on the wrist, the user can feel the real feeling corresponding to the picture or video content related to fire alarm first aid when the user touches the picture or video content related to fire alarm first aid displayed on the image display and feedback module of the information interaction system, thereby enabling the user to obtain the "touch reading" experience with real feeling feedback as if the user is reading "audio book". First, the information interaction system identifies and encodes the picture or video content related to fire alarm first aid. Then, when the user touches the picture or video content related to fire alarm first aid, the information interaction system generates the electric stimulation signal corresponding to the touched picture or video content according to the encoding information of the content at the touched position, and applies the electric stimulation signal to the wrist of the user through the bracelet with the function of the sensory evoking module of the system worn on the wrist of the user, so that the user can feel the real feeling, such as vibration, tingling, burning, etc., corresponding to the touched content, thereby enhancing the experience and learning efficiency of the user with impaired hearing, such as the deaf and mute, and the user with incomplete auditory and visual functions, such as children and adolescents, in recognizing new things.

[0047] Embodiment 3

[0048] The information interaction system provided by the application can make the user with normal visual and auditory function experience the picture or video content with real feeling feedback. Specifically, the user with normal visual and auditory function wears the signal electrode on the wrist, and the user can feel the real feeling corresponding to the picture or video content when the user touches the picture or video displayed on the image display and feedback module of the information interaction system. First, the information interaction system identifies and encodes the picture or video content. Then, when the user touches the picture or video, the information interaction system generates the electric stimulation signal corresponding to the touched picture or video content according to the encoding information of the picture or video content, and the electric stimulation output unit of the feeling evoking module of the system acts on the relevant stimulation part of the user, so as to evoke the user's brain to produce the real feeling such as vibration, tingling and burning corresponding to the touched picture or video content, thereby enabling the user to obtain the real feeling experience corresponding to the picture or video content. According to the above information interaction process, the user with normal visual and auditory function can not only realize the tactile experience with real feeling feedback, but also realize the high-efficiency hand touch interaction action without looking at the interactive object, thereby enabling the user to realize the high-efficiency information interaction in the case of "eye-hand separation".

[0049] Embodiment 4

[0050] The information interaction system provided by the application can make the user with normal visual and auditory function experience the picture or video content with real feeling feedback. Specifically, the user with normal visual and auditory function wears the signal electrode on the wrist, and the user can feel the real feeling corresponding to the picture or video content when the user touches the picture or video displayed on the image display and feedback module of the information interaction system. First, the information interaction system identifies and encodes the picture or video content. Then, when the user touches the picture or video, the information interaction system generates the electric stimulation signal corresponding to the touched picture or video content according to the encoding information of the picture or video content, and the electric stimulation output unit of the feeling evoking module of the system acts on the relevant stimulation part of the user, so as to evoke the user's brain to produce the real feeling such as vibration, tingling and burning corresponding to the touched picture or video content, thereby enabling the user to obtain the real feeling experience corresponding to the picture or video content. According to the above information interaction process, the user with normal visual and auditory function can not only realize the tactile experience with real feeling feedback, but also realize the high-efficiency hand touch interaction action without looking at the interactive object, thereby enabling the user to realize the high-efficiency information interaction in the case of "eye-hand separation".

[0051] In summary, the information interaction system and method based on sensory enhancement provided by the present application can enhance the acquisition and feedback effect of information for users including the blind, the weak-sighted, the deaf-mute, children and teenagers, and people with normal vision and hearing, etc. when they touch the media such as images. For example, when touching the image media such as pictures and videos, the users can feel the real stinging and burning sensation caused by the sharp objects and flames in the image content acting on the touching part, so as to enhance the efficiency and accuracy of the users in the information interaction process in a sensory enhancement manner.

[0052] The present application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0053] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, a magneto-optical storage device, or any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0054] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0055] Computer readable program instructions for carrying out operations of the present application can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0056] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0057] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0058] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0059] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0060] Embodiments of the present application have been described above, and the description is intended to be illustrative, and not restrictive, of the disclosed embodiments. Many modifications and variations of the disclosed embodiments are possible in light of the above teachings. It is therefore to be understood that within the scope of the disclosed embodiments, modifications and variations of the disclosed embodiments can be practiced. It is also to be understood that the specific order or hierarchy of steps in the processes disclosed is an illustration of exemplary processes. Based upon the description and illustrations provided herein, those skilled in the art will understand that changes can be made to the order of steps in the processes and that many of the individual steps can be modified or eliminated. Additionally, the description and illustrations provided herein are not meant to limit the scope of the disclosed embodiments. The scope of the disclosed embodiments is limited only by the claims.

Claims

1. A sensory enhancement-based information interaction system, comprising an image display and feedback module, an image content recognition and coding module, and a sensory induction module, wherein: the image display and feedback module is configured to display an image, obtain force signal features through interactive touch with a user, and then transmit the force signal features and image feature signals to the image content recognition and coding module, and transmit reference signal quantities to the sensory induction module; the image content recognition and coding module is configured to obtain image content information based on the image feature signals using a classification model, obtain contact pressure information and contact position information when the user touches the image based on the force signal features using a deep learning model, and then generate sensory coding information according to the image content information, the contact pressure information, and the contact position information; and the sensory induction module is configured to generate electrical stimulation signals capable of inducing the user's brain to produce real sensations according to the sensory coding information, and act on the nerve bundles and nerve endings of the user's stimulation sites. The image display and feedback module comprises an image display layer, a force detection layer, and a reference electrode layer, the image display layer is configured to display image content and output the image feature signals to the image content recognition and coding module, the force detection layer is configured to detect the force when the user touches the image display layer, and output the force signal features to the image content recognition and coding module, and the reference electrode layer is configured to transmit the reference signal quantities to the sensory induction module, and the reference signal quantities make the information interaction system and the human body have the same reference potential, so that an input-output loop is formed when the sensory induction module works. The image feature signals comprise color features, shape features, texture features, and spatial relationship features, and the force signal features comprise force peak values, force variation quantities, and finger touch positions. The sensory induction module comprises an encoding information conversion unit, a stimulation current generation unit, and an electrical stimulation output unit, the encoding information conversion unit is configured to convert the sensory coding information into electrical stimulation control signals, the stimulation current generation unit is configured to generate stimulation current signals according to the electrical stimulation control signals, and the electrical stimulation output unit is configured to combine the reference signal quantities, and non-invasively act on the nerve bundles and nerve endings of the user's stimulation sites. The image display layer is a display screen, the force detection layer comprises rigid force sensors or flexible force sensors, and the rigid force sensors are of a resistance type, a capacitance type, or a piezoelectric type.

2. The information interaction system according to claim 1, characterized in that, The reference electrode layer is made of transparent conductive materials.

3. The information interaction system according to claim 1, characterized in that, The classification model is a convolutional neural network, a depth-first search, a breadth-first search, or a Dijkstra algorithm image recognition processing model, the deep learning model is obtained through pre-training, and each sample in the training set reflects the corresponding relationship between sample force signal features, sample contact pressure information, and sample contact position information.

4. The information interaction system according to claim 1, characterized in that, ​ ​ 5. The information interaction system according to claim 1, characterized in that, ​ 6. The information interaction system according to claim 1, characterized in that, The stimulation site refers to the stimulation received by the receptors in the skin is transmitted to the nerve fibers connected with the receptors, thereby activating the nerve fibers to produce electrical activity, and then the electrical activity is transmitted to the brain along the upward sensory nerve transmission path, so that the brain produces a real feeling.

7. The information interaction system according to claim 3, characterized by The electrical stimulation control signal generated by the encoding information conversion unit is represented by stimulation characteristic parameters, including waveform, amplitude, width, interval, and different combinations of stimulation characteristic parameters correspond to stimulation current signals that induce the user's brain to generate different real feelings of knocking, vibration, tingling, and burning.

8. The information interaction system according to claim 2, characterized by The system is applied to information exchange, cognitive training, teaching guidance, game entertainment, or augmented reality scenarios.

9. A sensory enhanced information interaction method based on the information interaction system of any one of claims 1 to 8, comprising the following steps: The image is displayed to the user, and the force signal features are obtained through the interactive touch between the user and the image, and then the force signal features and the image feature signal are transmitted to the image content recognition and encoding module, and the reference signal quantity is transmitted to the sensory induction module; Based on the image feature signal, the image content information is obtained by using a classification model; based on the force signal features, the contact pressure information and the contact position information when the user touches the image are obtained by using a deep learning model; and then the sensory encoding information is generated according to the image content information, the contact pressure information and the contact position information; According to the sensory encoding information, an electrical stimulation signal capable of inducing the user's brain to produce a real feeling is generated, and is applied to the nerve bundle and nerve ending of the stimulation site of the user.

10. A computer readable storage medium having stored thereon a computer program, wherein, The computer program is executed by a processor to realize the steps of the information interaction method according to claim 9. The computer program is executed by a processor to realize the steps of the information interaction method according to claim 9.

Images