Information interaction system and method based on sensory enhancement

Abstract

An information interaction system and method based on sensory enhancement. In the system, an image display and feedback module obtains force signal features through interactive touch with a user, then transmits the force signal features and an image feature signal volume to an image content identification and encoding module, and transmits a reference signal volume to a sensory induction module. The image content identification and encoding module is configured to: on the basis of the image feature signals, use a classification model to obtain image content information; on the basis of the force signal features, use a deep learning model to obtain contact pressure information and contact position information of the user touching the image; and then on the basis of the image content information, the contact pressure information and the contact position information, generate sensory encoding information. The sensory induction module is configured to: on the basis of the sensory encoding information, generate an electrical stimulation signal capable of inducing the brain of the user to generate realistic sensations. The invention enhances the efficiency and accuracy of information interaction processes for users by means of sensory enhancement.

Description

An information interaction system and method based on sensory enhancement Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and more particularly, to an information interaction system and method based on sensory enhancement. Background Art

[0002] Traditional information interaction is generally achieved through vision, hearing, and other means. However, when the blind, partially sighted, deaf-mute, children, and adolescents are unable to obtain complete visual and auditory interaction information, they often interact through touch. In touch-based information interaction, relevant information is usually reflected through the three-dimensional structure or roughness of the surface of media such as Braille and images, or expressed through vibration feedback from the media. However, the three-dimensional structure and roughness of the surface of the media, or the vibration effects of the media feedback, do not allow users to interact with information efficiently and accurately. In particular, users with imperfect visual and auditory functions are unable to accurately obtain or feedback the information displayed by the media.

[0003] With the development of technologies such as cloud computing, big data, and artificial intelligence, touch interaction based on technologies such as virtual reality and augmented reality can provide users with a better information interaction experience. For example, patent application CN201910395324.2 provides an information sending method, a storage medium, and an electronic device. The method includes: when obtaining a first operation signal, determining a target touch electrode according to the first operation signal; outputting at least two preset electrical stimulation signals in sequence through the target touch electrode, each of the preset electrical stimulation signals corresponding to a user; when obtaining a second operation signal, determining the current preset electrical stimulation signal as a target preset electrical stimulation signal according to the second operation signal; obtaining a target user according to the target preset electrical stimulation signal, and sending information to the target user; by binding different electrical stimulation signals to different users, information can be quickly sent to the corresponding user, simplifying the information sharing process. However, the touch electrodes and electrical stimulation signals in this solution are used to feedback tactile signals, and only different electrical stimulation signals are used to allow users to feel different touch sensations.

[0004] Patent application CN201910400250.7 provides a control method, device, storage medium and electronic device for a touch screen. In this solution, when a touch instruction located in a first touch area is received, the touch electrodes in the first touch area are controlled to output corresponding electrical stimulation signals so that the user can feel the corresponding electrical stimulation touch sense and increase the diversity of touch sensing. However, this solution mainly enhances the user's touch sense of the touch screen through electrical stimulation, enriches the instructions of human-computer interaction through touch instructions, and its touch screen component structure only adds a layer of touch electrodes on the touch surface of the touch screen, and cannot obtain a real interactive feeling of the image content information through touch.

[0005] Patent application CN202010301023.1 discloses a system and method for using laser radar to transform an ordinary wall or ground into a touch screen, including a host, a power supply, a laser radar device, and a projector. The power terminals of the host, projector, and laser radar device are all electrically connected to the power output terminal through wires. The connection terminal of the laser radar device is connected to the connection terminal of the host via a network cable. The output terminal of the host is connected to the input terminal signal of the projector via a data cable. The transmitting end of the laser radar device emits a virtual touch screen. However, this solution only uses laser radar to detect gesture position, gesture action and other information to feedback the interactive action of touching the graphic content, and cannot enable the operator to feel the real feeling fed back by the image.

[0006] Patent application CN201310302106.2 proposes a multi-sensory interactive, multi-modal functional electrical stimulation system that organically combines functional electrical stimulation technology, assistive sports devices, the Internet of Things (IoT) technology, multi-sensory interactive games, and human vital sign parameters. The system comprises modules for functional electrical stimulation, vital sign parameter collection, assistive sports device information collection, information fusion, and IoT-based multi-sensory interactive game competitions. However, this solution only controls the functional electrical stimulation and the multi-sensory interactive game competition modules using human vital sign parameters and assistive sports device parameters. Not only is there no connection between the electrical stimulation and the multi-sensory interactive game, but the electrical stimulation used does not involve sensory feedback.

[0007] Patent application US12881376 discloses a touch panel module that can provide electrical stimulation sensory feedback. The module comprises a touch panel, a sensory feedback panel, a control unit, and a stimulation signal generation circuit. The sensory feedback panel is mounted on the touch panel and has multiple stimulation circuits arranged in a matrix or non-overlapping configuration. The stimulation signal generation circuit is connected to the stimulation circuit and the control unit. When the control unit receives a touch signal generated by the touch panel, the control unit controls the stimulation signal generation circuit to transmit a stimulation current to the stimulation circuit. When a user touches the stimulation circuit, the stimulation current flows through the user's finger to electrically stimulate the nerves, thereby achieving a sensory feedback effect and activating the function of the touch panel touched by the user. However, this solution simply integrates the stimulation circuit into the touch panel and provides feedback on the touch status by electrically stimulating the finger skin. It does not have the information interaction function of using the content on the touch panel to feedback the real feeling.

[0008] In summary, current touch-based information interaction typically uses the three-dimensional structure or roughness of media surfaces, such as Braille or images, to convey relevant information, or through vibration feedback from the media. However, users with limited visual and auditory functions cannot accurately capture or respond to the information presented by the media. For example, when touching a sharp object in a picture, the user cannot truly feel the stinging sensation caused by the sharp object; when touching a flame in a video, the user cannot truly feel the burning sensation caused by the flame. Technical issues

[0009] The purpose of the present invention is to overcome the above-mentioned defects of the prior art and provide an information interaction system and method based on sensory enhancement. Technical Solutions

[0010] According to a first aspect of the present invention, a sensory enhancement-based information interaction system is provided. The system comprises an image display and feedback module, an image content recognition and encoding module, and a sensory induction module, wherein:

[0011] The image display and feedback module is used to display images and obtain force signal characteristics through interactive touch with the user, and then transmit the force signal characteristics and image feature signal quantities to the image content recognition and encoding module, and transmit the reference signal quantity to the sensation induction module;

[0012] The image content recognition and encoding module is configured to: obtain image content information using a classification model based on the image feature signal; obtain contact pressure information and contact position information when a user touches the image using a deep learning model based on the force signal feature; and generate sensory encoding information based on the image content information, the contact pressure information, and the contact position information;

[0013] The sensory induction module is used to generate an electrical stimulation signal that can induce the user's brain to produce a real sensation based on the sensory coding information, and act on the nerve bundles and nerve endings at the stimulation site of the user;

[0014] Among them, the image display and feedback module is provided with an image display layer, a force detection layer and a reference electrode layer. The image display layer is used to display the image content and output the image feature signal to the image content recognition and encoding module; the force detection layer is used to detect the force when the user touches the image display layer content, and output the force signal feature to the image content recognition and encoding module; the reference electrode layer is used to transmit the reference signal quantity to the sensation induction module.

[0015] According to a second aspect of the present invention, a method for information interaction based on sensory enhancement is provided. The method comprises the following steps:

[0016] Displaying images to the user and acquiring force signal characteristics through interactive touch with the user, and then transmitting the force signal characteristics and image feature signal quantities to the image content recognition and encoding module, and transmitting the reference signal quantity to the sensation induction module;

[0017] Based on the image feature signal, a classification model is used to obtain image content information; based on the force signal feature, a deep learning model is used to obtain contact pressure information and contact position information when the user touches the image; and sensory encoding information is generated based on the image content information, the contact pressure information, and the contact position information.

[0018] Based on the sensory coding information, an electrical stimulation signal is generated that can induce the user's brain to produce real sensations, and acts on the nerve bundles and nerve endings at the stimulation site of the user. Beneficial effects

[0019] Compared with the existing technology, the advantage of the present invention is that the information interaction system and method based on sensory enhancement provided can enable users, including the blind, the visually impaired, the deaf-mute, children and adolescents, and those with normal visual and auditory functions, to enhance the user's acquisition and feedback of information through real sensations such as vibration, burning, and tingling feedback from the media when touching images and other media, thereby improving the user's efficiency and accuracy in the information interaction process in a sensory enhancement manner.

[0020] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0022] FIG1 is a schematic diagram of an information interaction system based on sensory enhancement according to an embodiment of the present invention;

[0023] FIG2 is a schematic diagram of signal output of an image display and feedback module according to an embodiment of the present invention;

[0024] FIG3 is a flow chart of signal processing of an image content recognition and encoding module according to an embodiment of the present invention;

[0025] FIG4 is a schematic diagram of the composition and signal relationship of a sensory induction module according to an embodiment of the present invention. Best Mode for Carrying Out the Invention

[0026] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention.

[0027] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0028] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0029] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0030] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0031] The present invention proposes an information interaction solution based on sensory enhancement. According to the recognition and encoding of the touch image content, an electrical stimulation signal corresponding to the encoded information is generated to induce the user's brain to generate real sensations such as vibration, tingling, burning, etc. corresponding to the image content.

[0032] As shown in Figure 1, the provided sensory enhancement-based information interaction system includes an image display and feedback module, an image content recognition and encoding module, and a sensory induction module. The image display and feedback module is used for image display and user touch interaction, while the image content recognition and encoding module is used to identify the image content displayed in the image display and feedback module and generate sensory induction coded information. The sensory induction module generates electrical stimulation signals based on the coded information from the image content recognition and encoding module, which can induce real sensations such as vibration, tingling, and burning in the user's brain and apply them to the user's wrist and other stimulation sites.

[0033] In one embodiment, the image display and feedback module comprises an image display layer, a force detection layer, and a reference electrode layer. The image display layer and the force detection layer are both connected to the image content recognition and encoding module, while the reference electrode layer is connected to the sensation induction module. It should be noted that the images in the information interaction system of the present invention include, but are not limited to, images such as pictures and videos.

[0034] The image display layer is used to display image content, including but not limited to LED, LCD, and other display screens. The force detection layer is used to detect the force applied when a user touches the image display layer, including but not limited to rigid or flexible force sensors such as resistors, capacitors, and piezoelectric sensors. The reference electrode layer is used to provide a reference signal to the sensory induction module when a user touches the image content, including but not limited to transparent conductive materials such as indium tin oxide.

[0035] Figure 2 is a schematic diagram of the signal output of the image display and feedback module. In the image display and feedback module, the image content in the image display layer is processed, including but not limited to grayscale, binarization, denoising, tilt correction, character cutting, normalization, etc., to output image feature signals. Image feature signals include but are not limited to color feature C O , shape feature S H , texture feature T E , spatial relationship feature S R For example, the image feature signal output by the image display layer is expressed as The force detection layer outputs the force signal when the user touches the module, including but not limited to the force peak F P , force changes , finger touch position , for example, the force signal output by the force detection layer is The reference electrode layer outputs a reference signal quantity, Rs, which provides a signal reference for the sensation induction module. For example, the reference signal quantity Rs can ensure that the information interaction system and the human body have the same reference potential, thereby forming an input-output loop when the sensation induction module is operating, thereby achieving effective electrical stimulation. It can also reduce interference from human static electricity on the information interaction system.

[0036] Figure 3 is a signal processing flow chart of the image content recognition and coding module. The image content recognition and coding module is used to recognize the image content displayed in the image display and feedback module and generate sensory induced coding information. First, the image content recognition and coding module generates sensory induced coding information based on the image feature signal quantity transmitted from the image display and feedback module. , using image recognition processing methods (or classification models) including but not limited to convolutional neural networks, depth-first search, breadth-first search, Dijkstra algorithm, etc., to classify and identify the image content displayed in the image display and feedback module to obtain the content information C of the displayed image I At the same time, the image content recognition and encoding module will be based on the force signal transmitted by the image display and feedback module. , combined with deep learning models and other processing methods, to obtain the contact pressure information F when the user touches the image T and contact position information P T ; Finally, according to the image content information C I , contact pressure information F T and contact position information P T After being processed by the sensory information coding algorithm, sensory coding information S is generated C .

[0037] For example, the process of using a convolutional neural network model to classify and identify image content is as follows: pre-training the convolutional neural network model, where each sample in the training set reflects the correspondence between the image feature signal and the image category; inputting the image feature signal acquired in real time into the pre-trained convolutional neural network model to obtain the corresponding image category. For another example, using a deep learning model to obtain the contact pressure information F when the user touches the image T and contact position information P T When pre-training the deep learning model, the training set reflects the correspondence between the sample force signal characteristics and the sample contact pressure information and the sample contact position information.

[0038] In one embodiment, the sensory encoding information S is generated C The process is a multimodal signal fusion process, which generates sensory coding information S by fusing multiple modal information such as image and force. C Specifically, the image content information C I , including the spatial information of the image displayed in the image display layer, the image object attributes, such as the image display layer showing a sharp cone object, the image content information C I It will include the spatial position information of the cone object, the contour information of the tip, the hardness of the physical material and other parameters; the contact pressure information F T and contact position information P TThese are interactive attribute parameters fed back by the force detection layer when the user touches the image in the image display layer, such as the force applied by the user when touching the image in the image display layer, the position information when the user touches the image in the image display layer during the touch process, etc. The spatial position information of the image content fed back by the image display layer corresponds to the spatial position information of the contact force fed back by the force detection layer, and the image content is associated with the spatial position of the image in the image display layer. Therefore, the image content information C is converted into the image content information according to the logic and association when the image is touched. I , contact pressure information F T and contact position information P T After multimodal fusion of information such as C .

[0039] Figure 4 shows the composition and signal relationship of the sensory induction module. Based on the image content recognition and encoding information from the encoding module, the sensory induction module generates electrical stimulation signals that can induce real sensations such as vibration, tingling, and burning in the user's brain, and then acts on the user's wrist and other stimulation sites.

[0040] In one embodiment, the sensory induction module includes a coding information conversion unit, a stimulation current generation unit, and an electrical stimulation output unit. The coding information conversion unit is used to convert the image content recognition into the sensory coding information S in the coding module. C Converted into electrical stimulation control signal U S Stimulation current generating unit is used according to the electrical stimulation control signal U S Generate stimulation current signal I S ; The signal reference R in the electrical stimulation output unit combined with the image display and feedback module S , through the signal electrode, wearable devices, etc., the stimulation current signal I S Acting on the wrist and other related stimulation parts. In the present invention, the user feels the real sensations such as vibration, stimulation, burning, etc. corresponding to the image content, not by stimulating the contact part with electric current to make the skin vibrate, tingle, burn, etc., but by stimulating the nerve bundles and nerve endings in the wrist and other related parts, inducing the user's brain to produce real sensations such as vibration, tingle, burning, etc. on the non-stimulated parts such as the fingers touching the image content. Therefore, the electrical stimulation control signal U generated by the coded information conversion unit in the sensation induction module is S The stimulation characteristic parameters include but are not limited to waveform, amplitude, width, interval, etc., and the combination of different stimulation characteristic parameters corresponds to the stimulation current signal I that induces the user's brain to generate different real sensations such as vibration, tingling, burning, etc. S .

[0041] It should be noted that the sensory induction in the present invention refers to the sensory receptors in the skin transmitting the electrical stimulation received to the nerve fibers connected to the receptors, thereby activating the nerve fibers to generate electrical activity, which is then transmitted along the upward sensory nerve transmission pathway to the brain, inducing the brain to produce proprioception such as vibration and tingling, that is, real feelings. However, the existing ordinary electrical stimulation signals cannot induce the brain to produce natural intuitive feelings. For example, under different stimulation current signals I S When the user's brain is stimulated by a current with a pulse interval of 200 μs, a pulse width of 200 μs, a current amplitude of 1 mA, and a frequency of 50 Hz, the user's middle finger, ring finger, and the corresponding palm position below the wrist will feel a real sense of tapping; when the user's wrist ulnar nerve is stimulated by a current with a pulse interval of 200 μs, a pulse width of 200 μs, a current amplitude of 0.75 mA, and a frequency of 200 Hz, the user's thumb palm and thenar position will feel a real sense of vibration; when the user's wrist radial nerve is stimulated by a current with a pulse interval of 200 μs, a pulse width of 350 μs, a current amplitude of 3 mA, and a frequency of 200 Hz, the user's little finger will feel a real sense of tingling. When the above different stimulation current signals I S By collecting the user's EEG maps and ERP (event-related potential) curves, the user's real feelings during the stimulation can be objectively verified as the natural intuitive feelings induced by the user's brain. For example, when electrical stimulation successfully induces the user's natural intuitive feelings, the EEG map shows that the activity of the scalp area corresponding to the user's motor sensory cortex is significantly increased within a specific time period. At the same time, there are obvious peaks in the ERP curves of specific EEG channels (especially the FZ, F2, FCZ, FC2, CZ, C2, and C4 channels). The ERP peak occurs within 200-300 ms after the onset of stimulation, which is consistent with the time response range of sensory induced by ERP.

[0042] After demonstration and preliminary experiments, the information interaction system based on sensory enhancement provided by the present invention can be applied to scenarios such as information exchange, cognitive training, teaching guidance, game entertainment, and augmented reality. Modes for Carrying Out the Invention

[0043] Example 1

[0044] The information interaction system provided by the present invention enables users with impaired visual function, such as the blind and the visually impaired, to read Braille or browse pictures. For example, after the blind and the visually impaired wear relevant signal electrodes on their wrists, the user will feel the real sensation of touching the Braille or picture content when their fingers touch the Braille or picture content displayed on the image display and feedback module of the information interaction system. First, the information interaction system identifies and encodes the Braille or picture content; then, when the user touches the Braille or picture, the information interaction system generates an electrical stimulation signal corresponding to the touching Braille or picture content based on the coded information of the content at the position touched by the user, and acts on the user's wrist and other parts through the electrical stimulation output unit of the sensory induction module, so that the user can feel the real sensations such as vibration, tingling, burning, etc. corresponding to the Braille or picture content they touch, thereby enhancing the reading ability and interactive experience of the blind and the visually impaired in obtaining text and picture information. For example, when blind people, people with low vision and other users touch the Braille that means "fire", they can feel the real sensations of burning and stinging when touching the flame, allowing users to understand and recognize the meaning and characteristics of "fire" more quickly.

[0045] Example 2

[0046] The information interaction system provided by the present invention can teach users with hearing impairments, such as the deaf-mute, and users with imperfect visual and auditory functions, such as children and adolescents, about fire safety knowledge, including first aid. For example, after a deaf-mute, child, or adolescent wears relevant signal electrodes on their wrist, similar to an "audiobook" function, when the user's finger touches the fire alarm and first aid related images or video content displayed on the image display and feedback module of the information interaction system, they will feel the corresponding real-life sensations of the fire alarm and first aid related images or video content, thus providing a "tactile reading" experience with realistic sensory feedback, just like reading an "audiobook." First, the information interaction system identifies and encodes fire alarm and first aid related pictures or video content; then, when the user touches the pictures or video content of fire alarm and first aid related knowledge, the information interaction system will generate an electrical stimulation signal corresponding to touching the fire alarm and first aid related pictures or video content according to the coded information of the content at the position touched by the user, and act on the user's wrist through a bracelet worn on the user's wrist and equipped with the sensory induction module function of this system, so that the user can obtain real sensations such as vibration, tingling, burning, etc. corresponding to the content they touch, thereby enhancing the experience and learning efficiency of the deaf and mute people, children and adolescents in learning new things.

[0047] Example 3

[0048] The information interaction system provided by the present invention enables users with normal visual and auditory functions to experience interactive browsing of images or videos with realistic sensory feedback. Specifically, after a user with normal visual and auditory functions wears relevant signal electrodes on their wrist, the user will experience the realistic sensations corresponding to the image or video content when their finger touches the image or video displayed on the image display and feedback module of the information interaction system. First, the information interaction system identifies and encodes the image or video content. Then, when the user touches the image or video, the information interaction system generates an electrical stimulation signal corresponding to the touching image or video content based on the encoded information of the image or video content. The electrical stimulation output unit of the sensory induction module of the system acts on the relevant stimulation area of ​​the user, thereby inducing the user's brain to produce realistic sensations such as vibration, tingling, and burning corresponding to the image or video content they touched, thereby allowing the user to experience the realistic sensory experience corresponding to the image or video content. According to the above information interaction process, users with normal visual and auditory functions can not only achieve a tactile experience with realistic sensory feedback, but also realize highly efficient hand touch interaction without looking at the interactive object, thus allowing users to achieve efficient information interaction under "eye-hand separation" conditions.

[0049] Example 4

[0050] The information interaction system provided by the present invention enables electronic game players to experience gaming with realistic sensory feedback. Specifically, after a gamer with normal or impaired visual and auditory function wears a wristband equipped with signal electrodes, the player will experience realistic sensations corresponding to the game screen content when their finger or palm touches the game screen displayed on the image display and feedback module of the information interaction system. First, the information interaction system identifies and encodes the game screen content. Then, when the gamer touches the game screen, the information interaction system generates an electrical stimulation signal corresponding to the touching of the game screen content based on the encoded information of the game screen content. The electrical stimulation output unit of the sensory induction module of the system acts on the relevant stimulation area of ​​the gamer, thereby inducing the gamer's brain to produce realistic sensations such as vibration, tingling, and burning corresponding to the touched game screen content, thereby allowing the gamer to experience electronic game playing with realistic sensory feedback. According to the above information interaction process, regardless of whether the visual and auditory functions are impaired, game players can obtain a real sensory experience of feedback on the game screen content during the game. For example, when game players touch the knife stabbing picture in the game, they can feel the real pain caused by the knife stabbing, thereby allowing game players to obtain a more realistic gaming experience.

[0051] In summary, the sensory-enhanced information interaction system and method proposed in the present invention can enhance information acquisition and feedback for users, including the blind, the visually impaired, the deaf-mute, children and adolescents, and those with normal vision and hearing, by providing realistic sensations such as vibration, burning, and tingling feedback when touching media such as images. For example, when touching image media such as pictures and videos, users experience realistic tingling and burning sensations caused by sharp objects, flames, and other objects in the image content impacting the touched area. This sensory enhancement improves the efficiency and accuracy of information interaction.

[0052] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0053] A computer-readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure within a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0054] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0055] The computer program instructions for performing the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, and conventional procedural programming languages ​​such as "C" or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, the state information of the computer-readable program instructions is used to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), so that the electronic circuit can execute the computer-readable program instructions, thereby implementing various aspects of the present invention.

[0056] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0057] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0058] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0059] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of an instruction, and the module, program segment or part of the instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

[0060] While various embodiments of the present invention have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. An information interaction system based on sensory enhancement, comprising an image display and feedback module, an image content recognition and encoding module, and a sensory induction module, wherein: The image display and feedback module is used to display images and obtain force signal characteristics through interactive touch with the user, and then transmit the force signal characteristics and image feature signal quantity to the image content recognition and encoding module, and transmit the reference signal quantity to the sensation induction module; The image content recognition and encoding module is used to: obtain image content information using a classification model based on the image feature signal; obtain contact pressure information and contact position information when the user touches the image using a deep learning model based on the force signal feature; and then generate sensory encoding information based on the image content information, the contact pressure information and the contact position information; The sensory induction module is used to generate an electrical stimulation signal that can induce the user's brain to produce a real feeling according to the sensory coding information, and act on the nerve bundles and nerve endings at the stimulation site of the user; Among them, the image display and feedback module is provided with an image display layer, a force detection layer and a reference electrode layer. The image display layer is used to display the image content and output the image characteristic signal to the image content recognition and encoding module; the force detection layer is used to detect the force when the user touches the image display layer content, and output the force signal characteristics to the image content recognition and encoding module; the reference electrode layer is used to transmit the reference signal quantity to the sensory induction module, and the reference signal quantity makes the information interaction system have the same reference potential as the human body, so that the input-output loop is formed when the sensory induction module is working.

2. The information interaction system according to claim 1, characterized in that: The image feature signal includes color feature, shape feature, texture feature and spatial relationship feature, and the force signal feature includes force peak value, force variation and finger touch position.

3. The information interaction system according to claim 1, characterized in that: The sensory induction module includes a coding information conversion unit, a stimulation current generating unit and an electrical stimulation output unit, wherein the coding information conversion unit is used to convert the sensory coding information into an electrical stimulation control signal; the stimulation current generating unit is used to generate a stimulation current signal according to the electrical stimulation control signal; and the electrical stimulation output unit is used to combine the reference signal quantity to non-invasively apply the stimulation current signal to the nerve bundles and nerve endings at the stimulation site of the user.

4. The information interaction system according to claim 1, characterized in that: The image display layer is a display screen; the force detection layer is provided with a rigid force sensor or a flexible force sensor, and the rigid force sensor is a resistance type, a capacitance type or a piezoelectric type; The reference electrode layer is made of transparent conductive material.

5. The information interaction system according to claim 1, characterized in that: The classification model is a convolutional neural network, depth-first search, breadth-first search or Dijkstra algorithm image recognition processing model; the deep learning model is obtained through pre-training, and each sample in the training set reflects the correspondence between the sample force signal characteristics and the sample contact pressure information and the sample contact position information.

6. The information interaction system according to claim 1, characterized in that: The stimulation site refers to the point where the stimulation is transmitted to the nerve fibers connected to the receptors through the skin, thereby activating the nerve fibers to produce electrical activity. The electrical activity is then transmitted to the brain along the upward sensory nerve transmission pathway, causing the brain to produce real sensations.

7. The information interaction system according to claim 3, characterized in that: The electrical stimulation control signal generated by the coding information conversion unit is characterized by stimulation characteristic parameters, which include waveform, amplitude, width, and interval, and the combination of different stimulation characteristic parameters corresponds to stimulation current signals that induce the user's brain to generate different real sensations of knocking, vibration, tingling, and burning.

8. The information interaction system according to claim 2, characterized in that: The system is used for information exchange, cognitive training, teaching guidance, game entertainment, and augmented reality scenarios.

9. A sensory enhanced information interaction method based on the information interaction system according to any one of claims 1 to 8, comprising the following steps: Displaying images to the user, and acquiring force signal characteristics through interactive touch with the user, and then transmitting the force signal characteristics and image feature signal quantities to the image content recognition and encoding module, and transmitting the reference signal quantity to the sensation induction module; Based on the image feature signal, the image content information is obtained by using a classification model; based on the force signal feature, the contact pressure information and the contact position information when the user touches the image are obtained by using a deep learning model; and then the sensory coding information is generated according to the image content information, the contact pressure information and the contact position information; Based on the sensory coding information, an electrical stimulation signal is generated that can induce the user's brain to produce real sensations, and acts on the nerve bundles and nerve endings at the stimulation site of the user.

10. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the information interaction method according to claim 9 are implemented.

Images