Method for training bionic vision in virtual reality

Abstract

The invention relates to the field of neurosurgical implants, and more particularly to technologies for the functional rehabilitation of patients fitted with cortical visual prosthesis systems. The present method includes the steps of reading the geometric parameters of an environment, transmitting said parameters to a personal computer on which editable virtual rooms are created, transmitting data about said virtual rooms to a computing unit of a cortical prosthesis system equipped with software that makes it possible to control virtual rooms and to visualize objects with the aid of bionic vision, immersing a user into a virtual room, continuously capturing data about the user's position in space and synchronizing said data on the computing unit with a digital copy of the user in the virtual room, and having the user perform exercises of a rehabilitation programme using virtual reality control devices. The technical result of the invention consists in reducing rehabilitation times for users of cortical visual prosthesis systems, increasing the safety of the rehabilitation process and broadening the functionality of systems for adapting to bionic vision.

Description

[0001] A METHOD FOR TRAINING BIONIC VISION IN VIRTUAL REALITY

[0002] The claimed invention relates to the field of neurosurgical implants, namely to technologies for the functional rehabilitation of patients using cortical visual prosthetic systems, and can be implemented in systems intended for use by people with disabilities in order to improve the perception and recognition of the surrounding space by the user.

[0003] Cortical neuroimplants are systems that can replace motor, sensory, or cognitive functions that may have been damaged by injury or disease. Cortical neuroimplants are located in direct communication with the cerebral cortex and can consist of a combination of external and internal (implantable) devices. By directly interacting with various areas of the cerebral cortex, a cortical neuroimplant can provide stimulation to the immediate area and provide various benefits depending on its design and placement. One type of cortical neuroimplant is a cortical visual prosthesis system, responsible for restoring visual function in profoundly blind individuals.

[0004] However, using a cortical visual prosthesis system requires a rather challenging adaptation period for the patient, during which they learn new bionic vision, including exploring various spaces in which they are expected to navigate independently, such as the geometric features of their own apartment or house, stores, libraries, residential buildings, and so on. Exploring a particular public space can present organizational and functional challenges, as during the initial familiarization phase, there should be no moving objects (people, animals, cars, etc.) nearby. Initially, the user should familiarize themselves with stationary elements of the space (walls, cash registers, kiosks, etc.) to avoid confusing them with moving ones in real-world conditions.Navigating through a given space can be challenging due to the technical limitations of visual prosthetic systems and the bionic vision they provide to the user. For example, due to limited spatial resolution (primarily dependent on the number of electrodes), identifying details can be difficult. Due to a lack of color or, if object contouring algorithms are used during the preprocessing stage, a lack of grayscale in the background, objects in the environment may be perceived as a single object, although in reality they are a conglomerate of several. This situation reduces the overall safety of cortical visual prosthetic systems, as the user may not see a potentially dangerous object or part of one. It can also cause difficulties with orientation, reducing overall user satisfaction and socialization.

[0005] To date, there are practically no known solutions in the state of the art for improving the quality of adaptation and reducing the rehabilitation period for users of cortical visual prosthetic systems.

[0006] The closest approach to the claimed technical solution is a method for implementing visualization for the blind using virtual reality technology (patent application CN 104510568 A, published April 15, 2015). This method transmits information about the surrounding space to the patient through brain stimulation with electrodes, with the source of the information being a helmet with various sensors. This solution is used to create a three-dimensional scene of the environment in real time, which is used to generate a stimulating signal to the electrode brain matrix. A disadvantage of this solution is the inability to use three-dimensional models for static training of the user of a cortical prosthetic system in safe conditions. Furthermore, the created models cannot be adjusted, and there is no guarantee that they are accurate and the user can trust them in crowded areas, around cars, and other potentially dangerous situations.

[0007] The objective of the claimed invention is to develop a method for creating editable three-dimensional models of various locations in which a user of a cortical visual prosthetic system may be located, for orientation training in safe and controlled conditions as part of rehabilitation programs.

[0008] The technical result of the claimed invention consists in reducing the rehabilitation period for users of cortical visual prosthetic systems, increasing the safety of the rehabilitation process, and expanding the functionality of bionic vision adaptation systems. This technical result is achieved by a method for teaching bionic vision in virtual reality, consisting of stages in which the geometric parameters of the space surrounding the user of the cortical visual prosthetic system are read using technical means to obtain information about the surrounding space, and transmitted to a personal computer, where, using software, virtual rooms with the ability to edit are created;transmit data about virtual rooms from a personal computer to the computing unit of the cortical visual prosthetic system with pre-installed software that provides the ability to control virtual rooms and visualize objects located in them using bionic vision;immerse the user of the cortical visual prosthetic system in a selected virtual room, continuously record data on the user's position in space using sensors for the position of parts of his body, synchronizing on the computing unit of the cortical visual prosthetic system the information on the position of the user's body in space with its digital copy in the virtual room, while the user of the cortical visual prosthetic system, surrounded by virtual objects, performs exercises of the rehabilitation program through scenarios simulating adapted everyday actions in the form of an individual or multi-user mode, using control tools in virtual reality.

[0009] Controllers or biological control instruments can be used as control instruments in virtual reality. Biological control instruments can include a voice control device, an electroencephalograph, an eye tracker, or any other similar solution.

[0010] Below we will examine in detail the terms and their definitions used in the description of the technical solution.

[0011] The cortical visual prosthetic system is a hardware and software complex that replaces lost visual functions by processing images from a camera and subsequently stimulating the visual cortex of the brain.

[0012] Bionic vision is vision that functions as a result of the cortical visual prosthetic systems. A virtual room is a digital three-dimensional model of an enclosed space and three-dimensional models of objects located within it.

[0013] The hardware basis of the proposed bionic vision training method is a computing unit of a cortical visual prosthetic system with installed software containing virtual rooms and controllers or biological control tools that enable the user's movement within the virtual space. These controllers immerse the user in an immersive virtual environment where they are present in the virtual room, can see digital copies (avatars) of other users (if multiple users are participating in the training), and perform exercises under the remote supervision of a rehabilitation instructor. The virtual environment replicates the user's familiar environment (apartment, store, gym, etc.).

[0014] The method for teaching bionic vision to a user of cortical visual prosthetic systems in virtual reality consists of several stages and is implemented as follows.

[0015] At the initial stage, a virtual copy of any space surrounding the user of the cortical visual prosthesis is created on the computer of a rehabilitation therapist or rehabilitation center engineer using any known method. For this purpose, the surrounding space can be reconstructed manually from scratch, based on the room layout and using software for modeling 3D objects or information about the geometric data of the space, obtained by any technical means for obtaining information about the surrounding space, such as a laser scanner, a laser locator (LIDAR), a camera, an ultrasound scanner, and other similar devices. This software can also be used to create and edit 3D models based on the data obtained from such a technical means (including intelligent algorithms that create 3D reconstructions from photographs or video frames).

[0016] Next, the files with the resulting 3D objects (the virtual room and the objects within it) are transferred to pre-installed computer software, which creates "rehabilitation virtual space" files containing a digital copy of the cortical visual prosthesis user. Rehabilitation virtual spaces are virtual rooms with 3D reconstructions of spaces that allow the user to navigate and interact with virtual objects within them, which can be further modified if necessary.

[0017] Connected to the computer via a communication channel is a computing unit of the cortical visual prosthetic system with pre-installed software capable of loading files of rehabilitation virtual spaces from the computer, or already containing virtual rooms with virtual objects located within them, as well as at least one tool for controlling the user's movement within the virtual room, connected via a communication channel to the computing unit of the cortical visual prosthetic system. The software pre-installed on the computing unit is capable of controlling virtual rooms (their selection, launching and exiting, selecting lighting conditions, and any other settings), as well as visualizing objects located within them using bionic vision.Using virtual reality control tools (controllers or biocontrol devices) connected via a communication channel to the computing unit of the cortical visual prosthetic system, the user performs actions prescribed by the rehabilitation program. Known technical solutions for controlling actions in virtual reality are used as controllers, such as gloves with sensors and external video cameras, including those controlled by artificial intelligence. When using virtual rooms in spacious spaces monitored by rehabilitation specialists (for example, in specialized rehabilitation centers), controllers can be replaced with external tracking sensors that enable spatial location of the person. A voice control device, an electroencephalograph (hereinafter referred to as an EEG device), an eye tracker, or any other similar solution are used as biocontrol tools.Several computing units of cortical visual prosthetic systems can be connected via a communication channel to the computer used to create and edit virtual rooms, thereby enabling bionic vision training for multiple users of visual prosthetic systems. Moreover, different virtual rooms can be installed on the computing units of different users. To ensure the accuracy and reliability of bionic vision training in virtual rooms, information about the spatial position of the actual user's body parts and their digital counterpart in the virtual room must be synchronized. For this purpose, sensors for tracking the position of body parts (e.g., external video cameras, scanners, wearable sensors, etc.) are placed outside the user's body and connected via a communication channel to the computing unit of the prosthetic system.The sensors can be used in various variations depending on the specific configuration of the cortical visual prosthetic system.

[0018] The bionic vision training process is as follows. The user of a cortical visual prosthesis is immersed in a virtual environment displaying a digital copy of the user and tracking the progress of exercises in training scenarios. The user is then asked to complete tasks outlined in the rehabilitation program. In the virtual room, the user of the cortical visual prosthesis sees their digital copy, as well as digital copies of other users in the case of a group rehabilitation session. Using a controller or biological control instruments, the user manipulates objects in the virtual room according to the specified tasks. Sensors for tracking the user's body position continuously record changes in the position of their body parts.Signals from these sensors are sent to the computing unit, where, using software, the corresponding changes in the position of body parts are synchronized with a digital copy of the user.

[0019] Virtual rooms are configured and managed using software installed on the computer. To do this, the necessary changes are made to existing virtual rooms on the computer, after which information about the adjusted virtual rooms is transmitted via a communication channel to the computing unit.

[0020] The proposed invention, when implemented, enables the creation of editable virtual spaces of various locations (e.g., apartments, stores, etc.) in which a user of a cortical visual prosthesis system can reside. These virtual spaces are used for orientation training in safe and controlled environments (e.g., at home or in a rehabilitation center) as part of rehabilitation programs.

Claims

CLAUSES OF THE INVENTION 1. A method for teaching bionic vision in virtual reality, comprising the steps of a) reading the geometric parameters of the space surrounding the user of the cortical visual prosthetic system by means of technical means for obtaining information about the surrounding space, transmitting them to a personal computer, where virtual rooms with the ability to edit are created using software, b) transmitting data about the virtual rooms from the personal computer to the computing unit of the cortical visual prosthetic system with pre-installed software that provides the ability to control the virtual rooms and visualize objects located in them using bionic vision, c) immersing the user of the cortical visual prosthetic system in the selected virtual room, continuously recording data about the user's position in space using sensors for the position of parts of his body,synchronizing on the computing unit of the cortical visual prosthetic system the information about the user's body position in space with its digital copy in the virtual room, while the user of the cortical visual prosthetic system, surrounded by virtual objects, performs the tasks of the rehabilitation program through scenarios simulating adapted everyday actions in the form of an individual or multi-user mode, using control tools in virtual reality.

2. The method according to paragraph 1, characterized in that controllers or biological control instruments are used as control instruments in virtual reality.

3. The method according to paragraph 2, characterized in that a voice control device, or an electroencephalograph, or an eye tracker, or any other similar solution is used as a biological control instrument. 7 SUBSTITUTE SHEET (RULE 26)