Method for adapting an environment for users of visual prosthesis systems

Abstract

The invention relates to technologies for the functional rehabilitation of patients fitted with cortical visual prosthesis systems. The method includes the steps of reading geometric parameters of an environment using technical means in order to obtain information about said environment, and transmitting said parameters over a communication channel to a computer, where three-dimensional space representations are created; transmitting two-dimensional images from a video camera to the computer, and reducing said images to the resolution determined by a visual prosthesis system; comparing on the computer the three-dimensional space representations and the images from the video camera to identify regions of the spaces which are invisible to bionic vision; editing the three-dimensional space representations and providing recommendations for adjusting the user's environment. The technical result of the invention consists in reducing rehabilitation times for users of visual prosthesis systems, increasing the safety of the rehabilitation process and broadening the functionality of systems for adapting to bionic vision.

Description

[0001] METHOD OF ADAPTATION OF THE ENVIRONMENT FOR USERS OF VISUAL PROSTHETIC SYSTEMS

[0002] The proposed method relates to the field of neurosurgical implants, namely to technologies for the functional rehabilitation of patients using cortical visual prosthetic systems and can be implemented in systems intended for use by people with disabilities, with the aim of improving the perception and recognition of the surrounding space by the user.

[0003] Cortical neuroimplants are systems that can replace human motor, sensory, or cognitive functions that may have been damaged by injury or disease. Cortical neuroimplants are located in direct communication with the cerebral cortex and may consist of a combination of external and internal (implantable) devices. By directly interacting with different areas of the cerebral cortex, a cortical neuroimplant can provide stimulation to the immediate area and provide different benefits depending on its design and placement. One type of cortical neuroimplant is the cortical visual prosthesis system, which is responsible for restoring visual function in totally blind people.

[0004] However, the use of a cortical visual prosthetic system requires a rather difficult adaptation period for the patient, during which he learns new bionic vision, including studying various spaces in which his independent orientation is assumed, for example, the geometric features of his own apartment or house, shops, libraries, residential buildings, etc. Studying a particular public space can be associated with organizational and functional difficulties, since at the stage of familiarization with any space there should be no moving objects nearby (people, animals, cars, etc.). At the initial stage, the user must familiarize himself with and study the stationary elements of the space (walls, cash desks, kiosks, etc.), so as not to confuse them with moving ones in real conditions.Moving around a particular space may be associated with problems related to the technical limitations of visual prosthetic systems and the bionic vision that it provides to the user. Thus, due to the limited spatial resolution (depending primarily on the number of electrodes), identification of details may be difficult, and due to the lack of color or, if object contouring algorithms are used at the pre-processing stage, the lack of gray gradations in the background, objects in the environment may be perceived as one object, although in reality they are a conglomerate of several. This situation reduces the overall safety level of using cortical visual prosthetic systems, because the user may not see a dangerous object or part of it. It may also cause difficulties in orientation, reducing the overall level of satisfaction with the user experience and the level of socialization.

[0005] The state of the art includes solutions aimed at creating three-dimensional models from real objects in the environment, be it household items, buildings, equipment, etc. Today, there are many technical approaches and their combinations in this area, and each of them solves the problem of recreating real objects in the form of three-dimensional digital models in its own way.

[0006] For example, patents US10823955B2 (Grating-based spatial mode filter for laser scanning, published 03.11.2020) and US9523850B2 (Beam scanning using an interference filter as a turning mirror, published 20.12.2016) describe Apple's solutions with original LiDAR technology, which is used to scan space by processing the reflected optical signal in real time. A similar principle underlies the method of scanning space and methods using other types of waves, such as ultrasound.

[0007] Patent applications EP2852932A1 (A method and a system for generating a realistic 3D reconstruction model for an object or being, published 01.04.2015) and EP4266257A1 (3D reconstruction from images, published 25.10.2023) describe another approach for 3D reconstruction of virtual objects from real ones - based on their images. In this case, an array of images of a particular object, taken from different sides and at different angles relative to the object, is processed in order to record the features of all its sides.

[0008] The disadvantage of all the presented and similar solutions in relation to the proposed method is the limited functionality until the 3D model is created, which does not allow them to be used to improve the visibility of surrounding objects with bionic vision when using cortical visual prosthetic systems. The objective of the claimed invention is to create a method for creating editable three-dimensional models of various locations to identify objects and zones in various locations that are difficult to perceive with bionic vision for the purpose of their adaptation or modification within the framework of clinical and functional rehabilitation programs for users of cortical visual prosthetic systems.

[0009] The technical result of the claimed invention consists in reducing the rehabilitation period for users of cortical visual prosthetic systems, increasing the safety of the rehabilitation process, and expanding the functionality of systems for adaptation to bionic vision.

[0010] The technical result is achieved by a method of adapting the surrounding space for users of visual prosthetic systems and includes the stages of reading the geometric parameters of the space surrounding the user of the cortical visual prosthetic system by means of technical means for obtaining information about the surrounding space, transmitting them via a communication channel to a personal computer, where three-dimensional images of spaces are created using software; transmitting two-dimensional images from a video camera to a personal computer, where they are reduced using software to the resolution provided by the visual prosthetic system; comparing three-dimensional images of spaces and images obtained from a video camera on a personal computer, revealing areas of spaces invisible to bionic vision using software;after which they edit three-dimensional images of spaces using software with the ability to create objects visible to bionic vision, and provide recommendations for making changes to the space surrounding the user of the cortical visual prosthetic system.

[0011] A laser scanner, or a laser locator, or a photo camera, or an ultrasonic scanner, or any other similar solution can be used as technical means for obtaining information about the surrounding space.

[0012] At the stage of reducing two-dimensional images to the resolution provided by the visual prosthetic system, image data can be converted from the "color" mode to the "grayscale" and "binary black and white" modes. Intelligent algorithms can be used as software for creating and editing three-dimensional models of virtual spaces.

[0013] At the stages of reading the geometric parameters of the surrounding space and editing three-dimensional images of spaces, they can interact with three-dimensional images of spaces using a monitor and input devices connected to a personal computer, or portable electronic devices, or virtual reality devices.

[0014] Below we will examine in detail the terms and their definitions used in the description of the technical solution.

[0015] The cortical visual prosthetic system is a hardware and software complex that is used to replace lost visual functions by processing images from a camera and subsequently stimulating the visual cortex of the brain.

[0016] Bionic vision is vision that functions as a result of the work of cortical visual prosthetic systems.

[0017] A virtual room is a digital three-dimensional model of a closed space and the objects located inside it.

[0018] The method of adapting the surrounding space for users of visual prosthetic systems consists of several stages and is implemented as follows.

[0019] At the initial stage, a virtual copy of any space surrounding the user of the cortical visual prosthetic system is created on the computer of the rehabilitation specialist or the rehabilitation center engineer using any of the known methods. For this purpose, the surrounding space can be reconstructed manually from scratch, i.e. based on the layout of the room and using software for modeling three-dimensional objects or information about the geometric data of the space received on the computer using any technical means for obtaining information about the surrounding space, such as a laser scanner, laser locator (LIDAR), camera, ultrasound scanner and others like that, as well as software that allows, based on the data received from such a technical means, to create and edit three-dimensional models (including intelligent algorithms that create three-dimensional reconstructions from photographs or video frames).

[0020] Then the files with the obtained three-dimensional objects (the virtual room and the objects in it) are transferred to the software pre-installed on the computer, with the help of which the files of "rehabilitation virtual spaces" are created - virtual rooms with 3D reconstructions of spaces, which, if necessary, can be changed later. Rehabilitation specialists and engineers interact with the virtual elements of the spaces using a monitor and input devices (mouse, keyboard), portable electronic devices connected to this computer (tablet, smartphone, etc.), or virtual reality devices (VR helmet with controllers and other components) for the purpose of studying them. In these rehabilitation virtual spaces, in addition to rehabilitation specialists and engineers, the user of the cortical visual prosthetic system can safely move and interact with the virtual elements of the environment using controllers.

[0021] A 360-degree video camera is connected to the specified computer via a communication channel, transmitting images to it, or a regular video camera transmitting a series of images, which are then reconstructed into a panoramic 360-degree image, thus obtaining images of the entire surrounding space at its different points, for further analysis by rehabilitation specialists and / or engineers for the purpose of identifying areas of space inaccessible (invisible) to bionic vision. On the computer, using pre-installed software that analyzes two-dimensional images of scanned spaces, the images obtained from the camera are reduced to the resolution provided in the cortical visual prosthetic system (depending on the number of electrodes on the implanted matrix).In addition, this software translates the color image into the "grayscale" and "binary black and white" modes, since direct stimulation of the visual cortex may not provide the user of the cortical visual prosthetic system with correct color vision. Then, "weak" areas of spaces, objects or their parts that cannot be seen with bionic vision are identified. In order to ensure the technical implementation of the task of highlighting (designating, indicating) areas of the analyzed space that are invisible to bionic vision, it is necessary to "match" the ZE reconstruction of the space with images obtained from the video camera.This solution can be implemented either manually using specialized software tools in a computer graphical environment, or in an automated mode, which is also provided by segmentation algorithms (machine vision), which determine objects and their visible and invisible (predicted) boundaries, and also graphically designate them in an editable format when visualizing 3D spaces (drawing lines, changing them, deleting them, etc.). The result of the software at this stage is a 3D file with identified areas of space that are “invisible” to bionic vision, or protruding parts of objects in space, or boundaries between objects that are obvious to a sighted person, but not to bionic vision (in the form of contours).

[0022] The final stage of the method implementation is the use of software that allows for manual and / or automatic creation of elements of space visible to bionic vision. For this purpose, the functionality of this software includes the corresponding tools for editing the geometric parameters of such spaces in the form of drawing on 3D models of space using raster and / or vector methods (changing shape, size, deleting, adding, etc.), creating text and graphic notes on them to explain their significance, features, placement rules, materials, etc., as well as tools for editing a two-dimensional image from a video camera (color, brightness, resolution, etc.), contours obtained earlier.In addition, the software contains bionic vision simulation tools, with the help of which a rehabilitation therapist, relative or other sighted person can “enter” the virtual space (via a personal computer, mobile devices or VR) and see what will be available to bionic vision before and after modifications to the space. The result of the method is a file with recommendations on how to change a particular space surrounding the user of the visual prosthetic system in which he is located, so that his bionic vision can “see” important elements of the space and, thus, be as useful as possible.The virtual rooms are controlled (their selection, launch and exit, selection of lighting conditions and any other settings) using pre-installed software on the computer, which may be a separate software product, or may be part of any other with the ability to use and edit three-dimensional files. In this case, the movement of the rehabilitation therapist or engineer in the virtual space is controlled using a keyboard and mouse, if a computer with the specified software is used, or a touch screen, if mobile devices (smartphones and tablets) are used, or controllers connected via a communication channel to a computer with the specified software, if a VR helmet is used.In the case of using virtual rooms in spacious and rehabilitator-controlled premises (for example, in specialized rehabilitation centers), the controllers can be replaced by external tracking sensors that provide for the determination of a person in space. All of the above software elements that allow the invention to be implemented can be parts of one or several software and hardware complexes.

[0023] This ensures the achievement of a technical result consisting in reducing the rehabilitation period for users of cortical visual prosthetic systems, increasing the safety of the rehabilitation process, and expanding the functionality of systems for adaptation to bionic vision.

Claims

CLAUSE OF INVENTION 1. A method for adapting the surrounding space for users of visual prosthetic systems, comprising the steps of a) reading the geometric parameters of the space surrounding the user of the cortical visual prosthetic system by means of technical means for obtaining information about the surrounding space, transmitting them via a communication channel to a personal computer, where three-dimensional images of spaces are created using software, b) transmitting two-dimensional images from a video camera to a personal computer, where they are reduced using software to the resolution provided by the visual prosthetic system, c) comparing three-dimensional images of spaces and images obtained from a video camera on a personal computer, revealing areas of spaces invisible to bionic vision using software,d) after which they edit three-dimensional images of spaces using software with the ability to create objects visible to bionic vision, and provide recommendations for making changes to the space surrounding the user of the cortical visual prosthetic system.

2. The method according to paragraph 1, characterized in that a laser scanner or laser locator, or a photo camera, or an ultrasonic scanner, or any other similar solution is used as technical means for obtaining information about the surrounding space.

3. The method according to paragraph 1, characterized in that at the stage of reducing two-dimensional images to the resolution provided by the visual prosthetic system, the image data is transferred from the “color” mode to the “grayscale” and “binary black and white” modes.

4. The method according to paragraph 1, characterized in that intelligent algorithms are used as software for creating and editing three-dimensional models of virtual spaces.

5. The method according to paragraph 1, characterized in that at stages (a) and (d) they interact with three-dimensional images of spaces using a monitor and input devices connected to a personal computer, or portable electronic devices, or virtual reality devices.