A blind image perception auxiliary system and method based on electrohaptics

Abstract

The application relates to the technical field of blind person perception assistance, in particular to a blind person image perception assistance system and method based on electro-tactile sense. The system comprises an image acquisition and basic processing module, a key object identification and mask generation module, an edge and gray information processing module, an edge-guided dynamic scanning module, a multi-modal electro-tactile coding and output module and a user feedback adjustment module. The method collects images, identifies key objects, extracts edge and gray information, divides grids based on edge directions for dynamic scanning, and finally maps stimulation information to a dual-mode electro-tactile array output of hands and brains. The application effectively solves the problems of inaccurate image structure perception and low information transmission efficiency of traditional methods by fusing hand precise contour tactile sense and brain overall imaging stimulation, and introducing a closed-loop adjustment mechanism based on key object identification and user feedback, and significantly improves the perception efficiency and recognition accuracy of blind people on image information.

Description

Technical Field

[0001] This invention relates to the field of sensory assistance technology for the blind, and in particular to an image sensory assistance system and method for the blind based on electrotactile sensing. Background Technology

[0002] With the development of information technology and artificial intelligence, image information plays an important role in daily life and social activities. However, for visually impaired individuals, directly acquiring image information presents significant difficulties. In recent years, electro-tactile assistive technology has been gradually applied to image perception for the blind, primarily through two technical approaches:

[0003] The first type of technology primarily relies on peripheral tactile substitution, transmitting image contour, grayscale, or structural information to the user by generating electrical stimulation on the skin of areas such as the palm and fingers. However, these traditional methods mostly depend on fixed stimulation patterns and low-resolution tactile arrays. Because the stimulation information lacks hierarchy and sequence, it fails to accurately perceive complex image structures. Furthermore, the resolution of hand touch is limited, making it difficult to convey rich color and fine texture information. Users need to integrate continuous tactile signals into an image in their brains, resulting in a high cognitive load and low efficiency.

[0004] The second type of technical approach explores central nervous system stimulation, attempting to induce image perception by directly stimulating the visual cortex of the brain through brain electrode arrays. However, this approach currently has significant drawbacks: the mapping relationship between stimulation parameters and image content is often coarse (such as a simple brightness-frequency mapping), lacking the ability to identify and focus on key objects in the image; more importantly, most systems are open-loop stimulation, unable to provide feedback and adjustment based on the user's real-time perception state, resulting in unstable perception effects and poor reliability; and the devices are often bulky and lack portability.

[0005] Therefore, both existing approaches have significant shortcomings, failing to combine the precise contour localization capabilities of peripheral tactile sensation with the intuitive holistic imaging advantages of central stimulation. There is an urgent need for a novel image perception assistance system that can integrate the advantages of both, achieving high efficiency, accuracy, intuitiveness, and closed-loop feedback capabilities. Summary of the Invention

[0006] To overcome the above shortcomings, this invention provides an image perception assistance system and method for the blind based on electrotactile sensation. It aims to address the problems of inaccurate image structure perception and insufficient information transmission levels in existing peripheral tactile alternative methods, as well as the coarse stimulus mapping relationships and lack of closed-loop feedback in central nervous system stimulation methods. This invention integrates a dual-modal channel of hand tactile sensation and brain stimulation, and introduces an information focusing and closed-loop adjustment mechanism based on key object recognition, ultimately achieving efficient, accurate, intuitive, and adaptive image perception assistance.

[0007] In a first aspect, the present invention provides the following technical solution: a blind image perception assistance system based on electrotactile sensing includes:

[0008] The image acquisition and basic processing module is used to acquire images through a camera device and perform grayscale conversion and noise reduction on the images to obtain a basic image;

[0009] The key object recognition and mask generation module is used to detect key objects in the basic image, generate key object region masks, and compress or discard non-key regions.

[0010] The edge and grayscale information processing module is used to extract continuous contour edge information in the key object area and generate electrotactile stimulation data from the grayscale information in the area, while clarifying the order of edge stimulation over grayscale stimulation;

[0011] The edge-guided dynamic scanning module is used to divide the key object area into grids along the edge direction and present stimuli on the electrotactile array in the order of the grids. The duration of the stimulus corresponding to each grid is adjusted according to the complexity of the object.

[0012] The edges of each grid and grayscale stimulus data are mapped to an electrotactile array of the palm or finger, and the intensity and frequency of the stimulus are controlled.

[0013] The image information of the key object is encoded into signal parameters suitable for electrical stimulation of the cerebral cortex based on a predetermined color-frequency mapping model, and the head electrotactile array is driven to output the signal.

[0014] The user feedback adjustment module is used to adjust the stimulation sequence, the stimulation duration of each grid, and the scanning strategy based on user feedback.

[0015] Furthermore, generating the key object region mask includes the following steps:

[0016] Perform object detection on the base image to obtain candidate regions containing key objects;

[0017] The candidate regions are semantically segmented to obtain the corresponding pixel-level object regions;

[0018] The object region is subjected to boundary smoothing, hole filling and shape regularization to form a key object region mask;

[0019] The process of compressing or discarding non-critical areas includes the following steps:

[0020] Based on the key object region mask, the base image is divided into key regions and non-key regions;

[0021] Image data is compressed for the pixel blocks in the non-critical areas according to a preset compression ratio. The compression adopts discrete cosine transform or wavelet transform.

[0022] When the boundary pixels of the non-critical region are adjacent to the boundary pixels of the critical region, the boundary transition pixels are retained.

[0023] When the amount of data in a non-critical area is lower than a preset threshold after compression, a discard operation is performed and the area index is recorded.

[0024] Furthermore, the step of generating electrotactile stimulation data from grayscale information within the region includes the following steps:

[0025] The pixel grayscale values ​​within the key object region are subjected to linear or piecewise linear normalization to obtain a standardized grayscale matrix.

[0026] The standardized grayscale matrix is ​​mapped to an electrical stimulation intensity matrix, where the grayscale value corresponds proportionally to the current amplitude.

[0027] Spatial sampling and temporal encoding are performed on the electrical stimulation intensity matrix to form an electrotactile stimulation signal sequence;

[0028] The electrotactile stimulation signal sequence is converted into corresponding voltage pulse data by the electro-stimulation driving unit, thereby driving the electrotactile array to generate tactile perception.

[0029] Furthermore, dividing the key object region into a mesh along the edge includes the following steps:

[0030] Curve fitting is performed on the contour edges of the key object region to obtain the principal direction vector field of the edge direction.

[0031] Based on the principal direction vector field, a local coordinate system is established in the normal and tangential directions of the edge curve;

[0032] According to the preset grid size parameters, the local coordinate system is segmented along the tangential direction of the edge curve and layered in the normal direction to form regular grid units;

[0033] Record the spatial location index of each grid cell and its corresponding edge feature parameters;

[0034] The adjustment of the stimulus duration for each grid according to the object complexity includes the following steps:

[0035] Calculate the edge density, grayscale change rate, and local texture gradient within each grid to obtain the object complexity index;

[0036] The object complexity index is normalized to obtain the complexity weight of each grid.

[0037] The preset total scanning cycle is weighted according to the complexity weight to determine the stimulation duration corresponding to each grid.

[0038] During the control phase of the electrotactile array, electrotactile stimulation signals are output sequentially according to the stimulation duration to complete the timing scheduling of dynamic scanning.

[0039] Furthermore, controlling the intensity and frequency of the stimulus includes the following steps:

[0040] Extract the edge amplitude and average grayscale value within each grid, and use them as the intensity control factor and frequency control factor, respectively.

[0041] The intensity control factor is normalized and mapped to the voltage amplitude control parameters of the electro-tactile array;

[0042] The frequency control factor is quantified and graded, and mapped to the electrical stimulation pulse frequency control parameter;

[0043] Based on the voltage amplitude control parameters and pulse frequency control parameters, an electrical stimulation control signal is generated;

[0044] The electrical stimulation control signal is sent to the electrode unit of the corresponding electrotactile array to achieve independent electrotactile output for different grids.

[0045] Furthermore, the adjustment of the stimulation sequence, the stimulation duration of each grid, and the scanning strategy includes the following steps:

[0046] Collect user feedback data during electrotactile stimulation, including tactile recognition accuracy and reaction time;

[0047] Statistical analysis is performed on the feedback data to calculate the recognition error rate and response delay value for each grid.

[0048] The stimulus sequence is reordered according to the recognition error rate, and the grids with recognition error rates exceeding a preset threshold are arranged in descending order of error rate at the beginning of the scanning sequence;

[0049] The stimulation duration of each grid is corrected based on the response delay value, and the time allocation parameters are updated using a proportional adjustment method.

[0050] Based on the reordered grid sequence and the updated time allocation parameters, a new scan path and schedule are generated.

[0051] Furthermore, the multimodal electrotactile encoding and output module also includes a central stimulation encoding unit, used to encode the image information of the key object into signal parameters suitable for electrical stimulation of the cerebral cortex based on a predetermined color-frequency mapping model.

[0052] Furthermore, the user feedback adjustment module also includes an EEG feedback acquisition unit, used to acquire and analyze the EEG signals of the user when receiving brain electrical stimulation to determine the perceptual state; the system also includes a central perception enhancement module, used to drive the head electrotactile array to stimulate, and to perform closed-loop adjustment of the stimulation parameters based on the EEG feedback.

[0053] Secondly, the present invention provides an image perception assistance method for the blind based on electrotactile sensing, the method comprising the following steps:

[0054] Images are acquired using a camera device, and the images are then converted to grayscale and denoised to obtain a base image.

[0055] For the base image, key objects are detected, a key object region mask is generated, and non-key regions are compressed or discarded.

[0056] Continuous contour edge information is extracted from the key object area, and grayscale information within the area is used to generate electrotactile stimulation data. At the same time, the order of edge stimulation taking precedence over grayscale stimulation is clearly defined.

[0057] The key object area is divided into grids along the edge direction, and stimuli are presented on the electrotactile array in the order of the grids. The duration of the stimulus corresponding to each grid is adjusted according to the complexity of the object.

[0058] The edges of each grid and grayscale stimulus data are mapped to an electrotactile array of the palm or finger, and the intensity and frequency of the stimulus are controlled.

[0059] The stimulation sequence, stimulation duration for each grid, and scanning strategy were adjusted based on user feedback.

[0060] Furthermore, the method further includes the step of:

[0061] The image information of the key object is encoded into a cortical electrical stimulation signal based on a predetermined color-frequency mapping model;

[0062] The stimulation is applied by driving the head electro-tactile array;

[0063] Collect the electroencephalogram (EEG) signals generated by the user when receiving the stimulation;

[0064] The user's perceptual state is determined based on the electroencephalogram (EEG) signals.

[0065] Based on the determined sensory state, the parameters of the electrical stimulation signal of the cerebral cortex are adjusted in a closed-loop manner.

[0066] The present invention has the following beneficial effects:

[0067] 1. In this invention, an image acquisition and basic processing module acquires images and performs grayscale conversion and noise reduction processing; a key object recognition and mask generation module detects key objects and generates key object region masks; an edge and grayscale information processing module extracts continuous contour edge information and generates electrotactile stimulation data; an edge-guided dynamic scanning module divides the key object region into grids along the edge direction and presents stimuli sequentially according to the grid order; a multimodal electrotactile encoding and output module maps the gridded information to the electrotactile array and controls the intensity and frequency of the stimulation; and a user feedback adjustment module adjusts the stimulation strategy according to user feedback, thereby enabling blind people to efficiently and accurately perceive image structures and key features through electrotactile sensing.

[0068] 2. By using a key object recognition and mask generation module to perform object detection on the base image and generate key object region masks, non-key regions are compressed or discarded, thus alleviating the problem of tactile stimulation information overload caused by traditional electro-tactile assistive systems transmitting full image data. At the same time, by directly imaging brain stimulation, the cognitive load on users who need to piece together images from continuous tactile signals is reduced.

[0069] 3. The edge-guided dynamic scanning module divides the key object area into grids along the edge direction and presents stimulation on the electrotactile array in grid order. The stimulation duration of each grid is adjusted according to the object complexity, thus solving the problem of low user perception efficiency and recognition accuracy caused by the traditional method of stimulating all areas with fixed time slices without considering the differences in object complexity.

[0070] 4. By introducing cortical electrotactile stimulation and closed-loop regulation of EEG feedback, this system breaks through the traditional mode of indirectly transmitting information only through peripheral senses. It can directly map the overall outline and color information of key objects into brain stimulation signals, allowing users to "see the image in their minds," resulting in a more intuitive and realistic perception.

[0071] 5. By employing a dual-modal collaborative approach of tactile sensation and brain stimulation, the system achieves a complementary balance between "precise local perception" and "holistic intuitive imaging." Users can perceive precise object outlines and textures through tactile touch, while simultaneously perceiving the object's color and overall shape through brain stimulation. This mutual reinforcement of perception significantly enhances the completeness, accuracy, and efficiency of environmental object recognition. The system offers multiple configurable operating modes, including parallel, sequential, and adaptive modes, allowing it to flexibly adapt to different application scenarios and user needs. Attached Figure Description

[0072] Figure 1 This is a schematic diagram of the architecture of an image perception assistance system for the blind based on electrotactile sensing proposed in this invention.

[0073] Figure 2This is a flowchart illustrating an image perception assistance method for the blind based on electrotactile sensing proposed in this invention. Detailed Implementation

[0074] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0075] Example 1:

[0076] In a first embodiment of the present invention, the present invention provides an image perception assistance system for the blind based on electrotactile sensing, such as... Figure 1 As shown, it includes an image acquisition and basic processing module, which is used to acquire images through a camera device and perform grayscale conversion and noise reduction on the images to obtain a basic image;

[0077] The key object recognition and mask generation module is used to detect key objects in the basic image, generate key object region masks, and compress or discard non-key regions.

[0078] Furthermore, generating a key object region mask includes the following steps:

[0079] Perform object detection on the base image to obtain candidate regions containing key objects;

[0080] Semantic segmentation is performed on the candidate regions to obtain the corresponding pixel-level object regions;

[0081] The object region is processed by boundary smoothing, hole filling and shape regularization to form a mask for the key object region.

[0082] Specifically, firstly, object detection is performed on the base image, with the input being the base image matrix obtained after grayscale conversion and denoising. The grayscale value of each pixel is The image is scanned using a sliding window or full-image feature extraction algorithm through a pre-trained convolutional neural network, and the target probability of each candidate region is calculated. It can be expressed by the following formula:

[0083] ;

[0084] in For convolution kernel weights, For bias terms, For activation function, Image coordinates, through a threshold Candidate regions with probabilities higher than a threshold are selected to obtain a set of candidate regions containing the key object. The second step is to perform semantic segmentation on the candidate region set, with the input being an image sub-block for each candidate region. The probability of each pixel belonging to a key object is calculated using a pixel-level classification network or a conditional random field model. After binarization, pixel-level object regions are obtained. In the first step, a pixel value of 1 indicates that the object belongs to the key object, and 0 indicates that it does not. The second step involves smoothing the boundaries of the pixel-level object regions, filling holes, and regularizing their shapes. Morphological operations such as dilation, erosion, and closing operations, as well as connected component analysis, are used to adjust the mask boundaries, making the mask region continuous and removing isolated holes, thus obtaining the final key object region mask. Each pixel value indicates whether the location belongs to a critical object, and the output mask is used for subsequent non-critical area compression, grayscale mapping, and electrotactile stimulation encoding steps.

[0085] By performing key object detection and pixel-level mask generation on the base image, it is possible to clearly distinguish between key object regions and non-key regions in the image, providing spatial reference information for the accurate presentation of subsequent electrotactile stimuli.

[0086] Furthermore, compressing or discarding non-critical areas includes the following steps:

[0087] Based on the key object region mask, the base image is divided into key regions and non-key regions;

[0088] Image data is compressed for non-critical pixel blocks according to a preset compression ratio, and the compression method is discrete cosine transform or wavelet transform.

[0089] When a non-critical region is adjacent to the boundary pixels of a critical region, the boundary transition pixels are retained.

[0090] When the amount of data in a non-critical area is lower than a preset threshold after compression, a discard operation is performed and the area index is recorded.

[0091] Specifically, the base image is divided into key and non-key regions based on the key object region mask. Image data compression is performed on the non-key regions, and discarding is performed when necessary. The base image is first represented as a matrix. ,in The image height in pixels. Image width in pixels, pixel value Take integers within the grayscale range and apply them through the key object mask matrix. Key areas Defined as Non-critical areas Defined as The non-critical areas were then divided into several pixel blocks. Each block is [size missing] Perform discrete cosine transform on each block or wavelet transform Calculate the compression coefficient matrix or According to the preset compression ratio Before keeping Set one coefficient and the rest to zero to obtain the compressed block. When a non-critical region block is adjacent to the boundary pixels of a critical region, the boundary transition pixels are preserved to maintain edge continuity; if the amount of data stored in the compressed block is less than the threshold... If so, discard the block and record its index. The final output is the compressed or discarded non-critical area. , and key areas Combined to form the processed image This result was used for subsequent edge extraction and grayscale information mapping to electrotactile stimulation data generation.

[0092] By compressing or discarding non-critical areas, image data redundancy is reduced, and the processing efficiency of critical objects is improved. This addresses the problem of slow response speed caused by excessive overall image processing data in traditional image perception assistive systems for the blind.

[0093] The edge and grayscale information processing module is used to extract continuous contour edge information in key object areas and generate electrotactile stimulation data from the grayscale information in the area, while clarifying the order of edge stimulation over grayscale stimulation;

[0094] Furthermore, generating electrotactile stimulation data from the grayscale information within the area includes the following steps:

[0095] The pixel grayscale values ​​within the key object area are linearly or piecewise linearly normalized to obtain a standardized grayscale matrix.

[0096] The standardized grayscale matrix is ​​mapped to the electrical stimulation intensity matrix, where the grayscale value corresponds proportionally to the current amplitude.

[0097] Spatial sampling and temporal encoding of the electrical stimulation intensity matrix are performed to form an electrotactile stimulation signal sequence;

[0098] The electrotactile stimulation signal sequence is converted into corresponding voltage pulse data by the electrotactile stimulation driving unit, which drives the electrotactile array to generate tactile perception.

[0099] Specifically, a standardized grayscale matrix is ​​obtained by linearly normalizing or piecewise linearly normalizing the pixel grayscale values ​​of the key object region. The input data is the grayscale matrix of the key object region. Each Represents pixels The grayscale value, the normalization calculation formula is as follows: Or according to piecewise linear functions Obtain standardized grayscale values. and These represent the minimum and maximum grayscale values ​​of the key object region, respectively; then, the standardized grayscale matrix is ​​mapped to the electrical stimulation intensity matrix. ;in , and The minimum and maximum values ​​of the current amplitude are used as the intensity information of the electrotactile stimulation signal; spatial sampling and temporal encoding of the electrotactile stimulation intensity matrix yield the electrotactile stimulation signal sequence. Each Corresponding to the stimulation intensity and duration at a certain moment on the electrotactile array, the sampling and encoding methods can be generated according to the row and column scanning order or the edge priority order; finally, the signal sequence is processed by the electrical stimulation driving unit. Converted to voltage pulse data The drive electro-tactile array outputs corresponding tactile feedback, forming continuous and perceptible edge and grayscale stimuli, allowing users to perceive the shape and texture information of key objects, and the voltage amplitude. Directly from the corresponding current amplitude The time slice length is determined by the signal sequence index interval after being converted by the driving unit.

[0100] By normalizing the grayscale information of key object areas and mapping it into electrotactile stimulation signals, voltage pulse data is output in edge priority order, enabling users to perceive the contour and grayscale texture information of key objects through touch.

[0101] The edge-guided dynamic scanning module is used to divide the key object area into grids along the edge direction and present stimuli on the electrotactile array in the order of the grids. The duration of the stimulus corresponding to each grid is adjusted according to the complexity of the object.

[0102] Furthermore, dividing the key object area into meshes along the edge includes the following steps:

[0103] Curve fitting is performed on the contour edges of the key object region to obtain the principal direction vector field of the edge direction.

[0104] Based on the principal direction vector field, establish a local coordinate system in the normal and tangential directions of the edge curve;

[0105] According to the preset grid size parameters, the grid is segmented along the tangential direction of the edge curve and layered in the normal direction within the local coordinate system to form regular grid units;

[0106] Record the spatial location index of each grid cell and its corresponding edge feature parameters.

[0107] Specifically, the edge-guided dynamic scanning module obtains the principal direction vector field by curve fitting the contour edges of the key object region, using the formula:

[0108] ;

[0109] in For the parameterization function of the edge contour curve, For the arc length of the curve, For the tangent vector, As the normal vector, after obtaining the principal direction vector field, a local coordinate system is established at each edge point, and the coordinates are determined according to the preset mesh size parameters. and The grid is divided into segments along the tangential direction and layers along the normal direction to form grid cells, and the spatial location index of each grid cell is output. With corresponding edge feature parameters The meshed information is then used for sequential stimulation of the electrotactile array to achieve dynamic scanning along the edges, with the input data being a binary edge image of the key object region. With grid size parameters The output is a spatial index matrix. With edge feature matrix It is used to guide the sequence and location of electrotactile stimulation.

[0110] By dividing the edges of key objects into regular grids and recording the spatial location and edge features of each grid, the electrotactile array presents stimuli in grid order, thereby improving the problem that traditional image tactile presentation cannot accurately guide the hand scanning contour.

[0111] Furthermore, adjusting the stimulus duration for each grid based on object complexity includes the following steps:

[0112] Calculate the edge density, grayscale change rate, and local texture gradient within each grid to obtain the object complexity index;

[0113] The object complexity index is normalized to obtain the complexity weight of each grid.

[0114] The preset total scanning cycle is weighted according to the complexity weight to determine the stimulus duration corresponding to each grid.

[0115] During the control phase of the electrotactile array, electrotactile stimulation signals are output sequentially according to the stimulation duration to complete the timing scheduling of dynamic scanning.

[0116] Specifically, this is achieved by calculating the complexity index of the grid cells of the key object region. The input data is first the gray-level matrix of the key object region. and contour edge matrix ,in Represents pixels grayscale value, Represents pixels Binary identifier for whether it belongs to the edge; calculate for each grid cell. Edge density Grayscale change rate and local texture gradient These metrics are combined to form an object complexity metric:

[0117] ;

[0118] in The complexity weights are obtained by normalization based on the preset weights. Then, the total scan cycle is adjusted according to the complexity weight. Perform a weighted allocation to obtain the stimulus duration for each grid. In the electro-haptic array control phase, the system follows the grid... The electrotactile stimulation signals are output sequentially to achieve dynamic scanning scheduling. This result is used to adjust the stimulation duration based on the local complexity of the object in hand tactile perception to enhance the hierarchy and continuity of information presentation. Input parameters include a grayscale matrix. Edge matrix Grid set and total scan cycle The output is the stimulus time series corresponding to each grid. The next step is to use this time series to control the electrotactile drive unit to generate stimulation signals of corresponding durations from the electrotactile array.

[0119] By calculating the object complexity based on the edge density, grayscale change rate, and local texture gradient of each grid and adjusting the stimulation duration, the scanning cycle is weighted according to complexity and the electrotactile signals are output sequentially, thereby improving the problem of insufficient information transmission levels caused by the uniform stimulation time in traditional blind image tactile presentation.

[0120] The multimodal electrotactile coding and output module, including a hand coding output unit and a central coding output unit, is used for:

[0121] The edges of each grid and grayscale stimulus data are mapped to an electrotactile array of the palm or finger, and the intensity and frequency of the stimulus are controlled.

[0122] The image information of the key object is encoded into signal parameters suitable for electrical stimulation of the cerebral cortex based on a predetermined color-frequency mapping model, and the head electrotactile array is driven to output the signal.

[0123] Furthermore, controlling the intensity and frequency of stimuli includes the following steps:

[0124] Extract the edge amplitude and average grayscale value within each grid, and use them as the intensity control factor and frequency control factor, respectively.

[0125] The intensity control factor is normalized and mapped to the voltage amplitude control parameters of the electrotactile array;

[0126] The frequency control factor is quantified and classified, and mapped to the electrical stimulation pulse frequency control parameter;

[0127] Based on the voltage amplitude control parameters and pulse frequency control parameters, an electrical stimulation control signal is generated;

[0128] The electrical stimulation control signal is sent to the electrode unit of the corresponding electrotactile array to achieve independent electrotactile output for different grids.

[0129] Specifically, by extracting the edge magnitude within each grid. Compared with the average gray level Using the intensity control factor and frequency control factor respectively, the intensity control factor is first normalized to obtain the voltage amplitude control parameter:

[0130] ;

[0131] in and These are the minimum and maximum edge magnitudes for all grid cells, respectively. To determine the maximum allowable voltage amplitude of the electro-haptic array, the frequency control factor is obtained as the pulse frequency parameter through quantization and gradation.

[0132] ;

[0133] in and These are the minimum and maximum values ​​of the average grayscale value for all grid cells, respectively. For frequency levels, For frequency step size, This indicates rounding to the nearest integer, followed by generating an electrical stimulation control signal based on the voltage amplitude control parameters and pulse frequency control parameters. in For frequency The standard pulse function, then the signal The signals are sent to the corresponding electrode units to complete independent electrotactile output for each grid. The output is a sequence of electrical stimulation signals of corresponding intensity and frequency on the electrotactile array. The next step is for the user's palm or fingers to perceive and generate tactile feedback. This result is used to transmit the edge and grayscale information of the image to the user's perception. Each parameter is as follows: All were obtained through the aforementioned image analysis and normalization or quantization calculations.

[0134] By mapping the edges and grayscale information of each grid to electrical stimulation signals with adjustable voltage and pulse frequency, distinguishable tactile perception of image information by the palm or fingers can be achieved, thereby improving the problem that traditional assistive systems for the blind have difficulty in transmitting detailed image features.

[0135] The user feedback adjustment module is used to adjust the stimulation sequence, the stimulation duration of each grid, and the scanning strategy based on user feedback.

[0136] Further adjustments to the stimulation sequence, stimulation duration for each grid, and scanning strategy include the following steps:

[0137] Collect user feedback data during electrotactile stimulation, including tactile recognition accuracy and reaction time;

[0138] Perform statistical analysis on the feedback data to calculate the recognition error rate and response delay value for each grid.

[0139] The stimulus order is reordered based on the recognition error rate, and grids with recognition error rates exceeding a preset threshold are arranged in descending order of error rate at the beginning of the scanning sequence;

[0140] The stimulus duration of each grid is adjusted based on the response delay value, and the time allocation parameters are updated using a proportional adjustment method.

[0141] Based on the reordered grid sequence and the updated time allocation parameters, a new scan path and schedule are generated.

[0142] Specifically, by collecting user feedback data during the electrotactile stimulation process, including tactile recognition accuracy... With reaction time For each grid The recognition performance was quantitatively evaluated, among which For users to the first The correct recognition ratio of each grid cell For users to the first The average reaction time of each grid cell was obtained by recording electrotactile stimulation and user responses in real time; firstly, the recognition error rate was calculated. With response delay value The formulas are as follows:

[0143]

[0144] ;

[0145] in The average reaction time for all grids; based on The stimulus sequences are reordered to construct a new scanning sequence. ,satisfy And according to Stimulus duration for each grid The formula for adjusting the ratio is:

[0146] ;

[0147] in For adjustment coefficients, The maximum value of all grid delays; the final output is the updated scan path. Stimulus time allocation These results were used to optimize the presentation order and duration of each grid in subsequent electrotactile stimulation to improve user tactile recognition efficiency and response consistency. Each parameter was obtained through statistical calculation of real-time user feedback data, with adjustment coefficients... Based on experimental experience, this setting is used to balance the delay correction magnitude and the overall scan cycle.

[0148] By dynamically adjusting the grid stimulation sequence and duration based on the user's tactile recognition accuracy and reaction time, the scanning path and timing are optimized, thereby improving the user's tactile perception efficiency and recognition consistency for key objects.

[0149] Example 2:

[0150] Based on Example 1, this embodiment further elaborates on the enhanced functions of the multimodal electrotactile coding and output module and the user feedback adjustment module, and introduces a central perception enhancement module to form a complete brain stimulation closed-loop system.

[0151] The central stimulation encoding unit in the multimodal electrotactile encoding and output module is used to encode the image information of key objects into signal parameters suitable for electrical stimulation of the cerebral cortex based on a predetermined color-frequency mapping model.

[0152] Specifically, the central stimulation coding unit converts the color and brightness information of the acquired image into current signals of different intensities and frequencies, using the following conversion formula: ,in , and These are the frequency conversion coefficients for the three primary colors R, G, and B, where R, G, and B are the red, green, and blue primary color values ​​acquired by the image acquisition module, respectively. The synthesized frequency for each pixel. Based on the frequency range that the human body can distinguish, the synthesized frequency for each pixel is determined. The frequency range should be between 0 and 1110 Hz. Therefore, according to the principle of three-primary-color synthesis, any color can be created by mixing red (R), green (G), and blue (B) in different proportions. However, due to the limitation of human skin's ability to distinguish frequencies from 0 to 1110 Hz, the maximum value of the three primary colors R, G, and B can be set to 10, thus determining the maximum frequency coefficient as: =100, =10、 =1. When setting the frequency conversion coefficient of the three primary colors. =100, =10、 When =1, the red, green, and blue primary color values ​​acquired by the image acquisition module can vary arbitrarily within the range of 0-10. The pixel synthesis frequency is calculated using the conversion formula. All are below 1110Hz, which is within the frequency range that the human body can distinguish.

[0153] The EEG feedback acquisition unit in the user feedback adjustment module is used to collect and analyze the EEG signals of the user when receiving brain electrical stimulation in order to determine the perceptual state.

[0154] Specifically, a conductive ointment is applied to reduce the impedance between the scalp and the electrodes, eliminating artifacts in the original EEG signal. Adaptive filtering is used to denoise the signal, extracting its time-domain, frequency-domain, time-frequency, and nonlinear features. These features are then automatically extracted using a deep learning algorithm based on a convolutional neural network, which is used to determine whether the user has successfully achieved image perception. For example, if the detected P300 event-related potential or power spectral density change in a specific frequency band exceeds a preset threshold, perception is considered successful.

[0155] The central perception enhancement module is used to drive the head electrotactile array to stimulate the brain and to perform closed-loop adjustment of the stimulation parameters based on the electroencephalogram feedback.

[0156] Specifically, the EEG signal conversion section consists of an array of multiple conductors. The column connection of each conductor in each column is connected to the output of the current generation module via a column electronic switch. The row connection of each conductor in each row is grounded via a row electronic switch. The control signal input terminals of each row and column electronic switch are connected to the control signal output of the central processing unit. All conductors in the EEG signal conversion section are fitted with conductive rubber that contacts the scalp.

[0157] To optimize stimulation parameters, the system is adjusted based on a skin resistance model. The skin consists of the epidermis, dermis, and subcutaneous tissue. The dermis and subcutaneous tissue, due to the abundance of blood vessels, possess good conductivity and can be simulated as a pure resistance R0. The outer layer of the epidermis is the stratum corneum, and the skin's impedance primarily depends on it. The stratum corneum has a relatively high impedance, acting like a thin insulating film, similar to the dielectric of a capacitor. The dermis and electrode pads are the two plates of the capacitor. Furthermore, because a small number of ions pass through the sweat gland pores in the stratum corneum, this report simulates the epidermis as a leaky capacitor, whose impedance can be considered as a parallel combination of capacitance C and pure resistance Rp. Based on this model, the central perception enhancement module dynamically adjusts the stimulation current, frequency, and pulse width according to the results from the EEG feedback acquisition unit, ensuring the user is always in the optimal perceptual state between the sensory threshold and the pain threshold, forming an effective closed-loop control.

[0158] Furthermore, the central sensing enhancement module works in conjunction with the hand electrotactile array, and the system can be configured to multiple operating modes, including but not limited to:

[0159] Parallel stimulation mode: Hand tactile stimulation and brain cortex stimulation are performed simultaneously, allowing users to receive both local contour and overall image information at the same time, resulting in the strongest perceptual complementarity.

[0160] Sequence mode: First, the outline of key objects is scanned through a hand tactile array. After the user establishes a preliminary spatial cognition, brain stimulation is then activated to induce the formation of an image that includes color and overall structure, which conforms to the cognitive law from part to whole.

[0161] Task-Complexity-Based Adaptive Mode: The system automatically selects the optimal modal combination based on the complexity of key objects (such as edge density and number of colors). For example, it only enables the hand mode for simple contours to save energy, while automatically enabling the dual-modal parallel mode for complex scenes to improve perception efficiency. Through unified spatial-temporal coding, the system ensures spatial consistency and temporal synchronization between the local contour information scanned by the hand tactile array and the overall shape and color information induced by brain stimulation in the user's perceptual experience, thereby promoting the fusion of the two perceptual modalities.

[0162] Example 3:

[0163] In the daily lives of blind people, tactile recognition of environmental objects and text information suffers from low efficiency and incomplete information acquisition. To address these issues, the present invention employs a multimodal electrotactile image perception assistance method for the blind. The specific implementation process of this method is as follows:

[0164] First, scene images are acquired through camera equipment, and the images are then converted to grayscale and denoised to obtain a basic image, thereby improving image quality and reducing noise interference, which facilitates the subsequent identification of key objects.

[0165] Subsequently, key objects are detected in the basic image, key object region masks are generated, and non-key regions are compressed or discarded to highlight important information and reduce data processing volume.

[0166] Extract continuous contour edge information within the key object area and generate electrotactile stimulation data from the grayscale information within the area. At the same time, clarify the priority of edge stimulation over grayscale stimulation, thereby achieving effective encoding of edge structure and grayscale details.

[0167] The key object area is divided into grids along the edge, and stimuli are presented sequentially on the hand electrotactile array according to the grid order. The duration of the stimulus corresponding to each grid is adjusted according to the complexity of the object, realizing the dynamic distribution of tactile information and allowing complex areas to have more time for perception. Users can clearly perceive the precise outline and surface texture of the object through hand touch.

[0168] Simultaneously, the brain stimulation channel is activated: the central stimulation encoding unit in the multimodal electrotactile encoding and output module, based on the overall outline and color information of the key object (e.g., a red round apple), follows a color-frequency mapping model. This generates corresponding frequency signals and spatial stimulation patterns. The central perception enhancement module drives the head electrotactile array to apply these signals to the user's visual association area of ​​the cerebral cortex.

[0169] Subsequently, closed-loop feedback verification was initiated: the EEG feedback acquisition unit in the user feedback adjustment module collected the user's EEG signals in real time when receiving brain stimulation. Through analysis, the system detected that characteristic EEG patterns related to "red" and "circular" objects were successfully induced, determining that the brain stimulation successfully induced perception. The user's subjective feedback of "a blurry red circular image in the brain" corroborated the "smooth circular object" perceived by touch.

[0170] Ultimately, the system dynamically adjusts the order, duration, and scanning strategy of hand stimulation, as well as the signal parameters of brain stimulation, based on user feedback (including the accuracy of hand tactile recognition, reaction time, and EEG perception status), thereby optimizing the tactile scanning path and timing, and improving recognition accuracy and operational efficiency.

[0171] Ultimately, this enables blind people to perceive fine details in complex environments through tactile sensation and directly induce the formation of an overall image through brain stimulation. The two sensory channels corroborate each other, greatly improving the accuracy, completeness, and intuitiveness of perception.

[0172] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A blind image perception assistive system based on electrotactile sensing, characterized in that, include: The image acquisition and basic processing module is used to acquire images through a camera device and perform grayscale conversion and noise reduction on the images to obtain a basic image; The key object recognition and mask generation module is used to detect key objects in the basic image, generate key object region masks, and compress or discard non-key regions. The edge and grayscale information processing module is used to extract continuous contour edge information in the key object area and generate electrotactile stimulation data from the grayscale information in the area, while clarifying the order of edge stimulation over grayscale stimulation; The edge-guided dynamic scanning module is used to divide the key object region into grids along the edge direction and present stimuli on the electro-tactile array in the order of the grids. The duration of the stimulus corresponding to each grid is adjusted according to the object complexity, including: calculating the edge density, grayscale change rate and local texture gradient in each grid to obtain the object complexity index; and allocating the stimulus duration of each grid according to the object complexity index. Multimodal electrotactile encoding and output module, used for: The edges of each grid and grayscale stimulus data are mapped to an electrotactile array of the palm or finger, and the intensity and frequency of the stimulus are controlled. The image information of the key object is encoded into signal parameters suitable for electrical stimulation of the cerebral cortex based on a predetermined color-frequency mapping model, and the head electrotactile array is driven to output the signal. The user feedback adjustment module is used to adjust the stimulation order, the stimulation duration of each grid, and the scanning strategy according to user feedback; wherein, the adjustment of the stimulation order includes: reordering according to the user's recognition error rate of each grid, and adjusting the grids with recognition error rates exceeding a preset threshold to the beginning of the scanning sequence.

2. The system according to claim 1, characterized in that, The process of generating the key object region mask includes the following steps: Perform object detection on the base image to obtain candidate regions containing key objects; The candidate regions are semantically segmented to obtain the corresponding pixel-level object regions; The object region is subjected to boundary smoothing, hole filling and shape regularization to form a key object region mask; The process of compressing or discarding non-critical areas includes the following steps: Based on the key object region mask, the base image is divided into key regions and non-key regions; Image data is compressed for the pixel blocks in the non-critical areas according to a preset compression ratio. The compression adopts discrete cosine transform or wavelet transform. When the boundary pixels of the non-critical region are adjacent to the boundary pixels of the critical region, the boundary transition pixels are retained. When the amount of data in a non-critical area is lower than a preset threshold after compression, a discard operation is performed and the area index is recorded.

3. The system according to claim 1, characterized in that, The process of generating electrotactile stimulation data from grayscale information within the region includes the following steps: The pixel grayscale values ​​within the key object region are subjected to linear or piecewise linear normalization to obtain a standardized grayscale matrix. The standardized grayscale matrix is ​​mapped to an electrical stimulation intensity matrix, where the grayscale value corresponds proportionally to the current amplitude. Spatial sampling and temporal encoding are performed on the electrical stimulation intensity matrix to form an electrotactile stimulation signal sequence; The electrotactile stimulation signal sequence is converted into corresponding voltage pulse data by the electrostimulation driving unit, thereby driving the electrotactile array to generate tactile perception.

4. The system according to claim 1, characterized in that, The process of dividing the key object region into a mesh along the edge includes the following steps: Curve fitting is performed on the contour edges of the key object region to obtain the principal direction vector field of the edge direction. Based on the principal direction vector field, a local coordinate system is established in the normal and tangential directions of the edge curve; According to the preset grid size parameters, the local coordinate system is segmented along the tangential direction of the edge curve and layered in the normal direction to form regular grid units; The adjustment of the stimulus duration for each grid according to the object complexity also includes the following steps: The object complexity index is normalized to obtain the complexity weight of each grid. The preset total scanning cycle is weighted according to the complexity weight to determine the stimulation duration corresponding to each grid. During the control phase of the electrotactile array, electrotactile stimulation signals are output sequentially according to the stimulation duration to complete the timing scheduling of dynamic scanning.

5. The system according to claim 1, characterized in that, The control of the intensity and frequency of the stimulus includes the following steps: Extract the edge amplitude and average grayscale value within each grid, and use them as the intensity control factor and frequency control factor, respectively. The intensity control factor is normalized and mapped to the voltage amplitude control parameters of the electro-tactile array; The frequency control factor is quantified and graded, and mapped to the electrical stimulation pulse frequency control parameter; Based on the voltage amplitude control parameters and pulse frequency control parameters, an electrical stimulation control signal is generated; The electrical stimulation control signal is sent to the electrode unit of the corresponding electrotactile array to achieve independent electrotactile output for different grids.

6. The system according to claim 1, characterized in that, The adjustment of the stimulation sequence, the stimulation duration of each grid, and the scanning strategy includes the following steps: Collect user feedback data during electrotactile stimulation, including tactile recognition accuracy and reaction time; Statistical analysis is performed on the feedback data to calculate the recognition error rate and response delay value for each grid. The stimulus sequence is reordered according to the recognition error rate, and the grids with recognition error rates exceeding a preset threshold are arranged in descending order of error rate at the beginning of the scanning sequence; The stimulation duration of each grid is corrected based on the response delay value, and the time allocation parameters are updated using a proportional adjustment method. Based on the reordered grid sequence and the updated time allocation parameters, a new scan path and schedule are generated.

7. The system according to any one of claims 1 to 6, characterized in that, The multimodal electrotactile encoding and output module also includes a central stimulation encoding unit, which encodes the image information of the key object into signal parameters suitable for electrical stimulation of the cerebral cortex based on a predetermined color-frequency mapping model.

8. The system according to claim 7, characterized in that, The user feedback adjustment module also includes an EEG feedback acquisition unit, used to acquire and analyze the EEG signals of the user when receiving brain electrical stimulation to determine the perceptual state; the system also includes a central perception enhancement module, used to drive the head electrotactile array to stimulate, and to perform closed-loop adjustment of the stimulation parameters based on the EEG feedback.

9. A method for assisting blind people in image perception based on electrotactile sensing, characterized in that, Includes the following steps: Images are acquired using a camera device, and the images are then converted to grayscale and denoised to obtain a base image. For the base image, key objects are detected, a key object region mask is generated, and non-key regions are compressed or discarded. Continuous contour edge information is extracted from the key object area, and grayscale information within the area is used to generate electrotactile stimulation data. At the same time, the order of edge stimulation taking precedence over grayscale stimulation is clearly defined. The key object area is divided into grids along the edge direction, and stimuli are presented on the electro-tactile array in grid order. The duration of the stimulus corresponding to each grid is adjusted according to the complexity of the object. The edges of each grid and grayscale stimulus data are mapped to an electrotactile array of the palm or finger, and the intensity and frequency of the stimulus are controlled. The stimulation sequence, stimulation duration for each grid, and scanning strategy were adjusted based on user feedback.

10. The method according to claim 9, characterized in that, The method further includes the following steps: The image information of the key object is encoded into a cortical electrical stimulation signal based on a predetermined color-frequency mapping model; The stimulation is applied by driving the head electro-tactile array; Collect the electroencephalogram (EEG) signals generated by the user when receiving the stimulation; The user's perceptual state is determined based on the electroencephalogram (EEG) signals. Based on the determined sensory state, the parameters of the electrical stimulation signal of the cerebral cortex are adjusted in a closed-loop manner.

Images