Bionic visual information perception method and system based on semantic driving

Abstract

The invention discloses a bionic visual information perception method and system based on semantic driving. The method comprises the steps that scene image information containing target content is acquired; performing semantic analysis on the scene image information to extract high-level semantic information, and calculating a high-level semantic weight based on the high-level semantic information; identifying visual features of the scene image information, and fusing the visual features and the high-level semantic weight to screen out key information pixels from the scene image information; mapping the key information pixel into a corresponding cortical electrode number based on a topological corresponding relation between a preset visual space and a visual cortical layer, and generating a stimulation vector corresponding to the cortical electrode number; and outputting the cortical electrode number and the stimulation vector to bionic visual stimulation equipment so as to realize perception of the scene image information by a user. Through a semantic-driven key information pixel screening and structured expression mechanism, efficient transmission of facial expressions and text information is realized under the limitation of a low-pixel channel.

Description

Technical Field

[0001] This invention relates to the field of biomimetic visual stimulation technology, and in particular to a semantically driven biomimetic visual information perception method and system. Background Technology

[0002] In the field of artificial bionic visual systems, especially in visual substitution scenarios based on electrical stimulation of the visual cortex or surface electrical stimulation devices, existing technologies are limited by the bandwidth constraints of information channels, making it difficult to present high-resolution images completely to blind or low-vision users. This results in severe obstacles for users in social interaction and reading ability. Current visual substitution systems typically use simple edge detection or contour extraction methods to process images. Although these methods retain basic structural information, with only a few dozen pixel channels, they not only consume a large amount of stimulation channel resources, increase energy consumption and computational burden, but more importantly, they cannot effectively convey key social information such as facial expressions, nor can they solve the reading difficulties caused by high text information density. This makes it difficult for blind users to recognize others' emotions and complete basic reading tasks, severely limiting their social participation and quality of life. At the same time, existing technologies lack a mechanism to effectively combine high-level semantic information with limited pixel channels and fail to establish a personalized topological mapping from visual space to the visual cortex, resulting in a mismatch between stimulation signal generation and user perceptual characteristics. Therefore, there is an urgent need for an innovative technical solution that can efficiently convey facial expressions and text information under low-pixel conditions. Summary of the Invention

[0003] In view of this, this invention proposes a semantically driven biomimetic visual information perception method and system, which can efficiently perceive facial expressions and text information under low-pixel conditions. This invention provides the following technical solution: A semantically driven biomimetic visual information perception method, the method comprising: Obtain scene image information containing the target content; Semantic parsing is performed on the scene image information to extract high-level semantic information, and high-level semantic weights are calculated based on the high-level semantic information; The visual features of the scene image information are identified, and the visual features and the high-level semantic weights are fused to filter out key information pixels from the scene image information. Based on the preset topological correspondence between visual space and visual cortex, the key information pixels are mapped to the corresponding cortical electrode numbers, and a stimulation vector corresponding to the cortical electrode number is generated. The cortical electrode number and stimulation vector are output to the bionic visual stimulation device.

[0004] Optionally, the target content includes facial expression information or text information.

[0005] Optionally, performing semantic parsing on the scene image information to extract high-level semantic information, and calculating high-level semantic weights based on the high-level semantic information includes: When the scene image information contains facial expression information, the face region is located by a face detection algorithm, and the expression category label is extracted as high-level semantic information by a preset expression recognition model. Based on the expression category labels and the confidence level output by the expression recognition model, the facial region is divided into the eye region and the mouth region. The high-level semantic weights of the corresponding regions are calculated based on the geometric feature parameters of the eye region and the mouth region.

[0006] Optionally, performing semantic parsing on the scene image information to extract high-level semantic information, and calculating high-level semantic weights based on the high-level semantic information includes: When the scene image information contains text information, the text content of the text information is extracted as high-level semantic information using an optical character recognition algorithm. Based on the character sequence and timestamp information of the text content, high-level semantic weights are assigned to characters at different positions in the character sequence.

[0007] Optionally, the visual features of the scene image information are identified, and the visual features and the high-level semantic weights are fused to filter out key information pixels from the scene image information. Identify low-level visual features of the scene image information, including brightness distribution and edge intensity; The low-level visual features and the high-level semantic weights are weighted and fused to generate a pixel comprehensive weight distribution map; The maximum number of displayable pixels N is determined based on the pixel channel limit of the bionic vision system; Based on the pixel comprehensive weight distribution map, the top N pixels with the highest weight values ​​are selected as key information pixels.

[0008] Optionally, the step of mapping the key information pixels to corresponding cortical electrode numbers based on a preset topological correspondence between visual space and the visual cortex, and generating stimulation parameters corresponding to the cortical electrode numbers, includes: Load the topological correspondence between visual space and visual cortex, and establish the initial mapping relationship between image pixels and electrode channels; Based on the user's individual electrode implantation data, the initial mapping relationship is personalized and modified to generate a unique mapping matrix adapted to the user. Based on the exclusive mapping matrix, the selected key information pixels are mapped to the corresponding cortical electrode numbers; Based on the pixel brightness values ​​of key information pixels and the user's perception threshold curve, each cortical electrode number is converted into stimulation parameters containing amplitude, frequency, and duration, generating the final stimulation vector.

[0009] Optionally, outputting the cortical electrode number and stimulation vector to a bionic visual stimulation device to enable the user to perceive the scene image information includes: The cortical electrode numbers and stimulation parameters are converted into electrical stimulation control signals, and stimulation frames are output to the stimulation device to directly stimulate the user's visual cortex or surface nerves. A double-buffered rendering mechanism is constructed to maintain two display buffers. The current stimulus frame is rendered in the first buffer, and the next stimulus frame is pre-rendered in the second buffer. The time-stamp-driven switching mechanism records the start and end timestamps of each stimulus update. When the stimulus frame rate is lower than a preset threshold, a linear interpolation algorithm is used to generate a transition frame between the two buffers. When the interval between two consecutive stimulus frames exceeds a preset threshold, a dynamic frame interpolation mechanism is activated, reducing the transparency of the stimulus content in the previous frame to 0, while simultaneously overlaying the stimulus content in the new frame to ensure a smooth transition in visual perception.

[0010] This invention further discloses a semantically driven biomimetic visual information perception system, comprising: The image acquisition module is used to acquire scene image information containing the target content; The semantic parsing module is used to perform semantic parsing on the scene image information to extract high-level semantic information and calculate high-level semantic weights based on the high-level semantic information. The key information pixel filtering module is used to identify the visual features of the scene image information, and fuse the visual features and the high-level semantic weights to filter out key information pixels from the scene image information. The pixel mapping module is used to map the key information pixels to corresponding cortical electrode numbers based on a preset topological correspondence between visual space and visual cortex, and to generate a stimulation vector corresponding to the cortical electrode number. The output control module is used to output the cortical electrode number and stimulation vector to the bionic visual stimulation device.

[0011] The present invention further discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0012] The present invention further discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described method.

[0013] This invention designs a semantically driven key information pixel selection mechanism. By fusing high-level semantic information with low-level visual features, it can select key information pixels from scene image information. Furthermore, through a preset topological correspondence between visual space and visual cortex, the selected key information pixels are accurately mapped to the corresponding cortical electrode numbers and stimulation vectors are generated. This not only significantly reduces the computational and energy consumption pressure of transmitting image data, but also achieves personalized adaptation to the individual electrode implantation of users, significantly improving the accuracy and comfort of visual stimulation. In addition, it effectively solves the technical problem of conveying facial expressions and text information under extremely low pixel channel conditions, achieving efficient expression of target content in scene images under low pixel channel conditions. Attached Figure Description

[0014] For illustrative and not limiting purposes, the present invention will now be described in conjunction with embodiments and accompanying drawings, wherein: Figure 1 This is a flowchart illustrating the semantically driven biomimetic visual information perception method in an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of a semantically driven bionic visual information perception system according to an embodiment of the present invention. Figure 3 This is a schematic diagram of the structure of the electronic device in an embodiment of the present invention; Figure 4 This is an exemplary facial expression recognition diagram in an embodiment of the present invention; Figure 5 This is an exemplary text recognition diagram in an embodiment of the present invention. Detailed Implementation

[0015] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present application.

[0016] It should be noted that, where there is no conflict, the embodiments and features of the embodiments in this application can be combined with each other. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0017] refer to Figure 1 This embodiment discloses a semantically driven biomimetic visual information perception method, which includes the following steps: S100: Acquire scene image information containing the target content.

[0018] In this embodiment, the scene image information is a virtual scene. The user selects the training type through an interface with virtual scene generation functionality and generates a scene containing target content. The target content includes facial expressions and text for targeted training of the user. After scene generation, a preset camera is activated for real-time image capture. For the two types of target content in this embodiment—facial expression information and text information—a preliminary analysis of the scene is performed during the image acquisition stage. When a facial area or dense text area is detected in the image, the camera's focus and exposure strategies are automatically adjusted to ensure image quality in these areas.

[0019] S200: Perform semantic parsing on the scene image information to extract high-level semantic information, and calculate high-level semantic weights based on the high-level semantic information.

[0020] First, the received scene image is preprocessed, including color space conversion and brightness normalization, to provide standardized input for subsequent semantic parsing. In this embodiment, the target content includes facial expression information or text information, and a dual-channel mechanism for facial expression recognition and text recognition is constructed through parallel processing.

[0021] refer to Figure 4 This diagram illustrates face detection, emotion extraction, and the extraction of key information pixels after weighting within the expression recognition pathway. In the expression recognition pathway, OpenCV DNN or MTCNN is used for face detection. After accurately locating the face region, a pre-trained FER (Face Expression Recognition) model is used to classify the detected face region into emotions, identifying basic emotion categories such as happiness, anger, and surprise as high-level semantic information. Subsequently, based on the emotion classification results, the face region is divided into two key regions: the eye region and the mouth region. High-level semantic weights are then calculated based on the confidence scores output by the FER model.

[0022] Specifically, given a face detection bounding box And perform the following region division: Left eye circle The center of the circle is located at ,radius ; Right eye circle The center of the circle is located at Its radius is the same as that of the left eye's circular area; osteal region of the mouth The center is located at axial length .

[0023] in, This means restricting x to the interval Inside, and These represent the width and height of the face detection bounding box, respectively. Each region is further defined using the following indicator functions: ; ; .

[0024] The formula for calculating the semantic weight of the face region is: ,in, and These are the weighting coefficients for the eye region and the mouth region, respectively, with values ​​ranging from [value range missing]. The default value is 0.5. The accuracy of the region definition is ensured by the indicator function 1, and the face detection box is cropped to the image boundary before the above geometric region is calculated.

[0025] Furthermore, the brightness enhancement coefficient is calculated based on the probability distribution output by the FER model. Define positive emotion set and negative emotions The formula for calculating the brightness enhancement factor is: ,in, K represents the confidence level of the sentiment category c output by the FER model, and K is the moderation coefficient. and The minimum and maximum enhancement coefficients are respectively calculated. Apply semantic weights to obtain enhanced facial region weights: If the confidence level of the main emotion category output by the FER model... , Then, the pixel brightness of the corresponding area is enhanced, and the enhancement amount is... ,in and These are the preset maximum and minimum brightness values, respectively. Simultaneously, the pixel position offset is dynamically adjusted based on the upward angle θ of the corners of the mouth. This ensures that facial features can still be accurately expressed even under low-resolution conditions.

[0026] In the text recognition process, the PaddleOCR engine is invoked to recognize text regions in the image, extracting complete text content as high-level semantic information. The recognized text sequence is then segmented into character units, and dynamically changing high-level semantic weights are assigned to characters at different positions within the sequence based on timestamp information. Specifically, the system defines the distance from each character to its boundary. The spatial weights are calculated using a cosine window function. ,in, This is the preset transition width. Combining a timestamp-driven scrolling window mechanism, the system records the time when text recognition is completed as a baseline. Based on the difference between the current time and the baseline time, it calculates the elapsed time and then determines the starting index of the character to be displayed. Where T is the character interval, and the time weight is calculated using the following formula: ,in, The attenuation coefficient is... Indicates character index With display window The distance. Therefore, the final high-level text weights are: ,in, For the first A bounding box for each character.

[0027] If the current scene image contains both facial and text information, a dual-path conflict resolution mechanism is used to determine the priority. Specifically, a path scoring system is defined: ,in, This represents the highest confidence level of the emotion output by the model within the face detection area. A collection of face regions. This represents the area of ​​the face detection bounding box. For a single face region, For OCR confidence, and These are the weighting coefficients. Further, the pathway mixing weights are calculated as follows: ,in, For the mixed weight of the pathway, This is the slope parameter. Therefore, the comprehensive high-level semantic weights are determined: .

[0028] In summary, for overlapping regions, this implementation provides the following strategy: If In this case, face masks are prioritized (the maximum value is used for synthesis), while text masks are prioritized. For non-overlapping regions, the combined high-level semantic weights of the face region and the text region are calculated separately to achieve independent expression and parallel transmission of the two types of information within their respective regions.

[0029] refer to Figure 5This diagram illustrates text extraction, character segmentation, and scrolling display within the text recognition pathway. In the text recognition pathway, the PaddleOCR engine is invoked to recognize text regions in the image, extracting complete text content as high-level semantic information. The recognized text sequence is then segmented into character units, and dynamically changing high-level semantic weights are assigned to characters at different positions based on timestamp information. Specifically, the time point at which text recognition is completed is recorded as a baseline. The elapsed time is calculated based on the difference between the current time and the baseline time, thus determining the starting index of the character to be displayed. The three characters in the currently displayed area are assigned the highest weight value, while characters in the upcoming and already displayed areas are assigned gradually changing weight values, ensuring that text information is fully expressed within a limited pixel area through temporal changes. Furthermore, the dominant pathway is automatically selected based on the scene content. When both faces and text are detected simultaneously, the resource allocation between the two pathways is dynamically adjusted according to the user-preset priority or the current task type.

[0030] S300: Identify the visual features of the scene image information, and fuse the visual features and the high-level semantic weights to filter out key information pixels from the scene image information.

[0031] The Retinex algorithm is employed for brightness equalization to compensate for uneven illumination under different lighting conditions, ensuring that key visual information is preserved even under low-contrast or complex lighting conditions, and obtaining the brightness distribution of scene image information. Subsequently, the system uses a combination of the Canny edge detection algorithm and the Sobel operator to extract structural features of the image. The Canny algorithm smooths the image with Gaussian filtering and then calculates the gradient magnitude and direction, obtaining accurate edge information through non-maximum suppression and double thresholding. The Sobel operator further enhances the edge features in the horizontal and vertical directions. The complementary use of these two algorithms ensures accurate capture of object contours even with limited pixels, thus obtaining the edge intensity features of scene image information. Brightness distribution and edge intensity features constitute the low-level visual features of scene image information.

[0032] For the facial expression recognition pathway, the high-level semantic weights of the facial regions obtained in step S200 (including the weight values ​​of the eye and mouth regions) are weighted and fused with the extracted brightness distribution and edge intensity features. For the text recognition pathway, the text regions are determined based on the OCR results, and dynamic semantic weights based on character positions are assigned to each pixel within the text regions. Simultaneously, the text edge features extracted using the MSR+Canny algorithm are combined to generate a comprehensive weight distribution map. Specifically, to achieve feature standardization, this implementation normalizes the extracted brightness and edge features: Normalized brightness: Normalized edge: The synthesis of low-level visual features employs a weighted combination method: ,in, and These are the weighting coefficients. , ,and For high-level semantic weights, the weights obtained in step S200 are... With brightness enhancement coefficient By combining these, we obtain the enhanced high-level semantic weights: The pixel comprehensive weight is calculated by weighted fusion of low-level features and high-level semantic weights: ,in The fusion coefficient is λ. To optimize the fusion effect, an adaptive mechanism is used to dynamically adjust the fusion coefficient λ. ,in, Based on the fusion coefficient, , , For adjustment coefficients, This represents the confidence level of the current main pathway. For the facial expression pathway, it is... For text paths, the average confidence level of OCR is used, and noise is the image noise or motion blur estimation.

[0033] After weighted fusion of low-level visual features and high-level semantic weights, the maximum number of displayable pixels N is determined based on the actual pixel channel limit of the bionic vision device. All pixels are sorted according to the pixel comprehensive weight distribution map, and the top N pixels with the highest weight values ​​are selected as key information pixels. These pixels retain sufficient structural information while highlighting the key areas emphasized by the high-level semantics. In expression recognition scenarios, they can clearly express emotional features, and in text reading scenarios, they ensure the continuity and recognizability of character strokes.

[0034] S400: Based on the preset topological correspondence between visual space and visual cortex, the key information pixels are mapped to the corresponding cortical electrode numbers, and a stimulation vector corresponding to the cortical electrode number is generated.

[0035] First, a standard topological mapping table from visual space to primary visual cortex electrodes is loaded. This table establishes an initial mapping relationship between visual field coordinates and electrode channels in the V1 region of the primary visual cortex, mapping each pixel in the image space to its corresponding cortical electrode location. After acquiring the user's individual electrode implantation data, the actual position information of the electrodes in the patient's brain is obtained through CT registration or functional imaging techniques. Then, affine or nonlinear corrections are performed on the standard mapping to generate a user-specific mapping matrix. This matrix reflects the precise correspondence between the user's specific visual cortical tissue structure and the electrode implantation location. Further, the coordinates of the key information pixels selected in step S300 are substituted into the specific mapping matrix to obtain the cortical electrode number corresponding to each key information pixel.

[0036] When generating the stimulus vector, the brightness values ​​of key information pixels are first converted into preliminary stimulus amplitudes. This conversion process must consider the influence of the perception threshold curve, which is obtained through prior psychophysical measurements of the user and describes the user's perceptual characteristics to different stimulus parameters. Specifically, the user's perception threshold curve is retrieved to determine the stimulus amplitude range that matches the pixel brightness values, and adjustments are made based on the user's comfort threshold to ensure that the stimulus intensity is within a perceptible range without causing discomfort. Subsequently, corresponding stimulus parameters are configured for each cortical electrode number, including amplitude, frequency, and duration. The amplitude parameter is directly related to the pixel brightness information, while the frequency and duration parameters are dynamically adjusted based on the user's historical feedback data and the current task type. Finally, the system combines the electrode number with the corresponding stimulus parameters to form a complete stimulus vector. This stimulus vector contains spatial location information (electrode number), stimulus intensity information (amplitude), and time dimension parameters (frequency and duration), together constituting a complete electrical stimulation instruction.

[0037] S500: Output the cortical electrode number and stimulation vector to the bionic visual stimulation device to enable the user to perceive the scene image information.

[0038] In this embodiment, a two-stage application of the bionic visual stimulation device is disclosed, including a preoperative stage and a postoperative stage. In the preoperative stage, a simulated optical illusion matrix is ​​generated using a universal mapping relationship for user training and parameter pre-calibration; in the postoperative stage, the personalized mapping matrix modified in step S400 is used to drive the actually implanted electrodes to achieve precise visual stimulation.

[0039] First, the system automatically determines the output channel type based on its current operating status. When in the preoperative training or volunteer testing phase, the cortical electrode number and stimulation vector are converted into a graphical display signal and output to a screen projection device or virtual reality headset to generate a simulated light illusion matrix. At this time, the system maps the electrode number to the corresponding light spot, converts the amplitude parameter in the stimulation vector into the brightness of the light spot, and converts the frequency and duration parameters into the flickering characteristics of the light spot.

[0040] When in the postoperative clinical application stage, the cortical electrode numbers and stimulation vectors are converted into electrical stimulation control signals and transmitted to implantable or surface electrical stimulation devices via a serial communication interface to directly stimulate the user's visual cortex or surface nerves. At this point, the electrode number corresponds to the actual stimulation electrode channel, the amplitude parameter controls the current intensity, the frequency parameter controls the stimulation pulse frequency, and the duration parameter controls the duration of a single stimulation. To ensure the continuity and smoothness of the output, a dual-buffering rendering mechanism is adopted, maintaining two independent display buffers. The current stimulation frame is fully rendered in the first buffer, while the next frame's stimulation content is preprocessed in the second buffer. After rendering is complete, the output buffer is switched instantaneously through a synchronization mechanism to avoid stuttering and overlap caused by direct replacement. Furthermore, a timestamp-driven switching strategy is implemented. The start and end timestamps are accurately recorded each time stimulation is updated. When the system frame rate is detected to be lower than a preset threshold (e.g., 5fps), a linear interpolation algorithm is automatically activated to generate several transition frames between the two buffers, achieving a gradual switching of stimulation content and effectively eliminating the feeling of frame skipping. Specifically, when the interval between two consecutive stimulus frames exceeds 200 milliseconds, a dynamic frame interpolation mechanism is immediately activated. The transparency of the previous stimulus frame is linearly reduced to 0, while the new stimulus frame is superimposed and its transparency is gradually increased. This smooth transition technique ensures a consistent visual experience for the user, even under high computational load or system response latency. The entire output control process strictly adheres to the perceptual threshold curve, ensuring all stimulus parameters are within a perceptible and comfortable range for the user. User feedback is recorded in real-time for subsequent optimization of stimulus parameters and adjustment of mapping relationships. Specifically, this implementation maintains two independent display buffers, BufferA and BufferB. The current stimulus frame is fully rendered in BufferA, while the next stimulus content is prepared in advance in BufferB. Once rendering is complete, the output buffer is switched instantaneously via GPU synchronization, avoiding stuttering and overlap caused by direct replacement. The start and end timestamps are precisely recorded for each stimulus update. and When the system frame rate is detected to be lower than a preset threshold... At that time, the linear interpolation algorithm is activated to generate a transition frame: , ,in, and These are the stimulus contents for the current frame and the next frame, respectively. These are the interpolation coefficients, which change linearly with time.

[0041] When the interval between two consecutive frames of stimulation When the preset threshold is exceeded, dynamic frame interpolation is initiated: The transparency of older frames decays exponentially. Or linear decay: New frame transparency overlay: .in, This is the decay time constant.

[0042] This embodiment exemplifies a method for face recognition and expression recognition based on the above method: The system receives real-time video frames from a camera in BGR color space format as input and performs face detection using a deep neural network. After initializing the facial region list and emotion name list, the system obtains the height and width parameters of the current frame image to provide coordinate references for subsequent processing. During the face detection phase, the system iterates through the detection results returned by the deep neural network, filters out detection items with a confidence level higher than 0.5, converts relative coordinates to absolute coordinates, and performs boundary cropping to ensure the detection boxes are completely within the image area. Finally, the effective face detection regions are stored in the facial region list in the format (x, y, w, h). When the system detects a face, it marks the number of detected faces on the image; if no face is detected, it displays the message "No face detected." For each detected face region, the system extracts the corresponding region of interest (ROI) and converts the color space from BGR to RGB format to adapt to the input requirements of the emotion recognition model. Subsequently, it calls a pre-loaded facial expression recognition (FER) model to classify the converted RGB face image for emotion and obtain the emotion probability distribution. During processing, the system incorporates an exception handling mechanism. When a runtime error occurs during emotion recognition, the results are automatically set to an empty list, and the error information is recorded. From the emotion probability distribution, the system selects the emotion category with the highest probability as the final emotion recognition result. Emotion categories include basic emotion types such as "happy," "angry," and "surprised," and the recognition result is added to the emotion name list. In the emoji generation stage, the system dynamically adjusts the visual expression of facial features based on the recognized emotion category. When an emoji overlay image is configured, it adjusts the image to a size that matches the detected face area and overlays it onto the corresponding position on the processed image. If no overlay image is configured, the system generates emojis using a pixel-level drawing method: first, two white circles are drawn in the face area as eye representations, and then the mouth shape drawing method is dynamically adjusted according to the emotion category. Specifically, when the identified emotion belongs to the positive category (such as "happy"), the system draws an upward-curving arc of the corners of the mouth (an upward elliptical arc); when the identified emotion belongs to the negative category (such as "anger" or "sadness"), the system draws a downward-curving arc of the corners of the mouth (a downward elliptical arc); for neutral emotions, the system draws a horizontal straight line as the mouth shape representation. The pseudocode for this example is as follows: Input: frame ←Current webcam frame (BGR image) detections ←DNN face detection results overlay_img ←Optional overlay image (eg emoji) emotion_detector←Pre-loaded FER emotion detector processed_img ←Image buffer for visual output Output: processed_img ←Image with emotion-dependent face rendering emotion_names[ ]←List of detected emotion labels Procedure: Initialize face_boxes←[ ] Initialize emotion_names←[ ] Get image size: (h, w)←frame.shape for each detection i in detections do confidence←detections[0, 0, i, 2] if confidence>0.5 then box←detections[0, 0, i, 3:7]×[w, h, w, h] (startX, startY, endX, endY)←box.astype(int) Clip to image bounds: startX←max(0, startX) startY←max(0, startY) endX←min(w-1, endX) endY←min(h-1, endY) Append face box to face_boxes: face_boxes.append((startX, startY, endX - startX, endY- startY)) if face_boxes is not empty then print("Detected " + face_boxes.length + " faces") Draw text "Faces: N" on frame else print("No faces detected") Draw text "No Face" on frame for each face_box ∈ face_boxes do (x, y, fw, fh) ← face_box face_roi ← crop(frame, region=(x, y, fw, fh)) rgb_face ← convert_BGR_to_RGB(face_roi) try results ← emotion_detector.detect_emotions(rgb_face) catch RuntimeError then results ← print("Emotion detection failed") if results not empty then emo_dict ← results[0]["emotions"] emo ← argmax(emo_dict) else emo ← "Unknown" emotion_names.append(emo) for each face_box, emo in zip(face_boxes, emotion_names) do (x, y, fw, fh) ← face_box if overlay_img is not None then small ← resize(overlay_img, (fw, fh)) place small image on processed_img at (x, y) else Draw two eyes as white circles Calculate mouth_center and mouth_axes if emo∈negative then Draw sad mouth arc (downward ellipse) else if emo∈positive then Draw happy mouth arc (upward ellipse) else Draw straight mouth line This embodiment exemplifies a text recognition and stimulation method based on the above method: First, the system receives PaddleOCR command-line arguments and the output directory as input. A subprocess is executed to call the PaddleOCR engine for text recognition processing. The system captures and records PaddleOCR's standard and error output information for subsequent debugging and performance analysis. During recognition, the system specifies the storage path for the result file as `system_results.txt` in the output directory and initializes OCR text variables to store the recognition results. When the result file exists and is not empty, the system reads the last line of valid data in the file. This line is tab-delimited, and the second part contains the recognition results in JSON format. The system parses this JSON data, extracts the transcribed text content of all recognized items, and concatenates them into a complete OCR recognition result string. The system then verifies the recognition result. If the recognized text content is not empty, the string is converted into a list of individual characters, serving as the basic data structure for subsequent scrolling display; if the recognition result is empty, an empty character list is returned. Throughout the process, the system accurately records the timestamp of text recognition completion as the baseline time point for subsequent timing control algorithms, ensuring the accuracy and consistency of character scrolling display.

[0043] The pseudocode for this example is as follows: Input: args ←Subprocess command-line arguments for PaddleOCR out_dir ←Directory where PaddleOCR stores output Output: text_chars[ ] ← List of recognized characters text_start_time ← Timestamp when text recognition completed Procedure: result ← execute_subprocess(args, capture_output = True) print("===== PaddleOCR STDOUT =====") print(result.stdout) print("===== PaddleOCR STDERR =====") print(result.stderr) res_path ← out_dir + " / system_results.txt" ocr_text ← "" if file_exists(res_path) then lines ← read_non_empty_lines(res_path) if lines is not empty then last_line ← lines[-1] json_str ← split(last_line, delimiter = "\t")[1] items ← parse_json(json_str) ocr_text ← join_text(item.transcription for item in items) print("===== OCR Recognition Result =====") print(ocr_text) if ocr_text ≠ "" then text_chars ← convert_to_char_list(ocr_text) else text_chars ← [ ] text_start_time←current_time() This embodiment further exemplifies a text scrolling playback method: The system dynamically selects the processing path based on the current operating mode: when the system is in text reading mode, it executes the text scrolling playback algorithm; when the system is in the normal visual perception mode, it executes edge enhancement processing. In text reading mode, a completely black background image of the same size as the original camera frame is first created as the display basis. This design ensures maximum readability of characters under low contrast conditions. Then, the system accurately calculates the elapsed time since the start of reading. By dividing the elapsed time by the character switching interval (1.0 second) and rounding down, the starting index of the character to be displayed is determined. This character switching interval parameter can be dynamically adjusted according to the individual needs of visually impaired users. The optimal range determined in tests with non-visually impaired users is 1.2 seconds ± 0.3 seconds. The system checks whether the current index is still within the character sequence range. If it is, it extracts three consecutive characters starting from the current index to form the display content. This grouping strategy of three characters per group ensures the semantic coherence of the phrases. If the index exceeds the character sequence range, "END" is displayed to indicate that reading is complete. Next, the system converts the black background image into PIL (Python Imaging). The system uses a PIL (Platform Library) format to obtain the drawing context and draws the currently displayed character content at the center of the image, using high-contrast white (brightness value 255) to ensure clear visibility under limited pixel conditions. Finally, the processed PIL image is converted back to NumPy array format as the final output. In the normal visual perception mode, the system applies an edge enhancement filtering algorithm to the original image, using specific parameter configuration (mode='object', param1=7, param2=3.0, param3=50, param4=1) to highlight the outlines of the main objects in the image, preserving key structural information while reducing unnecessary visual interference.

[0044] The pseudocode for this example is as follows: INPUT: FRAME ←RAW ​​CAMERA FRAME (RGB OR GRAYSCALE) READING_MODE ←BOOLEAN FLAG INDICATING READING MODE TEXT_CHARS[] ←CHARACTER ARRAY FOR DYNAMIC READING TEXT TEXT_START_TIME ←TIMESTAMP WHEN READING BEGAN FONT ←FONT USED FOR TEXT RENDERING OUTPUT: PROCESSED_IMG ←FINAL PROCESSED IMAGE (TEXT OR EDGE-ENHANCED) PROCEDURE: if reading_mode == True then processed_img←black image with same dimensions as frame elapsed ←current time − text_start_time index ←floor(elapsed / 1.0) if index<length(text_chars) then disp←concatenate(text_chars[index : index+3]) else disp←"END" pil_img←convert_to_PIL(processed_img) draw ←get_drawing_context(pil_img) draw_text_centered(draw, disp, font, color = 255) processed_img←convert_to_numpy(pil_img) else processed_img←filter_main_body_edges( input = frame, mode ='object', param1 = 7, param2 = 3.0, param3 = 50, param4 = 1 ) return processed_img In summary, this implementation method utilizes structured facial expression technology to transform complex facial expressions into pixel-level symbolic representations. Combined with a dynamic mapping mechanism between the upward angle of the corners of the mouth (θ) and the confidence level (p), it accurately conveys multiple basic emotions within the limitation of only a few dozen pixel channels, enabling users to gain the social ability to recognize others' expressions. Secondly, addressing the challenge of text reading, a scrolling reading strategy is proposed. Through a dual-buffering rendering mechanism and timestamp-driven switching technology, the OCR recognition results are converted into a character-interval scrolling mode, solving the problems of text information stacking and poor readability under low-resolution conditions in traditional methods. Furthermore, through topological mapping of the visual cortex, standard mapping is combined with personalized correction to achieve precise mapping from image pixels to cortical electrodes, significantly reducing the computational and energy consumption pressure of transmitting image data. Finally, the system adopts a dual-mode output mechanism, supporting both preoperative training for screen projection devices and driving implantable or surface electrical stimulation devices for real electrical stimulation, and is compatible with multiple mainstream bionic visual stimulation systems, significantly improving user efficiency.

[0045] refer to Figure 2 This embodiment further discloses a semantically driven biomimetic visual information perception system, including an image acquisition module 21, a semantic parsing module 22, a key information pixel filtering module 23, a pixel mapping module 24, and an output control module 25, which are described in detail below: Image acquisition module 21 is used to acquire scene image information containing the target content; The semantic parsing module 22 is used to perform semantic parsing on the scene image information to extract high-level semantic information and calculate high-level semantic weights based on the high-level semantic information. The key information pixel filtering module 23 is used to identify the visual features of the scene image information, and fuse the visual features and the high-level semantic weights to filter out key information pixels from the scene image information. The pixel mapping module 24 is used to map the key information pixels to corresponding cortical electrode numbers based on a preset topological correspondence between visual space and visual cortex, and to generate a stimulation vector corresponding to the cortical electrode number. The output control module 25 is used to output the cortical electrode number and stimulation vector to the bionic visual stimulation device so as to enable the user to perceive the scene image information.

[0046] Figure 3 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as... Figure 3 As shown, the electronic device 50 includes: a processor 501, a memory 502, and a bus 503; The processor 501 and the memory 502 communicate with each other via the bus 503; the processor 501 is used to call the program instructions in the memory 502 to execute the methods provided in the above-described embodiments.

[0047] This embodiment provides a non-transitory computer-readable storage medium that stores computer instructions that cause a computer to execute the methods provided in the above-described embodiments.

[0048] Those skilled in the art will understand that all or part of the steps of the above-described method implementation can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above-described method implementation. The aforementioned storage medium includes various storage media capable of storing program code, such as ROM, RAM, magnetic disk, or optical disk.

[0049] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0050] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of each embodiment or some parts of the embodiments.

[0051] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A semantic-driven based bionic visual information perception method, characterized in that, The method includes: Obtain scene image information containing the target content; Semantic parsing is performed on the scene image information to extract high-level semantic information, and high-level semantic weights are calculated based on the high-level semantic information; The visual features of the scene image information are identified, and the visual features and the high-level semantic weights are fused to filter out key information pixels from the scene image information. Based on the preset topological correspondence between visual space and visual cortex, the key information pixels are mapped to the corresponding cortical electrode numbers, and a stimulation vector corresponding to the cortical electrode number is generated. The cortical electrode number and stimulation vector are output to the bionic visual stimulation device.

2. The method according to claim 1, wherein, The target content includes facial expression information or text information.

3. The biomimetic visual information perception method according to claim 2, characterized in that, The step of performing semantic parsing on the scene image information to extract high-level semantic information, and calculating high-level semantic weights based on the high-level semantic information includes: When the scene image information contains facial expression information, the face region is located by a face detection algorithm, and the expression category label is extracted as high-level semantic information by a preset expression recognition model. Based on the expression category labels and the confidence level output by the expression recognition model, the facial region is divided into the eye region and the mouth region. The high-level semantic weights of the corresponding regions are calculated based on the geometric feature parameters of the eye region and the mouth region.

4. The biomimetic visual information perception method according to claim 2, characterized in that, The step of performing semantic parsing on the scene image information to extract high-level semantic information, and calculating high-level semantic weights based on the high-level semantic information includes: When the scene image information contains text information, the text content of the text information is extracted as high-level semantic information using an optical character recognition algorithm. Based on the character sequence and timestamp information of the text content, high-level semantic weights are assigned to characters at different positions in the character sequence.

5. The biomimetic visual information perception method according to claim 1, characterized in that, The step of identifying visual features of the scene image information and fusing the visual features with the high-level semantic weights to filter out key information pixels from the scene image information includes: Identify low-level visual features of the scene image information, including brightness distribution and edge intensity; The low-level visual features and the high-level semantic weights are weighted and fused to generate a pixel comprehensive weight distribution map; The maximum number of displayable pixels N is determined based on the pixel channel limit of the bionic vision system; Based on the pixel comprehensive weight distribution map, the top N pixels with the highest weight values ​​are selected as key information pixels.

6. The biomimetic visual information perception method according to claim 3, characterized in that, The process of mapping key information pixels to corresponding cortical electrode numbers based on a preset topological correspondence between visual space and the visual cortex, and generating stimulation parameters corresponding to the cortical electrode numbers, includes: Load the topological correspondence between visual space and visual cortex, and establish the initial mapping relationship between image pixels and electrode channels; Based on the user's individual electrode implantation data, the initial mapping relationship is personalized and modified to generate a unique mapping matrix adapted to the user. Based on the exclusive mapping matrix, the selected key information pixels are mapped to the corresponding cortical electrode numbers; Based on the pixel brightness values ​​of key information pixels and the user's perception threshold curve, each cortical electrode number is converted into stimulation parameters containing amplitude, frequency, and duration, generating the final stimulation vector.

7. The biomimetic visual information perception method according to claim 1, characterized in that, The step of outputting the cortical electrode number and stimulation vector to the bionic visual stimulation device includes: The cortical electrode numbers and stimulation parameters are converted into electrical stimulation control signals, and stimulation frames are output to the stimulation device to directly stimulate the user's visual cortex or surface nerves. A double-buffered rendering mechanism is constructed to maintain two display buffers. The current stimulus frame is rendered in the first buffer, and the next stimulus frame is pre-rendered in the second buffer. The time-stamp-driven switching mechanism records the start and end timestamps of each stimulus update. When the stimulus frame rate is lower than a preset threshold, a linear interpolation algorithm is used to generate a transition frame between the two buffers. When the interval between two consecutive stimulus frames exceeds a preset threshold, a dynamic frame interpolation mechanism is activated, reducing the transparency of the stimulus content in the previous frame to 0, while simultaneously overlaying the stimulus content in the new frame to ensure a smooth transition in visual perception.

8. A semantically driven biomimetic visual information perception system, characterized in that, include: The image acquisition module is used to acquire scene image information containing the target content; The semantic parsing module is used to perform semantic parsing on the scene image information to extract high-level semantic information and calculate high-level semantic weights based on the high-level semantic information. The key information pixel filtering module is used to identify the visual features of the scene image information, and fuse the visual features and the high-level semantic weights to filter out key information pixels from the scene image information. The pixel mapping module is used to map the key information pixels to corresponding cortical electrode numbers based on a preset topological correspondence between visual space and visual cortex, and to generate a stimulation vector corresponding to the cortical electrode number. The output control module is used to output the cortical electrode number and stimulation vector to the bionic visual stimulation device so as to enable the user to perceive the scene image information.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method of any one of claims 1-7.

Images