A first-person view angle auxiliary perception method for people with visual field defects

Abstract

The application provides a first-person visual angle auxiliary perception method for people with visual field defects, and relates to the technical field of visual technology.The application realizes automatic recording and auxiliary perception of target objects based on initial calibration and recording habits of users, and comprises the following steps: initialization calibration, user habit learning, multi-modal data acquisition, automatic recording and auxiliary perception.The method combines personalized needs of users and characteristics of visual field defects, and improves the perception efficiency and integrity of target objects for people with visual field defects through active learning and automatic recording.

Description

Technical Field

[0001] This invention relates to the technical field of visual technology, and more particularly to a first-person perspective-assisted perception method for people with visual impairments. Background Technology

[0002] People with visual impairments often experience narrowed visual fields and localized blind spots due to eye diseases, injuries, or other reasons. In daily life, including outings, home life, work, and study, they frequently fail to notice obstacles, signs, objects, and other key targets, increasing the risk of safety hazards. Furthermore, they struggle with basic tasks such as information recording and environmental assessment, severely impacting their autonomy and quality of life. Therefore, there is an urgent need for targeted assistive perception methods to address this visual challenge. These assistive perception methods primarily involve real-time monitoring of the user's surroundings, capturing key targets and environmental information both inside and outside the field of vision. They proactively identify and alert users to objects they might not have noticed due to visual impairments through voice, touch, and other means. Simultaneously, they adapt to individual user habits to record and organize information, helping people with visual impairments navigate daily scenarios more safely and conveniently, enhancing their ability to live independently and participate in society.

[0003] Currently, existing visual recording and assisted perception technologies have significant limitations: most technologies rely on users manually triggering the marking function, which is cumbersome and requires users to actively focus on the target, making it impossible to adapt to the visual limitations of people with visual impairments; while some technologies support automatic recording, they lack the ability to learn and adapt to users' personalized needs, making it difficult to match different users' recording habits and focus points, thus greatly reducing their practicality; at the same time, existing technologies do not fully consider the blind spots of people with visual impairments, and cannot actively identify and prompt users to identify target objects that they have not noticed due to visual impairments, which can easily lead to the omission of key information, ultimately failing to meet the diverse and refined daily assisted perception needs of people with visual impairments. Summary of the Invention

[0004] In response to the technical problems mentioned in the background section, this invention provides a first-person perspective-assisted perception method for people with visual impairments. This invention achieves automatic recording and assisted perception of target objects based on the user's initial calibration and recording habits.

[0005] The technical means employed in this invention are as follows:

[0006] A first-person perspective-assisted perception method for people with visual impairment includes the following steps: Step 1: Initialize target object calibration and visual field defect area division; the user inputs the object name through smart glasses or touch interface, and fuzzy matching and information storage are achieved through multimodal fusion and vector database; at the same time, the visual field range is calibrated by flashing the equidistant pixels point by point, and the initial visual field defect area is divided. Step 2: User Habit Learning; Collect multi-dimensional feature data of user marking operations, use clustering algorithms to subdivide scene types, extract user recording habit rules through gradient boosting tree algorithm, build a habit prediction model based on long short-term memory network combined with attention mechanism, and introduce forgetting curve model to dynamically adjust object matching weight; at the same time, predict the user's gaze landing point through gaze estimation model, and merge it with the original visual field defect area data to update the current visual field defect area; Step 3: Multimodal data acquisition; real-time acquisition of first-person perspective visual images via an RGB camera; object selection and category recognition based on target detection algorithms; extraction of local texture features and global morphological features using deep neural networks and fusion to generate a multimodal fusion feature vector; comparison with the calibrated target via vector similarity retrieval to trigger the object recording process; storage of the original object image, fusion feature vector, and structured metadata in a dual-backup mode. Step 4: Automatic recording and assisted perception; based on the gaze estimation model, predict the gaze point and manually mark it for assistance; combine the gaze point and the data of the visual field defect area to determine the visibility of the target object. When a marked object or a high-priority object that matches the user's habitual characteristics is detected and the object is in the visual blind spot, the recording process is automatically started and stored in the database in a hybrid storage method, while triggering vibration or voice reminder.

[0007] Further, in step 1, the fuzzy matching includes the following steps: First, construct a keyword system with a three-layer structure of "core category - attribute - related words", generate semantic embedding vectors through a large language model, calculate the cosine similarity with the embedding vectors of each core category in the core category library, take the core category with the highest similarity as the normalized label, and automatically associate the corresponding related words to form a fuzzy keyword set.

[0008] Furthermore, in step 1, a contrastive language-image pre-training model is used to map visual features and semantic embeddings to the same space, and the original language is condensed and pre-processed using a large language model.

[0009] Furthermore, in step 2, the multi-dimensional feature data includes: marking time, location, object type, marking frequency, associated object selection, dwell time, and scene context.

[0010] Furthermore, the clustering algorithm is the HDBSCAN algorithm, and the pre-trained table algorithm is the TabPFN algorithm.

[0011] Furthermore, in step 2, the habit prediction model updates the user habit feature weights in real time using the sliding window method; the gaze estimation model adopts a CNN+GRU network structure that combines a convolutional neural network and a gated recurrent unit.

[0012] Furthermore, in step 3, the target detection algorithm is the YOLOv11 algorithm; the deep neural network includes a ResNet-50 network for extracting local texture and color feature vectors, and a Swing Transformer-Tiny network for capturing global shape and relative position feature vectors of the environment.

[0013] Furthermore, in step 3, the structured metadata includes a timestamp, core category ID, priority, surrounding related objects, and a fuzzy keyword set; wherein, the surrounding related objects are detected by an object detection algorithm and obtained by filtering through a large language model; the unstructured descriptive text is generated by concatenating the structured data with the visual detection results through a large language model.

[0014] Furthermore, in step 4, the manual marking assistance is triggered by double-clicking the Bluetooth ring or by voice command; the visual field defect detection utilizes the principle of perspective projection to determine the visibility of the target object in the user's field of vision, and if the object is in a blind spot or semi-blind spot, the key monitoring mechanism is triggered.

[0015] Furthermore, the automatic recording triggering conditions in step 4 also include: when a target object appears in a non-visual loss area and the user does not manually record within a fixed time, an automatic vibration or voice reminder is triggered; the hybrid storage method is a "text + image patch" storage method.

[0016] Compared with the prior art, the present invention has the following advantages: This invention presents a first-person perspective-assisted perception method that specifically addresses the visual perception challenges faced by individuals with visual field defects. Through multi-module collaboration, it achieves personalized and intelligent recording and perception of target objects. The initialization calibration module employs structured storage and multi-interaction methods to ensure accurate calibration information and convenient operation, providing a reliable target benchmark for the system. The user habit learning module integrates various machine learning algorithms to accurately capture users' personalized habits and scene preferences, making automatic recording more tailored to user needs. The multimodal data acquisition module integrates multi-dimensional data such as visual, eye-tracking, and positional data, providing high-quality data support for target recognition and matching through high-precision synchronization and preprocessing. The automatic recording and assisted perception module actively captures target objects that the user may not perceive, filling the information gaps caused by visual field blind spots, through technologies such as blind spot adaptation, two-stage matching, and multimodal feedback. The overall method significantly reduces the burden of manual operation for users. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the overall process of the present invention. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0020] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0021] like Figure 1 As shown, this invention provides a first-person perspective-assisted perception method for people with visual impairments, comprising the following steps: Step 1: Initialize target object calibration and visual field defect region segmentation; the user inputs the object name through smart glasses or a touch interface, and fuzzy matching and information storage are achieved through multimodal fusion and a vector database; simultaneously, the visual field range is calibrated by using equidistant pixel-by-pixel flashing to segment the initial visual field defect region. As a preferred embodiment, in this application, the fuzzy matching includes the following steps: first, constructing a keyword system with a three-layer structure of "core category - attribute - associated word"; generating semantic embedding vectors through a large language model; calculating the cosine similarity with the embedding vectors of each core category in the core category library; using the core category with the highest similarity as the normalized label; and automatically associating corresponding associated words to form a fuzzy keyword set. Simultaneously, in Step 1, a contrastive language-image pre-training model is used to map visual features and semantic embeddings to the same space, and the original language is condensed and pre-processed using a large language model.

[0022] Step 2: User Habit Learning; Collect multi-dimensional feature data of user marking operations (marking time, location, object type, marking frequency, associated object selection, dwell time, and scene context), use a clustering algorithm (HDBSCAN algorithm) to subdivide scene types, extract user recording habit rules through a pre-trained table algorithm (TabPFN algorithm), construct a habit prediction model based on a long short-term memory network combined with an attention mechanism, and introduce a forgetting curve model to dynamically adjust object matching weights; simultaneously, predict the user's gaze landing point through a gaze estimation model, and merge it with the original visual field defect area data to update the current visual field defect area; the habit prediction model updates the user habit feature weights in real time using the sliding window method; the gaze estimation model adopts a CNN+GRU network structure that combines a convolutional neural network and a gated recurrent unit.

[0023] Step 3: Multimodal Data Acquisition; Real-time acquisition of first-person perspective visual images via an RGB camera; Object selection and category recognition based on a target detection algorithm (YOLOv11 algorithm); Extraction of local texture features and global morphological features using a deep neural network, and fusion to generate a multimodal fusion feature vector; Comparison with the calibrated target via vector similarity retrieval to trigger the object recording process; The original object image, fusion feature vector, and structured metadata (calibration timestamp, core category ID, priority, surrounding related objects, and fuzzy keyword set; wherein, the surrounding related objects are detected by the target detection algorithm and filtered by a large language model; the unstructured descriptive text is generated by concatenating the structured data with the visual detection results using a large language model) are stored in a dual-backup mode; In this application, preferably, the deep neural network includes a ResNet-50 network for extracting local texture and color feature vectors, and a Swin Transformer-Tiny network for capturing global morphological and environmental relative position feature vectors.

[0024] Step 4: Automatic recording and assisted perception; based on the gaze estimation model, predict the gaze point and manually mark it for assistance; combine the gaze point and the data of the visual field defect area to determine the visibility of the target object. When a marked object or a high-priority object that matches the user's habitual characteristics is detected and the object is in the visual blind spot, the recording process is automatically started and stored in the database in a hybrid storage method, while triggering vibration or voice reminder.

[0025] In a preferred embodiment, in this application, in step 4, the manual marking assistance is triggered by double-clicking a Bluetooth ring or a voice command; the detection of visual field defects utilizes the principle of perspective projection to determine the visibility of the target object in the user's field of vision, and if the object is in a blind spot or semi-blind spot, a key monitoring mechanism is triggered. Furthermore, the automatic recording triggering conditions in step 4 also include: when a target object appears in a non-visual field defect area and the user does not manually record within a fixed time, an automatic vibration or voice reminder is triggered; the hybrid storage method is a "text + image patch" storage method.

[0026] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In the above embodiments of the present invention, the descriptions of each embodiment have their own emphasis; parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. It should be understood that the disclosed technical content in the several embodiments provided in this application can be implemented in other ways.

[0027] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A first-person perspective-assisted perception method for people with visual impairments, characterized in that, Includes the following steps: Step 1: Initialize target object calibration and visual field defect area division; the user inputs the object name through smart glasses or touch interface, and fuzzy matching and information storage are achieved through multimodal fusion and vector database; at the same time, the visual field range is calibrated by flashing the equidistant pixels point by point, and the initial visual field defect area is divided. Step 2: User habit learning; Multi-dimensional feature data of user marking operations are collected, and clustering algorithms are used to subdivide scene types. User recording habit rules are extracted through pre-trained table algorithms. A habit prediction model is built based on long short-term memory network combined with attention mechanism, and a forgetting curve model is introduced to dynamically adjust object matching weights. At the same time, the user's gaze landing point is predicted through gaze estimation model, and the current vision loss area is updated by merging with the original vision loss area data. Step 3: Multimodal data acquisition; The system acquires first-person visual images in real time using an RGB camera, performs object selection and category recognition based on object detection algorithms, extracts local texture features and global morphological features using deep neural networks and fuses them to generate a multimodal fusion feature vector, and compares it with the calibrated target through vector similarity retrieval to trigger the object recording process; the original object image, fusion feature vector and structured metadata are stored in a dual backup mode. Step 4: Automatic recording and assisted perception; The system uses a line-of-sight estimation model to predict the point of view and manually marks it for assistance. It combines the line-of-sight point with data on the visual field defect area to determine the visibility of the target object. When a marked object or a high-priority object that matches the user's habitual characteristics is detected and the object is in the blind spot, the system automatically starts the recording process and stores it in the database in a hybrid storage method, while triggering a vibration or voice reminder.

2. The first-person perspective-assisted perception method for people with visual impairments according to claim 1, characterized in that, In step 1, the fuzzy matching includes the following steps: First, a keyword system with a three-layer structure of "core category - attribute - related words" is constructed. Semantic embedding vectors are generated through a large language model. The cosine similarity with the embedding vectors of each core category in the core category library is calculated. The core category with the highest similarity is used as the normalized label, and the corresponding related words are automatically associated to form a fuzzy keyword set.

3. A first-person perspective-assisted perception method for people with visual impairments according to claim 1 or 2, characterized in that, In step 1, a contrastive language-image pre-training model is used to map visual features and semantic embeddings to the same space, and a large language model is used to condense and summarize the original language for preprocessing.

4. The first-person perspective-assisted perception method for people with visual impairments according to claim 1, characterized in that, In step 2, the multi-dimensional feature data includes: marking time, location, object type, marking frequency, associated object selection, dwell time, and scene context.

5. A first-person perspective-assisted perception method for people with visual impairments according to claim 1, characterized in that, The clustering algorithm is the HDBSCA algorithm, and the pre-trained table algorithm is the TabPFN algorithm.

6. A first-person perspective-assisted perception method for people with visual impairments according to claim 1, characterized in that, In step 2, the habit prediction model updates the user habit feature weights in real time using the sliding window method; the gaze estimation model adopts a CNN+GRU network structure that combines convolutional neural networks and gated recurrent units.

7. A first-person perspective-assisted perception method for people with visual impairments according to claim 1, characterized in that, In step 3, the target detection algorithm is the YOLOv11 algorithm; the deep neural network includes a ResNet-50 network for extracting local texture and color feature vectors, and a Swing Transformer-Tiny network for capturing global shape and relative position feature vectors of the environment.

8. A first-person perspective-assisted perception method for people with visual impairments according to claim 1, characterized in that, In step 3, the structured metadata includes a timestamp, core category ID, priority, surrounding related objects, and a fuzzy keyword set; wherein, the surrounding related objects are detected by an object detection algorithm and obtained by filtering through a large language model; the unstructured descriptive text is generated by concatenating the structured data with the visual detection results through a large language model.

9. A first-person perspective-assisted perception method for people with visual impairments according to claim 1, characterized in that, In step 4, the manual marking assistance is triggered by double-clicking the Bluetooth ring or by voice command; the visual field defect detection uses the principle of perspective projection to determine the visibility of the target object in the user's field of vision. If the object is in a blind spot or semi-blind spot, the key monitoring mechanism is triggered.

10. A first-person perspective-assisted perception method for people with visual impairments according to claim 1, characterized in that, The automatic recording triggering conditions in step 4 also include: when a target object appears in a non-visual loss area and the user does not manually record within a fixed time, an automatic vibration or voice reminder is triggered; the hybrid storage method is a "text + image patch" storage method.

Images