Interaction system and method for visually impaired people to interact with environment

Abstract

The invention discloses an interaction system and method for visually impaired people to interact with an environment, relates to the technical field of image recognition, and aims to solve the technical problems that the visually impaired people are difficult to acquire and process information in an office administrative scene and are difficult to accurately associate the information, and the interaction system comprises an intelligent glasses terminal and a background service system; the intelligent glasses terminal comprises a multi-mode sensing module, an equipment space positioning module, a processing and communication module, an audio output module and a user interface, and is used for realizing user instruction input; the background service system is deployed in a local server and comprises an equipment connection and management unit, a computer vision processing unit, an interaction intention analysis and decision unit, a voice synthesis unit, a prediction model and a safety protection unit. The intelligent glasses terminal is combined with the background service system, so that the problem of information association between cross-equipment and a paper carrier in the prior art is thoroughly solved, and the autonomy of the visually impaired people for handling the office administrative affairs is remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, and more particularly to an interaction system and method for interaction between visually impaired people and the environment. BACKGROUND

[0002] With the popularization of information-based office work, office administrative affairs (such as OA system operation, reimbursement data checking, meeting notice transmission, special equipment state monitoring, etc.) have become one of the core daily work in the workplace. For the visually impaired, information acquisition and interaction in the office administrative scene are particularly urgent - they need to independently complete cross-carrier information reading from electronic devices (such as office computers, printers, meeting terminals) to paper documents (such as paper reimbursement forms, meeting agendas) to achieve reimbursement submission, meeting participation, equipment state confirmation, and other basic administrative affairs processing.

[0003] Currently, the visually impaired in the office administrative scene mainly rely on two types of assistive tools: one is a screen reader for open system devices, which reads device accessibility interface data to broadcast screen text and control information in the form of voice; the other is a Braille reader or general speech recognition tool for paper documents, which needs to manually scan the document and convert it to voice in batches.

[0004] The way the visually impaired acquire information in the office administrative scene has obvious limitations: On the one hand, traditional screen readers can only adapt to open system devices that support accessibility interfaces (such as regular office computers), and cannot be compatible with closed system devices that do not have accessibility interfaces (such as old printer displays, special meeting terminals), resulting in administrative information (such as printer "out of paper" prompts, meeting terminal "sign-in status") of such devices cannot be perceived by the visually impaired; On the other hand, for paper administrative documents (such as paper reimbursement forms, meeting agendas), general Braille readers or speech recognition tools lack the ability to associate with the gaze behavior of the visually impaired, and cannot accurately correspond the user's gaze area with the key administrative information (such as the "amount" column of the paper reimbursement form, the "location" item of the meeting agenda) in the document, lacking the ability to accurately locate the gaze point and associate with the text, making it difficult to correspond the visually impaired person's gaze area with the key administrative information (such as the reimbursement amount, the meeting location) in the document. Therefore, the visually impaired have difficulty in crossing open / closed system devices and paper document carriers in the office administrative scene, accurately associating their gaze points with the required administrative information, resulting in low administrative affairs processing efficiency and high dependence on others for assistance, and unable to independently handle office administrative affairs. In view of this, we propose an interaction system and method for interaction between visually impaired people and the environment. SUMMARY

[0005] The application aims to provide an interaction system and method for the visually impaired to interact with the environment, so as to solve the technical problems that the visually impaired have difficulty in obtaining, processing information and accurately associating information in an office administrative scene.

[0006] To solve the above technical problems, the application provides the following technical solutions: an interaction system for the visually impaired to interact with the environment, comprising an intelligent glasses terminal and a background service system, which performs encrypted data interaction through a local area network to assist the visually impaired in completing environmental information perception, administrative information acquisition and interactive operation. The intelligent glasses terminal comprises: A multi-modal sensing module acquires display device images, gaze point world coordinates and head movement data, and locates the gaze content of the visually impaired user. A device space positioning module receives a Bluetooth beacon broadcast signal and uses the received signal strength Real-time acquisition of device physical position parameters corrects coordinate mapping deviation. A processing and communication module is used for data processing, storage, encrypted transmission and local cache preloading. An audio output module is used to feed back the processed information to the user in the form of audio. A user interface is used to realize user instruction input. The background service system is deployed on a local server and comprises: A device connection and management unit is used to scan and authenticate display devices in the local area network and acquire device data based on open-closed interaction logic. A computer vision processing unit is used for image preprocessing, text recognition and graphic element recognition. An interactive intent analysis and decision unit is used to determine the interaction mode, associate the gaze point with the administrative information and generate a broadcast text. A speech synthesis unit is used to convert the broadcast text into natural speech. A prediction model is trained based on interaction data to improve the adaptability to user usage habits and scene requirements and optimize the broadcast priority and detail level. A security protection unit is used to ensure the security of data transmission and storage.

[0007] The application uses the multi-modal sensing module (acquiring gaze point world coordinates, device images and head movement data) and the device space positioning module (based on The signal strength correction position deviation) combined with the open-closed interaction logic of the background service system (open devices obtain structured UI data through UIAutomation / AT-SPI, and closed devices inversely analyze screen information through the LayoutLM model) and the gaze point-coordinate system mapping algorithm (convert world coordinates into device screen / image coordinates), the precise association of the gaze point of the visually impaired person with the UI control of the open system device, the screen information of the closed system device, and the text of the paper document can be realized. Without the assistance of others, the visually impaired person can independently obtain OA interface control attributes, printer status, and paper meeting notices through image recognition, and other administrative information, completely solving the information association problem across devices and paper carriers in the prior art, and significantly improving the independence of the visually impaired person in handling administrative affairs.

[0008] Preferably, the multi-modal sensing module comprises: a miniature camera for collecting images of office display devices; a gaze tracking module based on the near-infrared pupil corneal reflection principle, combined with a special pupil calibration algorithm, to estimate the gaze direction by establishing a personalized mapping model through multi-frame pupil feature collection , the corneal reflection point coordinates , the gaze vector is calculated by the formula , in which is a proportional coefficient; an inertial measurement unit for correcting head movement deviation.

[0009] Preferably, the device space positioning module calculates the distance between the intelligent glasses terminal and the display device based on the received signal strength , and the distance calculation formula is , in which is the distance between the display device and the intelligent glasses; is the reference value at a distance of 1 meter; is the environmental attenuation index.

[0010] Preferably, the processing and communication module comprises: an embedded processor for compressing and de-redundant processing of collected images to reduce local area network transmission volume; a wireless communication unit supporting Wi-Fi6 protocol and establishing a TLS1.3 encrypted link with the background service system; a local cache and preloading submodule for preloading user high-frequency administrative interface data when the network is stable and generating a basic broadcast based on the cache when the network is interrupted.

[0011] ​​Preferably, the user interface includes a touch panel and a voice wake-up unit; the touch panel is integrated into one temple of the glasses to realize interactive operation based on operation logic, and the voice wake-up unit supports users to trigger scene tag selection through voice commands.

[0012] Preferably, the open-closed interaction logic specifically includes: For open system devices, structured UI data is obtained through the UIAutomation / AT-SPI interface; For closed system devices, visual analysis is triggered. The device images captured by smart glasses are combined with the LayoutLM document layout analysis model to reverse analyze the text, position and state of UI controls.

[0013] Preferably, the computer vision processing unit specifically includes: The image preprocessing subunit eliminates light reflections through denoising, corrects oblique gaze distortion through perspective transformation, and enhances contrast to highlight the differences between icons and text. The OCR subunit, based on the Transformer OCR model, recognizes administrative text and outputs the text and its location bounding box. The graphic element recognition subunit uses a pre-trained YOLOv8 model, combined with enterprise-customized icons and semantic tags uploaded by the administrator for transfer learning fine-tuning, and automatically retrains when a new icon appears.

[0014] Preferably, the interaction intent parsing and decision-making unit specifically includes the following process: A: Interaction mode determination: Based on the device connection status, if it is connected to an open or closed system device, it is determined to be in IoT mode; if it is looking at a paper document, it is determined to be in visual mode. B: Focal point information association: For IoT mode: Combining the dynamic position parameters obtained by the device's spatial positioning module, a transformation matrix is ​​used. world coordinates of the gaze point Mapped to the display device screen coordinate system The formula is: ; In the formula, the transformation matrix It includes translation, rotation, and scaling transformation parameters, and allows querying structured UI data or visual analysis results to extract control properties. For visual patterns: map the world coordinates of the gaze point to the image coordinate system, and match them with the OCR results and graphic element recognition results to determine gaze information; C: Broadcast text generation: Combine scene tags to generate semantically complete natural language descriptions.

[0015] Preferably, the training of the prediction model specifically includes the following process: Data Acquisition: Anonymously record interaction events, including scene labels, gaze duration, and number of repetitions. Combine this with IMU data to filter noise, and analyze head angular velocity. When this happens, mark it as invalid data; Model learning: The XGBoost algorithm was used, with effective data of fixation duration > 2 seconds and repeated playback ≥ 2 times as the training set, to build a prediction model. The objective function is: In the formula, The number of samples; The loss function; The number of decision trees; This is a regularization term used to control the regularization of each tree. The complexity; Strategy Update: Apply model parameters to the interaction intent parsing and decision-making units to optimize the broadcast priority and level of detail.

[0016] An interaction method for visually impaired individuals to interact with their environment, comprising the following steps: S1: System initialization and scene calibration; The user wears the smart glasses terminal and starts the system. The near-infrared camera collects pupil features to complete personalized eye tracking calibration; The smart glasses terminal receives the Bluetooth beacon signal from the display device and initializes the device location parameters; The background service system scans and authenticates devices in the local area network; The user confirms the administrative scene label through voice commands or system scene recommendations; S2: Real-time perception and interaction mode determination; The multimodal sensing module continuously collects images from the display device, world coordinates of the gaze point, and head motion data, and updates the device position parameters; The interaction intent parsing and decision-making unit determines the interaction mode based on the device connection status. Specifically, when connected to an open system device: it is determined to be IoT mode - UI interface parsing; when connected to a closed system device: it is determined to be IoT mode - visual parsing; when gazing at a paper document: it is determined to be visual mode. S3: Dual-mode administrative information acquisition; if it is IoT mode - UI interface parsing, obtain structured UI data and extract control attributes by associating gaze points; if it is IoT mode - visual parsing, trigger the visual parsing process to obtain UI control information; if it is visual mode, identify text and graphic elements in paper documents and match gaze points; S4: Administrative content broadcasting and feedback; the interactive intent parsing and decision-making unit combines scene tags to generate broadcast text, which is then converted into streaming speech and played by the speech synthesis unit; users can adjust the feedback content via touch panel or voice commands, and basic information is broadcast based on local cache when the network is interrupted; S5: Scenario-based adaptive optimization, which involves filtering effective interaction data and updating prediction models and strategies.

[0017] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention utilizes a multimodal sensing module (collecting world coordinates of the gaze point, device image, and head motion data) and a device spatial positioning module (based on...) of the smart glasses terminal. By combining signal strength correction for positional deviations with the open-closed interaction logic of the back-end service system (open devices acquire structured UI data through UIAutomation / AT-SPI, while closed devices reverse-parse screen information through the LayoutLM model) and the gaze point-coordinate system mapping algorithm (converting world coordinates into device screen / image coordinates), it is possible to achieve accurate association between the gaze point of visually impaired individuals and UI controls of open system devices, screen information of closed system devices, and text in paper documents. Without the assistance of others, they can independently obtain administrative information such as OA interface control attributes, printer status, and paper meeting notices through image recognition, completely solving the problem of information association between devices and paper carriers in existing technologies, and significantly improving the autonomy of visually impaired individuals in handling office and administrative affairs.

[0018] 2. This invention also optimizes the priority and level of detail of broadcasts through the XGBoost prediction model of the back-end service system (trained based on effective data such as user gaze duration and number of repeated broadcasts). For example, in the reimbursement scenario, the approval results are broadcast first. At the same time, the local caching and preloading sub-modules of the smart glasses terminal pre-store high-frequency administrative interface data when the network is stable and generate basic broadcasts when the network is interrupted. After achieving accurate information association, this invention avoids the problems of insufficient adaptability of broadcast content due to differences in user habits and information acquisition stagnation due to network interruption. This ensures that information broadcasts are in line with user habits and do not rely on a continuous network connection, thereby further improving the efficiency of administrative affairs processing.

[0019] 3. This invention also ensures the security of administrative data transmission and storage by using a TLS1.3 encrypted transmission link between smart glasses and the back-end service system, dual authentication of device UUID (administrator QR code + user voice password), and anonymization of interactive data (removal of sensitive information such as names and amounts). This solves the administrative data security risks (such as OA reimbursement amounts and user identity information leakage) that exist in the process of information transmission and storage, avoids the risk of privacy leakage, and further enhances the applicability and reliability of the system in office scenarios. Attached Figure Description

[0020] Figure 1 This is a system framework diagram of the present invention; Figure 2 This is a flowchart of the method of the present invention. Detailed Implementation

[0021] Example 1: As Figure 1As shown, the present invention relates to an interactive system for visually impaired individuals to interact with their environment, comprising a smart glasses terminal and a back-end service system. The two systems exchange encrypted data via an office local area network (Wi-Fi), as detailed below: The smart glasses terminal is used to provide visually impaired individuals with environmental information perception and interactive feedback, assisting them in completing office administrative tasks. Its hardware includes: The multimodal sensing module integrates multiple sensors and outputs the world coordinates of the gaze point through data fusion, enabling precise positioning of the content gazed at by visually impaired users and obtaining image information of the office display device they are interested in, as well as their own head movement data. In an embodiment of the present invention, the sensor includes: Miniature cameras are used to capture images from office display devices, such as OA interfaces and meeting lists; The gaze tracking module, based on the near-infrared pupil-corneal reflection principle and combined with a special pupil calibration algorithm, is designed for visually impaired individuals with iris lesions or varying pupil sizes. It establishes a personalized mapping model through multi-frame pupil feature acquisition to estimate the gaze direction. Let the pupil center coordinates be... The coordinates of the corneal reflection point are The line-of-sight vector is calculated using the following formula. : ; In the formula, This is a scaling factor, obtained through pupil calibration experiments on different users. This factor is used to adjust the length and direction of the gaze vector to accommodate the differences in the ocular physiological characteristics of different users, ensuring the accuracy of gaze direction estimation.

[0022] Inertial measurement unit (IMU) for correcting head motion deviation; In embodiments of this invention, the initial source of the above formula comes from the principle of calculating the direction of a two-point vector in analytic geometry, combined with the pupil-corneal reflection method in the field of computer vision—a classic foundation of gaze tracking technology. Its core logic is to determine the gaze direction by the relative position of the pupil center and the corneal reflection point. Its purpose is to accurately estimate the gaze direction for visually impaired individuals with iris lesions or varying pupil sizes, providing a directional reference for subsequent gaze point-display device association. This is a core prerequisite for the system to interpret the user's gaze as reflected in the image. Original principle derivation: In the standard pupil-corneal reflection method, the gaze direction originates from the pupil center... The reflection point formed on the cornea by the near-infrared light source The difference vector directly determines the original vector, i.e., the original vector is... Its direction represents the approximate direction of gaze. Due to differences in the physiological characteristics of the eyes of visually impaired users (such as iris lesions causing reflective point shifts, and different pupil sizes causing vector scale distortion), directly using... This will produce significant errors. Therefore, a scaling factor is introduced to calibrate the original vector: Step 1: Data is collected through a user-personalized pupil calibration experiment—the user gazes at multiple marker points with known coordinates on the screen, and the corresponding coordinates of each marker point are recorded. And the actual gaze direction; Step 2: Calculate the direction of each marker point value( The average value is taken as the user's exclusive value. Step 3: Multiplying by the difference vector yields the calibrated line-of-sight vector. This enables gaze direction estimation that adapts to individual differences in eye shape.

[0023] The device's spatial positioning module uses a low-power Bluetooth receiver unit to receive Bluetooth beacon broadcast signals built into the office display device, utilizing... (Received signal strength) Real-time acquisition of device physical location parameters, correction of coordinate mapping deviations (such as automatic update of location data after the conference screen moves), ensuring that the smart glasses can accurately associate the user's gaze point with the actual location of the display device, and adapt to changes in device location.

[0024] In embodiments of the present invention, based on The distance calculation formula is: ; In the formula, The calculated distance between the device and the smart glasses, in meters, is used to determine the device's location in physical space. It is at a distance of 1 meter Reference value, which is obtained by actual measurement under specific conditions, is used to provide a reference signal strength for distance calculation; The Environmental Attenuation Index (AIA) is derived from on-site measurements and statistics under different office environments. This index reflects the attenuation characteristics of a signal propagating in the current environment and is used to correct the relationship between signal strength and distance. The formula is used to convert received signal strength into actual distance, thereby determining the physical location of the device.

[0025] In the embodiments of the present invention, the initial source of the above formula comes from the log-DistancePathLoss Model in the principles of communication (electronic information science), which is a classic formula for signal strength-distance conversion in wireless communication. Its original general form is: ( Signal strength at 1 meter (This refers to the environmental degradation index). Its function is to measure the strength of the Bluetooth beacon signal received by the smart glasses. This is converted into the actual distance between the smart glasses and the display device. This provides physical location parameters for the device's spatial positioning module, solving the problem of automatic position updates after the conference screen is moved. The original formula is derived by algebraically transforming the general logarithmic distance model to solve for the distance. This application modifies the following: In the original model and For general parameters, this application focuses on optimization for office environments: : Data was obtained through actual measurements in a specific office environment (e.g., at a distance of 1 meter in both open and obstructed areas of the office). (take the average). : Determined through on-site statistics in office settings (considering signal attenuation caused by walls and furniture, and the signal strength in different offices). Value differences can be automatically calibrated via pre-scanning to ensure distance calculations are adapted to complex office environments.

[0026] The processing and communication module processes, stores, and transmits the collected data, optimizes data transmission efficiency, ensures data communication security, and improves the stability and response speed of the interactive system. In an embodiment of the present invention, the processing and communication module includes: Embedded processors enable image compression and redundancy removal, reducing local area network transmission volume. The wireless communication unit supports the Wi-Fi 6 protocol and establishes a TLS 1.3 encrypted link with the backend. The local caching and preloading submodule is used to preload frequently used administrative interface data (such as commonly used OA page control properties) when the network is stable, and to generate basic broadcasts based on the cache when the network is interrupted.

[0027] The audio output module is used to feed back the information processed by the system to the user in audio form. It uses bone conduction speakers to improve the listening effect in noisy office environments and adaptively adjusts the volume and speech rate according to the network conditions to ensure the timeliness and accuracy of information broadcasting. The user interface is used to provide users with diverse interaction methods, realize command input and information feedback between users and the system, and meet the convenient operation needs of visually impaired users. In an embodiment of the present invention, the user interface includes a touch panel and a voice wake-up unit; The touch panel is integrated into one temple of the glasses and implements interactive operations based on operation logic, which includes: single-finger tap to confirm; two-finger swipe to repeat the announcement. The voice wake-up unit allows users to enter the expense reimbursement process, switch meeting scenarios, and trigger scenario tag selection via voice commands.

[0028] The back-end service system, serving as the core computing and decision-making center, is deployed on a local office server to ensure administrative data security. It provides visually impaired users with accurate environmental information analysis, interactive decision-making, and personalized services, including: Device Connection and Management Unit: Used to scan and authenticate display devices within the local area network. For different types of devices, it adopts corresponding data acquisition methods based on open-closed interaction logic to provide basic data support for subsequent information parsing and interaction using UPnP and mDNS IoT protocols. In embodiments of the present invention, corresponding data acquisition methods are adopted for different types of devices, specifically including: For open system devices (such as office computers): obtain structured UI data (control tree, text, status, position) through the UIAutomation / AT-SPI interface; For closed system devices (such as old printer displays): trigger visual analysis, use the device image captured by smart glasses, and combine it with the LayoutLM document layout analysis model to reverse analyze the text, position (based on the device spatial positioning module) and status of UI controls (such as judging whether a button is clickable by the color change).

[0029] Computer vision processing unit: used to process and analyze images captured by smart glasses, extract text and graphic information from images, and provide visual information support for interactive intent parsing; In an embodiment of the present invention, the computer vision processing unit specifically includes: Image preprocessing subunit: used for denoising to eliminate glare, perspective transformation to correct oblique gaze distortion, and contrast enhancement to highlight the differences between icons and text. Taking contrast enhancement as an example, a linear transformation formula is used: ; In the formula, These are the original image pixel values, typically ranging from 100 to 1000 pixels. ; The enhanced image pixel values ​​are also taken from... interval; This is the contrast adjustment factor, when Enhancing image contrast makes the difference between light and dark areas more pronounced; when Reduce contrast at times; This is the brightness adjustment factor; changing this value adjusts the overall brightness of the image. Adjusting these two parameters optimizes image contrast and brightness, making the difference between icons and text more pronounced.

[0030] OCR Subunit: Based on the Transformer OCR model, it recognizes administrative texts (reimbursement amounts, approval opinions) and outputs the text and location bounding boxes; Graphic element recognition subunit: It adopts a pre-trained YOLOv8 model, combined with user-defined training. The administrator uploads enterprise-defined icons (such as department-approved icons) and semantic tags. The system quickly fine-tunes the model through transfer learning and supports incremental updates (automatic retraining when new icons appear).

[0031] In embodiments of the present invention, the initial source of the above formula comes from the principle of linear grayscale transformation in digital image processing (computer science), which belongs to the mathematical discipline of linear functions (…). The direct application of this transformation—a fundamental method for image enhancement—is to adjust contrast and brightness through linear scaling and offsetting of pixel values. Its purpose is to address issues such as image glare, slant-gaze distortion, and low text-to-background contrast in office environments, highlighting differences in icons and text within devices or paper documents, and providing high-quality image input for subsequent OCR text recognition and YOLOv8 graphic element recognition. Derivation of the original principle: Pixel values ​​of digital images... The range is (0 represents black, 255 represents white). The core logic of linear transformation is: contrast adjustment: coefficient. Control the scaling ratio of pixel values ​​- when At that time, pixel value differences are amplified (e.g.) , , Time becomes , (As the difference increases from 100 to 120, the contrast is enhanced); when Contrast decreases; brightness adjustment: offset. Controlling the overall offset of pixel values—when At that time, all pixel values ​​are raised (e.g.) , (Changes to 70, brightness increases); when At that time, the brightness decreases. Modification in this application: The original linear transformation lacks scene adaptability; this application optimizes parameter settings for office administrative scenes: For low-contrast images caused by light reflections, the default settings... (Enhance contrast) (Increase brightness); Automatically adjust brightness for images on projection screens or other surfaces that are too bright. , To avoid text overexposure; ensure all parameters are adjusted to ensure... (Automatically truncated to 0 or 255 when outside the range), conforming to digital image display standards.

[0032] Interactive Intent Analysis and Decision Unit: Based on dual-mode interaction logic, it achieves precise correlation between gaze point and information, generates broadcast text that meets user needs, and provides efficient interactive services; In an embodiment of the present invention, the interaction intent parsing and decision-making unit specifically includes the following process: A: Interaction mode determination: Based on the device connection status, determine whether it is IoT mode (connected to open / closed system devices) or visual mode (looking at paper documents). B: Focal point information association: For IoT mode: Combine the dynamic position parameters obtained by the device's spatial positioning module to determine the world coordinates of the gaze point. Mapped to the display device screen coordinate system Assume the transformation matrix from world coordinates to screen coordinates is The conversion formula is: ; In the formula, the transformation matrix It includes transformation parameters such as translation, rotation, and scaling. This formula transforms the user's gaze point in the three-dimensional coordinates of the real world to the two-dimensional screen coordinate system of the display device, and then queries structured UI data or visual analysis results to extract control attributes (text, state, context).

[0033] For visual patterns: map the world coordinates of the gaze point to the image coordinate system, and match them with the OCR results and graphic element recognition results to determine gaze information; C: Broadcast text generation: Combine scene tags (such as expense reimbursement processing) to generate semantically complete natural language descriptions (such as expense reimbursement processing scenario: submit button, clickable, amount 1200 yuan, 3 attachments).

[0034] In the embodiments of the present invention, the initial source of the above formula comes from the principle of homogeneous coordinate 3D to 2D projection transformation in computer graphics (computer science), and the core is through a 4×3 transformation matrix. It integrates four transformations—translation, rotation, scaling, and projection—to achieve the mapping from 3D spatial coordinates to 2D screen coordinates—homogeneous coordinates (such as 3D point coordinates). The representation () is the mathematical foundation of this transformation, and it can uniformly handle linear transformations and translation transformations. Its function is to convert the user's gaze point into the world's three-dimensional coordinates. Precisely mapped to the two-dimensional screen coordinates of the display device This mechanism establishes the association between the user's gaze point and screen controls (such as OA buttons and meeting list text), serving as the core bridge for information parsing in the Internet of Things (IoT) model. Derivation of the original principle: Transformation matrix. It is a 4×3 matrix containing four types of transformation parameters, and its original general form is: ; in : Rotation parameter, which corresponds to the orientation of the display device in the world coordinate system (e.g., whether the screen is tilted). : Scaling parameter, which corresponds to the conversion ratio between world coordinate units (such as meters) and screen coordinate units (such as pixels); Translation parameters correspond to the position of the display device's screen center in the world coordinate system; 3D point. and After multiplication, we obtain the homogeneous coordinates of the screen. After normalization, they become screen coordinates. Modification of this application: The original transformation matrix required manual parameter setting. This application integrates the device spatial positioning module to achieve automatic parameter adaptation: Step 1: Through Distance formula for calculating the distance between smart glasses and display device Based on the device dimensions (such as screen width and height) pre-stored by the Bluetooth beacon, the scaling parameters are automatically calculated. Step 2: The display device's Bluetooth beacon broadcasts its pre-stored location in the office (e.g., the conference screen is at (5, 3, 0) meters), and automatically updates the translation parameters. Step 3: When the display device moves (e.g., the conference screen is pushed away), the device spatial positioning module updates in real time. and Synchronous adjustment Ensure coordinate mapping deviation Pixels (meeting the accuracy requirements for control recognition in administrative scenarios).

[0035] Speech synthesis unit: used to convert the broadcast text into natural speech (delay ≤300ms), ensuring the real-time and smoothness of speech output, providing users with clear speech feedback, adopting streaming TTS technology, and supporting speech rate adjustment.

[0036] Predictive model: Optimize personalized experience based on interaction data, train to improve adaptability to user habits and scenario needs, and optimize broadcast priority and level of detail; In an embodiment of the present invention, the training of the prediction model specifically includes the following process: Data Acquisition: Anonymously record interaction events, including scene labels, gaze duration, and number of repetitions, and combine this with IMU data to filter noise (head angular velocity). (When this happens, mark it as invalid data). Model Learning: The XGBoost algorithm is used, with valid data of gaze duration > 2 seconds and repeated playback ≥ 2 times as the training set to build a prediction model (e.g., prioritizing approval results in reimbursement scenarios). The objective function of XGBoost is: ; In the formula, The number of samples is the total amount of interactive event data used in training. This is the loss function, used to measure the predicted value. Compared with the true value The degree of difference between them is minimized by this function to make the model prediction more accurate; The number of trees represents the total number of decision trees in the XGBoost model, affecting the model's complexity and expressive power. This is a regularization term used to control the regularization of each tree. This reduces model complexity, prevents overfitting, and improves generalization ability. The model parameters are trained by minimizing this objective function. To better adapt to the usage habits and scenario needs of different users.

[0037] Strategy Update: Apply model parameters to the interaction intent parsing and decision-making units to optimize the broadcast priority and level of detail.

[0038] In the embodiments of this invention, the initial source of the above formula comes from the classic objective function of the XGBoost (Extreme Gradient Boosting Tree) algorithm in machine learning (computer science). Its core is the combination of a loss function and a regularization term—this function is the optimization objective for training the gradient boosting tree model, and its original form is: ( For the first The number of leaf nodes in a tree. The weight of the leaf node. , (This is the regularization coefficient). Its function is to train a user interaction preference prediction model. By analyzing the interaction data of visually impaired users (such as gaze duration and number of repetitions), it uncovers their contextual needs (such as prioritizing approval results in reimbursement scenarios), providing personalized decision-making basis for interaction intent analysis. Original principle derivation: The objective function consists of two parts: the loss function... : Measure the model's predicted values (e.g., whether a user is paying attention to a certain control) and the actual value (such as gaze duration) And repeated broadcasts ≥ 2 times are marked as Otherwise The difference in regularization terms is commonly represented by squared loss or logistic loss; regularization terms are also used. Control the first Decision Tree To reduce complexity and avoid model overfitting (e.g.) , The smaller The closer to 0, the simpler the tree; model training minimizes the gradient descent method. ( (Parameters for all trees), iteratively generating the optimal decision tree set. Modifications in this application: The original XGBoost model lacks adaptability to visually impaired user interaction scenarios; this application makes customized modifications in three aspects: Step 1: Data preprocessing optimization—combining IMU data to filter noise: when head angular velocity... When this happens, it is marked as invalid data (excluding accidental interactions caused by unintentional head turning by the user), and only the gaze duration is retained. And repeated broadcasts The first step is to use valid data as the training set; the second step is to adapt the loss function—for scenario-based preference prediction, a weighted logistic loss is adopted: samples from high-frequency administrative scenarios such as expense reimbursement and meetings are given a weight of 1.5 times to improve the model's sensitivity to core scenarios; the third step is to simplify the regularization term—due to the small sample size of office interaction data (approximately 50-100 interactions per user per day), the regularization term is simplified. Simplified to (neglect This helps avoid underfitting of the model.

[0039] The security protection unit, based on data security mechanisms, safeguards system data security, prevents data leakage and unauthorized access, and ensures the security and privacy of user information and administrative data. In embodiments of the present invention, the data security mechanism specifically includes: Communication encryption: All data transmissions are encrypted using the TLS 1.3 protocol; Device authentication: The smart glasses have a built-in unique UUID. For the first connection, dual authentication is required, which includes a QR code scan by the administrator and a voice password from the user. Data anonymization: Sensitive information such as user names and amounts is removed from the interactive data stored in the backend, and only scene tags and control types are retained for training.

[0040] Example 2: Figure 2 As shown, an interaction method for visually impaired individuals to interact with their environment includes the following steps: S1: System initialization and scene calibration, used to complete personalized configuration of user equipment, initialization of equipment location parameters, authentication of local area network equipment and confirmation of administrative scene, and to provide basic environment settings for subsequent interactions; S101: When the user wears smart glasses, press and hold the power button to start the system. The near-infrared camera collects pupil features and completes personalized calibration of eye tracking (takes about 30 seconds). S102: Displays the built-in Bluetooth beacon broadcast device ID and physical location. The smart glasses device's spatial positioning module receives the signal and initializes the location parameters. S103: The background device connection and management unit scans the local area network and authenticates available devices (open system devices obtain UI interface data, and closed system devices are marked as visual resolution mode). S104: Users confirm the current administrative scenario tag through voice commands (such as entering the expense reimbursement process) or system scenario recommendations (such as automatically recommending expense reimbursement scenarios at the end of each month), and the system records the scenario information.

[0041] S2: Real-time perception and interaction mode determination, used to continuously collect user environment and behavior data, determine the interaction mode based on the device connection status, and provide a pattern basis for information acquisition; S201: The multimodal sensing module of the smart glasses continuously collects data: the camera collects images from the display device, the gaze tracking module outputs the world coordinates of the gaze point and head motion data, and the device spatial positioning module updates the device position parameters. S202: The interaction intent parsing and decision-making unit determines the interaction mode based on the device connection status (connecting to open system devices: determined to be IoT mode - UI interface parsing; connecting to closed system devices: determined to be IoT mode - visual parsing; looking at paper documents: determined to be visual mode).

[0042] S3: Dual-mode administrative information acquisition, used to acquire administrative information corresponding to the user's gaze point based on different interaction modes and corresponding data acquisition and processing methods, providing content support for information broadcasting; S301: If it is IoT mode (open system device): The device connection and management unit obtains structured UI data in real time (such as the control tree of the OA reimbursement system), and the decision unit combines dynamic position parameters to map the gaze point and query the properties of the hit control (such as the submit button, which can be clicked). S302: If it is IoT mode (closed system device): The decision unit triggers visual analysis, the computer vision processing unit preprocesses the collected device images, parses the UI controls through the LayoutLM model, and obtains information by matching the gaze point (such as the paper shortage status of the printer display). S303: If it is a visual mode (paper notice): The computer vision processing unit recognizes the text and graphic elements of the paper notice, and the decision unit matches the gaze point to determine the information (such as the meeting location: meeting room A).

[0043] S4: Administrative content broadcasting and feedback, used to convert the acquired administrative information into voice feedback to the user, and support the user to adjust the feedback content, realize the information interaction closed loop, and ensure that the user obtains effective information; S401: The decision unit combines scene tags to generate broadcast text, which is then converted into streaming speech by the speech synthesis unit and transmitted to the smart glasses via an encrypted link for playback by a bone conduction speaker. S402: Users can adjust the feedback content via the touch panel (two-finger swipe to repeat) or voice commands (simplified broadcast). When the network is interrupted, the terminal broadcasts basic information based on local cache (such as the current gaze: submit button).

[0044] S5: Scenario-based adaptive optimization, used to improve the system's ability to adapt to user needs in different scenarios and optimize user interaction experience by filtering effective interaction data, updating prediction models and strategies; The logical relationships and derivation chains between the formulas, with the core logical thread being: gaze point localization → information parsing → personalized broadcasting closed loop. All formulas revolve around the core need of visually impaired users to obtain administrative information through gaze, forming a logical chain of layered dependencies and reverse optimization, as detailed below: Phase 1: Precise fixation point localization; Input: Pupil features, Bluetooth beacon signals, and head movement data collected by the smart glasses; Formula Collaboration: Line-of-Sight Vector Formula ( ):pass Calibration ensures an accurate gaze direction; Distance formula ( ): Calculate the distance d between the smart glasses and the display device, and determine the world coordinates of the display device; Coordinate matrix transformation formula ( ): Combine the gaze direction and device world coordinates to obtain the world coordinates of the gaze point, then map them to screen coordinates, and finally associate them with screen controls (such as OA expense reimbursement buttons).

[0045] Output: Screen control properties corresponding to the gaze point (e.g., submit button, clickable, amount 1200 yuan).

[0046] Phase Two: Low-quality image information restoration; Triggering scenario: When the interaction mode is IoT mode (closed system device, such as old printer) or visual mode (paper document), the captured image has problems such as glare and low contrast. Formula function: Image contrast enhancement formula ( ): Preprocess the original image to enhance the difference between text and background; Subsequent process: The processed image is input into the OCR submodule (to identify reimbursement amount and approval comments) and the YOLOv8 graphic recognition submodule (to identify departmental special approval icons) to obtain text / graphic information that matches the gaze point.

[0047] Phase 3: Personalized interaction optimization; Input: Interaction data generated in the first two stages (scene labels, gaze duration, number of repeated broadcasts, with noise filtered by IMU); Formula function: XGBoost objective function ( ): Train prediction models to uncover user preferences (e.g., prioritizing approval results in reimbursement scenarios); Output: Model parameters are fed back to the interaction intent parsing and decision-making unit to optimize the broadcast priority (e.g., broadcast the approval result first, then the number of attachments), forming a closed loop of data collection → model training → experience optimization.

[0048] The formula design of this system is not an isolated innovation, but rather based on classical principles from disciplines such as mathematics, computer science, and electronic information, adapted for office administrative scenarios for visually impaired individuals—the core modifications focus on individual difference calibration (such as gaze vector). Optimization of scene parameters (such as values) of The system addresses three aspects: data noise filtering (e.g., IMU data filtering in XGBoost), and data noise filtering. Formulas are deeply coupled through a logical chain of gaze point localization → information parsing → personalized optimization. Ultimately, this achieves the core goal of enabling visually impaired users to accurately obtain administrative information and complete office interactions without visual dependence. The derivation and modification of each formula serve to improve accuracy, adaptability, and personalization—three core requirements.

[0049] The embodiments disclosed in this invention are preferred embodiments, but are not limited thereto. Those skilled in the art can easily understand the spirit of this invention based on the above embodiments and make different extensions and variations, but as long as they do not depart from the spirit of this invention, they are all within the protection scope of this invention.

Claims

1. An interactive system and method for visually impaired individuals to interact with their environment, characterized in that, It includes smart glasses terminals and a back-end service system, which use a local area network for encrypted data interaction to assist visually impaired people in completing environmental information perception, administrative information acquisition and interactive operations. The smart glasses terminal includes: The multimodal sensing module collects images from the display device, world coordinates of the gaze point, and head movement data to locate the content being gazed at by visually impaired users. The device's spatial positioning module receives Bluetooth beacon broadcast signals and utilizes the received signal strength. Real-time acquisition of device physical location parameters to correct coordinate mapping deviations; The processing and communication module is used for data processing, storage, encrypted transmission, and local cache preloading. The audio output module is used to feed back the processed information to the user in audio form; User interface, used to implement user command input; The backend service system is deployed on a local server, including: The device connection and management unit is used to scan and authenticate display devices within the local area network and acquire device data based on open-closed interaction logic. The computer vision processing unit is used for image preprocessing, text recognition, and graphic element recognition. The interaction intent parsing and decision-making unit is used to determine the interaction mode, associate the gaze point with administrative information, and generate the broadcast text. The speech synthesis unit is used to convert the broadcast text into natural speech; The predictive model, trained based on interactive data, improves its adaptability to user habits and scenario needs, and optimizes the priority and level of detail of the broadcast. The security protection unit is used to ensure the security of data transmission and storage.

2. The interactive system and method for visually impaired individuals to interact with their environment according to claim 1, characterized in that, The multimodal sensing module includes: Miniature cameras are used to capture images from office display devices; The gaze tracking module, based on the near-infrared pupil-corneal reflection principle and combined with a special pupil calibration algorithm, establishes a personalized mapping model through multi-frame pupil feature acquisition to estimate the gaze direction based on the pupil center coordinates. Coordinates of corneal reflection point Through formula Calculate the line-of-sight vector In the formula, This is the proportionality coefficient; An inertial measurement unit is used to correct head movement deviations.

3. The interactive system and method for visually impaired individuals to interact with their environment according to claim 2, characterized in that, The device spatial positioning module is based on the received signal strength. The distance between the smart glasses terminal and the display device is calculated using the following formula: In the formula, This refers to the distance between the display device and the smart glasses. It is at a distance of 1 meter Reference value; This is the environmental degradation index.

4. The interactive system and method for visually impaired individuals to interact with their environment according to claim 3, characterized in that, The processing and communication module includes: Embedded processors are used to compress and de-redundancy the acquired images, reducing the amount of data transmitted over the local area network. The wireless communication unit supports the Wi-Fi 6 protocol and establishes a TLS 1.3 encrypted link with the backend service system; The local caching and preloading submodule is used to preload frequently used administrative interface data for users when the network is stable, and to generate basic broadcasts based on the cache when the network is interrupted.

5. The interactive system and method for visually impaired individuals to interact with their environment according to claim 4, characterized in that, The user interface includes a touch panel and a voice wake-up unit; the touch panel is integrated into one side of the temple of the glasses to realize interactive operation based on operation logic, and the voice wake-up unit supports users to trigger scene tag selection through voice commands.

6. The interactive system and method for visually impaired individuals to interact with their environment according to claim 5, characterized in that, The open-closed interaction logic specifically includes: For open system devices, structured UI data is obtained through the UIAutomation / AT-SPI interface; For closed system devices, visual analysis is triggered. The device images captured by smart glasses are combined with the LayoutLM document layout analysis model to reverse analyze the text, position and state of UI controls.

7. The interactive system and method for visually impaired individuals to interact with their environment according to claim 6, characterized in that, The computer vision processing unit specifically includes: The image preprocessing subunit eliminates light reflections through denoising, corrects oblique gaze distortion through perspective transformation, and enhances contrast to highlight the differences between icons and text. The OCR subunit, based on the Transformer OCR model, recognizes administrative text and outputs the text and its location bounding box. The graphic element recognition subunit uses a pre-trained YOLOv8 model, combined with enterprise-customized icons and semantic tags uploaded by the administrator for transfer learning fine-tuning, and automatically retrains when a new icon appears.

8. The interactive system and method for visually impaired individuals to interact with their environment according to claim 7, characterized in that, The interaction intent parsing and decision-making unit specifically includes the following process: A: Interaction mode determination: Based on the device connection status, if it is connected to an open or closed system device, it is determined to be in IoT mode; if it is looking at a paper document, it is determined to be in visual mode. B: Focal point information association: For IoT mode: Combining the dynamic position parameters obtained by the device's spatial positioning module, a transformation matrix is ​​used. world coordinates of the gaze point Mapped to the display device screen coordinate system The formula is: ； In the formula, the transformation matrix It includes translation, rotation, and scaling transformation parameters, and allows querying structured UI data or visual analysis results to extract control properties. For visual patterns: map the world coordinates of the gaze point to the image coordinate system, and match them with the OCR results and graphic element recognition results to determine gaze information; C: Broadcast text generation: Combine scene tags to generate semantically complete natural language descriptions.

9. An interactive system and method for visually impaired individuals to interact with their environment according to claim 8, characterized in that, The training of the prediction model specifically includes the following process: Data Acquisition: Anonymously record interaction events, including scene labels, gaze duration, and number of repetitions. Combine this with IMU data to filter noise, and analyze head angular velocity. When this happens, mark it as invalid data; Model learning: The XGBoost algorithm was used, with effective data of fixation duration > 2 seconds and repeated playback ≥ 2 times as the training set, to build a prediction model. The objective function is: In the formula, The number of samples; The loss function; The number of decision trees; This is a regularization term used to control the regularization of each tree. The complexity; Strategy Update: Apply model parameters to the interaction intent parsing and decision-making units to optimize the broadcast priority and level of detail.

10. An interaction method for visually impaired individuals to interact with their environment, applicable to the interaction system for visually impaired individuals to interact with their environment as described in claim 9, characterized in that, Includes the following steps: S1: System initialization and scene calibration; The user wears the smart glasses terminal and starts the system. The near-infrared camera collects pupil features to complete personalized eye tracking calibration. The smart glasses terminal receives Bluetooth beacon signals from the display device and initializes the device location parameters. The background service system scans and authenticates devices within the local area network. The user confirms the administrative scene label through voice commands or system scene recommendations. S2: Real-time perception and interaction mode determination; The multimodal sensing module continuously collects images from the display device, world coordinates of the gaze point, and head motion data, and updates the device position parameters; The interaction intent parsing and decision-making unit determines the interaction mode based on the device connection status. Specifically, when connected to an open system device: it is determined to be IoT mode - UI interface parsing; when connected to a closed system device: it is determined to be IoT mode - visual parsing; when gazing at a paper document: it is determined to be visual mode. S3: Dual-mode administrative information acquisition; if it is IoT mode - UI interface parsing, obtain structured UI data and extract control attributes by associating gaze points; if it is IoT mode - visual parsing, trigger the visual parsing process to obtain UI control information; if it is visual mode, identify text and graphic elements in paper documents and match gaze points; S4: Administrative content broadcasting and feedback; the interactive intent parsing and decision-making unit combines scene tags to generate broadcast text, which is then converted into streaming speech and played by the speech synthesis unit; users can adjust the feedback content via touch panel or voice commands, and basic information is broadcast based on local cache when the network is interrupted; S5: Scenario-based adaptive optimization, which involves filtering effective interaction data and updating prediction models and strategies.

Images