Amblyopia and strabismus recovery training equipment and system

Abstract

The invention discloses amblyopia and strabismus recovery training equipment and system, and the method comprises the steps: collecting a face image of a user through a camera, extracting face key feature points from the face image of the user through employing an image analysis and processing technology based on machine vision, and calculating the head posture data of the user; performing data normalization on the head posture data of the user for embedded coding, and performing facial feature extraction and enhancement on the facial image of the user at the same time; therefore, adaptive adjustment of the distance between two identical object images and the display angle of the object images is realized based on cross-modal intersection prompt joint representation between enhanced user facial features and user head posture embedding features. Through the mode, the normal eye position interpupillary distance and the head posture of the user can be matched more accurately, the object image distance and the display angle can be adjusted in real time, manual intervention of the user is not needed, and therefore it is ensured that training parameters are consistent with actual requirements of the user, and the effect of amblyopia and strabismus recovery training is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent training, and more specifically, to a device and system for training amblyopia and strabismus recovery. Background Art

[0002] Patients with amblyopia and strabismus usually face the problem of poor binocular coordination, which affects their depth perception and stereoscopic vision. Traditional amblyopia treatment only provides a single recovery training mode with a limited field of view, usually only including red light flashing and picture observation. This training method is relatively monotonous and difficult to attract patients to concentrate for a long time, so the treatment effect is not always satisfactory.

[0003] In this regard, patent CN113786277B discloses a device and system for restoring amblyopia and strabismus, which uses VR technology to display virtual reality restoration training items and can generate two identical images on the display screen to be displayed simultaneously at a predetermined distance. The user can adjust the distance between the two images to match their normal pupil distance, thereby helping to correct strabismus. This method improves the traditional defects, not only expanding the field of vision, but also increasing the amount of stimulation to the eyes, thereby improving the restoration training effect.

[0004] In the above patent, the user manually adjusts the distance between the two objects to match the user's normal eye-pupillary distance. However, manual adjustment relies on the user's subjective judgment and may not achieve the best accuracy. It is especially difficult for children or the elderly, who have weaker operating abilities, to master the correct adjustment method, which increases the threshold for use. In addition, during the training process, the user's head position may change from time to time, and the manual adjustment method cannot respond to these changes in real time, which may cause the distance between the objects and the display angle to be inconsistent with the user's current actual needs, thereby affecting the consistency and effectiveness of the training effect.

[0005] Therefore, an optimized amblyopia and strabismus recovery training program is desired, which can adaptively adjust the distance and display angle between two objects to ensure precise matching with the actual needs of users, so as to provide more personalized and accurate vision recovery training. Summary of the invention

[0006] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides an amblyopia and strabismus recovery training device and system, which collects the user's facial image by a camera, and uses machine vision-based image analysis and processing technology to extract facial key feature points from the user's facial image and calculate the user's head posture data, and then performs data regularization and embedding coding on the user's head posture data, and at the same time extracts and enhances the user's facial image. The adaptive adjustment of the distance between two identical objects and the display angle of the object image is achieved based on the cross-modal cross-cue joint representation between the enhanced user's facial features and the embedded features of the user's head posture. In this way, the user's normal eye position pupil distance and head posture can be matched more accurately, and the object image distance and display angle can be adjusted in real time without the need for manual intervention by the user, thereby ensuring that the training parameters are consistent with the actual needs of the user, and effectively improving the effect of amblyopia and strabismus recovery training.

[0007] According to one aspect of the present application, a device for restoring amblyopia and strabismus is provided, comprising:

[0008] A display screen, a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program execution process is as follows:

[0009] Presenting a VR recovery training item selection interface on the display screen;

[0010] Acquire a VR recovery training item selection instruction and a VR recovery training item training operation instruction input by a user;

[0011] Creating two identical object images based on the VR recovery training item selection instruction and the VR recovery training item training operation instruction, and displaying them simultaneously on the display screen with a predetermined distance between them;

[0012] Capturing a user's facial image through a camera and storing the user's facial image in a memory;

[0013] The user facial image is retrieved from the memory, and feature analysis is performed on the user facial image to adjust the distance and display angle between the two identical objects in real time, including: extracting facial key feature points from the user facial image, and calculating user head posture data based on the facial key feature points; performing data regularization and embedding coding on the user head posture data to obtain user head posture embedded features; performing facial feature extraction and multi-posture perception enhancement modulation on the user facial image to obtain enhanced user facial features; performing cross-modal interactive prompt optimization combination on the enhanced user facial features and the user head posture embedded features to obtain user facial semantics-head posture cross-modal joint coding features; based on the user facial semantics-head posture cross-modal joint coding features, obtaining an object image display adjustment instruction, the object image display adjustment instruction including a recommended value for the distance between two identical objects and a recommended value for the object image display angle.

[0014] According to another aspect of the present application, a system for restoring amblyopia and strabismus is provided, comprising:

[0015] A user head posture data calculation module is used to extract key facial feature points from the user's facial image and calculate the user's head posture data based on the key facial feature points;

[0016] A data regularization and embedding coding module, used for performing data regularization and embedding coding on the user head posture data to obtain a user head posture embedding feature;

[0017] A facial feature extraction and multi-posture perception enhancement modulation module, used for performing facial feature extraction and multi-posture perception enhancement modulation on the user facial image to obtain enhanced user facial features;

[0018] A cross-modal interaction prompt optimization joint module, used for performing cross-modal interaction prompt optimization joint on the enhanced user facial features and the user head posture embedded features to obtain a user facial semantics-head posture cross-modal joint encoding feature;

[0019] The object image display adjustment instruction generation module is used to obtain the object image display adjustment instruction based on the user's facial semantics-head posture cross-modal joint coding features, and the object image display adjustment instruction includes a recommended value of the distance between two identical objects and a recommended value of the object image display angle.

[0020] Compared with the prior art, the present application provides a device and system for amblyopia and strabismus recovery training, which collects the user's facial image through a camera, and uses machine vision-based image analysis and processing technology to extract key facial feature points from the user's facial image and calculate the user's head posture data, and then performs data regularization and embedding coding on the user's head posture data, and extracts and enhances the facial features of the user's facial image at the same time, thereby realizing adaptive adjustment of the distance between two identical images and the display angle of the image based on the cross-modal cross-cue joint representation between the enhanced user's facial features and the embedded features of the user's head posture. In this way, the user's normal eye position pupil distance and head posture can be matched more accurately, and the image distance and display angle can be adjusted in real time without manual intervention by the user, thereby ensuring that the training parameters are consistent with the user's actual needs, and effectively improving the effect of amblyopia and strabismus recovery training. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0022] Figure 1 is a flow chart of a device for recovering amblyopia and strabismus according to an embodiment of the present application;

[0023] Figure 2 A schematic diagram of data flow of a device for recovering amblyopia and strabismus according to an embodiment of the present application;

[0024] Figure 3 This is a flowchart of sub-step S5 of the amblyopia and strabismus recovery training device according to an embodiment of the present application;

[0025] Figure 4 4 is a block diagram of a system for recovering amblyopia and strabismus according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0027] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0028] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0029] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. Meanwhile, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0030] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0031] In the technical solution of the present application, a device for training amblyopia and strabismus recovery is proposed. Figure 1 Flow chart of amblyopia and strabismus recovery training device according to an embodiment of the present application. Figure 2 Schematic diagram of data flow of amblyopia and strabismus recovery training device according to an embodiment of the present application. Figure 1 and Figure 2 As shown, according to the embodiment of the present application, the amblyopia and strabismus recovery training device includes: a display screen, a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the execution process of the computer program is as follows: S1, presenting a VR recovery training project selection interface on the display screen; S2, obtaining a VR recovery training project selection instruction and a VR recovery training project training operation instruction input by a user; S3, creating two identical images based on the VR recovery training project selection instruction and the VR recovery training project training operation instruction, and displaying them simultaneously at a predetermined distance on the display screen; S4, capturing a user's facial image through a camera, and storing the user's facial image in a memory; S5, retrieving the user's facial image from the memory, and performing feature analysis on the user's facial image to adjust the spacing and display angle between the two identical images in real time.

[0032] In particular, S1 presents a VR recovery training program selection interface on the display screen. The main purpose of presenting the VR recovery training program selection interface on the display screen is to enable users to easily select and start different recovery training programs. Through this interface, users can browse and select various types of VR recovery training programs, such as VR games and VR videos, to help them choose appropriate recovery training programs according to their needs. Through a well-designed interface and a variety of interactive methods, users can easily select and start training programs, thereby improving the efficiency and effectiveness of training.

[0033] In particular, the step S2 obtains the VR recovery training item selection instruction and the VR recovery training item training operation instruction input by the user. The main purpose of obtaining the instructions input by the user is to achieve two-way interaction between the user and the system, so that the user can select a suitable recovery training item according to their needs and perform various operations during the training process. In this way, the user can independently control the progress of the training and improve the flexibility and personalization of the training.

[0034] In particular, the S3 creates two identical images based on the VR recovery training project selection instruction and the VR recovery training project training operation instruction, and displays them on the display screen at a predetermined distance. In the amblyopia and strabismus recovery training device, creating two identical images based on the VR recovery training project and training operation instruction selected by the user, and displaying them on the display screen at a predetermined distance, is a key step to achieve accurate training. The main purpose of this function is to provide users with an accurate training environment, and to help users correct strabismus problems by creating two identical images and adjusting their spacing. These images are generated based on the content of the selected VR recovery training project, which can be game screens, video clips or other training content. The system determines the initial spacing between the two images according to the preset default value or the user's initial setting. After the user enters the project, various operations can be performed through the touch screen or the control handle. For example, the user can slide the screen to adjust the image spacing, click a button to control the game progress, etc. During the training process, if the user finds that the image spacing is not appropriate, the user can enter the spacing adjustment instruction through the specific button of the control handle. After receiving the spacing adjustment instruction, the system will calculate the new spacing value and adjust the distance between the two images. The two object images on the display will update their display positions in real time according to the new spacing value, ensuring that users can see the adjusted effects.

[0035] In particular, the S4, collects the user's facial image through the camera and stores the user's facial image in a memory. In the amblyopia and strabismus recovery training device, collecting the user's facial image through the camera and storing the image in the memory is an important step to achieve adaptive object-image distance and display angle adjustment. This process ensures that the system can obtain the user's facial information in real time, and then perform accurate head posture estimation and pupil distance calculation, thereby optimizing the training effect.

[0036] In particular, the step S5 retrieves the user's facial image from the memory and performs feature analysis on the user's facial image to adjust the distance and display angle between the two identical images in real time. Figure 3 As shown, the S5 includes: S51, extracting facial key feature points from the user facial image, and calculating the user head posture data based on the facial key feature points; S52, performing data regularization and embedding coding on the user head posture data to obtain user head posture embedded features; S53, performing facial feature extraction and multi-posture perception enhancement modulation on the user facial image to obtain enhanced user facial features; S54, performing cross-modal interactive prompt optimization combination on the enhanced user facial features and the user head posture embedded features to obtain user facial semantics-head posture cross-modal joint coding features; S55, based on the user facial semantics-head posture cross-modal joint coding features, obtaining an object image display adjustment instruction, the object image display adjustment instruction including a recommended value for the spacing between two identical objects and a recommended value for the object image display angle.

[0037] Specifically, the S51 extracts key facial feature points from the user's facial image, and calculates the user's head posture data based on the key facial feature points. Wherein, the user's head posture data includes pitch angle, yaw angle and roll angle. Considering that when performing vision recovery training, the eyes are key feature information, which can reflect the user's gaze direction and focus, and ensure the accuracy and effectiveness of visual stimulation. At the same time, when adjusting the object-image distance, it is also necessary to comprehensively consider the user's head posture, including pitch angle, yaw angle and roll angle, to achieve more accurate content positioning and display. Based on this, in the technical solution of the present application, the key facial feature points are extracted from the user's facial image, and the user's head posture data is calculated based on the key facial feature points, and the user's head posture data includes pitch angle, yaw angle and roll angle. It should be understood that the user's head posture data can calculate the specific posture of the user's head relative to the camera by utilizing the relative position relationship between the key facial feature points. Specifically, the pitch angle is estimated by calculating the angle between the tip of the nose and the line connecting the centers of the two eyes. The yaw angle is estimated by calculating the angle between the line connecting the centers of the two eyes and the horizontal line. The roll angle is estimated by calculating the angle between the line connecting the centers of the two eyes and the vertical line.

[0038] In one example, the process of extracting facial key feature points from the user's facial image and calculating the user's head posture data based on the facial key feature points involves multiple links such as image preprocessing, key feature point positioning, and posture estimation. Specifically, first, the user's facial image is captured by a camera. The image captured by the camera is usually a color image. In order to reduce the amount of calculation for subsequent processing, the image needs to be grayed. Graying can convert the color value of each pixel of the color image into a gray value, which can simplify the complexity of image processing. Then, the image is normalized to ensure that the brightness and contrast of the image remain consistent so that the subsequent facial detection and feature point extraction steps can be performed more accurately. After completing the image preprocessing, the facial detection stage is entered. The goal of facial detection is to locate the position of the face from the image. Common facial detection algorithms include Haar feature cascade classifiers, HOG+SVM detectors in the Dlib library, etc. These algorithms can efficiently detect the facial area in the image by training a large number of face samples. Once the facial area is detected, the key feature points can be further extracted. The extraction of key feature points is achieved through a specific feature point detection algorithm. Common feature point detection algorithms include the 68-point facial landmark detector in the Dlib library and MediaPipe Face Mesh. These algorithms can detect multiple key points on the face, such as the specific locations of the eyes, eyebrows, nose, mouth, etc. Each key point detected has a two-dimensional coordinate (x, y) in the image, and these coordinate information will be used for subsequent head posture estimation. After extracting the key facial feature points, the user's head posture data needs to be calculated next. The head posture data mainly includes three Euler angles: pitch, yaw, and roll. These angles represent the degree of up and down tilt, left and right rotation, and left and right tilt of the head, respectively. The method of calculating these angles is usually implemented through a pre-trained head posture estimation model. These models can be deep learning-based models, such as MTCNN, OpenPose, etc. These models output the posture parameters of the head by inputting the extracted facial key feature point coordinates. Specifically, the pitch angle (Pitch) represents the angle of up and down tilt of the head, which can be estimated by calculating the vertical distance of the center point of the eye. The yaw angle (Yaw) indicates the angle of the head turning left and right, which can be estimated by calculating the horizontal distance of the eye center. The roll angle (Roll) indicates the angle of the head tilting left and right, which can be estimated by calculating the tilt angle of the line connecting the two eye centers.

[0039] Specifically, the S52 performs data regularization and embedding coding on the user head posture data to obtain the user head posture embedding feature. That is, in the technical solution of the present application, after the user head posture data is regularized into a head posture vector according to the head posture sample dimension, the head posture vector is embedded and coded by a head posture embedding matrix to obtain the user head posture embedding coding vector as the user head posture embedding feature. Here, in order to regularize the head posture data of each user into a vector of a unified format so that it is consistent in numerical range and distribution, in the technical solution of the present application, the user head posture data is regularized into a head posture vector according to the head posture sample dimension. Next, considering that the head posture vector contains key semantic information about the user's head posture, and in order to better understand and utilize complex head posture information, so as to more accurately adjust the training content to match the user's actual viewing conditions, further, the head posture vector is embedded and coded by a head posture embedding matrix to capture and extract key head posture features, and obtain the user head posture embedding coding vector. In particular, the head posture embedding matrix is ​​obtained by training through a machine learning model.

[0040] Specifically, the S53 performs facial feature extraction and multi-gesture perception enhancement modulation on the user facial image to obtain enhanced user facial features. That is, in the technical solution of the present application, first, the user facial image is passed through a human face facial feature extractor based on a convolutional neural network model to obtain a user facial feature map. Considering that the user facial image includes key point position information of the eyes in the face, it may also contain more complex patterns, such as the gaze direction of the eyes. And considering that the convolutional neural network (CNN) is a deep learning model, it is particularly suitable for processing data with a grid structure, such as images. The core of CNN lies in its unique structural design, which can efficiently capture local features and maintain spatial invariance. Therefore, in the technical solution of the present application, the user facial image is passed through a human face facial feature extractor based on a convolutional neural network model to utilize the powerful feature extraction capability of the convolutional neural network to automatically identify and extract important facial feature points from the facial image to obtain a user facial feature map. Then, the user facial feature map is subjected to multi-gesture perception enhancement modulation to obtain an enhanced user facial feature map as the enhanced user facial feature. Here, it is considered that when the user's head posture changes, it will not only affect the appearance of the face, but also cause significant changes in facial features and facial states. For example, the pitch (up and down tilt), yaw (left and right rotation) and roll (rotation along its own axis) of the head will change the position and relative relationship of key facial features such as the eyes. Therefore, in order to maintain accurate recognition and effective processing of facial features when facing changes in the user's head posture, in the technical solution of the present application, the user's facial feature map is subjected to multi-posture perception enhancement modulation to obtain an enhanced user facial feature map.Specifically, in an embodiment of the present application, the specific steps of performing multi-gesture perception enhancement modulation on the user facial feature map include: first, upsampling the user facial feature map to capture finer facial details, obtaining an upsampled user facial feature map with a higher resolution, and fusing the original user facial feature map and the upsampled user facial feature map to obtain features containing facial information at different levels, so as to obtain a multi-scale user facial feature map; then, convolutionally encoding the multi-scale user facial feature map using convolutional layers with convolutional kernels of 1×3, 3×1, and 3×3, respectively, to obtain a first user facial convolutional encoding feature map, a second user facial convolutional encoding feature map, and a third user facial convolutional encoding feature map; and convolutionally encoding the fused feature map by using convolutional kernels of different shapes (1×3, 3×1, and 3×3) to utilize Convolution kernels of different shapes are used to capture facial feature information in different directions and scales; then, the means of the corresponding positions in the first user face convolution coding feature map, the second user face convolution coding feature map and the third user face convolution coding feature map are calculated to synthesize the feature information extracted by different convolution kernels, and the obtained multi-pose user face convolution coding feature map is subjected to hole convolution coding to further expand the receptive field, capture context information in a larger range, and obtain a multi-pose perception user face feature map, so as to obtain a multi-pose perception user face feature map; that is, the hole convolution can increase the receptive field of the network without increasing the number of parameters, which helps to better understand the changes of facial features in different postures; finally, the user face feature map and the multi-pose perception user face feature map are residually connected to obtain the enhanced user face feature map. It should be understood that the residual connection can help retain the important information in the original feature map, while introducing the features of multi-pose perception, so that the enhanced feature map retains the details of the original features and contains information about posture changes.

[0041] Specifically, the S54 performs a cross-modal interactive prompt optimization combination on the enhanced user facial features and the user head posture embedded features to obtain a user facial semantics-head posture cross-modal joint coding feature. Considering that the enhanced user facial feature map is a facial image feature representation after multi-posture perception enhancement modulation processing, it contains detailed visual information of the user's facial key points (such as eyes, etc.). The user head posture embedded coding vector is a vector representation after encoding the user's head posture data (such as pitch angle, yaw angle, roll angle), capturing the posture information of the head in three-dimensional space. Therefore, in order to be able to integrate the information of the two, to create a unified representation that integrates visual appearance and spatial position information, so as to more accurately capture the correlation between facial expression changes and head posture. In the technical solution of the present application, the enhanced user facial features and the user head posture embedded features are cross-modal interactive prompt optimization combined to obtain the user facial semantics-head posture cross-modal joint coding feature, so that a representation that contains both rich visual details and powerful context perception can be constructed, thereby improving the accuracy of object image display adjustment.

[0042] Specifically, the specific process of optimizing the combination of the enhanced user facial features and the user head posture embedded features for cross-modal interactive prompts is as follows: First, the product between the user head posture embedded coding vector and the transposed vector of the user head posture embedded coding vector is calculated to obtain the user head posture autocorrelation coding matrix. Here, by utilizing the concept of outer product in matrix operations, the obtained user head posture autocorrelation coding matrix can reveal the relationship strength between different feature elements within the user head posture, that is, autocorrelation. The user head posture autocorrelation coding matrix generated in this way not only contains the information of the user head posture embedded coding vector, but also adds the correlation measure between features, providing a basis for subsequent inter-modal interaction.

[0043] More specifically, the product between the user head posture embedding coding vector and the transposed vector of the user head posture embedding coding vector is calculated by the following autocorrelation coding formula to obtain the user head posture autocorrelation coding matrix; wherein the autocorrelation coding formula includes:

[0044]

[0045] Wherein, V1 is the user head posture embedding coding vector, is matrix multiplication, V1 T is the transposed vector of V1, M z is the user's head posture autocorrelation encoding matrix.

[0046] Next, feature decoupling is performed on the enhanced user facial feature map to obtain a set of enhanced user facial local feature matrices. Feature decoupling of the enhanced user facial feature map is conducive to implementing fine-grained cross-modal interaction, that is, improving the model's attention to local details, while reducing redundant information between features and enhancing feature differentiation.

[0047] More specifically, the enhanced user facial feature map is feature decoupled using the following feature decoupling formula to obtain a set of enhanced user facial local feature matrices; wherein the feature decoupling formula includes:

[0048] Decouple(F2)={M1,...,M i ,...,M n}

[0049] Wherein, F2 is the enhanced user facial feature map, Decouple(F2) is the feature decoupling operation performed on F2, M1, M i and M n They are respectively the 1st, ith and nth enhanced user facial local feature matrices in the set of enhanced user facial local feature matrices.

[0050] Then, each enhanced user facial local feature matrix in the set of the user head posture autocorrelation coding matrix and the enhanced user facial local feature matrix is ​​respectively cross-modal prompt coded to obtain a set of user facial semantics-head posture cross-modal prompt information semantic coding matrices. Specifically, first, the user head posture autocorrelation coding matrix is ​​linearly transformed to obtain a user head posture query coding matrix and a user head posture value coding matrix; through a linear mapping method, the user head posture autocorrelation coding matrix is ​​converted into a form suitable for attention mechanism calculation, wherein the user head posture query coding matrix is ​​used to find the correlation with other modal features, and the user head posture value coding matrix saves important information of the user head posture features. This conversion is the key to achieving cross-modal interaction, so that information from different modalities can be compared and fused in a unified way. It is worth mentioning that linear transformation is not just a simple mathematical operation, but by introducing trainable parameters to adjust the spatial distribution of feature representation, so that the converted query coding matrix and value coding matrix are more suitable for cross-modal comparison and fusion. Then, each enhanced user facial local feature matrix in the set of the user head posture query encoding matrix, the user head posture value encoding matrix and the enhanced user facial local feature matrix is ​​respectively input into the cross-modal prompt information encoder based on the converter structure to obtain the set of user facial semantics-head posture cross-modal prompt information semantic encoding matrices. In this process, the converter structure allows the model to perform well in processing long-distance dependencies and can effectively capture the complex interaction patterns between different modal user head posture features. The set of user facial semantics-head posture cross-modal prompt information semantic encoding matrices generated in this way not only retains the information of the original modal user head posture features, but also integrates supplementary knowledge from other modalities, thereby improving the richness and accuracy of feature representation. The core of the converter structure lies in the self-attention mechanism, which allows the model to directly focus on the relationship between all elements without considering the order when processing sequence data. In the cross-modal scenario, this mechanism can help the model cross the gap between different modalities and establish deep semantic connections.

[0051] More specifically, the following cross-modal prompt coding formula is used to perform cross-modal prompt coding on each enhanced user facial local feature matrix in the set of the user head posture autocorrelation coding matrix and the enhanced user facial local feature matrix to obtain a set of user facial semantics-head posture cross-modal prompt information semantic coding matrices; wherein the cross-modal prompt coding formula includes:

[0052]

[0053] Among them, W q is the query embedding matrix, Wq is the user head pose query encoding matrix, W v is the value embedding matrix, M v is the encoding matrix of the user's head posture value, M i T It is M i The transposed matrix of M i The scale is the matrix width multiplied by the matrix height, softmax(·) is the softmax function, I i It is M q and M i The semantic encoding matrix of user facial semantics-head posture cross-modal prompt information.

[0054] Then, the set of semantic encoding matrices of the user's facial semantics-head posture cross-modal prompt information is gated and decoded to obtain a set of user facial semantics-head posture cross-modal interactive attention weights. That is, the set of semantic encoding matrices of the user's facial semantics-head posture cross-modal prompt information is input into the decoder-based information gating unit to obtain a set of user facial semantics-head posture cross-modal interactive attention weights. The information gating unit is responsible for adjusting the importance of different modal features and determining which information is more critical by calculating the attention weights. In multimodal learning, this mechanism helps the model to flexibly allocate attention resources according to the context, avoiding over-reliance on a certain modal information and ignoring other clues that may be equally important. By optimizing the attention weights, the model can more accurately capture the complementarity and synergy between different modalities.

[0055] More specifically, the set of user facial semantics-head posture cross-modal prompt information semantic encoding matrices is gated and decoded using the following gated decoding formula to obtain a set of user facial semantics-head posture cross-modal interaction attention weights; wherein the gated decoding formula is:

[0056] a i =softmax{decoder(I i ,W a )}

[0057] Among them, W a is the weight matrix, decoder is the decoder, softmax(·) is the softmax function, a i It is M i Corresponding user facial semantics-head pose cross-modal interaction attention weights.

[0058] Further, the user head posture autocorrelation coding matrix and the set of enhanced user facial local feature matrices are cross-modally interacted to obtain a set of user facial semantic-head posture cross-modal interaction local feature matrices. That is, the set of user head posture autocorrelation coding matrix and enhanced user facial local feature matrix are input into the cross-modal interaction unit to obtain a set of user facial semantic-head posture cross-modal interaction local feature matrices. In this step, direct interaction of features between modalities is achieved by positional point multiplication. This method can effectively capture the complementary information of the two modalities of user facial semantics-head posture at the same spatial position, thereby enhancing the spatial consistency of feature representation. Finally, the set of user facial semantics-head posture cross-modal interaction attention weights and the set of user facial semantics-head posture cross-modal interaction local feature matrices are cross-modally optimized to obtain the user facial semantics-head posture cross-modal joint coding feature vector as the user facial semantics-head posture cross-modal joint coding feature. That is, the set of attention weights of the user's facial semantics-head posture cross-modal interaction and the set of local feature matrices of the user's facial semantics-head posture cross-modal interaction are input into the cross-modal interaction optimization unit to obtain the user's facial semantics-head posture cross-modal joint coding feature vector. The cross-modal interaction optimization unit combines the attention mechanism, interaction features and pooling operations, and generates a joint coding feature with stronger expression ability and less redundant information by weighted fusion of different modal features.

[0059] More specifically, the set of the user facial semantics-head posture cross-modal interaction attention weights and the set of the user facial semantics-head posture cross-modal interaction local feature matrices are cross-modally optimized using the following cross-modal interaction formula to obtain a user facial semantics-head posture cross-modal joint encoding feature vector; wherein the cross-modal interaction formula is:

[0060] F 1-2 =couple{a1·M1⊙M z ,...,a i ·M i ⊙M z ,...,a n ·M n ⊙M z}

[0061] V 1-2 =GAP(F 1-2 )

[0062] Among them, ⊙ is the position point product, couple{·,·,...,·} is the feature coupling along the channel dimension, and F 1-2is the user's facial semantics-head posture cross-modal joint encoding feature map, GAP(·) represents the global mean pooling of each feature matrix along the channel dimension in the feature map, V 1-2 It is the user's facial semantics-head posture cross-modal joint encoding feature vector.

[0063] Specifically, the S55, based on the user facial semantics-head posture cross-modal joint coding feature, obtains an object image display adjustment instruction, and the object image display adjustment instruction includes a recommended value for the spacing between two identical objects and a recommended value for the display angle of the object image. In the technical solution of the present application, the user facial semantics-head posture cross-modal joint coding feature vector is passed through a decoder-based object image display adjustment module to obtain the object image display adjustment instruction. The user facial semantics-head posture cross-modal joint coding feature vector is passed through a decoder-based object image display adjustment module to obtain the object image display adjustment instruction, and the object image display adjustment instruction includes a recommended value for the spacing between two identical objects and a recommended value for the display angle of the object image. That is, the user facial semantics-head posture cross-modal joint coding feature obtained by hinting the joint is decoded and processed using the enhanced user facial feature map and the user head posture embedded coding vector, so as to realize the adaptive adjustment of the spacing between two identical objects and the display angle of the object image. In this way, the user's normal eye position, pupil distance and head posture can be matched more accurately, and the object-image distance and display angle can be adjusted instantly without the need for manual intervention by the user, thereby ensuring that the training parameters are consistent with the user's actual needs, effectively improving the effect of amblyopia and strabismus recovery training.

[0064] In particular, the decoder module usually contains multiple neural network layers. During the decoding process, these layers gradually parse the user's facial semantics-head posture cross-modal joint encoding feature vector to gradually restore the corresponding physical parameters. Specifically: in the calculation of the recommended spacing value, the decoder will calculate the most suitable object-image spacing for the current user based on the eye spacing information in the head posture data. During the calculation process, the decoder will consider factors such as the user's pupil distance change and head tilt to ensure that the recommended spacing value can accurately match the user's current eye position pupil distance; and in the calculation of the recommended value of the object display angle, the decoder will also calculate the most suitable object display angle for the current user based on the tilt angle and rotation angle information in the head posture data. During the calculation process, the decoder will comprehensively consider the user's head posture changes to ensure that the recommended angle value can enable the user to comfortably view the object in any head posture. Further, after obtaining the recommended spacing value and the recommended value of the object display angle, the system will dynamically adjust the distance between the two objects according to the recommended spacing value to ensure that they are consistent with the user's pupil distance; and adjust the display angle of the object according to the recommended value of the object display angle to ensure that the user can clearly see the object regardless of the posture of the head. Through the above decoding process, the system can adjust the distance and display angle of objects and images in real time and accurately, ensuring that the training parameters are always consistent with the actual needs of the user.

[0065] In one example, since the enhanced user facial feature map and the user head posture embedded coding vector represent the image semantic features of the user facial image and the embedded coding features of the user head posture data respectively, after cross-modal interactive optimization encoding based on a cross-modal prompt gate, the user facial semantics-head posture cross-modal joint coding feature vector will also have interactive fusion distribution space structure differences due to the differences in cross-modal feature suggestiveness, affecting the decoding regression convergence consistency, and thus affecting the accuracy of the object display adjustment instructions obtained by the decoder-based object display adjustment module.

[0066] Therefore, it is preferred to optimize the user facial semantics-head posture cross-modal joint encoding feature vector, including:

[0067] Calculate the absolute value sum of all eigenvalues ​​of the user facial semantics-head posture cross-modal joint encoding feature vector to obtain the first user facial semantics-head posture cross-modal joint encoding spatial structure value, and calculate the square root of the sum of squares to obtain the second user facial semantics-head posture cross-modal joint encoding spatial structure value, that is:

[0068]

[0069] Among them, v irepresents each eigenvalue of the feature vector of the cross-modal joint encoding of the user's facial semantics and head posture, w1 represents the spatial structure value of the cross-modal joint encoding of the first user's facial semantics and head posture, and w2 represents the spatial structure value of the cross-modal joint encoding of the second user's facial semantics and head posture;

[0070] For each eigenvalue of the user facial semantics-head posture cross-modal joint encoding feature vector, the first user facial semantics-head posture cross-modal joint encoding spatial structure value minus the product of the eigenvalue and the total number of eigenvalues ​​of the user facial semantics-head posture cross-modal joint encoding feature vector is calculated to obtain the first user facial semantics-head posture cross-modal joint encoding long-range dependency value x i =w1-v i ×n;

[0071] Calculate the square root of the total number of eigenvalues ​​multiplied by the product of the eigenvalues ​​minus the second user facial semantics-head posture cross-modal joint coding spatial structure value to obtain the second user facial semantics-head posture cross-modal joint coding long-range dependency value

[0072] The exponential value calculated by taking the first user's facial semantics-head posture cross-modal joint coding long-range dependency value as the exponent of the natural constant and the inverse of the second user's facial semantics-head posture cross-modal joint coding long-range dependency value are weighted summed to obtain the optimized eigenvalue corresponding to each eigenvalue as well as

[0073] The optimized feature values ​​are combined into an optimized user facial semantics-head posture cross-modal joint encoding feature vector.

[0074] Then, the optimized user facial semantics-head posture cross-modal joint encoding feature vector is passed through a decoder-based object image display adjustment module to obtain an object image display adjustment instruction.

[0075] Therefore, in order to address the convergence inconsistency of decoding regression based on feature-based implicit inference of spatial structure information due to the possible spatial structure loss of the feature set of the user facial semantics-head posture cross-modal joint encoding feature vector in the high-dimensional space, a long-distance feature dependency relationship is established based on the overall feature scale of the user facial semantics-head posture cross-modal joint encoding feature vector relative to the spatial structure representation of the user facial semantics-head posture cross-modal joint encoding feature vector, so as to establish the feature local connectivity of the user facial semantics-head posture cross-modal joint encoding feature vector, and capture the spatial ambiguous information of the object feature value through the unstructured feature value point prediction of the user facial semantics-head posture cross-modal joint encoding feature vector, thereby improving the spatial inductive bias perception ability of the feature set of the user facial semantics-head posture cross-modal joint encoding feature vector, improving the convergence consistency of decoding regression, and improving the accuracy of the object display adjustment instruction obtained by the decoder-based object display adjustment module.

[0076] In summary, the amblyopia and strabismus recovery training device according to the embodiment of the present application is explained, which collects the user's facial image by a camera, and uses image analysis and processing technology based on machine vision to extract key facial feature points from the user's facial image and calculate the user's head posture data, and then performs data regularization and embedding coding on the user's head posture data, and at the same time extracts and enhances the facial features of the user's facial image, so as to realize the adaptive adjustment of the distance between two identical objects and the display angle of the object image based on the cross-modal cross-cue joint representation between the enhanced user's facial features and the embedded features of the user's head posture. In this way, the user's normal eye position pupil distance and head posture can be matched more accurately, and the object image distance and display angle can be adjusted in real time without the need for manual intervention by the user, thereby ensuring that the training parameters are consistent with the actual needs of the user, and effectively improving the effect of amblyopia and strabismus recovery training.

[0077] Furthermore, a system for recovering from amblyopia and strabismus is provided.

[0078] Figure 4 FIG. 1 is a block diagram of a system for restoring amblyopia and strabismus according to an embodiment of the present application. Figure 4As shown, according to the embodiment of the present application, the amblyopia and strabismus recovery training system 300 includes: a user head posture data calculation module 310, which is used to extract facial key feature points from the user facial image and calculate the user head posture data based on the facial key feature points; a data regularization and embedding coding module 320, which is used to perform data regularization and embedding coding on the user head posture data to obtain user head posture embedded features; a facial feature extraction and multi-posture perception enhancement modulation module 330, which is used to perform facial feature extraction and multi-posture perception enhancement modulation on the user facial image to obtain enhanced user facial features; a cross-modal interaction prompt optimization combination module 340, which is used to perform cross-modal interaction prompt optimization combination on the enhanced user facial features and the user head posture embedded features to obtain user facial semantics-head posture cross-modal joint coding features; an object image display adjustment instruction generation module 350, which is used to obtain an object image display adjustment instruction based on the user facial semantics-head posture cross-modal joint coding features, and the object image display adjustment instruction includes a recommended value for the spacing between two identical objects and a recommended value for the object image display angle.

[0079] As described above, the amblyopia and strabismus recovery training system 300 according to the embodiment of the present application can be implemented in various wireless terminals, such as a server with an amblyopia and strabismus recovery training algorithm. In a possible implementation, the amblyopia and strabismus recovery training system 300 according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or a hardware module. For example, the amblyopia and strabismus recovery training system 300 can be a software module in the operating system of the wireless terminal, or can be an application developed for the wireless terminal; of course, the amblyopia and strabismus recovery training system 300 can also be one of the many hardware modules of the wireless terminal.

[0080] Alternatively, in another example, the amblyopia and strabismus recovery training system 300 and the wireless terminal may be separate devices, and the amblyopia and strabismus recovery training system 300 may be connected to the wireless terminal via a wired and / or wireless network and transmit interactive information in an agreed data format.

[0081] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A device for training amblyopia and strabismus, characterized in that: include: A display screen, a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program execution process is as follows: Presenting a VR recovery training item selection interface on the display screen; Acquire a VR recovery training item selection instruction and a VR recovery training item training operation instruction input by a user; Creating two identical object images based on the VR recovery training item selection instruction and the VR recovery training item training operation instruction, and displaying them simultaneously on the display screen with a predetermined distance between them; Capturing a user's facial image through a camera and storing the user's facial image in a memory; The user facial image is retrieved from the memory, and feature analysis is performed on the user facial image to adjust the distance and display angle between the two identical objects in real time, including: extracting facial key feature points from the user facial image, and calculating user head posture data based on the facial key feature points; performing data regularization and embedding coding on the user head posture data to obtain user head posture embedded features; performing facial feature extraction and multi-posture perception enhancement modulation on the user facial image to obtain enhanced user facial features; performing cross-modal interactive prompt optimization combination on the enhanced user facial features and the user head posture embedded features to obtain user facial semantics-head posture cross-modal joint coding features; based on the user facial semantics-head posture cross-modal joint coding features, obtaining an object image display adjustment instruction, the object image display adjustment instruction including a recommended value for the distance between two identical objects and a recommended value for the object image display angle.

2. The amblyopia and strabismus recovery training device according to claim 1, characterized in that: The user's head posture data includes a pitch angle, a yaw angle, and a roll angle.

3. The amblyopia and strabismus recovery training device according to claim 2, characterized in that: The user head posture data is subjected to data regularization and embedding coding to obtain a user head posture embedding feature, including: regularizing the user head posture data into a head posture vector according to the head posture sample dimension, and then embedding coding the head posture vector through a head posture embedding matrix to obtain a user head posture embedding coding vector as the user head posture embedding feature.

4. The amblyopia and strabismus recovery training device according to claim 3, characterized in that: Performing facial feature extraction and multi-gesture perception enhancement modulation on the user facial image to obtain enhanced user facial features, including: The user's facial image is passed through a human face facial feature extractor based on a convolutional neural network model to obtain a user's facial feature map; Multi-gesture perception enhancement modulation is performed on the user facial feature map to obtain an enhanced user facial feature map as the enhanced user facial feature.

5. The amblyopia and strabismus recovery training device according to claim 4, characterized in that: Performing multi-gesture perception enhancement modulation on the user facial feature map to obtain an enhanced user facial feature map, including: Upsampling the user facial feature map to obtain an upsampled user facial feature map, and fusing the user facial feature map and the upsampled user facial feature map to obtain a multi-scale user facial feature map; Using convolution layers with convolution kernels of 1×3, 3×1, and 3×3, respectively, to perform convolution encoding on the multi-scale user facial feature map to obtain a first user facial convolution encoding feature map, a second user facial convolution encoding feature map, and a third user facial convolution encoding feature map; Calculating the mean of corresponding positions in the first user face convolutional coding feature map, the second user face convolutional coding feature map, and the third user face convolutional coding feature map, and performing hole convolution coding on the obtained multi-pose user face convolutional coding feature map to obtain a multi-pose perceived user face feature map; The user facial feature map and the multi-pose-aware user facial feature map are residually connected to obtain the enhanced user facial feature map.

6. The amblyopia and strabismus recovery training device according to claim 5, characterized in that: The enhanced user facial features and the user head posture embedded features are combined by cross-modal interactive prompt optimization to obtain user facial semantics-head posture cross-modal joint encoding features, including: Calculate the product between the user head posture embedding coding vector and the transposed vector of the user head posture embedding coding vector to obtain a user head posture autocorrelation coding matrix; Performing feature decoupling on the enhanced user facial feature map to obtain a set of enhanced user facial local feature matrices; Performing cross-modal prompt coding on each enhanced user facial local feature matrix in the set of the user head posture autocorrelation coding matrix and the enhanced user facial local feature matrix to obtain a set of user facial semantics-head posture cross-modal prompt information semantic coding matrices; Performing gated decoding on the set of semantic encoding matrices of the user's facial semantics-head posture cross-modal prompt information to obtain a set of user's facial semantics-head posture cross-modal interaction attention weights; Performing cross-modal interaction between the user head posture autocorrelation coding matrix and the set of enhanced user facial local feature matrices to obtain a set of user facial semantics-head posture cross-modal interaction local feature matrices; The set of user facial semantics-head posture cross-modal interaction attention weights and the set of user facial semantics-head posture cross-modal interaction local feature matrices are cross-modal interaction optimized to obtain a user facial semantics-head posture cross-modal joint encoding feature vector as the user facial semantics-head posture cross-modal joint encoding feature.

7. The amblyopia and strabismus recovery training device according to claim 6, characterized in that: Cross-modal prompt encoding is performed on each enhanced user facial local feature matrix in the set of the user head posture autocorrelation coding matrix and the enhanced user facial local feature matrix to obtain a set of user facial semantics-head posture cross-modal prompt information semantic coding matrices, including: Performing a linear transformation on the user head posture autocorrelation coding matrix to obtain a user head posture query coding matrix and a user head posture value coding matrix; The user head posture query encoding matrix, the user head posture value encoding matrix and each enhanced user facial local feature matrix in the set are respectively input into the cross-modal prompt information encoder based on the converter structure to obtain the set of user facial semantics-head posture cross-modal prompt information semantic encoding matrices.

8. The amblyopia and strabismus recovery training device according to claim 7, characterized in that: Based on the user's facial semantics-head posture cross-modal joint coding features, an object image display adjustment instruction is obtained, which is used to: pass the user's facial semantics-head posture cross-modal joint coding feature vector through a decoder-based object image display adjustment module to obtain the object image display adjustment instruction.

9. A system for restoring amblyopia and strabismus, characterized in that: include: A user head posture data calculation module is used to extract key facial feature points from the user's facial image and calculate the user's head posture data based on the key facial feature points; A data regularization and embedding coding module, used for performing data regularization and embedding coding on the user head posture data to obtain a user head posture embedding feature; A facial feature extraction and multi-posture perception enhancement modulation module, used for performing facial feature extraction and multi-posture perception enhancement modulation on the user facial image to obtain enhanced user facial features; A cross-modal interaction prompt optimization joint module, used for performing cross-modal interaction prompt optimization joint on the enhanced user facial features and the user head posture embedded features to obtain a user facial semantics-head posture cross-modal joint encoding feature; The object image display adjustment instruction generation module is used to obtain the object image display adjustment instruction based on the user's facial semantics-head posture cross-modal joint coding features, and the object image display adjustment instruction includes a recommended value of the distance between two identical objects and a recommended value of the object image display angle.

Images