Self-service vision testing method and system based on infrared distance measurement and gesture recognition

Abstract

The invention discloses a self-service vision testing method and system based on infrared distance measurement and gesture recognition. The method comprises the steps that the testing distance between a testee and testing equipment is obtained through an infrared distance measurement method; calculating the size of a sighting mark based on the test distance, and generating a random sighting mark with the corresponding size through test equipment; collecting a target image sequence of a testee and extracting a gesture image sequence; segmenting the gesture image sequence based on the continuity of the action, and extracting an image to be recognized; and obtaining a vision test result based on the to-be-recognized image. According to the method, infrared distance measurement, dynamic sighting mark generation and a gesture recognition algorithm are combined, a collected image sequence is simplified, the problems that vision testing depends on a fixed distance, the size is large, cost is high, individuation is lacked, operation is complex, and the real-time data processing amount is large are solved, and convenient, efficient and accurate self-service vision testing is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of vision testing, and particularly to a self-service vision testing method and system based on infrared ranging and gesture recognition. Background Art

[0002] Vision testing is a very important part of ophthalmic medical treatment. With the development of technology, traditional manual vision testing methods have gradually been replaced by intelligent devices. These devices can improve the detection accuracy, efficiency, and provide a more convenient experience for the tested person through modern technology. However, there are still many limitations in the current vision testing devices on the market, such as high equipment cost, complex operation, and the need for professional assistance. Therefore, in daily life scenarios such as at home and in the community, the tested person cannot conveniently and economically conduct self-vision testing. In order to meet the needs of people's daily vision health management, there is an urgent need for a convenient, accurate, and self-service intelligent vision testing system. Summary of the Invention

[0003] In view of the above problems existing in the prior art, the present invention provides a self-service vision testing method and system based on infrared ranging and gesture recognition. The purpose is to break the dependence of traditional devices on a fixed testing distance by introducing an infrared ranging module, split the short reaction actions of the tested person based on the continuity of the actions, and thus extract a small number of key image frames for real-time processing, realizing real-time, efficient, fully automated and self-service vision testing.

[0004] To achieve the above purpose, the present invention provides the following technical solutions:

[0005] In the first aspect, the present invention provides a self-service vision testing method based on infrared ranging and gesture recognition, including:

[0006] Obtaining the testing distance between the tested person and the testing device by using the infrared ranging method;

[0007] Calculating the size of the visual target based on the testing distance, and generating a random visual target of the corresponding size through the testing device;

[0008] Collecting a sequence of gesture images of the tested person observing the visual target;

[0009] Segmenting the sequence of gesture images based on the continuity of the actions, and extracting the image to be recognized, where the image to be recognized is the direction discrimination result of the tested person for the visual target;

[0010] Obtaining the vision testing result based on the image to be recognized.

[0011] Preferably, the obtaining the testing distance between the tested person and the testing device by using the infrared ranging method includes:

[0012] Convert the voltage analog signal collected by the infrared ranging sensor into a digital quantity;

[0013] According to the following formula

[0014]

[0015] Calculate the test distance; where D is the test distance and V o is the digital quantity of the voltage analog signal collected by the infrared ranging device.

[0016] Preferably, calculating the size of the visual target based on the test distance includes:

[0017] Calculate the minimum pixel length based on the side length and the number of pixel points of the test device: where dx is the minimum pixel length, x is the side length of the test device, and m is the number of pixel points of the test device;

[0018] Calculate the side length of the visual target based on the standard visual acuity value level and the test distance:

[0019]

[0020] where H is the side length of the visual target and L is the visual acuity value level in the 5-point visual acuity recording method;

[0021] Convert the size of the visual target into the number of pixel points:

[0022]

[0023] where count is the number of pixel points into which the side length of the visual target is converted, H is the side length of the visual target, and dx is the minimum pixel length.

[0024] Preferably, generating a random visual target of a corresponding size by the test device includes:

[0025] Set the analog input pin on the main control of the test device to the floating input mode;

[0026] Real-time collect the data value of the analog input pin and convert it into a random number from 0 to 3;

[0027] Correspond the random numbers from 0 to 3 to the visual targets in the up, down, left, and right four directions respectively, and the test device fills in the corresponding number of pixel points to generate a random visual target.

[0028] Preferably, extracting the gesture image sequence according to the target image sequence includes:

[0029] Record any frame image in the target image sequence as the image to be processed;

[0030] For any of the to-be-processed images, obtain an initial hand contour based on OTSU threshold segmentation and morphological operations;

[0031] Perform multi-level pixel binary classification on the to-be-processed image based on different pixel point detection methods to obtain the binary classification results of each level for each pixel point in the to-be-processed image; the different pixel point detection methods include multiple different levels of pixel point detection methods, and the pixel point range for the action of the next-level pixel point detection method is determined based on the binary classification results of the upper-level pixel point detection method; the binary classification result is the probability that the pixel point belongs to any one of the two categories in the binary classification;

[0032] Calculate the target classification result of each pixel point in the to-be-processed image according to the binary classification results of each level of the pixel points;

[0033] Verify each pixel point on the initial hand contour based on the target classification result to determine the final hand contour;

[0034] Extract a gesture image sequence based on the final hand contour in each frame of the to-be-processed image in the target image sequence.

[0035] Preferably, the different pixel point detection methods are edge detection, skin color detection, and moving point detection in sequence.

[0036] Preferably, segmenting the gesture image sequence based on the continuity of the action and extracting the to-be-recognized image includes:

[0037] Extract the hand key points of each frame image in the gesture image sequence;

[0038] Perform key point matching on the hand key points of two adjacent frames of images in time, and obtain a set of change vectors of the two adjacent frames of images based on the spatial position differences of a set of matched key points; the change vector includes an angle and a distance;

[0039] Calculate the mean value of all the change vectors of the two adjacent frames of images, and denote it as the change difference vector of the two adjacent frames of images;

[0040] Connect the endpoints of the change difference vectors of any two adjacent frames of images in chronological order;

[0041] If there is no inflection point in the connection line of the endpoints of the change difference vectors of the (n - 1)-th frame image, the n-th frame image, and the (n + 1)-th frame image, then arrange the (n - 1)-th frame image, the n-th frame image, and the (n + 1)-th frame image in chronological order and denote them as a local action group;

[0042] Wherein the inflection point means that both the angle difference and the distance difference of the change difference vectors at the connection of the endpoints of two adjacent frames of images are greater than a preset threshold;

[0043] Extract the last frame image of each of the local action groups to obtain a number of images to be recognized, forming a group of images to be recognized.

[0044] Preferably, the method further includes:

[0045] When the end frame gesture of any local action group m i can be obtained by performing an integer multiple transformation of rotating 90° on the initial frame gesture of the local action group m i then delete the image to be recognized obtained from the previous local action group m i of the local action group m from the group of images to be recognized. i-1

[0046] Preferably, obtaining the visual acuity test result based on the image to be recognized includes: using the deep residual network model ResNet-18 to recognize the gestures in the image to be recognized;

[0047] The ResNet-18 model has 18 weighted layers, including an input convolutional layer, four residual modules, a global average pooling layer, and a fully connected layer; the input layer receives a 224x224 RGB image, and after passing through the first 7x7 convolutional layer and 3x3 max pooling layer, it enters the residual module; each residual module is composed of two residual blocks, and each residual block contains two 3x3 convolutional layers. After each convolution, batch normalization and ReLU activation functions are connected, and a skip connection is added to directly connect the input to the output.

[0048] In a second aspect, the present invention provides a self-service visual acuity test system based on infrared ranging and gesture recognition, including:

[0049] An infrared ranging module, configured to obtain the test distance between the subject and the test device by using an infrared ranging method;

[0050] A visual target generation module, configured to calculate the size of the visual target based on the test distance, and generate a random visual target of the corresponding size through the test device;

[0051] A gesture acquisition module, configured to collect a sequence of target images when the subject observes the visual target, and extract a sequence of gesture images according to the sequence of target images;

[0052] An image screening module, configured to segment the sequence of gesture images based on the continuity of the actions, and extract the image to be recognized, where the image to be recognized is the direction discrimination result made by the subject for the visual target;

[0053] A result analysis module, configured to obtain the visual acuity test result based on the image to be recognized.

[0054] A self-service vision testing method and system based on infrared ranging and gesture recognition according to the present invention has the following beneficial effects:

[0055] Through the infrared ranging technology, the present invention realizes dynamic adjustment of the size of the visual target, can adjust the size and display mode of the visual target according to the actual distance between the tested person and the device, and ensures the detection accuracy in different environments. At the same time, before combining the gesture recognition technology, the system first screens the images processed by the model, only selects the real response actions of the tested person to the visual target and filters out repeated or irrelevant actions, greatly reducing the amount of real-time data processing for gesture recognition and realizing lightweight self-service vision testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a schematic flow chart of a self-service vision testing method based on infrared ranging and gesture recognition according to the present invention;

[0057] Figure 2 is the algorithm flow of the vision detection device provided in an embodiment of the present invention;

[0058] Figure 3 is a schematic diagram of the loss rate and accuracy rate during the training and verification of the ResNet-18 gesture recognition model adopted in an embodiment of the present invention;

[0059] Figure 4 is a structural block diagram of a self-service vision testing system based on infrared ranging and gesture recognition according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0060] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without making creative efforts fall within the scope of protection of the present invention.

[0061] For the convenience of understanding this embodiment, first, a simple introduction to the application scenario of a self-service vision testing method based on infrared ranging and gesture recognition disclosed in the embodiments of the present invention is given:

[0062] Traditional vision testing devices often require a fixed testing distance, are cumbersome to operate, and difficult to adapt to different testing environments. To solve these problems, the present invention proposes a single-person self-service vision testing device based on the STM32F103ZET6 single-chip microcomputer. The STM32F103ZET6 single-chip microcomputer is used to form the main control unit of the testing device. This device integrates multiple functions such as liquid crystal display, vision evaluation, infrared ranging, and visual target display. It can not only dynamically adjust the size of the visual target according to the actual ranging result, but also introduce gesture recognition technology, enabling the test taker to independently conduct vision tests at any distance.

[0063] The specific process of the test start is as follows: The test taker presses the start button, the device starts to be powered on, and self-check is performed. The self-check process includes detecting the connection status and functional status of each module to ensure that all parts of the device can operate normally. After the self-check is completed, the liquid crystal display screen will display a welcome interface, greeting the test taker and guiding the test taker to perform the next operation.

[0064] Next, a detailed introduction to a self-service vision testing method based on infrared ranging and gesture recognition disclosed in the embodiments of the present invention will be given.

[0065] As Figure 1 、 Figure 2 shown, the method includes:

[0066] S1, obtaining the test distance between the test taker and the testing device by using the infrared ranging method.

[0067] Through the infrared ranging module, the present invention can measure the distance between the test taker and the device in real time and automatically adjust the size of the visual target to ensure the accuracy of the vision test. Whether the test taker is at the standard distance of 5 meters or other custom distances from the device, the device can dynamically adjust the display size of the visual target according to the actual distance, thus meeting the test requirements in various environments. The accuracy of the test is ensured, so that the test results of the test taker are consistent and comparable regardless of the distance at which the test is conducted.

[0068] Traditional vision testing devices usually require vision testing to be carried out at a fixed distance (such as 5 meters or 6 meters), and cannot flexibly adjust the test distance. This limitation not only affects the convenience and flexibility of the device, but also cannot meet the personalized needs of different test takers. For example, the home environment may not be able to provide a fixed test distance, and the needs of different test takers (such as children, the elderly, etc.) are diverse.

[0069] Specifically, obtaining the test distance between the test taker and the testing device by using the infrared ranging method may include:

[0070] Converting the voltage analog signal collected by the infrared ranging sensor into a digital quantity;

[0071] The infrared ranging module consists of a red light-emitting diode and a signal processing circuit. Its working principle is that the infrared sensor emits an infrared beam and receives the reflected light signal, and calculates the distance between the subject and the device based on the triangulation principle.

[0072] The infrared detection module converts the analog signal detected by the infrared sensor into a digital quantity, and the digital quantity is processed in the single-chip microcomputer of the main control unit to obtain the analog value output by the sensor.

[0073] Then, according to the following formula

[0074]

[0075] calculate the test distance; where D is the test distance, and V o is the digital quantity of the voltage analog signal collected by the infrared ranging device.

[0076] S2. Calculate the size of the visual target based on the test distance, and generate a random visual target of the corresponding size through the test device.

[0077] Specifically, calculate the minimum pixel point length based on the side length and the number of pixel points of the test device: where dx is the minimum pixel point length, x is the side length of the test device, and m is the number of pixel points of the test device;

[0078] Calculate the side length of the visual target based on the standard visual acuity value level and the test distance:

[0079]

[0080] where H is the side length of the visual target, and L is the visual acuity value level in the 5-point visual acuity recording method;

[0081] Convert the size of the visual target into the number of pixel points:

[0082]

[0083] where count is the number of pixel points obtained by converting the side length of the visual target, H is the side length of the visual target, and dx is the minimum pixel point length.

[0084] After calculating the number of pixel points, after the liquid crystal display module of the self-service test device fills the corresponding number of pixel points in the horizontal and vertical directions, the screen will generate a black square with no gaps but the same size.

[0085] In order to create the void part of the visual target so that the human eye can distinguish the display direction of the visual target, according to the type of visual target (i.e., the different openings of the "E", including four directions: up, down, left, and right), this module divides different drawing ranges according to different random numbers (i.e., 0 - 3). If the current pixel point is within the drawing range, it is skipped; if the current pixel point is not within the range, it is drawn as black.

[0086] For random number generation, the pins on the main control unit are selected as analog input pins, and the input is set to floating. In its floating input mode, the state is unstable when there is no specific external input, and it is greatly affected by electromagnetic interference, and is used for random number generation. Continuously collect the data values of the floating pins, and use the converted digital quantity as the generated random number. Establish a one-to-one correspondence between various types of visual targets and numbers for the randomly generated numbers to generate four different random visual targets.

[0087] S3. Collect the target image sequence when the subject observes the visual target, and extract the gesture image sequence according to the target image sequence.

[0088] After the subject presses the start button, the testing device will start to display visual targets in random directions. The directions of the visual targets (such as up, down, left, right) are generated by a random algorithm to prevent users from affecting the test results by memorizing the order of the visual targets. The camera captures the actions of the subject at a preset sampling frequency to form a target image sequence, that is, the image acquisition range of the target image sequence includes the whole body of the subject and the background. It is necessary to extract the corresponding directional gestures (pointing up, down, left, right) made by the subject in the direction of observing the visual target, that is, the gesture image sequence. In this embodiment, the discrimination time for one visual target is 3 s. When the discrimination time exceeds 3 s, even if it is correct, it is ignored. The gesture recognition module uses image recognition technology to capture and recognize the gesture actions of the user in real time, and transmits the recognition results to the main control unit for processing.

[0089] Preferably, the extracting the gesture image sequence according to the target image sequence includes:

[0090] Record any frame image in the target image sequence as the image to be processed;

[0091] For any of the images to be processed, obtain the initial hand contour based on OTSU threshold segmentation and morphological operations;

[0092] OTSU is a commonly used algorithm in threshold segmentation. It can automatically generate the optimal segmentation threshold according to the image, and its core idea is to maximize the between-class variance.

[0093] Morphological operations include morphological opening and closing operations, which refer to the process of first dilation and then erosion. It is used to fill small holes in the object, connect adjacent objects, and smooth the boundary without significantly changing its area. Through the above steps, a connected and relatively clear contour can be obtained.

[0094] Perform multi-level pixel point binary classification on the image to be processed based on different pixel point detection methods, and obtain the binary classification results of each level of each pixel point in the image to be processed; the different pixel point detection methods include various levels of pixel point detection methods.

[0095] Preferably, the different pixel point detection methods are edge detection, skin color detection, and moving point detection in sequence.

[0096] In this step, pixel point binary classification refers to dividing pixel points into two categories and calculating the probability that a pixel point belongs to one of the categories. The two parts corresponding to different levels of binary classification are different.

[0097] Edge detection determines the probability that a pixel point belongs to an edge point; skin color detection determines the human body area; since the actions during vision detection are basically generated by the hands, the hands are in a moving state, and moving point detection determines moving points and non-moving points by the change of the same pixel point among multiple frames of images.

[0098] The above three detection methods can all adopt conventional technical solutions, and the present application does not limit this. For example, skin color detection may include the following steps: converting the RGB image to the YCrCb color space and extracting the Cr component image; performing Gaussian filtering on the Cr component; performing self-binarization threshold segmentation on the filtered image for the Cr component to obtain the human skin color segmentation threshold.

[0099] Determine the pixel point range for the next-level pixel point detection method based on the binary classification result of the upper-level pixel point detection method; the binary classification result is the probability that a pixel point belongs to any one of the binary classifications.

[0100] For example, in edge detection, the binary classification result can be the probability that the pixel point belongs to an edge point; in skin color detection, the binary classification result can be the probability that the pixel point meets the Cr component threshold...

[0101] Among them, "determine the pixel point range for the next-level pixel point detection method based on the binary classification result of the upper-level pixel point detection method" specifically means, for example, according to the probability that a pixel point belongs to an edge point determined by edge detection, perform skin color detection on the pixel points with a relatively high probability of belonging to edge points, and then perform moving point detection on the pixel points with a relatively high probability of belonging to human skin.

[0102] Calculate the target classification result of each pixel point in the to-be-processed image according to the binary classification result of each level of the pixel point.

[0103] Each level of detection feature corresponds to a feature of the hand, and the latter-level feature is detected based on the former-level feature. Therefore, the probabilities obtained from the binary classification results of each level can be multiplied as the target classification result.

[0104] Verify each pixel point on the initial hand contour based on the target classification result to determine the final hand contour.

[0105] In a specific implementation manner, the pixel points with a target classification result greater than a preset probability are retained, and the pixel points with a target classification result less than the preset probability are replaced with other pixel points on the periphery of the current contour connection line that have a target classification result greater than the preset probability, so that the curve is coherent, and the final hand contour is obtained.

[0106] Extract a gesture image sequence based on the final hand contour in each frame of the to-be-processed image in the target image sequence.

[0107] Specifically, the minimum bounding rectangle of each final hand contour in the target image sequence can be obtained, and the final hand contour is cut out from each to-be-processed image according to the minimum bounding rectangle, and arranged in the time sequence of the corresponding original target image sequence to obtain the gesture image sequence.

[0108] The present invention first extracts an initial hand contour based on conventional threshold segmentation. Considering the influence of factors such as illumination and background, simple threshold segmentation may not be accurate enough. After morphological processing of the initial hand contour, multi-level binary classification of each pixel point is performed, and hierarchical detection is carried out according to the hand region characteristics respectively. Each level of detection corresponds to a characteristic of the hand. The possibility that the pixel point feature belongs to the hand feature is calculated at each level of detection, and the joint probability that the pixel point belongs to the hand contour under the next-level feature is further calculated under the probability that the previous-level feature conforms to the hand feature, so that the extraction of the hand contour has high anti-interference ability, realizes accurate extraction of the hand image, and lays a foundation for the extraction of the next hand key points.

[0109] Vision testing devices for household use are mostly resource-constrained. Therefore, in order to achieve real-time applications with low computational cost, the present invention splits all the responses of the testee within the time after the visual target appears. That is:

[0110] S4. Segment the gesture image sequence based on the continuity of the action, and extract the to-be-recognized image, where the to-be-recognized image is the direction discrimination result made by the testee for the visual target.

[0111] The present invention takes into account the low-computation-resource scheme required by vision testing devices for household or community use, and thus further reduces the amount of data to be processed.

[0112] Because the gestures for vision testing are fixed, only one result is needed for one test, and it needs to be maintained for a period of time. Therefore, only the end of the action needs to be concerned about whether it is a pointing gesture. The irrelevant or repeated gestures in the middle do not contribute much to whether the testee's gesture is correct.

[0113] As a preferred implementation manner, the segmenting the gesture image sequence based on the continuity of the action and extracting the to-be-recognized image includes:

[0114] Extract the hand key points of each frame image in the gesture image sequence.

[0115] The gesture image sequence is a series of hand movements made by the subject's hand after the visual target appears and the subject starts to observe the visual target. It may be pointing in the direction of the visual target or hand movements unrelated to the directivity of the visual target.

[0116] For example, a 21 - hand - key - point modeling method commonly used in the prior art can be adopted, that is, the Hands model of MediaPipe, to extract the hand key points of each frame image in the gesture image sequence according to the above - mentioned final hand contour.

[0117] To improve the real - time recognition efficiency of gestures, in continuous frame images, a complete action such as raising the hand will be divided into multiple frames. The present invention simplifies gesture recognition effectively and realizes real - time response by extracting the key points in the images of a group of the same actions and only extracting the last frame, that is, the image when the action is completed.

[0118] Perform key - point matching on the hand key points of two adjacent frames in time sequence, and obtain a group of change vectors of the two adjacent frames based on the spatial position differences of a group of matching key - point pairs; the change vectors include angles and distances.

[0119] By comparing the changes between adjacent frames to track the continuity of the action, if the action is continuous, the differences between adjacent multiple frames of images should change gradually.

[0120] Fix a key point in the previous - frame image and find its matching point in the subsequent - frame image to track the dynamic process. Among them, the vector includes magnitude and direction, that is, the distance change between two points in adjacent frames and the angle change in adjacent frames, which reflects the spatial position differences between the two key points in a matching key - point pair. For example, the difference between two vectors can be calculated to obtain a group of change vectors.

[0121] Calculate the mean value of all the change vectors of the two adjacent frames of images, and denote it as the change - difference vector of the two adjacent frames of images.

[0122] This step represents the overall change difference of a frame of image with the mean value. The direction quantifies the change angle of the action in adjacent frames, and the magnitude quantifies the moving distance of the action in adjacent frames, which overall reflects the change characteristics and trends of a local action.

[0123] Connect the endpoints of the change - difference vectors of any two adjacent frames of images in time sequence.

[0124] This step connects the endpoints of the change - difference vectors in space in time sequence, and determines whether the action is continuous by the smoothness of the connection line, that is:

[0125] If there is no inflection point in the connecting line of the endpoints of the change difference vectors of the (n - 1)-th frame image, the n-th frame image, and the (n + 1)-th frame image, then arrange the (n - 1)-th frame image, the n-th frame image, and the (n + 1)-th frame image in chronological order and denote them as a local action group; n represents the n-th gesture image sequence.

[0126] Wherein the inflection point means that both the angle difference and the distance difference of the change difference vectors at the connection of the endpoints of two adjacent frames are greater than a preset threshold.

[0127] Connect the endpoints of the change difference vectors of adjacent frames. If the change in the action amplitude is large, that is, the angle or distance of the change difference vector changes significantly, then obvious inflection points will appear in the two adjacent line segments, and the included angle is significantly greater than the preset threshold, indicating that the hand key points in the adjacent frames have moved significantly, which may be a new action trend.

[0128] Extract the last frame image of each local action group to obtain a number of images to be recognized, forming a group of images to be recognized.

[0129] It should be noted that if the subject does not change the action, that is, the key points of adjacent frames do not move, then only the last frame within the time period is extracted to represent the overall action.

[0130] Based on the continuity of the action, the present invention segments the actions of the subject within a short reaction time after the visual target appears. Considering that the reaction actions required for vision testing are short-term static actions, only the tail frame of a segment of action trend is extracted as the gesture to be recognized in the next step, which can effectively remove the irrelevant reaction actions or repeated actions of the subject, reduce the real-time processing volume of the model, and improve the timeliness of the analysis of vision test results.

[0131] In some cases, for example, the subject makes a pointing gesture immediately after the visual target appears, but changes the gesture direction midway. For example, the subject makes an upward pointing action reflexively for the first time, but then realizes that the action is incorrect and changes it to a rightward pointing action. Only the subject's last pointing action is used as the reaction result within the short reaction time of the vision test. Therefore, the method further includes:

[0132] When the end-frame gesture of any local action group m i can be obtained by rotating the initial-frame gesture of the local action group m i by an integer multiple of 90°, then delete the recognized image obtained from the previous local action group m i of the local action group m i-1 from the group of images to be recognized.

[0133] By comparing the similarity between the overall hand key points in the last frame and the overall hand key points in the initial frame after the rotation direction, the higher the similarity, the more it indicates that only the pointing direction has been changed, that is, the subject made a corrective action after a period of reaction to the visual target and timely changed the reaction result. The subsequent frame image should be used as the image to be recognized.

[0134] After effectively extracting the reaction actions, gesture recognition is performed on the image to be recognized to analyze the test results of the subject.

[0135] S5. Obtain the visual acuity test result based on the image to be recognized.

[0136] Specifically, in some embodiments, the Deep Residual Network model ResNet-18 can be used to recognize the gestures in the image to be recognized.

[0137] ResNet is a Deep Residual Network. ResNet introduces Batch Normalization to accelerate training and improve accuracy. By normalizing the input data in each batch, the network becomes more robust and can handle more complex data. The depth structure of the ResNet model with more than 1000 layers makes it perform excellently in tasks such as image classification and object detection, and it has become one of the very important models in the field of deep learning.

[0138] The ResNet-18 model has a total of 18 weighted layers, including an input convolutional layer, four residual modules, a global average pooling layer, and a fully connected layer; the input layer receives RGB images of 224x224. After passing through the first 7x7 convolutional layer and 3x3 max pooling layer, it enters the residual module; each residual module is composed of two residual blocks, and each residual block contains two 3x3 convolutional layers. After each convolution, batch normalization and ReLU activation functions are connected, and a skip connection is added to directly connect the input to the output.

[0139] The residual structure is better optimized than the traditional structure. Building traditional deep networks will cause problems such as gradient disappearance, gradient explosion, and information loss, while the residual structure can well avoid such problems. The residual block can skip several layers to achieve end-to-end connection. During this process, some intermediate layers will be skipped, so the training effect of the intermediate layer will not affect the training effect of other layers.

[0140] This residual structure enables the network to learn the change amount (residual) of each layer, effectively preventing the problem of gradient disappearance. The last fully connected layer is modified to output 4 categories for gesture classification. During the training process, the model uses the cross-entropy loss function, SGD optimizer, and learning rate scheduler. The loss and accuracy of the training and validation sets are recorded for each epoch, and the best model is saved.

[0141] Figure 3 Shows the loss and accuracy during the training and validation processes of the gesture recognition model.

[0142] As Figure 3 In the left figure, for the training and validation losses, both the training loss (Train Loss) and the validation loss (Val Loss) decrease as the number of training epochs increases, and gradually stabilize after 5 to 10 epochs. The validation loss has dropped to a relatively low level after 5 epochs and remains stable in subsequent epochs, slightly lower than the training loss, indicating that the model performs well on the validation set. The trends of the training loss and the validation loss are consistent, and the gap between them is small, indicating that the model does not show obvious overfitting or underfitting.

[0143] As Figure 3 In the right figure, for the training and validation accuracies, the training accuracy (Train Acc) and the validation accuracy (Val Acc) increase rapidly in the initial stage, reach over 90% around 5 epochs, and gradually approach 1.0. The validation accuracy remains above 0.9 after 7 epochs, slightly higher than the training accuracy, which may be due to the slightly different sample characteristics of the validation set from the training set, or the validation set itself being relatively simple. The training accuracy and the validation accuracy are close and stable, indicating that the model can effectively classify the data without obvious overfitting.

[0144] Overall, the model performs very well on both the training set and the validation set, with the accuracy reaching close to 1.0 and the loss remaining at a low level.

[0145] There is no phenomenon of loss or accuracy oscillation during the training process, and the model parameter settings and training strategies are reasonable.

[0146] Compare the input gesture result in the image to be recognized of the subject output by the model with the actual direction of the visual target to judge the vision result, and the result is displayed through the liquid crystal display module to provide a detailed vision assessment report. This vision chart uses the 5-point vision recording method as the vision value determination. The criteria for whether the subject can "recognize" are as follows:

[0147] 1. Answering correctly four times in a row with an interval not exceeding 3s in the middle indicates "recognized".

[0148] 2. When answering incorrectly twice in a row, it indicates "not recognized".

[0149] 3. When the recognition time exceeds 3s, even if it is correct, it is ignored.

[0150] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are sequentially shown according to the indications of the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0151] Based on the same inventive concept, an embodiment of the present application further provides a system for implementing the above-mentioned self-service vision test method based on infrared ranging and gesture recognition. The implementation solution provided by this system to solve problems is similar to the implementation solution described in the above method. Therefore, the specific limitations in the embodiments of the self-service vision test system provided below can refer to the limitations on this method in the above text, and will not be repeated here.

[0152] As Figure 4 shown, the present invention also provides a self-service vision test system based on infrared ranging and gesture recognition, including:

[0153] An infrared ranging module, configured to obtain a test distance between a subject and a test device by using an infrared ranging method;

[0154] A visual target generation module, configured to calculate the size of a visual target based on the test distance and generate a random visual target of a corresponding size through a test device;

[0155] A gesture acquisition module, configured to acquire a sequence of gesture images of a subject observing a visual target;

[0156] An image screening module, configured to segment the sequence of gesture images based on the continuity of actions and extract an image to be recognized, where the image to be recognized is the direction discrimination result of the subject on the visual target;

[0157] A result analysis module, configured to obtain a vision test result based on the image to be recognized.

[0158] It should be noted that: when the self-service vision testing system provided in this embodiment processes vision detection, only the above-mentioned division of each functional module is used as an example for illustration. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the system is divided into different functional modules. Each of these functional modules can be composed of a single execution unit, or can be integrated by two or more execution units into a functional module to achieve all the functions of the functional module.

[0159] Those skilled in the art can understand that all or part of the above-mentioned modules can be implemented through software, hardware, and their combinations. The above-mentioned modules can be embedded in the processor of the computer device in hardware form or independent of it, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

[0160] An embodiment of the present invention also provides an electronic device, including:

[0161] A memory for storing executable instructions;

[0162] A processor, when running the executable instructions stored in the memory, implements a self-service vision testing method based on infrared ranging and gesture recognition as described above. The processor can be a central processing unit, other general-purpose processors, digital signal processors, application-specific integrated circuits, field-programmable gate arrays, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. chips, or a combination of the above types of chips.

[0163] The specific details of the above computer device during self-service vision testing can be understood by referring to the relevant descriptions and effects corresponding to the above method, and will not be elaborated here.

[0164] Another embodiment of the present invention also provides a computer-readable storage medium storing executable instructions, and when the executable instructions are executed by a processor, a self-service vision testing method based on infrared ranging and gesture recognition as described above is implemented.

[0165] The storage medium can be various computer-readable storage media such as a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disc that can store program codes.

[0166] The present invention is not limited to the above specific implementation manners. Those of ordinary skill in the art starting from the above concepts and making various transformations without creative labor fall within the protection scope of the present invention.

Claims

1. A self-service vision test method based on infrared ranging and gesture recognition, characterized in that, The method includes the following steps: Obtain the test distance between the subject and the test device by using the infrared ranging method; Calculate the size of the visual target based on the test distance, and generate a random visual target of the corresponding size through the test device; Collect the target image sequence when the subject observes the visual target, and extract the gesture image sequence according to the target image sequence; Segment the gesture image sequence based on the continuity of the action, and extract the image to be recognized; the image to be recognized is the direction discrimination result made by the subject for the visual target; Obtain the visual acuity test result based on the image to be recognized.

2. The self-service vision testing method based on infrared ranging and gesture recognition according to claim 1, wherein The step of obtaining the test distance between the subject and the test device by using the infrared ranging method includes: Convert the voltage analog signal collected by the infrared ranging sensor into a digital quantity; According to the following formula Calculate the test distance; where D is the test distance and V o is the digital quantity of the voltage analog signal collected by the infrared ranging device.

3. The self-service vision testing method based on infrared ranging and gesture recognition according to claim 2, wherein The step of calculating the size of the visual target based on the test distance includes: Calculate the minimum pixel length based on the side length and the number of pixels of the test device: Where dx is the minimum pixel length, x is the side length of the test device, and m is the number of pixels of the test device; Calculate the side length of the visual target based on the standard visual acuity value level and the test distance: Wherein, H is the side length of the visual target, and L is the visual acuity value level in the 5-point visual acuity recording method; Convert the size of the visual target into the number of pixel points: Wherein, count is the number of pixel points obtained by converting the side length of the visual target, H is the side length of the visual target, and dx is the length of the smallest pixel point.

4. The self-service vision testing method based on infrared ranging and gesture recognition according to claim 3, characterized in that, The step of generating a random visual target of the corresponding size through the test device includes: Set the analog input pin on the main control of the test device to the floating input mode; Collect the data value of the analog input pin in real time and convert it into a random number from 0 to 3; Correspond the random numbers from 0 to 3 to the visual targets in the up, down, left, and right four directions respectively, and the test device fills the corresponding number of pixel points to generate a random visual target.

5. The self-service vision testing method based on infrared ranging and gesture recognition according to claim 1, wherein The step of extracting the gesture image sequence according to the target image sequence includes: Record any frame image in the target image sequence as the image to be processed; For any of the images to be processed, obtain the initial hand contour based on OTSU threshold segmentation and morphological operations; Perform multi-level pixel point binary classification on the image to be processed based on different pixel point detection methods, and obtain the binary classification result of each level of each pixel point in the image to be processed; the different pixel point detection methods include various pixel point detection methods at different levels and determine the pixel point range for the next-level pixel point detection method based on the binary classification result of the previous-level pixel point detection method; the binary classification result is the probability that the pixel point belongs to any one of the two classifications; Calculate the target classification result of each pixel point in the image to be processed according to the binary classification result of each level of the pixel point; Verify each pixel point on the initial hand contour based on the target classification result to determine the final hand contour; Extract the gesture image sequence based on the final hand contour in each frame of the image to be processed in the target image sequence.

6. The self-service vision testing method based on infrared ranging and gesture recognition according to claim 5, characterized in that, The different pixel point detection methods are edge detection, skin color detection, and moving point detection in sequence.

7. The self-service vision testing method based on infrared ranging and gesture recognition according to claim 6, wherein The step of segmenting the gesture image sequence based on the continuity of the action and extracting the image to be recognized includes: Extract the hand key points of each frame image in the gesture image sequence; Perform key point matching on the hand key points of two adjacent frames in time, and obtain a set of change vectors of the two adjacent frames of images based on the spatial position difference of a set of matching key point pairs; the change vector includes an angle and a distance; Calculate the mean value of all change vectors of the two adjacent frames of images, and denote it as the change difference vector of the two adjacent frames of images; Connect the endpoints of the change difference vectors of any two adjacent frames of images in chronological order; If there is no inflection point in the connection line of the endpoints of the change difference vectors of the (n - 1)-th frame image, the n-th frame image, and the (n + 1)-th frame image, then arrange the (n - 1)-th frame image, the n-th frame image, and the (n + 1)-th frame image in chronological order and denote them as a local action group; Wherein the inflection point means that both the angle difference and the distance difference of the change difference vectors at the connection of the endpoints of two adjacent frames of images are greater than a preset threshold; Extract the last frame image of each local action group to obtain a number of images to be recognized, and form a group of images to be recognized.

8. The self-service vision test method based on infrared ranging and gesture recognition according to claim 7, characterized in that, The method further includes: When any local action group m i 's end-frame gesture can be obtained by transforming the initial-frame gesture of this local action group m i through an integer multiple of 90° rotation, then the previous local action group m i of this local action group m i-1 The to-be-recognized image obtained is deleted from the to-be-recognized image group.

9. The self-service vision testing method based on infrared ranging and gesture recognition according to claim 1, characterized in that, The obtaining the vision test result based on the image to be recognized includes: using the deep residual network model ResNet-18 to recognize the gestures in the image to be recognized; The ResNet-18 model has 18 weighted layers, including an input convolutional layer, four residual modules, a global average pooling layer, and a fully connected layer; the input layer receives RGB images of 224x224, and enters the residual module after passing through the first 7x7 convolutional layer and the 3x3 max pooling layer; each residual module is composed of two residual blocks, each residual block contains two 3x3 convolutional layers, and after each convolution, batch normalization and ReLU activation function are connected, and a skip connection is added to directly connect the input to the output.

10. A self-service vision testing system based on infrared ranging and gesture recognition, characterized in that, Including: An infrared ranging module, configured to obtain a test distance between a subject and a test device by using an infrared ranging method; A visual target generation module, configured to calculate the size of a visual target based on the test distance, and generate a random visual target of the corresponding size through the test device; A gesture acquisition module, configured to acquire a target image sequence when the subject observes the visual target, and extract a gesture image sequence according to the target image sequence; An image screening module, configured to segment the gesture image sequence based on the continuity of actions, and extract an image to be recognized, where the image to be recognized is the direction discrimination result made by the subject for the visual target; A result analysis module, configured to obtain a vision test result based on the image to be recognized.

Images