Optical system, test method and equipment for measuring gesture recognition latency

Abstract

The present invention relates to the technical field of human-computer interaction performance evaluation, and discloses an optical system, test method and device for measuring gesture recognition delay. A gesture trigger light signal device is triggered by a gesture action and emits a gesture trigger light signal. At the same time, the head-mounted device recognizes the gesture action and generates content displayed by the display module. Then, the optical sensor arranged on the head-mounted device detects the content displayed by the display module of the head-mounted device and generates a light sensor signal. Then, the high-frequency photodetector receives the gesture trigger light signal and the light sensor signal synchronously transmitted by the synchronous transmission device. Finally, the data processing device detects and calculates the time difference based on the gesture trigger light signal and the light sensor signal received by the high-frequency photodetector to determine the delay time of gesture recognition. The above-mentioned optical system for measuring gesture recognition delay reduces the difficulty of gesture recognition delay testing, improves test efficiency, and measures the end-to-end delay of gesture actions.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction performance evaluation, and in particular to an optical system, a testing method, and a device for measuring gesture recognition delay. Background Art

[0002] Natural human-computer interaction in immersive virtual environments is currently a major concern. A key factor affecting user immersion is the time lag between the user performing a gesture and the system correctly recognizing and responding to it during gesture recognition, known as gesture recognition latency. Related methods for testing gesture recognition latency are constrained by various factors, including hardware, algorithms, and usage scenarios, and a truly universally applicable testing solution has yet to be established. Therefore, an optical system for measuring gesture recognition latency is needed. Summary of the Invention

[0003] The embodiments of this specification aim to at least partially solve one of the technical problems in the related art. To this end, the embodiments of this specification propose an optical system, a testing method, and an apparatus for measuring gesture recognition delay.

[0004] An embodiment of this specification provides an optical system for measuring gesture recognition delay, the system comprising:

[0005] A gesture-triggered light signal device, configured to be triggered by a gesture action and emit a gesture-triggered light signal;

[0006] an optical sensor, disposed on the head-mounted device, for detecting content displayed by a display module of the head-mounted device and generating an optical sensor signal, wherein the displayed content is generated by the head-mounted device recognizing the gesture action;

[0007] a synchronous transmission device for synchronously transmitting the gesture trigger light signal and the light sensor signal to a high-frequency photodetector;

[0008] a high-frequency photodetector, configured to receive the synchronously transmitted gesture trigger light signal and the light sensor signal;

[0009] A data processing device is used to detect and calculate the time difference between the gesture trigger light signal and the light sensor signal received by the high-frequency photodetector to determine the delay time of gesture recognition.

[0010] In one embodiment, the detecting and calculating the time difference between the gesture trigger light signal and the light sensor signal received by the high-frequency photodetector to determine the gesture recognition delay time includes:

[0011] determining a time difference between the first time and the second time based on a first time corresponding to the gesture trigger light signal and a second time corresponding to the light sensor signal;

[0012] Based on the time difference, a delay time for gesture recognition is determined.

[0013] In one embodiment, the gesture trigger light signal and the light sensor signal are fused to form a fused signal, wherein the fused signal represents a change in light brightness over time, and the first moment and the second moment are obtained by:

[0014] Identifying the fusion signal to obtain mutation information of the fusion signal;

[0015] When the mutation information indicates that a mutation value of the fusion signal at a moment is greater than a first preset threshold, determining the moment as the first moment;

[0016] When the mutation information indicates that a mutation value of the fusion signal at a moment is greater than a second preset threshold, the moment is determined as the second moment.

[0017] In one embodiment, the gesture-triggered light signal device includes:

[0018] A signal receiving device, configured to be triggered by a gesture action and generate a control signal;

[0019] A light signal generating device is used to generate and emit the gesture trigger light signal based on the control signal.

[0020] In one embodiment, the synchronous transmission device includes:

[0021] a first optical fiber connected between the gesture trigger light signal device and the high-frequency photoelectric detector, for receiving the gesture trigger light signal emitted by the gesture trigger light signal device and transmitting the gesture trigger light signal to the high-frequency photoelectric detector;

[0022] The second optical fiber is connected between the optical sensor and the high-frequency photodetector, and is used to receive the light sensor signal generated by the optical sensor and transmit the light sensor signal to the high-frequency photodetector.

[0023] In one embodiment, the signal receiving device includes a magnetic contact electrode and a hand-worn device, wherein, by wearing the hand-worn device and performing a gesture on the magnetic contact electrode, the series circuit formed by the hand-worn device, the magnetic contact electrode and the signal generating device is turned on or off; and the control signal is transmitted to the optical signal generating device through the series circuit.

[0024] In one embodiment, the magnetically attracted contact electrode includes a first component and a second component, wherein the first component and the second component are magnetically connected;

[0025] Performing a first gesture on the magnetic contact electrode by wearing the hand-worn device to conduct the series circuit, wherein the first gesture includes the hand-worn device contacting the second component;

[0026] The series circuit is disconnected by wearing the hand-worn device and performing a second gesture on the magnetic contact electrode, wherein the second gesture includes at least one of the hand-worn device leaving the second component and the second component being separated from the first component by the hand-worn device.

[0027] In one embodiment, when the second component is one:

[0028] The first gesture action includes: wearing the hand-worn device and pinching the second component with one hand;

[0029] The second gesture action includes: releasing the second component with one hand while wearing the hand-worn device, or pulling the second component with one hand while wearing the hand-worn device so that the second component is separated from the first component.

[0030] In one embodiment, when there are two second components:

[0031] The first gesture action includes: wearing the hand-worn device and pinching the two second components with both hands;

[0032] The second gesture action includes: wearing the hand-worn device with both hands to release the second component, or wearing the hand-worn device with both hands to pull the two second components so that at least one of the two second components is separated from the first component.

[0033] In one embodiment, when there is one second component, there is one hand-worn device, the first component is connected to the optical signal generating device via a wire, and the hand-worn device is connected to the optical signal generating device via a wire.

[0034] In one embodiment, when there are two second components, the two second components are magnetically connected to the two ends of the first component respectively, and there are two hand-worn devices, one hand-worn device is connected to the optical signal generating device through a wire, and the other hand-worn device is connected to the optical signal generating device through a wire.

[0035] In one embodiment, the gesture-triggered light signal includes at least one of the following:

[0036] Executing the gesture to generate a light signal representing a light change, wherein the light signal generating device includes a lighting device;

[0037] Executing the gesture to generate a light signal representing the shape change, wherein the light signal generating device includes a shape change sensing device;

[0038] The gesture action is performed to generate a light signal representing a position change, wherein the light signal generating device includes a position change sensing device.

[0039] In one embodiment, the gesture includes at least one of the following:

[0040] One-hand pinching gesture, two-hand pinching gesture, one-hand release gesture, one-hand pulling gesture, two-hand pulling gesture, two-hand rotation gesture.

[0041] The present disclosure provides a method for testing gesture recognition delay, the method comprising:

[0042] Receiving a gesture-triggered light signal and a light sensor signal from a high-frequency photodetector, wherein the gesture-triggered light signal is triggered and emitted by a gesture-triggered light signal device based on a gesture action, the light sensor signal is generated by an optical sensor provided on a head-mounted device detecting content displayed by a display module of the head-mounted device, and the displayed content is generated by the head-mounted device recognizing the gesture action, and the gesture-triggered light signal and the light sensor signal are synchronously transmitted to the high-frequency photodetector by a synchronous transmission device;

[0043] For the fusion signal of the gesture trigger light signal and the light sensor signal received from the high-frequency photodetector, mutation detection is performed on the fusion signal to obtain mutation information, and time difference is detected and calculated based on the mutation information to determine the delay time of gesture recognition.

[0044] The present invention provides an electronic device, which includes: a memory, and one or more processors communicatively connected to the memory; the memory stores instructions that can be executed by the one or more processors, and the instructions are executed by the one or more processors to enable the one or more processors to implement the steps of the method described in any of the above embodiments.

[0045] The present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method described in any one of the above embodiments are implemented.

[0046] The present invention provides a computer program product, which includes instructions. When the instructions are executed by a processor of a computer device, the computer device is enabled to perform the steps of the method described in any one of the above embodiments.

[0047] In the optical system for measuring gesture recognition delay described above, a gesture action triggers a gesture-triggering light signal device, which emits a gesture-triggering light signal. Simultaneously, a head-mounted device recognizes the gesture action and generates content displayed by the display module. An optical sensor disposed on the head-mounted device then detects the content displayed by the head-mounted device's display module and generates a light sensor signal. A high-frequency photodetector then receives the gesture-triggering light signal and light sensor signal synchronously transmitted by the synchronous transmission device. Finally, a data processing device detects and calculates the time difference between the gesture-triggering light signal and light sensor signal received by the high-frequency photodetector to determine the gesture recognition delay. In the above-described embodiment, on the one hand, the gesture-triggering light signal device is triggered by a gesture action and emits a gesture-triggering light signal, and the optical sensor disposed on the head-mounted device detects the content displayed by the head-mounted device's display module and generates a light sensor signal, thereby reducing the difficulty of gesture recognition delay testing and improving test efficiency. On the other hand, the gesture-triggered light signal device is triggered by a gesture action and emits a gesture-triggered light signal. Furthermore, an optical sensor provided on the head-mounted device detects the content displayed by the head-mounted device's display module and generates a light sensor signal. The end-to-end delay of the gesture action is measured. The measurement accuracy depends on the sampling frequency of the high-frequency photodetector and can reach the millisecond level. Furthermore, the optical system for measuring gesture recognition delay can test gesture recognition delay for a variety of gesture actions to suit different application scenarios. Furthermore, the synchronous transmission device synchronously transmits the gesture trigger light signal and the light sensor signal to the high-frequency photodetector, avoiding the problem of multi-level time synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1a A schematic diagram of an optical system for measuring gesture recognition latency according to an example scenario of this specification;

[0049] Figure 1b A flowchart of an optical system for measuring gesture recognition latency provided according to an example scenario of this specification;

[0050] Figure 2a A schematic diagram of an optical system for measuring gesture recognition delay according to an embodiment of this specification;

[0051] Figure 2b A schematic diagram of a flow chart for determining the delay time for gesture recognition provided in an embodiment of this specification;

[0052] Figure 2c A schematic diagram of a process for determining a first moment and a second moment provided in an embodiment of this specification;

[0053] Figure 2d A schematic diagram of determining a first moment and a second moment provided in an embodiment of this specification;

[0054] Figure 2e A schematic diagram of a gesture-triggered light signal device provided according to an embodiment of this specification;

[0055] Figure 3 A schematic diagram of a synchronous transmission device provided according to an embodiment of this specification;

[0056] Figure 4 A schematic diagram of a first gesture action provided in an embodiment of this specification when there is only one second component;

[0057] Figure 5 A schematic diagram of a first gesture action provided in an embodiment of this specification when there are two second components;

[0058] Figure 6 A schematic diagram of a gesture action circuit structure in the case where there is only one second component provided in an embodiment of this specification;

[0059] Figure 7 A schematic diagram of a gesture action circuit structure provided in an embodiment of this specification when there are two second components;

[0060] Figure 8a A flowchart of a method for testing gesture recognition delay provided in an embodiment of this specification;

[0061] Figure 8b A schematic diagram of a method for testing the delay time of a gesture performed by one hand according to an embodiment of this specification;

[0062] Figure 8c A schematic diagram of a method for testing the delay time of gestures performed by both hands according to an embodiment of this specification;

[0063] Figure 9 This is a diagram of the internal structure of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0064] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0065] Take XR devices (Extended Reality) as an example. XR devices can create virtual environments, allowing users to interact with real scenes. Gesture recognition uses algorithms to identify user gestures, allowing users to control or interact with devices using simple gestures, enabling computers to understand human behavior.

[0066] When using gesture recognition for human-computer interaction, real-world images are first acquired using the front-facing camera of the XR device. A hand detection network deployed in the XR device then processes each real-world image, identifying the hand within the image and generating a bounding box for the hand region. The real-world image is then cropped using the detected bounding box to produce an image retaining the hand region. This image retaining the hand region is then fed into the hand keypoint network, which augments the input with keypoint features generated from extrapolated hand poses, including estimated hand keypoint positions and other relevant information from previous frames. Extrapolation involves using estimated hand keypoint positions and other relevant information from previous frames to enhance the hand keypoint predictions for the current frame. This approach leverages temporal continuity to improve the accuracy and robustness of hand keypoint recognition. Based on the input image and keypoint features, the hand keypoint network predicts a 2D position heatmap and a 1D relative distance heatmap for 21 hand keypoints. These keypoints are used to fit a template hand model for each hand. Therefore, from the user making a gesture to the XR device recognizing the gesture and making a judgment, a series of complex special processing links are required to achieve human-computer interaction.

[0067] A key factor affecting user immersion is the time lag between the user performing a gesture and the XR device correctly identifying and responding to it during gesture recognition, known as gesture recognition latency. To improve the user experience, an efficient and detailed testing system is required to accurately determine the gesture recognition latency of XR devices, thereby clarifying device performance indicators and providing optimization strategies.

[0068] However, given that XR devices are integrated devices, their gesture recognition latency is constrained by multiple factors, including hardware, algorithms, and usage scenarios. Consequently, a universally applicable testing solution has not yet been developed. Therefore, a universally applicable optical system for measuring gesture recognition latency is needed.

[0069] In related technologies, the device delay time is determined by counting the number of times the subject's gestures are correctly completed and accurately recognized, the number of times they are incorrectly recognized, and the duration of gesture interaction.

[0070] However, related technologies cannot directly test the actual end-to-end delay of each gesture recognition, and the accuracy of the device delay time obtained needs to be improved.

[0071] Based on the above analysis, the embodiments of this specification provide an optical system for measuring gesture recognition delay. A gesture action triggers a gesture trigger light signal device and emits a gesture trigger light signal. Simultaneously, a head-mounted device recognizes the gesture action and generates content displayed by a display module. An optical sensor provided on the head-mounted device then detects the content displayed by the head-mounted device's display module and generates a light sensor signal. A high-frequency photodetector then receives the gesture trigger light signal and light sensor signal synchronously transmitted by the synchronous transmission device. Finally, a data processing device detects and calculates the time difference between the gesture trigger light signal and light sensor signal received by the high-frequency photodetector to determine the gesture recognition delay.

[0072] In the above-described embodiment, on the one hand, by triggering a gesture-triggered light signal device through a gesture action and emitting a gesture-triggered light signal, and by using an optical sensor disposed on a head-mounted device to detect the content displayed by the head-mounted device's display module and generate a light sensor signal, the difficulty of gesture recognition delay testing is reduced and test efficiency is improved. On the other hand, by triggering a gesture-triggered light signal device through a gesture action and emitting a gesture-triggered light signal, and by using an optical sensor disposed on the head-mounted device to detect the content displayed by the head-mounted device's display module and generate a light sensor signal, the end-to-end delay of the gesture action is measured. The measurement accuracy depends on the sampling frequency of the high-frequency photodetector and can reach the millisecond level. Furthermore, the optical system for measuring gesture recognition delay can test gesture recognition delay for a variety of gesture actions to suit different application scenarios. In addition, the synchronous transmission device synchronously transmits the gesture trigger light signal and the light sensor signal to the high-frequency photodetector, avoiding the problem of multi-level time synchronization.

[0073] The optical system for measuring gesture recognition delay provided in the embodiments of this specification may include six modules. Figure 1aThe gesture-triggered optical signal device includes a signal receiving module 1 and an optical signal generating module 2. The signal receiving module 1 may include a bionic glove with electrodes on the fingers and a special circuit module. The optical signal generating module 2 may be an LED light that switches on and off under gesture control. The head-mounted device module 3 may be an XR device under test whose display brightness changes in response to gesture triggering. The synchronous transmission device module 4 may be two optical fibers guiding light. One optical fiber may be a first optical fiber connected between the optical signal generating module 2 and the high-frequency photodetector module 5, and the other optical fiber may be a second optical fiber connected between the head-mounted device module 3 and the high-frequency photodetector module 5. The high-frequency photodetector module 5 may be a high-frame-rate luminance meter, which may be an Asteria luminance meter. It is understood that the device for detecting light brightness changes is not limited to a luminance meter; any other device capable of sensing light brightness changes may also be used. The data processing device module 6 may include a delay calculation program or a device or equipment for executing the delay calculation program. The data processing device module 6 may also display test results.

[0074] The optical system for measuring gesture recognition delay uses the brightness change of the light signal generating device module 2 to measure the true value moment of the gesture action and uses the brightness change of the screen of the head-mounted device module 3 itself to measure the moment when the head-mounted device module 3 recognizes the gesture action, thereby determining the delay time of gesture recognition.

[0075] Specifically, the optical system for measuring gesture recognition latency can include two paths. The signal receiving device module 1 can be a bionic glove with electrodes on the fingers, the optical signal generating device module 2 can be an LED light that switches on and off under gesture control, the head-mounted device module 3 can be the XR device to be tested, the synchronous transmission device module 4 can be a first optical fiber and a second optical fiber, and the high-frequency photodetector module 5 can be a high-frequency luminance meter.

[0076] See also Figure 1b The first path is the gesture truth path, which is used to determine the actual moment when the gesture is executed. The tester puts on a bionic glove containing electrodes in the circuit and performs a gesture. While performing the gesture, the circuit can be closed / opened, thereby controlling the LED in the circuit to turn on / off. According to the circuit principle and high-speed camera photography experiments, the execution of the gesture and the lighting of the LED occur at the same time or the difference between the two moments is negligible. Therefore, the moment when the LED turns on / off can be used to represent the true moment of the gesture. At the same time as the LED turns on / off, the gesture trigger light signal is transmitted to the high-frequency luminance meter through the first optical fiber. The luminance meter detects the change in the brightness of the LED and uses this moment as the true moment of the gesture.

[0077] The second path is the gesture recognition path, which is used to determine the moment when the XR device to be tested recognizes the gesture. The XR device to be tested is installed with a gesture test program that triggers the display brightness change with the gesture. The test program can be developed based on Unity. The test program installed on the XR device to be tested will call the gesture recognition function. Under the action of the gesture recognition function, if the XR device to be tested recognizes the gesture, the test program will give the XR device to be tested a signal, and the screen of the XR device to be tested will change from bright to dark / from dark to bright due to the action of the signal. Similarly, the light sensor signal generated by the screen of the XR device to be tested is transmitted to the high-frequency luminance meter through the second optical fiber. The luminance meter detects the brightness change of the screen of the XR device to be tested and uses this moment as the gesture recognition moment.

[0078] Finally, the delay calculation program in the data processing device module 6 performs calculation based on the detected gesture true value time and gesture recognition time, and uses the difference between the gesture recognition time and the gesture true value time as the delay time of gesture recognition.

[0079] This specification provides an optical system 200 for measuring gesture recognition delay. Figure 2a , the system 200 may include:

[0080] The gesture-triggered light signal device 210 is configured to be triggered by a gesture action and emit a gesture-triggered light signal.

[0081] The optical sensor 220 is provided on the head mounted device 230 and is used to detect the content displayed by the display module 240 of the head mounted device 230 and generate a light sensor signal.

[0082] The synchronous transmission device 250 is used to synchronously transmit the gesture trigger light signal and the light sensor signal to the high-frequency photodetector 260 .

[0083] The high-frequency photodetector 260 is used to receive the synchronously transmitted gesture trigger light signal and light sensor signal.

[0084] The data processing device 270 is used to detect and calculate the time difference between the gesture trigger light signal and the light sensor signal received by the high-frequency photodetector 260 to determine the delay time of gesture recognition.

[0085] The displayed content is generated by the head-mounted device 230 recognizing gestures. Gestures can be interactive actions, such as tapping, zooming, pinching, etc. The head-mounted device can be a device that implements action interaction, such as an XR device, an AR device (Augmented Reality device), or a VR device (Virtual Reality device). The display module can be a component of the head-mounted device, typically consisting of a screen and related control circuits.

[0086] Specifically, when a gesture action is performed, the gesture trigger light signal device 210 is triggered based on the gesture action. After the gesture trigger light signal device 210 is triggered, it determines a gesture trigger light signal generated according to the gesture action and emits the gesture trigger light signal.

[0087] When performing gestures, the headset 230 uses built-in sensors and algorithms to analyze and process information such as hand posture, finger position, and motion trajectory, achieving real-time motion recognition of the gestures. When the headset 230 successfully recognizes a specific gesture, it generates content for display on the headset's display module 240. The optical sensor 220, mounted on the headset 230, then detects the content displayed on the headset's display module 240 and generates a light sensor signal based on the detection results.

[0088] In some embodiments, the brightness of the light is changed by adjusting the brightness of the display module, and the display module generates a light sensor signal. For example, the head-mounted device can be an XR device that needs to be tested, that is, the XR device to be tested, and the display module can be the display screen of the XR device to be tested. The XR device to be tested is installed with a test program that triggers the brightness change of the display screen with gestures, and the test program can be developed based on Unity. The test program will call the gesture recognition function. Under the action of the gesture recognition function, if the XR device to be tested recognizes a gesture action (such as pinching, zooming), the test program will give the XR device to be tested a signal (for example, when a pinch gesture action is recognized, the test program will give signal 1, and when a zoom gesture action is recognized, the test program will give signal 2). Due to the action of the signal, the XR device to be tested will cause the display screen of the XR device to be tested to change from dark to bright, resulting in a change in brightness. If the XR device under test recognizes a gesture (such as a one-hand release gesture or a two-hand release gesture), the test program will send a signal to the XR device under test (for example, the test program will send signal 3 when a one-hand release gesture is recognized, and will send signal 4 when a two-hand release gesture is recognized). Due to the effect of the signal, the display screen of the XR device under test will change from bright to dark, resulting in a brightness change.

[0089] The synchronous transmission device 250 is a signal transmission device used to synchronously transmit the gesture trigger light signal and the light sensor signal to the high-frequency photodetector 260 for processing. The high-frequency photodetector 260 then receives the synchronously transmitted gesture trigger light signal and light sensor signal. The high-frequency photodetector 260 can be a high-frequency luminance meter, such as an Asteria luminance meter. It should be understood that the high-frequency photodetector 260 is not limited to a luminance meter; any other device capable of sensing changes in light brightness can also be used.

[0090] Data processing device 270 detects gesture-triggered light signals and light sensor signals received by high-frequency photodetector 260, capturing changes in the signal's brightness. It then calculates the time difference between the gesture-triggered light signals and light sensor signals to determine the gesture recognition delay.

[0091] It should be noted that data or signal transmission and interaction between different devices can be achieved through wired connections (such as cables) or through wireless communication technologies (such as Wi-Fi, Bluetooth).

[0092] In the optical system for measuring gesture recognition delay, a gesture triggering light signal device is triggered by a gesture action, which emits a gesture triggering light signal. Simultaneously, a head-mounted device recognizes the gesture action and generates content displayed by a display module. An optical sensor mounted on the head-mounted device then detects the content displayed by the head-mounted device's display module and generates a light sensor signal. A high-frequency photodetector then receives the gesture triggering light signal and light sensor signal synchronously transmitted by the synchronous transmission device. Finally, a data processing device detects and calculates the time difference between the gesture triggering light signal and light sensor signal received by the high-frequency photodetector to determine the gesture recognition delay.

[0093] In the above-described embodiment, on the one hand, by triggering a gesture-triggered light signal device through a gesture action and emitting a gesture-triggered light signal, and by using an optical sensor disposed on a head-mounted device to detect the content displayed by the head-mounted device's display module and generate a light sensor signal, the difficulty of gesture recognition delay testing is reduced and test efficiency is improved. On the other hand, by triggering a gesture-triggered light signal device through a gesture action and emitting a gesture-triggered light signal, and by using an optical sensor disposed on the head-mounted device to detect the content displayed by the head-mounted device's display module and generate a light sensor signal, the end-to-end delay of the gesture action is measured. The measurement accuracy depends on the sampling frequency of the high-frequency photodetector and can reach the millisecond level. Furthermore, the optical system for measuring gesture recognition delay can test gesture recognition delay for a variety of gesture actions to suit different application scenarios. In addition, the synchronous transmission device synchronously transmits the gesture trigger light signal and the light sensor signal to the high-frequency photodetector, avoiding the problem of multi-level time synchronization.

[0094] In some embodiments, see Figure 2b , based on the detection and calculation of the time difference between the gesture trigger light signal and the light sensor signal received by the high-frequency photodetector to determine the delay time of gesture recognition, the following steps may be included:

[0095] S202: Determine a time difference between the first time and the second time based on a first time corresponding to the gesture trigger light signal and a second time corresponding to the light sensor signal.

[0096] S204: Determine a delay time for gesture recognition based on the time difference.

[0097] The first moment may be the moment when the high-frequency photoelectric detector receives the gesture trigger light signal, and the second moment may be the moment when the high-frequency photoelectric detector receives the light sensor signal.

[0098] Specifically, a time difference can be determined by subtracting the first moment from the second moment. This time difference represents the time elapsed from the time the gesture triggers the light signal device to generate a brightness change until the head-mounted device generates a brightness change, reflecting the gesture recognition delay. Therefore, the gesture recognition delay can be determined based on the time difference. For example, the time difference can be directly determined as the gesture recognition delay.

[0099] In the above embodiment, based on the first moment corresponding to the gesture trigger light signal and the second moment corresponding to the light sensor signal, the moment difference between the first moment and the second moment is determined, and based on the moment difference, the delay time of gesture recognition is determined, and the end-to-end delay of the gesture action is measured to improve the accuracy of the test.

[0100] In some embodiments, see Figure 2cThe gesture trigger light signal and the light sensor signal are fused to form a fusion signal. The fusion signal represents the change of light brightness over time. The first moment and the second moment are obtained by the following method:

[0101] S210: Identify the fusion signal to obtain mutation information of the fusion signal.

[0102] S220: When the mutation information represents that the mutation value of the fusion signal at a moment is greater than a first preset threshold, determine the moment as a first moment.

[0103] S230: When the mutation information indicates that the mutation value of the fusion signal at a moment is greater than a second preset threshold, determine the moment as a second moment.

[0104] Specifically, the high-frequency photodetector acts as a light intensity signal accumulation device. Since the interval between the gesture-triggered light signal and the light sensor signal received by the high-frequency photodetector is short, the two signals are fused and accumulated to form a fused signal. The signal transmitted by the high-frequency photodetector to the data processing device is the fused signal. Because the fused signal represents changes in light brightness over time, the fused signal is identified to obtain information about its mutations. If, at a certain moment, the mutation value representing the fused signal corresponding to that moment exceeds a first preset threshold, that moment is determined as the first moment. The first moment can be considered the moment when the gesture triggers the gesture-triggered light signal device and emits the gesture-triggered light signal. If, at a certain moment, the mutation value representing the fused signal corresponding to that moment exceeds a second preset threshold, that moment is determined as the second moment. The second moment can be considered the moment when the head-mounted device recognizes the gesture and generates a light sensor signal via the optical sensor.

[0105] For example, see Figure 2d , the high-frequency photodetector continuously detects light signals. Before making a gesture, the light signal detected by the high-frequency photodetector is usually the ambient light signal, and the brightness of the ambient light is relatively small, such as Figure 2d The signal before time t1 is the detected ambient light signal. For example, the signal value of the ambient light signal is approximately 1 nits.

[0106] When a gesture is made, the gesture trigger light signal device generates and emits a gesture trigger light signal. After the high-frequency photodetector detects the gesture trigger light signal, the gesture trigger light signal is fused and superimposed with the ambient light signal, and the signal undergoes a sudden change, for example, from 1 nits to 10 nits. If the sudden change information of the fused signal indicates that the sudden change value of the fused signal at time t1 is greater than the first preset threshold, time t1 can be determined as the first time. Figure 2dThe mutation value shown is approximately 9 nits, and the first preset threshold value may be, for example, 3 nits, 4 nits, or 5 nits, etc. The first moment may be considered as the moment when the gesture action triggers the gesture trigger signal receiving device to generate a control signal, and then the LED light lights up based on the control signal and emits a gesture trigger light signal.

[0107] The high-frequency photodetector continues to detect the light signal. After detecting the light sensor signal generated by the optical sensor of the head-mounted device, the light sensor signal is fused and superimposed with the previously detected signal, and the signal undergoes another mutation, for example, from 10 nits to 12 nits. If the mutation information of the fused signal indicates that the mutation value of the fused signal at time t2 is greater than the second preset threshold, time t2 can be determined as the second time. Figure 2d The mutation value shown is approximately 2 nits. The second preset threshold value can be, for example, 0.5 nits, 1 nit, or 1.5 nits. The second moment can be considered the moment when the head-mounted device recognizes the gesture and generates a light sensor signal through the optical sensor. In some embodiments, the second moment, i.e., the gesture recognition moment t2, is subtracted from the first moment, i.e., the gesture true value moment t1, to obtain Δt as the gesture recognition delay.

[0108] In some embodiments, see Figure 2e , the gesture-triggered light signal device 210 includes:

[0109] The signal receiving device 280 is configured to be triggered by a gesture action and generate a control signal.

[0110] The light signal generating device 290 is used to generate and emit a gesture trigger light signal based on the control signal.

[0111] Specifically, when the user performs a gesture, the signal receiving device 280 is triggered. The trigger signal receiving device 280 generates a control signal corresponding to the gesture in response to the gesture. The signal receiving device 280 may include a bionic glove and a magnetic contact electrode.

[0112] After signal receiving device 280 generates a control signal, the control signal is transmitted to optical signal generating device 290. Upon receiving the control signal, optical signal generating device 290 adjusts its brightness based on the control signal. Optical signal generating device 290 generates a gesture-triggered optical signal and transmits it to high-frequency photodetector 260. Optical signal generating device 290 may be an LED light.

[0113] In some embodiments, see Figure 3 , the synchronous transmission device 250 may include:

[0114] The first optical fiber 310 is connected between the gesture trigger light signal device 210 and the high-frequency photodetector 260 , and is used to receive the gesture trigger light signal emitted by the gesture trigger light signal device 210 and transmit the gesture trigger light signal to the high-frequency photodetector 260 .

[0115] The second optical fiber 320 is connected between the optical sensor 220 and the high-frequency photodetector 260 , and is configured to receive the light sensor signal generated by the optical sensor 220 and transmit the light sensor signal to the high-frequency photodetector 260 .

[0116] Among them, optical fiber can be a special cable used to transmit light signals. It is made of glass or plastic and has a very high refractive index, which allows light signals to be transmitted through it with less signal strength loss.

[0117] Specifically, the gesture-triggered light signal device 210 generates light of varying brightness depending on the situation. When the circuit is on, the gesture-triggered light signal device 210 emits light of a certain intensity; when the circuit is off, the light dims or disappears completely. The gesture-triggered light signal emitted by the gesture-triggered light signal device 210 serves as the signal source for the first optical fiber 310. The first optical fiber 310 is connected between the gesture-triggered light signal device 210 and the high-frequency photodetector 260. The first optical fiber 310 receives the gesture-triggered light signal emitted by the gesture-triggered light signal device 210 and transmits it to the high-frequency photodetector 260.

[0118] The optical sensor 220 generates light of varying brightness depending on the situation. For example, when it detects a gesture that turns on a circuit, the optical sensor 220 emits light of a certain intensity. When it detects a gesture that turns off a circuit, the light dims or disappears completely. The light sensor signal generated by the optical sensor 220 is the signal source for the second optical fiber 320. The second optical fiber 320 is connected between the optical sensor 220 and the high-frequency photodetector 260. The second optical fiber 320 receives the light sensor signal generated by the optical sensor 220 and transmits it to the high-frequency photodetector 260.

[0119] The above-mentioned optical system for measuring gesture recognition delay uses a first optical fiber connected between the gesture trigger light signal device and the high-frequency photodetector to receive the gesture trigger light signal emitted by the gesture trigger light signal device and transmit the gesture trigger light signal to the high-frequency photodetector; a second optical fiber connected between the optical sensor and the high-frequency photodetector is used to receive the light sensor signal generated by the optical sensor and transmit the light sensor signal to the high-frequency photodetector, thereby facilitating signal transmission. The first optical fiber and the second optical fiber are used to merge the two optical signals into the high-frequency photodetector for detection. That is, the two signals can be detected by only one high-frequency photodetector, avoiding the problem of multi-level time synchronization caused by introducing too many other devices into the test system.

[0120] In some embodiments, the signal receiving device may include a magnetic contact electrode and a hand-worn device.

[0121] Among them, by wearing a hand-worn device and performing a gesture action on the magnetic contact electrode, the series circuit formed by the hand-worn device, the magnetic contact electrode and the signal generating device is turned on or off; the control signal is transmitted to the optical signal generating device through the series circuit.

[0122] Magnetic contact electrodes utilize magnetism as a driving force for connection and disconnection. These devices typically consist of two parts connected magnetically: an electrode portion fixed in a specific position and a movable portion. Hand-worn devices are conductive devices that can be worn directly on the human body and used for monitoring, recording, or performing other functions. For example, a hand-worn device could be a bionic glove containing electrodes.

[0123] Specifically, the magnetic contact electrode is usually installed in a fixed position and is part of the circuit. The state of the series circuit is controlled by connecting or disconnecting the magnetic contact electrode. After the magnetic contact electrode completes a measurement, it will automatically recover so that multiple measurements can be made repeatedly, improving the efficiency and reliability of the test. The hand-worn device acts as a "switch" or "trigger" of the circuit. By moving or operating the device, it connects or disconnects it to the magnetic contact electrode to form the conduction or disconnection of the series circuit.

[0124] The magnetic contact electrode, the hand-worn device, and the signal generating device can form a series circuit. When the hand-worn device is worn and the hand-worn device contacts the magnetic contact electrode to perform a gesture, the series circuit is turned on, and current can flow to the signal generating device through the magnetic contact electrode and the conductive part of the hand-worn device. The control signal generated by the magnetic contact electrode and the hand-worn device is transmitted to the optical signal generating device through the series circuit. When the hand-worn device leaves the magnetic contact electrode, the magnetic connection is broken, the series circuit is disconnected, and the current stops flowing.

[0125] The above-mentioned optical system for measuring gesture recognition delay may include a signal receiving device that includes a magnetic contact electrode and a hand-worn device. By wearing the hand-worn device and performing a gesture action on the magnetic contact electrode, the series circuit formed by the hand-worn device, the magnetic contact electrode, and the signal generating device is turned on or off; the control signal is transmitted to the optical signal generating device through the series circuit, providing a basis for the subsequent determination of the first moment.

[0126] In some embodiments, the series circuit can be turned on or off by performing a gesture on the magnetic contact electrode by wearing a hand-worn device, thereby generating a control signal.

[0127] Specifically, by wearing the hand wearable device and performing a gesture, the hand wearable device contacts the magnetic contact electrode, forming a conduction point in the series circuit, making the series circuit conductive, thereby generating a control signal. By wearing the hand wearable device and performing a gesture, the hand wearable device is separated from the magnetic contact electrode, breaking the magnetic connection and disconnecting the series circuit, thereby generating a control signal.

[0128] The optical system for measuring gesture recognition delay generates a control signal by performing a gesture on the magnetic contact electrode while wearing the wearable device on the hand, thereby turning the series circuit on or off and providing a data basis for the subsequent determination of the moment information.

[0129] In some embodiments, the magnetic contact electrode includes a first component and a second component, the first component and the second component being magnetically connected. A first gesture is performed on the magnetic contact electrode by wearing a hand-worn device, thereby completing the series circuit: the first gesture includes the hand-worn device contacting the second component.

[0130] The first component can be a fixed part connected to an end point or component of the entire series circuit system. The first component can be made of magnetic material and can generate sufficient magnetic force to ensure that the second component is attracted to it, thereby forming a stable series circuit connection. The second component can be made of conductive material and contain magnetic components.

[0131] Specifically, the magnetic contact electrode includes a first component and a second component, which are magnetically connected to facilitate the connection between the first component and the second component, thereby achieving conduction of the series circuit. When the user performs a first gesture on the magnetic contact electrode by wearing the hand wearable device, such as making the hand wearable device touch the second component, a loop is formed between the magnetic contact electrode and the hand wearable device, thereby achieving conduction of the series circuit.

[0132] The above-mentioned optical system for measuring gesture recognition delay makes the series circuit conductive by performing a first gesture action on the magnetic contact electrode by wearing a hand-worn device: the first gesture action includes the hand-worn device contacting the second component, providing a data basis for subsequent determination of time information.

[0133] In some embodiments, the magnetic contact electrode includes a first component and a second component, the first component and the second component being magnetically connected. The series circuit is disconnected by performing a second hand gesture on the magnetic contact electrode while wearing the hand-worn device. The second hand gesture includes at least one of the following: the hand-worn device moving away from the second component or the hand-worn device separating the second component from the first component.

[0134] Specifically, the magnetic contact electrode includes a first component and a second component, and the first component and the second component are magnetically connected to facilitate the separation of the first component and the second component, thereby disconnecting the series circuit. When the user performs a second gesture on the magnetic contact electrode by wearing a hand wearable device, the second component is separated from the first component by the hand wearable device, so that the loop formed between the magnetic contact electrode and the hand wearable device is disconnected, thereby disconnecting the series circuit. In other embodiments, the second gesture is performed on the magnetic contact electrode by wearing a hand wearable device, so that the bionic glove leaves the second component, and the loop formed between the magnetic contact electrode and the hand wearable device is disconnected, thereby disconnecting the series circuit. Among them, the hand wearable device can be a bionic glove.

[0135] The above-mentioned optical system for measuring gesture recognition delay disconnects the series circuit by performing a second gesture action on the magnetic contact electrode while wearing a hand-worn device. The second gesture action includes at least one of the hand-worn device leaving the second component and the second component being separated from the first component by the hand-worn device, providing a data basis for subsequent determination of moment information.

[0136] In some embodiments, when there is only one second component, the first gesture action includes pinching the second component with one hand while wearing the hand wearable device, and the second gesture action includes releasing the second component with one hand while wearing the hand wearable device, or pulling the second component with one hand while wearing the hand wearable device so that the second component is separated from the first component.

[0137] Specifically, when there is only one second component, the first gesture is performed by pinching the second component with one hand while wearing the hand wearable device, thereby forming good contact with the second component and ensuring conduction of the series circuit. The second gesture is performed by releasing the second component by removing the hand while wearing the hand wearable device, or by pulling the second component with one hand while wearing the hand wearable device so that the second component is separated from the first component, thereby breaking the loop formed between the magnetic contact electrode and the hand wearable device, thereby disconnecting the series circuit.

[0138] For example, see Figure 4 The first gesture action may include any one of Tap, Double tap, Pinch and hold, Pinch and drag, etc.

[0139] The above-mentioned optical system for measuring gesture recognition delay, when there is only one second component, the first gesture action includes pinching the second component with one hand of the wearable device on the hand, and the second gesture action includes releasing the second component with one hand of the wearable device on the hand, or pulling the second component with one hand of the wearable device on the hand so that the second component is separated from the first component, providing a data basis for subsequent determination of moment information.

[0140] In some embodiments, when there are two second components, the first gesture action includes pinching the two second components with both hands of the wearable device worn on the hand, and the second gesture action includes releasing the second components with both hands of the wearable device worn on the hand, or pulling the two second components with both hands of the wearable device worn on the hand so that at least one of the two second components is separated from the first component.

[0141] Specifically, when there are two second components, the first gesture is performed by pinching the two second components with both hands, thereby ensuring good contact between the hands and the respective second components and ensuring conduction of the series circuit. When the hands wearing the wearable device are released from the hand, that is, the second components are released, or the hands pull the two second components so that at least one of the two second components is separated from the first component, thereby performing the second gesture, breaking the loop formed between the magnetic contact electrode and the wearable device, and thus disconnecting the series circuit.

[0142] For example, see Figure 5 The first gesture action may include any one of Zoom, Rotate, etc.

[0143] In the above-mentioned optical system for measuring gesture recognition delay, when there are two second components, the first gesture action includes pinching the two second components with both hands of the wearable device on the hand, and the second gesture action includes releasing the second components with both hands of the wearable device on the hand, or pulling the two second components with both hands of the wearable device on the hand so that at least one of the two second components is separated from the first component, providing a data basis for subsequent determination of moment information.

[0144] In some embodiments, when there is one second component, there is one hand-worn device, the first component is connected to the optical signal generating device through a wire, and the hand-worn device is connected to the optical signal generating device through a wire. The "connection" in this example may include an electrical connection.

[0145] Specifically, if there is only one second component, it is magnetically connected to one side of the first component to ensure a secure and reliable connection. Because there is only one second component, a hand-worn device is required for testing. The first component is connected to the optical signal generating device via a wire, and the hand-worn device is also connected to the optical signal generating device via a wire, thus forming a loop to test gestures performed by a single hand.

[0146] For example, the light signal generating device may be an LED lamp. Figure 6 , the second component 602 is magnetically connected to the right end of the first component 604, the first component 604 is connected to the LED light 606 via a wire, and the hand-worn device 608 is also connected to the LED light 606 via a wire. It should be noted that the series circuit may also include a power supply 610, which can be connected anywhere in the circuit. In some embodiments, the power supply 610 may be located on the wire connecting the first component 604 and the LED light 606. In other embodiments, the power supply may be located on the wire connecting the hand-worn device 608 and the LED light 606.

[0147] The above-mentioned optical system for measuring gesture recognition delay, when the second component is one, the hand-worn device is one, the first component is connected to the optical signal generating device through a wire, and the hand-worn device is connected to the optical signal generating device through a wire, so as to test the gesture recognition delay of gesture actions performed by one hand.

[0148] In some embodiments, when there are two second components, the two second components are magnetically connected to the two ends of the first component respectively, and there are two hand-worn devices. One hand-worn device is connected to the optical signal generating device through a wire, and the other hand-worn device is connected to the optical signal generating device through a wire. The "connection" in this example may include electrical connection.

[0149] Specifically, if there are two second components, the two second components are magnetically connected to the ends of the first component to ensure a secure and reliable connection. Because there are two second components, two hand-worn devices are required for testing. One hand-worn device is connected to the optical signal generating device via a wire, and the other hand-worn device is also connected to the optical signal generating device via a wire, forming a loop to test gestures performed by both hands.

[0150] For example, the light signal generating device may be an LED lamp. Figure 7, the second component 702 is magnetically connected to the left end of the first component 712, and the second component 704 is magnetically connected to the right end of the first component 712. The hand-worn device 706 is connected to the LED light 710 via a wire, and the hand-worn device 708 is connected to the LED light 710 via a wire. It should be noted that the series circuit may also include a power supply 714, which can be connected to any position in the circuit. In some embodiments, the power supply 714 can be located on the wire connecting the hand-worn device 706 and the LED light 710. In other embodiments, the power supply can be located on the wire connecting the hand-worn device 708 and the LED light 710.

[0151] In the above-mentioned optical system for measuring gesture recognition delay, when there are two second components, the two second components are magnetically connected to the two ends of the first component respectively, and there are two hand-worn devices, one hand-worn device is connected to the optical signal generating device via a wire, and the other hand-worn device is connected to the optical signal generating device via a wire, so as to test the gesture recognition delay of gesture actions performed by both hands.

[0152] It can be understood that the present application is not limited to characterizing object changes based on the on and off of light sources. As long as the measurement equipment used allows, other forms of information can be used to characterize object changes without affecting accuracy, such as shape change information, position change information, etc.

[0153] In some embodiments, the gesture triggering the light signal may include: performing a gesture action to generate a light signal representing a light change.

[0154] Wherein, the optical signal generating device includes a lighting device.

[0155] Specifically, when a user performs a gesture, it triggers a signal receiving device and causes the device's lighting to change. This lighting change serves as a control signal to convey specific information. The lighting device can capture the control signal from the device's lighting change and convert it into a light signal representing the lighting change. This allows the optical system used to measure gesture recognition latency to recognize and understand the gesture, thereby determining the moment the gesture was performed.

[0156] The optical system for measuring gesture recognition delay generates light signals representing light changes by performing gesture actions, which can adapt to different application scenarios and achieve a personalized control experience.

[0157] In some embodiments, the gesture triggering the light signal may include: performing a gesture action to generate a light signal representing a shape change.

[0158] Wherein, the optical signal generating device includes a shape change sensing device.

[0159] Specifically, when a user performs a gesture, it triggers the signal receiving device and causes it to change shape. This shape change serves as a control signal to convey specific information. The shape change sensing device captures the control signal of the device's shape change and converts it into a light signal representing the shape change. This allows the optical system used to measure gesture recognition latency to recognize and understand the gesture, thereby determining the moment the gesture was performed.

[0160] The above-mentioned optical system for measuring gesture recognition delay performs gesture actions to generate light signals representing shape changes, which can be adapted to different application scenarios.

[0161] In some embodiments, triggering a light signal with a gesture may include: performing a gesture action to generate a light signal representing a position change.

[0162] Among them, the optical signal generating device includes a position change sensing device.

[0163] Specifically, when a user performs a gesture, it triggers the signal receiving device and causes it to change position. This position change serves as a control signal to convey specific information. The position change sensing device captures the control signal of the device's position change and converts it into an optical signal representing the position change. This allows the optical system used to measure gesture recognition latency to recognize and understand the gesture, thereby determining the moment the gesture was performed.

[0164] The optical system for measuring gesture recognition delay performs gesture actions to generate light signals representing position changes, and can be adapted to different application scenarios.

[0165] In some embodiments, the gesture-triggered light signal generated by the light signal generating device may include at least one of the following: a gesture-triggered light signal generated when the light signal generating device is lit; or a gesture-triggered light signal generated when the light device is off.

[0166] Specifically, based on the gesture action, the light signal generating device lights up, causing the ambient light intensity to increase, thereby generating a change in light brightness and generating a gesture trigger light signal. Based on the gesture action, the light signal generating device turns off, causing the ambient light intensity to decrease, thereby generating a change in light brightness and generating a gesture trigger light signal. This change in light brightness not only helps users clearly perceive that the gesture action has been recognized, but also provides intuitive visual feedback and enables real-time feedback on the gesture action.

[0167] The test system for the delay time of the above-mentioned device generates a gesture-triggered light signal when the light signal generating device is on; and generates a gesture-triggered light signal when the light device is off. The gesture action is characterized by the lighting of the light signal generating device, providing a data basis for the subsequent determination of the moment information.

[0168] In some embodiments, the gesture includes at least one of the following: a one-hand pinch gesture, a two-hand pinch gesture, a one-hand release gesture, a one-hand pull gesture, a two-hand pull gesture, and a two-hand rotation gesture.

[0169] Specifically, when the user's hand enters the field of view of the headset, the headset recognizes the gesture. Gestures are designed to interact with objects in the virtual environment, and different gestures produce different effects.

[0170] When a user pinches with one hand, the headset recognizes it as a one-hand pinch gesture. The one-hand pinch gesture can be used to edit icons, similar to a long press on a phone.

[0171] When the user releases their fingers on one hand, the headset recognizes it as a one-hand release gesture. This one-hand release gesture can be used to open apps or go to the next menu level, similar to double-clicking a mouse or clicking an icon on a phone.

[0172] When a user pulls with one hand, the headset recognizes it as a one-hand pull gesture. This can be used to move virtual objects. For example, in a game, a user can use this gesture to move a character.

[0173] When the user pinches with the fingers of both hands, the headset recognizes it as a pinching gesture.

[0174] When the user pulls with their fingers, the headset recognizes it as a two-hand pull gesture. This two-hand pull gesture can be used to zoom in or out on objects in a virtual environment.

[0175] When the user moves the left hand to the lower right and the right hand to the upper left (or the left hand to the upper right and the right hand to the lower left), the headset will recognize it as a two-hand rotation gesture. The two-hand rotation gesture can be used to rotate objects or images.

[0176] The above-mentioned optical system for measuring gesture recognition delay includes at least one of the gesture actions of pinching with one hand, pinching with both hands, releasing with one hand, pulling with one hand, pulling with both hands, and rotating with both hands, so as to provide a data basis for testing gesture recognition delay.

[0177] This specification provides a method for testing gesture recognition delay. Figure 8aThe gesture recognition delay testing method may include the following steps:

[0178] S810: Receive a gesture trigger light signal and a light sensor signal from a high-frequency photodetector.

[0179] Among them, the gesture trigger light signal is triggered and emitted by the gesture trigger light signal device based on the gesture action, the light sensor signal is generated by the optical sensor set on the head-mounted device to detect the content displayed by the display module of the head-mounted device, and the displayed content is generated by the head-mounted device recognizing the gesture action. The gesture trigger light signal and the light sensor signal are synchronously transmitted to the high-frequency photodetector by the synchronous transmission device.

[0180] Specifically, when a user performs a gesture, the signal receiving device is triggered. The trigger signal receiving device responds to the gesture by capturing and recording the changes caused by the gesture, identifying and processing these changes, and generating a control signal corresponding to the gesture. The brightness of the light signal generating device is then adjusted based on the control signal. The light signal generating device generates a gesture-triggered light signal and transmits it to a high-frequency photodetector.

[0181] When performing gestures, the head-mounted device uses built-in sensors and algorithms to analyze and process information such as hand posture, finger position, and motion trajectory to achieve real-time action recognition of gestures. When the head-mounted device successfully recognizes a specific gesture, the head-mounted device will generate the content displayed by the display module of the head-mounted device. Then, the optical sensor set on the head-mounted device detects the content displayed by the display module of the head-mounted device and generates a light sensor signal based on the detection result. Then, the synchronous transmission device synchronously transmits the gesture trigger light signal and the light sensor signal to the high-frequency photodetector. Finally, the gesture trigger light signal and the light sensor signal from the high-frequency photodetector are received.

[0182] S820: Perform mutation detection on the fused signal of the gesture trigger light signal and the light sensor signal received from the high-frequency photodetector to obtain mutation information, and detect and calculate the time difference based on the mutation information to determine the delay time of gesture recognition.

[0183] Specifically, the high-frequency photodetector acts as a light intensity signal accumulation device. Since the interval between the gesture-triggered light signal and the light sensor signal received by the high-frequency photodetector is short, the two signals are fused and accumulated to form a fused signal. The signal transmitted by the high-frequency photodetector to the data processing device is the fused signal. Because the fused signal represents changes in light brightness over time, the fused signal is identified to obtain information about its mutations. If, at a certain moment, the mutation value representing the fused signal at that moment is greater than a first preset threshold, that moment is determined as the first moment. If, at a certain moment, the mutation value representing the fused signal at that moment is greater than the first preset threshold, that moment is determined as the first moment. The first moment can be considered the moment when the gesture triggers the gesture-triggered light signal device and emits the gesture-triggered light signal. If, at a certain moment, the mutation value representing the fused signal at that moment is greater than a second preset threshold, that moment is determined as the second moment. The second moment can be considered the moment when the head-mounted device recognizes the gesture and generates a light sensor signal via the optical sensor.

[0184] The system calculates the difference between the first and second moments and determines the time difference between the second and first moments. This difference represents the time elapsed from the time the gesture triggers the light signal device to generate a brightness change until the head-mounted device generates a brightness change, reflecting the gesture recognition delay. Therefore, the gesture recognition delay is calculated based on the time difference.

[0185] In some embodiments, see Figure 8b By wearing a bionic glove containing electrodes, a one-hand pinching gesture (pinch) is performed on the magnetic contact electrode. Specifically, the thumb and index finger touch the finger pinch point on the second component of the magnetic contact electrode, thereby conducting the series circuit. After the series circuit is conducted, the LED light turns on, generating a gesture trigger light signal. The head-mounted device successfully recognizes the one-hand pinching gesture and generates a light sensor signal. The gesture trigger light signal and the light sensor signal are fused to form a fused signal. The fused signal is identified to obtain mutation information of the fused signal. If the mutation information indicates that the mutation value of the fused signal at a moment is greater than a first preset threshold, the moment is determined as the first moment, i.e., the moment when the LED light turns on, and is recorded as s1. If the mutation information indicates that the mutation value of the fused signal at a moment is greater than a second preset threshold, the moment is determined as the second moment, i.e., the moment when the head-mounted device screen turns on, and is recorded as s2. The difference between the second moment s2 and the first moment s1 is used as the gesture recognition delay time for the one-hand pinching gesture. It should be noted that the finger pinch point can be any position on the second component.

[0186] Please continue reading Figure 8bThen, by wearing a bionic glove containing electrodes and releasing the magnetic contact electrode, a single-hand release gesture (release) is performed. Specifically, the thumb and index finger release the finger pinch point on the second component of the magnetic contact electrode, disconnecting the action circuit. After the action circuit is disconnected, the LED light turns off, generating a gesture trigger light signal. The head-mounted device successfully recognizes the single-hand release gesture and generates a light sensor signal. The gesture trigger light signal and the light sensor signal are fused to form a fused signal. The fused signal is identified to obtain mutation information of the fused signal. If the mutation information indicates that the mutation value of the fused signal at a moment is greater than a first preset threshold, the moment is determined as the third moment, i.e., the moment when the LED light turns off, recorded as s3. If the mutation information indicates that the mutation value of the fused signal at a moment is greater than a second preset threshold, the moment is determined as the fourth moment, i.e., the moment when the head-mounted device screen turns off, recorded as s4. The difference between the fourth moment s4 and the third moment s3 is used as the gesture recognition delay time for the single-hand release gesture.

[0187] In other embodiments, please refer to Figure 8b After performing a pinch gesture ("pinch") on the magnetic contact electrode while wearing a bionic electrode-containing glove and determining the gesture recognition delay for the pinch gesture, the bionic electrode-containing glove then pulls the magnetic contact electrode to perform a drag gesture ("pull"), specifically, pinching the finger pinch point with the thumb and index finger and forcefully pulling the second component of the magnetic contact electrode outward, disconnecting the second component from the first component and breaking the series circuit. After the series circuit is broken, the LED turns off, generating a gesture trigger light signal. The head-mounted device successfully recognizes the pull gesture and generates a light sensor signal. The gesture trigger light signal and the light sensor signal are fused to form a fused signal. The fused signal is identified to obtain mutation information of the fused signal. If the mutation information indicates that the mutation value of the fused signal at a moment is greater than a first preset threshold, that moment is determined as the fifth moment, i.e., the moment when the LED turns off, denoted as s5. If the mutation value of the fusion signal at a certain moment, as represented by the mutation information, is greater than the second preset threshold, the moment is determined as the sixth moment, i.e., the moment when the head-mounted device screen turns off, and is recorded as s6. The difference between the sixth moment s6 and the fifth moment s5 is used as the gesture recognition delay for the single-handed pull gesture.

[0188] See also Figure 8cWearing a bionic glove containing electrodes, the user releases the magnetic contact electrodes and performs a two-handed pulling gesture (zoom). Specifically, the user pinches the fingertips with the thumbs and index fingers of both hands and pulls the second component of the magnetic contact electrode outward, disconnecting the second component from the first component and breaking the series circuit. After the series circuit is disconnected, the LED turns off, generating a gesture trigger light signal. The head-mounted device successfully recognizes the two-handed pulling gesture and generates a light sensor signal. The gesture trigger light signal and the light sensor signal are fused to form a fused signal. The fused signal is identified to obtain mutation information of the fused signal. If the mutation information indicates that the mutation value of the fused signal at a moment is greater than a first preset threshold, the moment is determined to be the seventh moment, i.e., the moment when the LED turns off, and is recorded as s7. If the mutation information indicates that the mutation value of the fused signal at a moment is greater than a second preset threshold, the moment is determined to be the eighth moment, i.e., the moment when the head-mounted device screen turns off, and is recorded as s8. The difference between the eighth moment s8 and the seventh moment s7 is used as the gesture recognition delay for the two-handed pulling gesture.

[0189] In the above embodiment, a gesture trigger light signal and a light sensor signal are received from a high-frequency photodetector. A fused signal of the gesture trigger light signal and the light sensor signal received from the high-frequency photodetector is subjected to mutation detection to obtain mutation information. Based on the mutation information, time differences are detected and calculated to determine the delay time of gesture recognition, measure the end-to-end delay of the gesture action, reduce the difficulty of gesture recognition delay testing, and improve test efficiency.

[0190] In some embodiments, an electronic device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9 As shown. The electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a test method for gesture recognition delay is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the housing of the electronic device, or an external keyboard, touchpad or mouse.

[0191] Those skilled in the art will understand that Figure 9 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution disclosed in this specification, and does not constitute a limitation on the computer device to which the solution disclosed in this specification is applied. Specifically, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0192] An embodiment of this specification provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor implements the method steps in the above embodiment when executing the computer program.

[0193] An embodiment of this specification provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method in any of the above embodiments are implemented.

[0194] The embodiments of this specification provide a computer program product, which includes instructions. When the instructions are executed by a processor of a computer device, the computer device can perform the steps of any method of the above embodiments.

[0195] It should be noted that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing it in a suitable manner if necessary, and then storing it in a computer memory.

Claims

1. An optical system for measuring gesture recognition delay, characterized in that: The system comprises: A gesture-triggered light signal device, configured to be triggered by a gesture action and emit a gesture-triggered light signal; an optical sensor, disposed on the head-mounted device, for detecting content displayed by a display module of the head-mounted device and generating an optical sensor signal, wherein the displayed content is generated by the head-mounted device recognizing the gesture action; a synchronous transmission device for synchronously transmitting the gesture trigger light signal and the light sensor signal to a high-frequency photodetector; a high-frequency photodetector, configured to receive the synchronously transmitted gesture trigger light signal and the light sensor signal; A data processing device is used to detect and calculate the time difference between the gesture trigger light signal and the light sensor signal received by the high-frequency photodetector to determine the delay time of gesture recognition.

2. The system according to claim 1, wherein: The detecting and calculating the time difference between the gesture trigger light signal and the light sensor signal received by the high-frequency photodetector to determine the delay time of gesture recognition includes: determining a time difference between the first time and the second time based on a first time corresponding to the gesture trigger light signal and a second time corresponding to the light sensor signal; Based on the time difference, a delay time for gesture recognition is determined.

3. The system according to claim 2, characterized in that The gesture trigger light signal and the light sensor signal are fused to form a fused signal, where the fused signal represents the change in light brightness over time. The first moment and the second moment are obtained in the following manner: Identifying the fusion signal to obtain mutation information of the fusion signal; When the mutation information indicates that a mutation value of the fusion signal at a moment is greater than a first preset threshold, determining the moment as the first moment; When the mutation information indicates that a mutation value of the fusion signal at a moment is greater than a second preset threshold, the moment is determined as the second moment.

4. The system according to claim 1, wherein: The gesture-triggered light signal device comprises: A signal receiving device, configured to be triggered by a gesture action and generate a control signal; A light signal generating device is used to generate and emit the gesture trigger light signal based on the control signal.

5. The system according to claim 1, wherein: The synchronous transmission device includes: a first optical fiber connected between the gesture trigger light signal device and the high-frequency photoelectric detector, for receiving the gesture trigger light signal emitted by the gesture trigger light signal device and transmitting the gesture trigger light signal to the high-frequency photoelectric detector; The second optical fiber is connected between the optical sensor and the high-frequency photodetector, and is used to receive the light sensor signal generated by the optical sensor and transmit the light sensor signal to the high-frequency photodetector.

6. The system according to claim 4, characterized in that: The signal receiving device includes a magnetic contact electrode and a hand-worn device, wherein, by wearing the hand-worn device and performing a gesture on the magnetic contact electrode, the series circuit formed by the hand-worn device, the magnetic contact electrode and the signal generating device is turned on or off; the control signal is transmitted to the optical signal generating device through the series circuit.

7. The system according to claim 6, characterized in that The magnetic contact electrode includes a first component and a second component, and the first component and the second component are magnetically connected; Performing a first gesture on the magnetic contact electrode by wearing the hand-worn device to conduct the series circuit, wherein the first gesture includes the hand-worn device contacting the second component; The series circuit is disconnected by wearing the hand-worn device and performing a second gesture on the magnetic contact electrode, wherein the second gesture includes at least one of the hand-worn device leaving the second component and the second component being separated from the first component by the hand-worn device.

8. The system according to claim 7, characterized in that In the case where the second component is one: The first gesture action includes: wearing the hand-worn device and pinching the second component with one hand; The second gesture action includes: releasing the second component with one hand while wearing the hand-worn device, or pulling the second component with one hand while wearing the hand-worn device so that the second component is separated from the first component.

9. The system according to claim 7, wherein: In the case where there are two second components: The first gesture action includes: wearing the hand-worn device and pinching the two second components with both hands; The second gesture action includes: wearing the hand-worn device with both hands to release the second component, or wearing the hand-worn device with both hands to pull the two second components so that at least one of the two second components is separated from the first component.

10. The system according to claim 8, wherein: In the case where there is one second component, there is one hand-worn device, the first component is connected to the optical signal generating device via a wire, and the hand-worn device is connected to the optical signal generating device via a wire.

11. The system according to claim 9, wherein: When there are two second components, the two second components are magnetically connected to the two ends of the first component respectively. When there are two hand-worn devices, one hand-worn device is connected to the optical signal generating device through a wire, and the other hand-worn device is connected to the optical signal generating device through a wire.

12. The system according to claim 4, wherein: The gesture trigger light signal includes at least one of the following: Executing the gesture to generate a light signal representing a light change, wherein the light signal generating device includes a lighting device; Executing the gesture to generate a light signal representing the shape change, wherein the light signal generating device includes a shape change sensing device; The gesture action is performed to generate a light signal representing a position change, wherein the light signal generating device includes a position change sensing device.

13. The system according to any one of claims 1 to 12, characterized in that: The gesture action includes at least one of the following: One-hand pinching gesture, two-hand pinching gesture, one-hand release gesture, one-hand pulling gesture, two-hand pulling gesture, two-hand rotation gesture.

14. A method for testing gesture recognition delay, characterized in that: The test method includes: Receiving a gesture-triggered light signal and a light sensor signal from a high-frequency photodetector, wherein the gesture-triggered light signal is triggered and emitted by a gesture-triggered light signal device based on a gesture action, the light sensor signal is generated by an optical sensor provided on a head-mounted device detecting content displayed by a display module of the head-mounted device, and the displayed content is generated by the head-mounted device recognizing the gesture action, and the gesture-triggered light signal and the light sensor signal are synchronously transmitted to the high-frequency photodetector by a synchronous transmission device; For the fusion signal of the gesture trigger light signal and the light sensor signal received from the high-frequency photodetector, mutation detection is performed on the fusion signal to obtain mutation information, and time difference is detected and calculated based on the mutation information to determine the delay time of gesture recognition.

15. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to claim 14 are implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 14 are implemented.

Images