Rendering method and device of wearable equipment, equipment and storage medium

Abstract

The invention discloses a rendering method and device of wearable equipment, equipment and a storage medium, relates to the technical field of wearable equipment, and discloses the rendering method of the wearable equipment, the method is applied to the wearable equipment, an eye tracking device is arranged, and the eye tracking device is used for monitoring eye movement. The method comprises the following steps: analyzing an eye tracking image collected by an eye tracking device, and determining a gazing point location and gazing parameters; determining a device offset state according to the gaze parameter; and when the state is not shifted, determining a target rendering area according to the fixation point location and the equipment interaction information, and performing image enhancement rendering in the area. Through the above mode, whether deviation exists between the equipment and the eyes of the user is determined based on the gaze parameter, the rendering area is determined by combining the gaze point location and the equipment interaction information when deviation does not occur, rendering is enhanced in the area, visual delay and scene deviation are effectively reduced, motion sickness of the user is relieved, and user experience is improved.

Description

Technical Field

[0001] This application relates to the field of wearable device technology, and in particular to rendering methods, apparatus, devices and storage media for wearable devices. Background Technology

[0002] When users play games on wearable devices, such as Virtual Reality (VR) devices, especially sports games like ball games and shooting games, prolonged use may cause dizziness or nausea. The root cause is that the image update speed cannot keep up with the user's head movement speed, resulting in visual delay and triggering motion sickness. Currently, the main reasons for this inability to keep up with head movement speed include: insufficient image rendering speed; and insufficient fit between the wearable device and the head during rapid head movements, causing the image to shift relative to the head due to physical inertia. Therefore, optimizing visual delay and scene shift to improve user experience is a pressing issue that needs to be addressed.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of this application is to provide a rendering method, apparatus, device, and storage medium for wearable devices, aiming to solve the technical problem of how to optimize visual latency and scene offset in wearable devices.

[0005] To achieve the above objectives, this application proposes a rendering method for a wearable device. The wearable device is equipped with an eye-tracking device for monitoring eye movements. The method includes:

[0006] The eye-tracking images acquired by the eye-tracking device are analyzed to determine the user's current gaze point and current gaze parameters;

[0007] Determine the device offset state based on the current gaze parameters;

[0008] When the device offset state is no offset, the target rendering area is determined based on the current gaze point and device interaction information;

[0009] Image enhancement rendering is performed within the target rendering area.

[0010] In one embodiment, the wearable device further includes an image acquisition device for monitoring the position information of a handle, the handle being communicatively connected to the wearable device;

[0011] The step of determining the target rendering region based on the current gaze point and device interaction information includes:

[0012] When the current interaction scenario is detected to be a hand interaction scenario, the image acquisition device captures the handle position image based on the device interaction information, and the current handle position is determined based on the handle position image.

[0013] The rendering path is determined based on the current position of the handle and the current gaze point.

[0014] The target rendering area is determined based on the target rendering width and the rendering path.

[0015] In one embodiment, the step of determining the target rendering region based on the current gaze point and device interaction information includes:

[0016] When the current interaction scenario is detected to be an audio interaction scenario, the audio source interaction information is determined based on the device interaction information;

[0017] The direction of the sound source interaction is determined based on the sound source interaction information.

[0018] The target rendering area is determined based on the sound source interaction location and the current gaze point.

[0019] In one embodiment, the wearable device further includes an attitude sensor and an inertial measurement unit;

[0020] The step of determining the target rendering region based on the current gaze point and device interaction information includes:

[0021] When the current interaction scenario is detected to be a head interaction scenario, the three-dimensional attitude information collected by the attitude sensor and the head motion information collected by the inertial measurement unit are determined based on the device interaction information.

[0022] The direction of head movement is determined based on the three-dimensional posture information and the head movement information;

[0023] The target rendering area is determined based on the current gaze point, the target rendering angle, and the head movement direction.

[0024] In one embodiment, the step of determining the device offset state based on the current gaze parameter includes:

[0025] Compare each eye positioning angle in the current gaze parameters with the corresponding historical positioning angles;

[0026] When there is a discrepancy between the eye positioning angle and the historical positioning angle corresponding to the eye positioning angle, the device offset state is determined to be an offset.

[0027] When each eye positioning angle is consistent with the corresponding historical positioning angle, the device offset status is determined to be no offset.

[0028] In one embodiment, after the step of determining that the device offset state has occurred when there is a discrepancy between the eye positioning angle and the historical positioning angle corresponding to the eye positioning angle, the method further includes:

[0029] The device offset and device offset direction are determined based on the eye positioning angle and the historical positioning angle corresponding to the eye positioning angle.

[0030] The rendering adjustment direction is determined based on the device offset direction;

[0031] The image rendering data is adjusted according to the device offset and the rendering adjustment direction.

[0032] In one embodiment, the step of analyzing the eye-tracking images acquired by the eye-tracking device to determine the user's current gaze point and current gaze parameters includes:

[0033] The eye-tracking images acquired by the eye-tracking device are analyzed to determine the current gaze point and the location of each corner of the eye.

[0034] Multiple auxiliary positioning points and their locations are determined based on the location of each corner of the eye;

[0035] Angle calculations are performed based on the location of each eye corner, the location of each auxiliary positioning point, and the location of the eye tracking device to determine multiple eye positioning angles.

[0036] Information from each eye positioning angle is collected to obtain the current gaze parameters.

[0037] In addition, to achieve the above objectives, this application also proposes a rendering device for a wearable device, the rendering device for the wearable device comprising: an analysis module, used to analyze the eye-tracking images collected by the eye-tracking device to determine the user's current gaze point and current gaze parameters;

[0038] The processing module is used to determine the device offset state based on the current gaze parameters;

[0039] The processing module is further configured to determine the target rendering area based on the current gaze point and device interaction information when the device offset state is no offset.

[0040] The rendering module is used to perform image enhancement rendering within the target rendering area.

[0041] In addition, to achieve the above objectives, this application also proposes a wearable device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the rendering method for the wearable device as described above.

[0042] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the rendering method for wearable devices as described above.

[0043] This application provides a rendering method for a wearable device. The method is applied to a wearable device equipped with an eye-tracking device for monitoring eye movements. The method includes: analyzing eye-tracking images captured by the eye-tracking device to determine the user's current gaze point and current gaze parameters; determining the device offset state based on the current gaze parameters; when the device offset state is no offset, determining a target rendering area based on the current gaze point and device interaction information; and performing image enhancement rendering within the target rendering area. By using the eye-tracking device to determine the current gaze point and current gaze parameters, and based on the current gaze parameters to determine whether there is an offset between the device and the user's eyes, and when no offset occurs, determining the target rendering area by combining the gaze point and device interaction information, and enhancing rendering within that area, visual delay and scene offset are effectively reduced, further alleviating motion sickness and improving the user experience. Attached Figure Description

[0044] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 A flowchart illustrating the rendering method for wearable devices provided in Embodiment 1 of this application;

[0047] Figure 2 A schematic diagram of the internal structure of the wearable device rendering method provided in Embodiment 1 of this application;

[0048] Figure 3A schematic diagram of the overall device for the rendering method of the wearable device provided in Embodiment 1 of this application;

[0049] Figure 4 A schematic diagram of gaze parameters for the rendering method of the wearable device provided in Embodiment 1 of this application;

[0050] Figure 5 A flowchart illustrating the second embodiment of the rendering method for wearable devices in this application;

[0051] Figure 6 A flowchart illustrating the rendering method for wearable devices provided in Embodiment 3 of this application;

[0052] Figure 7 A schematic diagram of device offset for the rendering method of the wearable device provided in Embodiment 3 of this application;

[0053] Figure 8 This is a schematic diagram of the module structure of the rendering device for a wearable device according to an embodiment of this application;

[0054] Figure 9 This is a schematic diagram of the hardware operating environment involved in the rendering method of the wearable device in this embodiment of the application.

[0055] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0056] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0057] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0058] The main solution of this application embodiment is: to analyze the eye-tracking images collected by the eye-tracking device to determine the user's current gaze point and current gaze parameters; to determine the device offset state based on the current gaze parameters; when the device offset state is no offset, to determine the target rendering area based on the current gaze point and device interaction information; and to perform image enhancement rendering within the target rendering area.

[0059] Currently, when using wearable devices, such as VR, the following situation occurs: the image update speed cannot keep up with the user's head movement speed, resulting in visual delay and causing motion sickness. The main reasons for this inability to keep up with head movement speed are: insufficient image rendering speed; and insufficient fit between the wearable device and the head during rapid head movements. Due to physical inertia, the image initially moves slower than the head, but later moves faster, meaning the image's position relative to the head shifts.

[0060] This application uses an eye-tracking device to determine the current gaze point and current gaze parameters. Based on the current gaze parameters, it determines whether there is a misalignment between the device and the user's eyes. When no misalignment occurs, it combines the gaze point and device interaction information to determine the target rendering area and enhances rendering within that area. This effectively reduces visual latency and scene misalignment, further alleviates motion sickness, and improves the user experience.

[0061] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or wearable device capable of performing the above functions. The following description uses a wearable device as an example to illustrate this embodiment and the subsequent embodiments.

[0062] Based on this, embodiments of this application provide a rendering method for wearable devices, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the rendering method for wearable devices according to this application.

[0063] In this embodiment, the rendering method for the wearable device is applied to the wearable device, which is equipped with an eye-tracking device for monitoring eye movements. The method includes steps S10 to S40:

[0064] Step S10: Analyze the eye-tracking images acquired by the eye-tracking device to determine the user's current gaze point and current gaze parameters.

[0065] It should be noted that the wearable device in this embodiment is a VR device, an AR device, or other head-mounted wearable device. This embodiment does not limit this; in this embodiment, a VR device is used as an example for illustration.

[0066] It is understood that wearable devices are equipped with at least one eye-tracking device, an IMU (Inertial Measurement Unit), an attitude sensor, an image acquisition device, and other functional devices. In this embodiment, the gyroscope in the IMU is used to monitor the rotational motion of the head, the accelerometer can measure the linear motion of the head, such as movement in the four directions of forward, backward, left, and right, and the magnetometer is used to measure the orientation of the head in a magnetic field to help determine the absolute orientation of the device; the attitude sensor is used to monitor the three-dimensional attitude of the device, including pitch, yaw, and roll angle changes, and the data collected by the attitude sensor and the data collected by the IMU can be combined to provide more comprehensive head motion information; the eye-tracking device is used to monitor eye movements, capture the user's gaze point and the position of the inner and outer corners of the eyes relative to the eye-tracking device, and the eye-tracking device includes a camera and an infrared light source, and may also include other modules that can achieve the above functions, which is not limited in this embodiment; the image acquisition device is used to monitor the position information of the handle and can calculate the path from the user's gaze point to the hand, and the image acquisition device can be an infrared camera or other image acquisition devices, which is not limited in this embodiment.

[0067] In this embodiment, as Figure 2 As shown, the posture sensor and IMU are located inside the wearable device; the eye-tracking device can be located at the lens barrel of the wearable device, or at other locations where eye images can be captured; such as Figure 3 As shown, a communication connection is established between the controller and the wearable device. The controller sends the controller motion information collected by the controller's built-in IMU to the controller of the wearable device, and sends the spatial coordinates of the controller relative to the device.

[0068] In a specific implementation, the wearable device is also equipped with a distance sensor. This sensor detects whether the device is being worn by the user. When power is detected, the distance sensor continuously monitors the distance between the wearable device and the user's face. When the distance is less than a set distance threshold, it indicates that the device is close enough to the face, and the wearable device is considered to be worn and properly adjusted. In this embodiment, the distance sensor can be a proximity sensor, which uses infrared or ultrasonic technology to measure distance; it can also be an optical sensor, which measures distance by emitting and receiving light; other sensors are also possible. This embodiment does not limit the specific device used for the distance sensor.

[0069] It should be noted that once the wearable device has been detected as being worn by the user and adjusted to the correct angle for use, the eye-tracking device acquires the user's eye-tracking image. During image acquisition, the eye-tracking device uses an infrared light source to illuminate the eye area in order to form a clear pupil and corneal reflection point in the image.

[0070] Understandably, the eye-tracking images captured by the eye-tracking device undergo preprocessing, including brightness adjustment, contrast enhancement, and noise removal. Feature extraction is performed on the preprocessed eye-tracking images, and distance and angle calculations are performed to obtain the user's current gaze point and current gaze parameters. In this embodiment, the current gaze parameters include, but are not limited to, the distance from the eye-tracking device to the user's inner and outer corners of the eyes, the distance between the inner and outer corners of the eyes, the distance between the eye-tracking device and the midpoint of the line connecting the inner and outer corners of the eyes, the distance between the eye-tracking device and the lower endpoint of the positioning line, and the angle formed between these lines. The positioning line is a line of a preset length that is perpendicular to the line connecting the inner and outer corners of the eyes and parallel to the eye.

[0071] In one feasible implementation, step S10 may further include steps A11 to A14:

[0072] Step A11: Analyze the eye-tracking images collected by the eye-tracking device to determine the current gaze point and the location of each corner of the eye.

[0073] It should be noted that the eye-tracking images captured by the eye-tracking device undergo preprocessing, followed by feature extraction. The feature extraction process can be as follows: using image processing algorithms, the position and size of the pupil, the location of the inner and outer corners of the eye are identified; reflective points on the cornea are identified, which help determine the direction of eye movement; the positions of the inner and outer corners of the eye are identified to assist in determining the width and direction of the eye. Based on the positional relationship between the pupil center and the corneal reflective points, the pupil-corneal centerline is calculated, and the user's current gaze point can be determined using this line.

[0074] Step A12: Determine multiple auxiliary positioning points and their locations based on the positions of each corner of the eye.

[0075] It should be noted that, based on the location of each corner of the eye, the inner and outer corners are connected, and the midpoint between the lines connecting the inner and outer corners is used as one of the auxiliary positioning points, and the position of the auxiliary positioning point is obtained. From this midpoint, a line of a preset length is drawn that is perpendicular to the line connecting the inner and outer corners and parallel to the eye. The lower end of this drawn line is used as another auxiliary positioning point, and the position of this auxiliary positioning point is determined. In this embodiment, the preset length can be set to 1 cm, or other lengths can be set; this embodiment does not impose any restrictions on this.

[0076] Step A13: Calculate the angles based on the positions of each eye corner, each auxiliary positioning point, and the position of the eye tracking device to determine multiple eye positioning angles.

[0077] It should be noted that, based on the location of each corner of the eye, the location of each auxiliary positioning point, and the location of the eye tracking device, the distance from the eye tracking device to the user's inner and outer corners of the eye, the distance between the inner and outer corners of the eye, the distance from the eye tracking device to each auxiliary positioning point, and the angle formed between each connecting line are determined, thereby obtaining multiple eye positioning angles.

[0078] In this embodiment, for ease of understanding, it is now referred to as Figure 4 To illustrate, Figure 4 For the user's left eye and the corresponding eye-tracking device, A represents the distance from the eye-tracking device to the outer corner of the eye, B represents the distance from the eye-tracking device to the inner corner of the eye, C represents the distance from the outer corner of the eye to the inner corner of the eye, D represents the distance from the eye-tracking device to the blocking positioning point 1, E represents the distance from the auxiliary positioning point 1 to the auxiliary positioning point 2 (this distance is a preset length), and F represents the distance from the eye-tracking device to the auxiliary positioning point 2. By determining the distances between these points, the angles AC and BC between A and C, DE between D and E, and EF between E and F can be clearly defined. In this embodiment, angles AC, BC, DE, and EF represent multiple eye positioning angles.

[0079] Step A14: Summarize the information of each eye positioning angle to obtain the current gaze parameters.

[0080] It should be noted that the current gaze parameters are obtained based on all eye positioning angles and the distances between each point. Figure 4 For example, the current gaze parameters include, but are not limited to, the distance A from the eye tracking device to the outer corner of the eye, the distance B from the eye tracking device to the inner corner of the eye, the distance C from the outer corner of the eye to the inner corner of the eye, the distance D from the eye tracking device to auxiliary positioning point 1, the distance E from auxiliary positioning point 1 to auxiliary positioning point 2, the distance F from the eye tracking device to auxiliary positioning point 2, the included angles AC, BC, DE, and EF.

[0081] Step S20: Determine the device offset state based on the current gaze parameters.

[0082] It should be noted that the device offset status can be determined based on the current gaze parameters. Each eye positioning angle in the current gaze parameters can be compared with the corresponding historical positioning angle. If all eye positioning angles are consistent with their corresponding historical positioning angles, it means that there is no offset between the wearable device and the user's eyes, and the device offset status is "no offset". If there are eye positioning angles that are inconsistent with their corresponding historical positioning angles, it means that there is an offset between the wearable device and the user's eyes, and the device offset status is "offset".

[0083] Step S30: When the device offset state is no offset, determine the target rendering area based on the current gaze point and device interaction information.

[0084] It should be noted that when the device offset state is "no offset," device interaction information is acquired. In this embodiment, device interaction information includes, but is not limited to, at least one of the following: handle position image acquired by the image acquisition device, audio source interaction information, head motion information acquired by the IMU, and three-dimensional posture information acquired by the posture sensor. In this embodiment, the image acquisition device performs image acquisition in real time. If the handle is not detected in the acquired image, the image is discarded, and the image in which the handle is detected is used as the handle position image.

[0085] Understandably, the controller sends its motion information to the wearable device in real time. This motion information is collected by the controller's built-in IMU. Analyzing this motion information, if the controller is detected moving, the user interacts with the device using the controller; this interaction scenario is a hand interaction scenario. The wearable device analyzes the head motion information collected by the IMU and the 3D posture information collected by the posture sensor in real time. If it detects head movement, this interaction scenario is a head interaction scenario. The wearable device can also anticipate the arrival of sudden sounds through system interaction information, such as in shooting games or ball games. When a sudden sound is about to occur, the interaction scenario is an audio interaction scenario. In this embodiment, a sudden sound can be defined as an audio sound with a rapid increase in volume or a large change in frequency within a short period, including but not limited to gunshots, racket hitting sounds, and screams.

[0086] In the specific implementation, after clarifying the current interaction scenario, the corresponding interaction position or direction is determined based on the device interaction information. The area between the current gaze point and the interaction position / direction is designated as the target rendering area, and enhanced rendering is performed within this area. Specifically: if the current interaction scenario is a hand interaction scenario, the current position of the handle is determined based on the device interaction information, and the path from the current gaze point to the current position of the handle is designated as the target rendering area, which is then enhanced. If the current interaction scenario is an audio interaction scenario, the source direction of a sudden sound is determined based on the device interaction information, and the path from the current gaze point to the source direction is designated as the target rendering area, which is then enhanced. If the current interaction scenario is a head interaction scenario, the head movement direction is determined based on the device interaction information. Using the current gaze point as the source point, the area within a preset angle range at the source point is designated as the target rendering area, and this area is then enhanced. In this embodiment, if multiple interaction scenarios exist simultaneously, the rendering order of each interaction scenario is determined based on the priority of their occurrence. The later an interaction scenario occurs, the earlier its corresponding rendering area is rendered. For example, if the current interaction scenario is a hand movement scenario, the controller is constantly moving, but a sudden sound is about to occur. At this time, an audio interaction scenario also occurs. In this case, the target rendering area corresponding to the audio interaction scenario is rendered first, and then the target rendering area corresponding to the hand movement scenario is rendered more strongly.

[0087] In one feasible implementation, the wearable device further includes an image acquisition device for monitoring the position information of the handle, the handle being communicatively connected to the wearable device; step S30 may further include steps B11 to B13:

[0088] Step B11: When the current interaction scenario is detected to be a hand interaction scenario, determine the handle position image captured by the image acquisition device according to the device interaction information, and determine the current position of the handle according to the handle position image.

[0089] It should be noted that the wearable device's controller analyzes the real-time motion information of the handle sent by the handle to determine whether the handle has moved. If so, it indicates that the current interaction scenario is a hand interaction scenario. At this time, based on the device interaction information, the controller determines the handle position image captured by the image acquisition device, processes the handle position image, extracts key features related to the handle to determine the handle's two-dimensional coordinates in the image, and then converts the handle's two-dimensional coordinates in the image into three-dimensional spatial coordinates relative to the wearable device. In this embodiment, the handle's current position refers to the handle's three-dimensional spatial coordinates relative to the wearable device.

[0090] It is understood that in this embodiment, by combining the handle position image with the handle motion information acquired by the handle's built-in IMU, the handle's movement trajectory can be quickly obtained. When the handle moves from the range that the image acquisition device can capture to outside the range, the handle's built-in IMU can calculate the distance the handle has moved, thereby calculating the handle's three-dimensional spatial coordinates relative to the wearable device.

[0091] Step B12: Determine the rendering path based on the current position of the handle and the current gaze point.

[0092] Step B13: Determine the target rendering area based on the target rendering width and the rendering path.

[0093] It should be noted that by taking either the current gaze point or the current position of the controller as the starting point and the other point as the ending point, the rendering path can be clearly defined. Using this rendering path as the center line of the rendering area, the target rendering area in the hand interaction scenario can be determined according to the set target rendering width and rendering path. In this embodiment, the target rendering width can be adjusted according to the gaze point range of different users, thereby reducing the occurrence of blurry images seen by the user due to untimely rendering.

[0094] It's understandable that one cause of motion sickness when using wearable devices is the blurring and delay in the image when the visual focus shifts to a new location during scene transitions. In ball games like badminton and basketball, besides rendering the user's gaze point, the player typically prepares their hands in advance based on the direction the ball is coming from, as they subconsciously keep their eyes on the ball while hitting it. Therefore, the final gaze point will inevitably settle on the controller. In this case, simply enhancing the rendering of the path from the user's gaze point to the controller in advance will reduce the perceived blurring and delay.

[0095] In one feasible implementation, step S30 may further include steps C11 to C13:

[0096] Step C11: When the current interaction scenario is detected to be an audio interaction scenario, determine the sound source interaction information based on the device interaction information.

[0097] It should be noted that wearable devices can anticipate the arrival of sudden sounds through system interaction information, such as in shooting games or ball games. When a sudden sound is about to occur, the current interaction scenario is determined to be an audio interaction scenario. At this point, the audio source interaction information is determined based on the device's interaction information. In this embodiment, the audio source interaction information includes, but is not limited to, the location, type, and duration of the sudden sound, as well as relevant information about other normal audio sources.

[0098] Step C12: Determine the direction of the sound source interaction based on the sound source interaction information.

[0099] It is understood that the location of the sudden sound source is determined based on the audio source interaction information. In this embodiment, the location of the audio source interaction refers to the location of the sudden sound source. In this embodiment, a sudden sound can be defined as an audio signal that rapidly increases in volume or changes in frequency within a short period of time, including but not limited to gunshots, racket hitting sounds, screams, etc.

[0100] Step C13: Determine the target rendering area based on the sound source interaction location and the current gaze point.

[0101] It should be noted that by taking either the current gaze point or the audio source interaction position as the starting point and the other point as the ending point, the rendering path in the audio interaction scene can be clearly defined. The rendering path is used as the center line of the rendering area, and the target rendering area in the audio interaction scene can be determined according to the set target rendering width and rendering path.

[0102] Understandably, one reason for motion sickness when using wearable devices is the blurring and delay in the image caused by the visual focus shifting to a new location during scene changes. In ball games like badminton and basketball, and shooting games, besides rendering the user's gaze point, if there's a sudden sound, especially in shooting games, the user's next step will inevitably be to look in the direction of the sound. This is because the glasses' system anticipates the sound and determines its direction based on sound analysis. In this case, simply enhancing the rendering of the path from the user's gaze point to the sound source in advance will reduce the perceived blurring and delay.

[0103] Step S40: Perform image enhancement rendering within the target rendering area.

[0104] It should be noted that the wearable device's controller acquires the image data that needs to be rendered within the target rendering area and enhances the rendering of the image data in the target rendering area in advance, thereby ensuring that the user perceives a clearer picture after the visual focus moves and reducing visual latency.

[0105] This embodiment provides a rendering method for a wearable device. The wearable device is equipped with an eye-tracking device to monitor eye movements. The method includes: analyzing eye-tracking images captured by the eye-tracking device to determine the user's current gaze point and current gaze parameters; determining the device offset state based on the current gaze parameters; when the device offset state is no offset, determining a target rendering area based on the current gaze point and device interaction information; and performing image enhancement rendering within the target rendering area. Through this method, the current gaze point and current gaze parameters are determined by the eye-tracking device. Based on the current gaze parameters, it is determined whether there is an offset between the device and the user's eyes. When no offset occurs, the target rendering area is determined by combining the gaze point and device interaction information, and rendering is enhanced within this area. This effectively reduces visual delay and scene offset, further alleviating motion sickness and improving the user experience.

[0106] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 5 In this embodiment, the wearable device further includes an attitude sensor and an inertial measurement unit; in step S30: determining the target rendering area based on the current gaze point and device interaction information includes steps S31 to S33:

[0107] Step S31: When the current interaction scenario is detected to be a head interaction scenario, determine the three-dimensional attitude information collected by the attitude sensor and the head motion information collected by the inertial measurement unit based on the device interaction information.

[0108] It should be noted that wearable devices analyze head motion information collected by the IMU and 3D posture information collected by the posture sensor in real time. Head motion information includes, but is not limited to, acceleration and angular velocity information, while 3D posture information includes, but is not limited to, head pitch and lateral rotation information. If data is generated for both 3D posture and head motion information, it indicates that the user's head is moving, and the current interaction scenario is a head interaction scenario.

[0109] Step S32: Determine the head movement direction based on the three-dimensional posture information and the head movement information.

[0110] It should be noted that the three-dimensional pose information and head motion information are fused together, and the motion direction is predicted based on the fused data to determine the user's head motion direction.

[0111] Step S33: Determine the target rendering area based on the current gaze point, the target rendering angle, and the head movement direction.

[0112] It should be noted that the target rendering angle is set based on the angles at which the user's head can rotate in various directions. In this embodiment, the target rendering angle is set to 60 degrees, but it can also be set to other values ​​as needed. This embodiment does not impose any restrictions on this. In this embodiment, the current gaze point is used as the source point, and the target rendering angle is rendered from the source point towards the direction of head movement. The area formed at this time is the target rendering area.

[0113] Understandably, user head movements follow certain patterns, such as approximately 60 degrees in each direction. Typically, the gaze point moves synchronously with the head. In this embodiment, the direction of head movement can be obtained using an IMU and a posture sensor. When the head begins to move, rendering starts in advance in the direction of head movement, using the gaze point as the source. The rendered area is a 60-degree angle from the source point. As the head position changes, the rendering range also changes, but always maintains a 60-degree angle from the gaze point.

[0114] This embodiment provides a rendering method for wearable devices. When the current interaction scene is detected to be a head-interaction scene, this embodiment determines the 3D posture information collected by the posture sensor and the head motion information collected by the inertial measurement unit based on the device interaction information; determines the head motion direction based on the 3D posture information and the head motion information; and determines the target rendering area based on the current gaze point, the target rendering angle, and the head motion direction. Through this method, the target rendering area in the head-interaction scene can be accurately obtained, laying the foundation for subsequent enhanced rendering.

[0115] Based on the first and / or second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to that in embodiments one and two above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 6 Step S20, the rendering method for the wearable device further includes steps S21 to S22:

[0116] Step S21: Compare each eye positioning angle in the current gaze parameters with the corresponding historical positioning angle.

[0117] Step S22: When there is an inconsistency between the eye positioning angle and the historical positioning angle corresponding to the eye positioning angle, the device offset state is determined to be offset.

[0118] It should be noted that when the user's head turns, if the wearable device does not keep pace with the head's movement, it will shift, and the detected eye positioning angle will change.

[0119] In this embodiment, for ease of understanding, it is referred to as Figure 7 For example, 1. When a user turns their head to the right, if the wearable device cannot keep up with the speed of head rotation, the length of line A will decrease, the length of line B will increase, and the corresponding angle AC will increase (BC will decrease). After the user completes the head rotation, if the distance the wearable device rotates is greater than the head rotation, the distance of line A will increase, the distance of line B will decrease, and the corresponding angle AC will decrease (BC will increase). 2. When the user tilts their head up or down, if the wearable device changes relative to the user's eyes, if the distance between the wearable device's glasses and the eyes is downward, line D will increase, and the angle between DE will decrease. If the distance between the wearable device and the eyes is upward, line D will decrease, and the angle between DE will increase. 3. If the user's movement is not left-right or up-down, but rather irregular movements such as upper left or lower right, then calculations 1 and 2 above will be applied simultaneously.

[0120] Understandably, by comparing each eye positioning angle in the current gaze parameters with its corresponding historical positioning angle, if all eye positioning angles are consistent with their corresponding historical positioning angles, it indicates that there is no offset between the wearable device and the user's eyes, and the device offset state is "no offset." If there is an eye positioning angle that is inconsistent with its corresponding historical positioning angle, it indicates that there is an offset between the wearable device and the user's eyes, and the device offset state is "offset." In this embodiment, the historical positioning angle can be obtained after the wearable device is initially detected to be worn by the user and the angle is adjusted for use; or it can be obtained from the analysis of the user's eye image at a historical time interval from the current time. This embodiment restricts the source of the historical positioning angle, but the historical positioning angle is obtained when there is no offset between the wearable device and the user's eyes.

[0121] In one feasible implementation, after step S22, steps D11 to D13 may also be included:

[0122] Step D11: Determine the device offset and device offset direction based on the eye positioning angle and the historical positioning angle corresponding to the eye positioning angle.

[0123] It should be noted that when the device shifts, an eye positioning angle that is inconsistent with the corresponding historical positioning angle is obtained. The difference between the eye positioning angle and the historical positioning angle is calculated, and the angle difference between the two is the device shift amount. At the same time, the direction of the device shift can be determined.

[0124] Step D12: Determine the rendering adjustment direction based on the device offset direction.

[0125] Step D13: Adjust the image rendering data according to the device offset and the rendering adjustment direction.

[0126] It should be noted that the rendering adjustment direction is the direction opposite to the device offset direction, which involves offsetting the image data rendered on the screen at this time towards the rendering adjustment direction by the device offset amount. In this embodiment, the image data rendered on the screen at this time is the image rendering data.

[0127] Understandably, for ease of understanding, the following is used... Figure 7 For example, when there is a misalignment between the wearable device and the user's eye, for instance... Figure 7 As shown, shifting to the right increases the angle between B and C, causing the user's gaze point to shift to the left. To maintain the user's gaze point in its original position, the entire scene needs to be shifted to the left by the angle of shift between B and C. Figure 7 The angle between line B and line B+ is set to 1 degree. When the angle between line B and line B+ is 1 degree, the device offset is 1 degree, and the scene is shifted 1 degree to the left. This ensures that the image aligns with the user's visual focus, reducing the inconsistency between visual and physical perception and decreasing the probability of motion sickness.

[0128] Step S23: When each eye positioning angle is consistent with the corresponding historical positioning angle, the device offset status is determined to be no offset.

[0129] This embodiment provides a rendering method for wearable devices. This embodiment compares each eye positioning angle in the current gaze parameters with its corresponding historical positioning angle. If an eye positioning angle is inconsistent with its historical counterpart, the device is determined to be in an offset state. If all eye positioning angles are consistent with their corresponding historical counterparts, the device is determined to be in an offset state. This method accurately identifies whether scene offset has occurred, allowing for appropriate adjustments and improved user experience.

[0130] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the rendering method of the wearable device of this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0131] This application also provides a rendering apparatus for a wearable device; please refer to [reference needed]. Figure 8 The rendering apparatus of the wearable device includes:

[0132] The analysis module 10 is used to analyze the eye-tracking images acquired by the eye-tracking device to determine the user's current gaze point and current gaze parameters.

[0133] Processing module 20 is used to determine the device offset state based on the current gaze parameters.

[0134] The processing module 20 is further configured to determine the target rendering area based on the current gaze point and device interaction information when the device offset state is no offset.

[0135] The rendering module 30 is used to perform image enhancement rendering within the target rendering area.

[0136] Optionally, the processing module 20 is further configured to:

[0137] When the current interaction scenario is detected to be a hand interaction scenario, the image acquisition device captures the handle position image based on the device interaction information, and the handle position image is used to determine the current position of the handle; the rendering path is determined based on the current handle position and the current gaze point; and the target rendering area is determined based on the target rendering width and the rendering path.

[0138] Optionally, the processing module 20 is further configured to:

[0139] When the current interaction scenario is detected to be an audio interaction scenario, the sound source interaction information is determined based on the device interaction information; the sound source interaction location is determined based on the sound source interaction information; and the target rendering area is determined based on the sound source interaction location and the current gaze point.

[0140] Optionally, the processing module 20 is further configured to:

[0141] The head movement direction is determined based on the three-dimensional pose information and the head movement information; the target rendering area is determined based on the current gaze point, the target rendering angle, and the head movement direction.

[0142] Optionally, the processing module 20 is further configured to:

[0143] Each eye positioning angle in the current gaze parameters is compared with the corresponding historical positioning angle. If there is a discrepancy between the eye positioning angle and the corresponding historical positioning angle, the device is determined to be in a state of offset. If each eye positioning angle is consistent with the corresponding historical positioning angle, the device is determined to be in a state of no offset.

[0144] Optionally, the processing module 20 is further configured to:

[0145] The device offset and device offset direction are determined based on the eye positioning angle and the corresponding historical positioning angle; the rendering adjustment direction is determined based on the device offset direction; and the image rendering data is rendered and adjusted according to the device offset and the rendering adjustment direction.

[0146] Optionally, the analysis module 10 is further configured to:

[0147] The eye-tracking images acquired by the eye-tracking device are analyzed to determine the current gaze point and the location of each corner of the eye. Based on the location of each corner of the eye, multiple auxiliary positioning points and their locations are determined. Based on the location of each corner of the eye, the location of each auxiliary positioning point, and the location of the eye-tracking device, angle calculations are performed to determine multiple eye positioning angles. The information of each eye positioning angle is summarized to obtain the current gaze parameters.

[0148] The rendering apparatus for wearable devices provided in this application, employing the rendering method for wearable devices in the above embodiments, can solve the technical problem of how to optimize visual delay and scene offset in wearable devices. Compared with the prior art, the beneficial effects of the rendering apparatus for wearable devices provided in this application are the same as those of the rendering method for wearable devices provided in the above embodiments, and other technical features in the rendering apparatus for wearable devices are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0149] This application provides a wearable device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to execute the rendering method of the wearable device in the first embodiment described above.

[0150] The following is for reference. Figure 9 The diagram illustrates a structural schematic suitable for implementing the embodiments of this application. The wearable devices in the embodiments of this application may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 9 The wearable device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0151] like Figure 9 As shown, the wearable device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the wearable device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the wearable device to communicate wirelessly or wiredly with other devices to exchange data. While wearable devices with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0152] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0153] The wearable device provided in this application, employing the rendering method of the wearable device in the above embodiments, can solve the technical problem of how to optimize visual delay and scene offset in wearable devices. Compared with the prior art, the beneficial effects of the wearable device provided in this application are the same as those of the rendering method of the wearable device provided in the above embodiments, and other technical features of the wearable device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0154] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0155] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0156] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the rendering method of the wearable device in the above embodiments.

[0157] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0158] The aforementioned computer-readable storage medium may be included in the wearable device; or it may exist independently and not assembled into the wearable device.

[0159] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a wearable device, cause the wearable device to: analyze the eye-tracking images acquired by the eye-tracking device to determine the user's current gaze point and current gaze parameters; determine the device offset state based on the current gaze parameters; when the device offset state is no offset, determine the target rendering area based on the current gaze point and device interaction information; and perform image enhancement rendering within the target rendering area.

[0160] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0161] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0162] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0163] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the rendering method of the wearable device described above. This addresses the technical problem of optimizing visual latency and scene offset in wearable devices. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the rendering method of the wearable device provided in the above embodiments, and will not be elaborated upon here.

[0164] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the rendering method for wearable devices as described above.

[0165] The computer program product provided in this application can solve the technical problem of how to optimize visual latency and scene shift in wearable devices. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the rendering method for wearable devices provided in the above embodiments, and will not be repeated here.

[0166] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A rendering method for a wearable device, characterized in that, The rendering method for the wearable device is applied to the wearable device, which is equipped with an eye-tracking device for monitoring eye movements. The method includes: The eye-tracking images acquired by the eye-tracking device are analyzed to determine the user's current gaze point and current gaze parameters; Determine the device offset state based on the current gaze parameters; When the device offset state is no offset, the target rendering area is determined based on the current gaze point and device interaction information; Image enhancement rendering is performed within the target rendering area.

2. The method as described in claim 1, wherein the wearable device further comprises an image acquisition device, the image acquisition device being used to monitor the position information of the handle, the handle being communicatively connected to the wearable device; The step of determining the target rendering region based on the current gaze point and device interaction information includes: When the current interaction scenario is detected to be a hand interaction scenario, the image acquisition device captures the handle position image based on the device interaction information, and the current handle position is determined based on the handle position image. The rendering path is determined based on the current position of the handle and the current gaze point. The target rendering area is determined based on the target rendering width and the rendering path.

3. The method as described in claim 1, characterized in that, The step of determining the target rendering region based on the current gaze point and device interaction information includes: When the current interaction scenario is detected to be an audio interaction scenario, the audio source interaction information is determined based on the device interaction information; The direction of the sound source interaction is determined based on the sound source interaction information. The target rendering area is determined based on the sound source interaction location and the current gaze point.

4. The method as described in claim 1, characterized in that, The wearable device also includes an attitude sensor and an inertial measurement unit; The step of determining the target rendering region based on the current gaze point and device interaction information includes: When the current interaction scenario is detected to be a head interaction scenario, the three-dimensional attitude information collected by the attitude sensor and the head motion information collected by the inertial measurement unit are determined based on the device interaction information. The direction of head movement is determined based on the three-dimensional posture information and the head movement information; The target rendering area is determined based on the current gaze point, the target rendering angle, and the head movement direction.

5. The method as described in claim 1, characterized in that, The step of determining the device offset state based on the current gaze parameter includes: Compare each eye positioning angle in the current gaze parameters with the corresponding historical positioning angles; When there is a discrepancy between the eye positioning angle and the historical positioning angle corresponding to the eye positioning angle, the device offset state is determined to be an offset. When each eye positioning angle is consistent with the corresponding historical positioning angle, the device offset status is determined to be no offset.

6. The method as described in claim 5, characterized in that, After the step of determining that the device offset state has occurred when there is a discrepancy between the eye positioning angle and the historical positioning angle corresponding to the eye positioning angle, the method further includes: The device offset and device offset direction are determined based on the eye positioning angle and the historical positioning angle corresponding to the eye positioning angle. The rendering adjustment direction is determined based on the device offset direction; The image rendering data is adjusted according to the device offset and the rendering adjustment direction.

7. The method according to any one of claims 1 to 6, characterized in that, The step of analyzing the eye-tracking images acquired by the eye-tracking device to determine the user's current gaze point and current gaze parameters includes: The eye-tracking images acquired by the eye-tracking device are analyzed to determine the current gaze point and the location of each corner of the eye. Based on the location of each corner of the eye, multiple auxiliary positioning points and the location of each auxiliary positioning point are determined; Angle calculations are performed based on the location of each eye corner, the location of each auxiliary positioning point, and the location of the eye tracking device to determine multiple eye positioning angles; Information from each eye positioning angle is collected to obtain the current gaze parameters.

8. A rendering apparatus for a wearable device, characterized in that, The rendering device for the wearable device includes: The analysis module is used to analyze the eye-tracking images acquired by the eye-tracking device to determine the user's current gaze point and current gaze parameters; The processing module is used to determine the device offset state based on the current gaze parameters; The processing module is further configured to determine the target rendering area based on the current gaze point and device interaction information when the device offset state is no offset. The rendering module is used to perform image enhancement rendering within the target rendering area.

9. A wearable device, characterized in that, The wearable device includes: a memory, a processor, and a rendering program for the wearable device stored in the memory and executable on the processor, wherein the rendering program for the wearable device is configured to implement the rendering method for the wearable device as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a rendering program for a wearable device, which, when executed by a processor, implements the rendering method for a wearable device as described in any one of claims 1 to 7.

Images