Display method and device and electronic equipment

Abstract

The invention provides a display method and device and electronic equipment, the method is applied to head-mounted display equipment, a display screen of the head-mounted display equipment comprises a first display screen and a second display screen, the method comprises the steps that the contour of a target three-dimensional object is extracted, the contour is formed based on a preset wireframe color and a preset wireframe thickness, the target three-dimensional object is a three-dimensional object which is determined based on eye movement data and watched by a user in a virtual reality VR scene, and the eye movement data is eye movement data of the user wearing the head-mounted display device; the contour is displayed on the first display screen, the target three-dimensional object is displayed on the second display screen, the first display screen corresponds to non-amblyopia eyes of the user, and the second display screen corresponds to amblyopia eyes of the user. And the training effect on amblyopia eyes is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of display, in particular to a display method, device and electronic equipment. BACKGROUND

[0002] Amblyopia, also known as "lazy eye", is a condition in which a user has poor vision in one eye due to certain reasons during visual development, and in some cases, the vision cannot reach a normal level even after correction (such as wearing glasses). Amblyopia is usually caused by abnormal visual experience or input, which causes the brain to prefer to use one eye and ignore the image of the other eye.

[0003] However, to train the amblyopic eye, the contrast, brightness, saturation and other display parameters of the image seen by the amblyopic eye are generally enhanced to strengthen the visual stimulation of the amblyopic eye, which easily causes visual discomfort of the user and leads to poor training effect on the amblyopic eye. SUMMARY

[0004] The embodiments of the present application provide a display method, device and electronic equipment to solve the problem of poor training effect on the amblyopic eye.

[0005] To solve the above technical problems, the present application is implemented as follows:

[0006] In a first aspect, the embodiments of the present application provide a display method, which comprises:

[0007] extracting a contour of a target three-dimensional object, the contour being formed based on a preset wireframe color and a preset wireframe thickness, wherein the target three-dimensional object is a three-dimensional object gazed at by a user in a virtual reality (VR) scene based on eye movement data, and the eye movement data is eye movement data of the user wearing a head-mounted display device;

[0008] displaying the contour on the first display screen and displaying the target three-dimensional object on the second display screen, the first display screen corresponding to a non-amblyopic eye of the user, and the second display screen corresponding to an amblyopic eye of the user.

[0009] In a second aspect, the embodiments of the present application provide a VR system, comprising a controller and a head-mounted display device, and the display screen of the head-mounted display device comprises a first display screen and a second display screen;

[0010] The controller is configured to:

[0011] extract a contour of a target three-dimensional object, the contour being formed based on a preset wireframe color and a preset wireframe thickness, wherein the target three-dimensional object is a three-dimensional object fixated by a user in a virtual reality (VR) scene based on eye movement data of the user wearing the head-mounted display device;

[0012] The head-mounted display device is configured to display the contour on the first display screen and the target three-dimensional object on the second display screen, the first display screen corresponding to a non-weak eye of the user, and the second display screen corresponding to a weak eye of the user.

[0013] In a third aspect, an embodiment of the present application provides a display device, applied to a head-mounted display device, a display screen of the head-mounted display device including a first display screen and a second display screen, and the device including:

[0014] a first extraction module configured to extract a contour of a target three-dimensional object, the contour being formed based on a preset wireframe color and a preset wireframe thickness, wherein the target three-dimensional object is a three-dimensional object fixated by a user in a virtual reality (VR) scene based on eye movement data of the user wearing the head-mounted display device;

[0015] a first display module configured to display the contour on the first display screen and the target three-dimensional object on the second display screen, the first display screen corresponding to a non-weak eye of the user, and the second display screen corresponding to a weak eye of the user.

[0016] In a fourth aspect, an embodiment of the present application provides an electronic device, which can be a head-mounted display device, a display screen of the head-mounted display device including a first display screen and a second display screen, and the electronic device including a transceiver and a processor,

[0017] The processor is configured to:

[0018] extract a contour of a target three-dimensional object, the contour being formed based on a preset wireframe color and a preset wireframe thickness, wherein the target three-dimensional object is a three-dimensional object fixated by a user in a virtual reality (VR) scene based on eye movement data of the user wearing the head-mounted display device;

[0019] display the contour on the first display screen and the target three-dimensional object on the second display screen, the first display screen corresponding to a non-weak eye of the user, and the second display screen corresponding to a weak eye of the user.

[0020] In a fifth aspect, an electronic device is provided, which includes a processor, a memory, and a program stored in the memory and executable on the processor, and when the program is executed by the processor, the steps of the display method in the first aspect are implemented.

[0021] In a sixth aspect, a computer readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the display method in the first aspect are implemented.

[0022] In a seventh aspect, a computer program product is provided, which includes computer instructions, and when the computer instructions are executed by a processor, the steps of the method in the first aspect are implemented.

[0023] In the display method, the user's non-weak eye can view the content displayed on the first display screen, and the user's weak eye can view the content displayed on the second display screen. The target three-dimensional object can be displayed on the second display screen, and the target three-dimensional object is a target three-dimensional object gazed at by the user determined based on the collected eye movement data of the user in the VR scene. In addition, the contour of the two-dimensional image of the target three-dimensional object can be extracted, and the contour of the target three-dimensional object can be displayed on the first display screen, i.e., the contour of the target three-dimensional object is displayed on the first display screen corresponding to the user's non-weak eye, while the target three-dimensional object is displayed on the second display screen corresponding to the user's weak eye, so that split-screen display is realized. In this way, different contents are displayed on the first display screen and the second display screen respectively, and the user's non-weak eye and weak eye can view different contents, so as to train the user's weak eye, reduce the user's visual discomfort caused by adjusting the display parameters, and improve the training effect on the user's weak eye. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0025] Figure 1 is one of the flowcharts of the display method provided by the embodiments of the present application;

[0026] Figure 2 is a principle diagram of the display method provided by the embodiments of the present application;

[0027] Figure 3 is one of the schematic diagrams of extracting the contour of the target three-dimensional object provided by the embodiments of the present application;

[0028] Figure 4 This is one of the display effect diagrams of a display method provided in an embodiment of this application;

[0029] Figure 5 This is a rendering of the fusion of a contour and a target 3D object provided in an embodiment of this application;

[0030] Figure 6 This is a second display effect diagram of a display method provided in an embodiment of this application;

[0031] Figure 7 This is a schematic diagram illustrating the principle of enhancing binocular stereoscopic perception training through a 3D jigsaw puzzle game, as provided in an embodiment of this application.

[0032] Figure 8 This is a schematic diagram of the structure of a display device provided in an embodiment of this application;

[0033] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] See Figure 1 , Figure 1 This is a flowchart of a display method provided in an embodiment of this application, applied to a head-mounted display device. The display screen of the head-mounted display device includes a first display screen and a second display screen, such as... Figure 1 As shown, the display method provided in this embodiment includes the following steps:

[0036] Step 101: Extract the outline of the target 3D object. The outline is formed based on the preset wireframe color and preset wireframe thickness.

[0037] The target 3D object is the 3D object that the user is looking at in the virtual reality (VR) scene, determined based on eye-tracking data. The eye-tracking data is the eye-tracking data of the user wearing the head-mounted display device. It can be understood that the user can wear the head-mounted display device to view in the VR scene, and the eye-tracking data of the user wearing the head-mounted display device can be collected. Based on this eye-tracking data, the target 3D object that the user is focusing on / gazing at in the VR scene can be determined.

[0038] In this embodiment, the outline of the target 3D object can be extracted. It should be understood that the preset wireframe color determines the color of the outline, and the preset wireframe thickness determines the thickness of the outline. Users can preset the wireframe color and preset wireframe thickness in the head-mounted display device; for example, the device can receive preset wireframe color and preset wireframe thickness input by the user in the head-mounted display device.

[0039] Step 102: Display the outline on the first display screen and the target 3D object on the second display screen. The first display screen corresponds to the user's non-impaired eyes, and the second display screen corresponds to the user's impaired eyes.

[0040] Displaying the target 3D object on the second display screen can be understood as projecting the target 3D object onto the second display screen, where a 2D image of the target 3D object after projection is shown. The outline displayed on the first display screen can be understood as the outline of the 2D image of the target 3D object projected onto the first display screen.

[0041] The user's non-impaired eyes can view the content displayed on the first screen, while the user's impaired eyes can view the content displayed on the second screen. A target 3D object can be displayed on the second screen. The target 3D object is determined by collecting the user's eye movement data in the VR scene, identifying the target 3D object being gazed upon. Additionally, the outline of the target 3D object's 2D image can be extracted and displayed on the first screen. That is, the outline of the target 3D object is displayed on the first screen corresponding to the user's non-impaired eyes, while the target 3D object is displayed on the second screen corresponding to the user's impaired eyes, achieving split-screen display. By displaying different content on the first and second screens respectively, the user's non-impaired and impaired eyes can view different content, thus training the user's impaired eyes. This reduces visual discomfort caused by adjusting display parameters and improves the training effect on the user's impaired eyes.

[0042] In some embodiments, the mesh of the target 3D object is composed of multiple triangles, and extracting the contour of the target 3D object includes:

[0043] The target 3D object is projected onto the first display screen to obtain a 2D image;

[0044] For each of the multiple triangles, obtain the coordinates of the three vertices of the triangle projected onto the 2D image;

[0045] Determine the distance from each of the three vertices in a 2D image to its opposite edge;

[0046] For each pixel within a triangle in a two-dimensional image, the distances from each pixel to the three sides of the triangle are determined based on the distances from each of the three vertices to the opposite side, thus obtaining the three distances of the pixel.

[0047] Based on the three distances of the pixel and the preset wireframe thickness, determine whether to draw a preset wireframe color on the pixel; the outline is composed of pixels with preset wireframe colors.

[0048] It should be noted that the mesh of the target 3D object can be understood as a mathematical representation of the surface of the target 3D object. In this embodiment, the display screen is a two-dimensional plane. Projecting the target 3D object onto the first display screen can be understood as the target 3D object being projected as a 2D image on the first display screen. After the triangle of the target 3D object is projected onto the first display screen, each point on the triangle has a corresponding 2D coordinate. The coordinates of the three vertices of the triangle projected onto the 2D image can be understood as the 2D coordinates of the three vertices after being projected onto the first display screen. Then, the distance from each vertex of the triangle in the 2D image after the target 3D object is projected onto the first display screen to the opposite side of that vertex can be calculated. The opposite side of a vertex of the triangle is the side of the triangle that does not include that vertex. In this way, the distance from each vertex of the triangle in the 2D image to the opposite side can be determined based on the distance from each vertex to the opposite side of the triangle, thus obtaining the three distances of each pixel in the triangle in the 2D image. For each pixel within a triangle, using the three distances of that pixel and the preset wireframe thickness, it is determined whether to draw a preset wireframe color for that pixel. By performing the above processing on each triangle of the target 3D object, it can be determined whether each pixel segment of each triangle should be drawn with a preset wireframe color, thereby determining the outline of the target 3D object. The outline is composed of pixels with preset wireframe colors.

[0049] In this embodiment, based on the distance from the pixel to the opposite side and the preset wireframe thickness, it is determined whether to draw a preset wireframe color on the pixel. In this way, it can be ensured that the thickness of the drawn outline meets the preset wireframe thickness and the color of the outline meets the preset wireframe color, so as to meet the requirements for outline drawing.

[0050] In some embodiments, determining whether to draw a preset wireframe color on a pixel based on three distances to the pixel and a preset wireframe thickness includes at least one of the following:

[0051] If the minimum distance among the three distances of a pixel is less than or equal to the preset wireframe thickness, draw the preset wireframe color on the pixel.

[0052] If the smallest of the three distances of a pixel is greater than the preset wireframe thickness, a preset fill color is drawn on the pixel. The preset fill color is different from the preset wireframe color.

[0053] In this embodiment, the outline color is a preset wireframe color, and the line width of the outline is determined by the preset wireframe thickness. For pixels outside the outline (i.e., pixels other than the outline within the triangle), a preset fill color different from the outline color is used to distinguish between the outline and non-outline pixels. This makes the outline stand out from the triangle, thus improving its display effect. Furthermore, it should be noted that the preset wireframe color and the preset fill color can be any different colors to differentiate the outline from other areas. For example, the preset wireframe color can be white, and the preset fill color can be transparent or black, etc.

[0054] In some embodiments, the method further includes:

[0055] Receive the user's first selection input for the first 3D model among multiple 3D models;

[0056] The first 3D model is divided into multiple image fragments;

[0057] Upon receiving a second selection input from the user for a first image fragment among multiple image fragments, the outline of the first image fragment is assembled to the assembly position associated with the first image fragment on the first display screen and the first image fragment is assembled to the assembly position on the second display screen; or, the assembly position associated with the first image fragment is prompted, and based on the user's operation command for the first image fragment, the outline of the first image fragment is moved to the assembly position on the first display screen and the first image fragment is moved to the assembly position on the second display screen.

[0058] In this embodiment, "to be assembled" can also be understood as "to be pieced together." It can be understood that if multiple image fragments of the first 3D model are successfully assembled, then the first 3D model is successfully assembled / pieced together. The first selection input is used to select the model to be assembled, and the second selection input is the fragment selection input during the piece-by-piece process, where the user selects the image fragment to be assembled. In one example, to facilitate user operation, after selecting an image fragment, the image fragment automatically appears in the next part of the first 3D model to be assembled; that is, the image fragment will be displayed at the assembly position associated with the image fragment. Additionally, it should be noted that if the image fragment and its assembly position do not match, resulting in unsuccessful assembly, the user can rotate the image fragment to match the assembly position, matching the image fragment with the edge of the position. In another example, after selecting an image fragment, visual cues can be provided for the assembly position, such as, but not limited to, shadows, highlights, and depth cues. Based on these cues, the user can determine the correct position and orientation of the image fragment and manipulate it to move it to the assembly position to complete the assembly of that image fragment. As an example, the operation command could be a move command, etc.

[0059] In this embodiment, a jigsaw puzzle function is also provided to enhance the user's binocular stereoscopic vision and improve the training effect on the user's amblyopic eyes.

[0060] In some embodiments, assembling the outline of a first image fragment to the assembly position associated with the first image fragment on a first display screen includes:

[0061] Obtain the outline of the first image fragment from the outline of the first 3D model;

[0062] The outline of the first image fragment is displayed at the assembly position on the first display screen.

[0063] It should be understood that the principle of extracting the contour of the first 3D model is similar to that of extracting the contour of the target 3D object, and will not be repeated here. After the user selects the first image fragment, the contour of the first image fragment can be obtained from the contour of the first 3D model. The contour of the first image fragment is displayed at the assembly position on the first display screen, and the first image fragment is assembled into the assembly position on the second display screen to improve the training effect on the user's amblyopic eyes.

[0064] In some embodiments, the method further includes:

[0065] When the outline of the first image fragment is successfully assembled into the position to be assembled on the first display screen, a first prompt message is output and the time when the outline of the first image fragment is successfully assembled is recorded. The first prompt message is used to indicate that the assembly is successful.

[0066] Each time an image fragment is successfully assembled, a notification is sent to the user, which enhances interaction with the user based on user feedback and encouragement. The time when the image fragment is successfully assembled is also recorded for later retrieval.

[0067] In some embodiments, assembling the outline of a first image fragment to the assembly position associated with the first image fragment on a first display screen includes:

[0068] Receive a user's request for help assembling the outline of the first image fragment;

[0069] In response to an assembly request, the outline of the first image fragment is assembled into the position to be assembled, and the number of times the user requests assistance is recorded.

[0070] During the jigsaw puzzle process, users may encounter situations where they are unable to assemble the puzzle successfully. This embodiment provides a help function, allowing users to request assistance in assembling the outlines of image fragments. In response to these requests, the outlines of the first image fragments are assembled into the desired positions on the first display screen, and the number of times the user requests assistance is requested is recorded. Additionally, in response to these requests, the first image fragments can be assembled into the desired positions on the second display screen. As an example, the assembly completion time for each image fragment can also be recorded for later use. As another example, the method may further include: calculating the total assembly time of the first 3D model, the number of times the user requests assistance in assembling the first 3D model, and the completion time of each image fragment of the first 3D model; and adjusting the task difficulty based on the calculated total assembly completion time, number of requests, and completion time of each image fragment of the first 3D model. For example, adjusting the fragment assembly task difficulty allows the user to be provided with multiple 3D models matching the fragment assembly difficulty for subsequent assembly. As an example, based on the total assembly time, the number of requests for assistance, and the completion time of each image fragment of the first 3D model, the user's binocular perception ability can be determined, and the task difficulty can be adjusted accordingly.

[0071] The process of the above method will be specifically described below with some specific embodiments.

[0072] The goal of training a user's amblyopic eye is to improve the vision of the affected eye and promote the development of binocular visual function. The basic principle is to enhance the use of the amblyopic eye and balance the visual abilities between the two eyes. For an object, a normal, non-amblyopic eye sees the object itself, while an amblyopic eye may see the object's outline. This application provides a display method that can improve the effectiveness of training a user's amblyopic eye. For example... Figure 2 As shown, the overall idea of ​​the display method in this embodiment is as follows: First, the outline of the three-dimensional (3D) object being viewed by the user is dynamically extracted. Specifically, the user's eye movement data is dynamically acquired. When the eyes focus on a certain 3D object, the outline of that 3D object is automatically extracted, such as... Figure 3 As shown; then the extracted contour is displayed as an independent entity and the original 3D object in a split-screen display (i.e., dual-screen display), as shown. Figure 4 As shown. Assume the user's left eye is the dominant eye (normal, not amblyopic) and their right eye is the amblyopic eye. In this case... Figure 4The display shows the outline on the left screen (second screen) and the 3D object on the right screen (first screen). The user's right eye, which is amblyopic, receives a stronger visual stimulus than the left eye, effectively inhibiting the development of the dominant eye and stimulating the growth of the amblyopic eye. Although the content seen by the user's left and right eyes through the VR headset (i.e., head-mounted display device) is not completely identical, the merged content still forms a complete 3D object. Figure 5 As shown, the fused 3D effect can ensure that the user's eyes receive sufficient stereoscopic perception stimulation, ensuring that visual function is exercised or even strengthened while amblyopia vision training is being conducted.

[0073] The display method in this embodiment is as follows:

[0074] (1) Identify the target 3D object. Obtain the direction of eye focus using eye-tracking technology. Construct a collision detection ray with the midpoint between the two eyes as the starting point and the direction of eye focus as the direction vector. The first 3D object touched by the ray is the target 3D object that the eye is staring at (focusing / gazing).

[0075] (2) Extract the outline of the current target 3D object. The mesh of the 3D model is composed of triangles, such as... Figure 5 The cube shown has two triangles on each face. First, the 3D object is projected onto a two-dimensional screen coordinate system. Then, pixels are filled into the two triangles to form a facet. The contour extraction process involves finding the edges of each triangle and deleting the rest. The contour extraction scheme used in this embodiment calculates the distance from each pixel to the three sides of the triangle in the GPU, obtaining three distances. Then, pixels with a minimum distance less than the preset wireframe thickness are deleted. The specific implementation process is as follows:

[0076] For example, the coordinates of the three vertices of the triangle projected onto the screen are as follows:

[0077] ;

[0078] The area of ​​the triangle is calculated using the formula for the area of ​​a triangle whose modulus is the cross product:

[0079] ;

[0080] The distances from the three vertices of a triangle to their opposite sides are calculated by multiplying the base by the altitude, and are as follows:

[0081] ;

[0082] ;

[0083] ;

[0084] in, Let be the distance from vertex A to the opposite side BC. Let be the distance from vertex B to the opposite side AC. Let C be the distance from vertex C to the opposite side AB.

[0085] After obtaining the distances from the three vertices to the edges, the engine (e.g., Unity) will automatically interpolate the distances by passing them as parameters from the shader to the pixel shader. Interpolation will be performed from each vertex to the opposite edge within each triangle region. Through interpolation, the distance from each pixel within the triangle to each edge of the triangle can be obtained.

[0086] In the pixel shader, the minimum value of the three distances from each pixel within the triangle to the three sides is calculated. ,if Then, draw a border at the position corresponding to the pixel within the triangle, where... To preset the wireframe width (i.e., preset the wireframe thickness), this application provides an interface for preset wireframe width, allowing users to adjust the value of the preset wireframe width themselves.

[0087] (3) This embodiment of the application is implemented in Unity using shader code, following the approach described in (2) above. An example of pseudocode is as follows:

[0088] 3.1 Attribute Definition:

[0089] Define the lineframe color _LineColor, which defaults to white (1,1,1,1);

[0090] Define the fill color _FillColor, which defaults to transparent (0,0,0,0);

[0091] Define the wireframe thickness _WireFramThickness, ranging from 0 to 800, with a default value of 100;

[0092] Define a material switching option UseDiscard to determine whether to use the discard command to delete fragments outside the wireframe;

[0093] 3.2 Vertex Shader (vert):

[0094] Input: Vertices in the model space;

[0095] Output: the vertex positions in projection space (projectionSpaceVertex) and world space (worldSpacePosition).

[0096] Steps (process):

[0097] Transform the vertex from model space to clip space: projectionSpaceVertex = UnityObjectToClipPos(vertex);

[0098] Transform the vertex from model space to world space: worldSpacePosition = mul(unity_ObjectToWorld, vertex);

[0099] 3.3 Geometry Shader (geom):

[0100] Input: Projected spatial coordinates i of the three vertices[3]

[0101] Output: the distance *dist* from each vertex to the sides of the triangle, and the vertex position in the projection space.

[0102] step:

[0103] Calculate the projected spatial coordinates of each vertex: A, B, C;

[0104] Calculate the area of ​​the triangle: S;

[0105] For each vertex:

[0106] Calculate the distance from a vertex to its opposite edge: dist.xyz = (2S / length(edge)) *projectionSpaceVertex.w; length(edge) represents the length of the edge opposite the vertex.

[0107] Calculate the reciprocal of the projection space (projectionSpaceVertex.w, i.e., the projection space depth): dist.w = 1.0 / projectionSpaceVertex.w;

[0108] Add the calculation results to the triangle stream;

[0109] It should be noted that although only the distance from the vertex of the triangle to the opposite side is calculated here, the result is automatically input into the fragment shader and interpolated to obtain the distance from each fragment to the triangle side as input. This process is all done automatically in Unity.

[0110] 3.4 Frag Shader:

[0111] Input: The distance i.dist from each segment to the edge of the triangle;

[0112] Output: The final pixel color;

[0113] step:

[0114] Calculate the minimum distance from the segment to the edge of the triangle: minDistanceToEdge = min(i.dist[0], min(i.dist[1], i.dist[2])) * i.dist[3];

[0115] If minDistanceToEdge is greater than WireframeThickness:

[0116] If UseDiscard is enabled, the fragment will be discarded;

[0117] Otherwise, return the fill color _FillColor;

[0118] Otherwise, return the lineframe color _LineColor;

[0119] (4) Dynamically adjust the wireframe width and fill color. The effect of customizing the fill color and border thickness (wireframe thickness) is as follows: Figure 6 As shown, users can adjust the VR training system themselves or under the guidance of professionals to meet their customized training needs.

[0120] In addition, embodiments of this application also provide training to enhance binocular stereo perception, such as... Figure 7 As shown, the specific content is as follows:

[0121] Design a 3D jigsaw puzzle game (task) where the user's goal is to assemble a set of 3D puzzle pieces (image fragments) into a complete 3D model. These fragment models contain various shapes. The core principle of the game is to allow the user to assemble the 3D model in a VR system using the aforementioned contour extraction dual-screen display method. This requires the user to accurately judge the relative positions and depths of the puzzle pieces, effectively training their binocular stereoscopic vision. The specific process includes:

[0122] (1) Model selection

[0123] Users select one model (corresponding to the first 3D model) from a series of preset 3D models to complete the puzzle. Each model has different difficulty levels to suit users with varying stereoscopic vision abilities.

[0124] (2) Select puzzle pieces

[0125] After selecting a model, the panel will automatically switch to the model's fragment panel. The selected model is divided into multiple puzzle pieces, each different in shape and size. Each puzzle piece has unique raised and recessed edges to ensure that only the correct stereoscopic vision can match it correctly.

[0126] (3) Stereoscopic visual cues

[0127] The game provides visual cues, such as shadows, highlights, and depth clues, to help users determine the correct position and orientation of puzzle pieces.

[0128] (4) Fragmentation control

[0129] To facilitate user operation, after selecting a puzzle piece, it automatically appears in the next part of the model to be assembled. Users simply need to rotate the puzzle piece and observe it to match its edges. This operation requires precise stereoscopic vision to determine the relative positions of the pieces.

[0130] (5) Feedback and incentives

[0131] Users need to assemble the puzzle pieces step by step into the correct positions. Each time a piece is successfully assembled, the user receives immediate feedback and encouragement, and the time taken to complete the puzzle is recorded.

[0132] (6) System Assistance

[0133] To enhance gameplay, the game will prompt users to request assistance when they are unable to assemble the current fragment. Clicking the system request will automatically complete the assembly, but the number of times the user requests assistance will be recorded. In future games, this data will be used to provide targeted training to help users improve their fine motor skills.

[0134] (7) Statistical score

[0135] Repeat steps (2) to (4) until all puzzle pieces are assembled. Record the user's completion time, number of requests for help, and completion time for each piece. These three parameters will serve as important references for the training effect. Based on the user's improved visual perception, automatically adjust the difficulty of the puzzle assembly task to ensure the training is challenging yet not overly difficult.

[0136] The display method provided in the application embodiments can achieve at least the following effects:

[0137] (1) Innovative dual-screen split display technology

[0138] Simultaneous growth of vision and visual acuity: Related technical solutions often sacrifice stereoscopic perception, focusing on enhancing graphic-level stimulation, such as through contrast, brightness, and saturation. Alternatively, they may employ a more direct approach: only the amblyopic eye can see the target content. This undoubtedly impairs the user's binocular stereoscopic perception ability. Although the user's amblyopic eye recovers after training, their visual perception ability declines. This application's embodiment, based on the dual-screen split-display training principle, solves the above problems, ensuring that the user receives sufficient stereoscopic perception stimulation during training.

[0139] To avoid visual discomfort and enhance training effectiveness: Related technologies, by creating a definite stereoscopic perception, can cause user discomfort, leading to dizziness or even vomiting, and resulting in strong user resistance. The method in this application, while addressing stereoscopic visual perception, allows users to customize the wireframe width and fill color, significantly reducing visual discomfort.

[0140] High content scalability: In addition to the specialized 3D model assembly game proposed in this application embodiment, the dual-screen split-display technology proposed in this application embodiment can be applied to any current VR amblyopia training content. Furthermore, for new training content for amblyopia users, it is not limited to image parameter configuration; only the game content itself needs to be considered, thus enhancing the scalability of the training content.

[0141] Better training results: The embodiments of this application possess the elements of current amblyopia training theory, such as enhancing stereoscopic perception of both eyes, reducing visual discomfort, and high compliance, all of which contribute to improved treatment results.

[0142] (2) Strengthen binocular stereo perception training

[0143] Enhanced Stereoscopic Perception: The specially designed stereoscopic perception training game requires users not only to have a precise understanding of the overall target model, but also to have a refined perception of each individual puzzle piece. This design enhances both the user's rough and fine stereoscopic perception, a crucial visual training element for users with visual impairments. Users can not only practice spatial reasoning but also develop their fine-grained control over three-dimensional objects.

[0144] Promoting binocular visual fusion: Users need binocular coordination to complete overall assembly and precise control of individual pieces. While related technologies allow for binocular coordination, they cannot achieve perceptual fusion. This application's embodiment, by stimulating the user to complete the task, ensures that the less visual eye is not "lazy," prompting the brain to stimulate binocular visual fusion.

[0145] Personalized customization enhances user compliance: Users completing training through game challenges is already a way to increase the enjoyment of training. In this embodiment, the game content provides system assistance when the user is unable to complete the predetermined goals, helping them complete the task and recording the requested information. The recorded data serves as a reference for the next game, helping the user to perform targeted training in areas of high-precision, three-dimensional training that are otherwise impossible to complete, greatly improving user compliance, which is a crucial factor in treatment.

[0146] like Figure 8 As shown, Figure 8 This is a schematic diagram of the structure of a display device 800 provided in an embodiment of this application, applied to a head-mounted display device. The display screen of the head-mounted display device includes a first display screen and a second display screen, such as... Figure 8 As shown, the display device 800 includes:

[0147] The first extraction module 801 is used to extract the outline of the target three-dimensional object. The outline is formed based on the preset wireframe color and preset wireframe thickness. The target three-dimensional object is a three-dimensional object that the user is looking at in a virtual reality (VR) scene, which is determined based on eye-tracking data. The eye-tracking data is the eye-tracking data of the user wearing a head-mounted display device.

[0148] The first display module 802 is used to display an outline on a first display screen and a target three-dimensional object on a second display screen. The first display screen corresponds to the user's non-impaired eyes, and the second display screen corresponds to the user's impaired eyes.

[0149] In some embodiments, the first extraction module 801 includes:

[0150] The projection module is used to project the target three-dimensional object onto the first display screen to obtain a two-dimensional image;

[0151] The acquisition module is used to obtain the coordinates of the three vertices of each triangle projected onto a two-dimensional image for each of multiple triangles;

[0152] The second determining module is used to determine the distance from each of the three vertices in the two-dimensional image to the opposite edge;

[0153] The third determining module is used to determine the distances from each pixel point inside a triangle in a two-dimensional image to the three sides of the triangle based on the distance from each vertex to the opposite side, thus obtaining the three distances of the pixel point.

[0154] The fourth determining module is used to determine whether to draw a preset wireframe color on a pixel based on the three distances of the pixel and the preset wireframe thickness; wherein, the outline is composed of pixels with the preset wireframe color drawn.

[0155] In some embodiments, the fourth determining module includes at least one of the following:

[0156] The first drawing module is used to draw a preset wireframe color on a pixel when the minimum distance among the three distances of the pixel is less than or equal to the preset wireframe thickness.

[0157] The second drawing module is used to draw a preset fill color on a pixel when the minimum distance among the three distances of the pixel is greater than the preset wireframe thickness. The preset fill color is different from the preset wireframe color.

[0158] In some embodiments, the device 800 further includes:

[0159] The first receiving module is used to receive the user's first selection input for the first three-dimensional model among multiple three-dimensional models;

[0160] The partitioning module is used to divide the first 3D model into multiple image fragments;

[0161] An assembly module is configured to, upon receiving a second selection input from a user for a first image fragment among multiple image fragments, assemble the outline of the first image fragment to an assembly position associated with the first image fragment on a first display screen and assemble the first image fragment to the assembly position on a second display screen; or, prompt the user for the assembly position associated with the first image fragment, and based on the user's operation command for the first image fragment, move the outline of the first image fragment to the assembly position on the first display screen and move the first image fragment to the assembly position on the second display screen.

[0162] In some embodiments, assembling the outline of a first image fragment to the assembly position associated with the first image fragment on a first display screen includes:

[0163] Extract the outline of the first image fragment;

[0164] The outline of the first image fragment is displayed at the assembly position on the first display screen.

[0165] In some embodiments, the apparatus further includes:

[0166] The output module is used to output a first prompt message when the outline of the first image fragment is successfully assembled into the assembly position on the first display screen.

[0167] The recording module is used to record the time when the outline of the first image fragment is successfully assembled, and the first prompt message is used to indicate that the assembly is successful.

[0168] In some embodiments, assembling the outline of a first image fragment to the assembly position associated with the first image fragment on a first display screen includes:

[0169] Receive a user's request for help assembling the outline of the first image fragment;

[0170] In response to an assembly request, the outline of the first image fragment is assembled into the position to be assembled, and the number of times the user requests assistance is recorded.

[0171] The display device 800 provided in this embodiment can implement the various processes of the above-described display method embodiments. The technical features correspond one-to-one and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0172] This application also provides a VR system (also known as a VR training system, etc.), including a controller and a head-mounted display device, wherein the display screen of the head-mounted display device includes a first display screen and a second display screen;

[0173] The controller is used for:

[0174] The outline of the target 3D object is extracted. The outline is formed based on the preset wireframe color and preset wireframe thickness. The target 3D object is the 3D object that the user is looking at in the virtual reality (VR) scene, which is determined based on eye-tracking data. The eye-tracking data is the eye-tracking data of the user wearing a head-mounted display device.

[0175] A head-mounted display device for displaying an outline on a first display screen and a target three-dimensional object on a second display screen, the first display screen corresponding to the user's non-impaired eyes and the second display screen corresponding to the user's impaired eyes.

[0176] In some embodiments, the mesh of the target 3D object is composed of multiple triangles, and extracting the contour of the target 3D object includes:

[0177] The target 3D object is projected onto the first display screen to obtain a 2D image;

[0178] For each of the multiple triangles, obtain the coordinates of the three vertices of the triangle projected onto the 2D image;

[0179] Determine the distance from each of the three vertices in a 2D image to its opposite edge;

[0180] For each pixel within a triangle in a two-dimensional image, the distances from each pixel to the three sides of the triangle are determined based on the distances from each of the three vertices to the opposite side, thus obtaining the three distances of the pixel.

[0181] Based on the three distances of the pixel and the preset wireframe thickness, determine whether to draw a preset wireframe color on the pixel; the outline is composed of pixels with preset wireframe colors.

[0182] In some embodiments, determining whether to draw a preset wireframe color on a pixel based on three distances to the pixel and a preset wireframe thickness includes at least one of the following:

[0183] If the minimum distance among the three distances of a pixel is less than or equal to the preset wireframe thickness, draw the preset wireframe color on the pixel.

[0184] If the smallest of the three distances of a pixel is greater than the preset wireframe thickness, a preset fill color is drawn on the pixel. The preset fill color is different from the preset wireframe color.

[0185] In some embodiments, the controller is further configured to:

[0186] Receive the user's first selection input for the first 3D model among multiple 3D models;

[0187] The first 3D model is divided into multiple image fragments;

[0188] Upon receiving a second selection input from the user for a first image fragment among multiple image fragments, the outline of the first image fragment is assembled to the assembly position associated with the first image fragment on the first display screen and the first image fragment is assembled to the assembly position on the second display screen; or, the assembly position associated with the first image fragment is prompted, and based on the user's operation command for the first image fragment, the outline of the first image fragment is moved to the assembly position on the first display screen and the first image fragment is moved to the assembly position on the second display screen.

[0189] In some embodiments, assembling the outline of a first image fragment to the assembly position associated with the first image fragment on a first display screen includes:

[0190] Extract the outline of the first image fragment;

[0191] The outline of the first image fragment is displayed at the assembly position on the first display screen.

[0192] In some embodiments, the controller is further configured to:

[0193] When the outline of the first image fragment is successfully assembled into the position to be assembled on the first display screen, a first prompt message is output and the time when the outline of the first image fragment is successfully assembled is recorded. The first prompt message is used to indicate that the assembly is successful.

[0194] In some embodiments, assembling the outline of a first image fragment to the assembly position associated with the first image fragment on a first display screen includes:

[0195] Receive a user's request for help assembling the outline of the first image fragment;

[0196] In response to an assembly request, the outline of the first image fragment is assembled into the position to be assembled, and the number of times the user requests assistance is recorded.

[0197] The VR system provided in this embodiment can realize each process of the above-described display method embodiments, with one-to-one correspondence of technical features and the same technical effect. To avoid repetition, it will not be described again here.

[0198] This application also provides an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, it implements the various processes of the above-described display method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0199] For details, see Figure 9 This application also provides an electronic device, which is a head-mounted display device. The display screen of the head-mounted display device includes a first display screen and a second display screen. The electronic device includes a bus 901, a transceiver 902, an antenna 903, a bus interface 904, a processor 905, and a memory 906.

[0200] The processor 905 is used for:

[0201] The outline of the target 3D object is extracted. The outline is formed based on the preset wireframe color and preset wireframe thickness. The target 3D object is the 3D object that the user is looking at in the virtual reality (VR) scene, which is determined based on eye-tracking data. The eye-tracking data is the eye-tracking data of the user wearing a head-mounted display device.

[0202] The outline is displayed on a first display screen and the target 3D object is displayed on a second display screen. The first display screen corresponds to the user's non-impaired eyes, and the second display screen corresponds to the user's impaired eyes.

[0203] exist Figure 9 In this document, a bus architecture (represented by bus 901) is used. Bus 901 can include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 905 and memory represented by memory 906. Bus 901 can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 904 provides an interface between bus 901 and transceiver 902. Transceiver 902 can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 905 is transmitted over a wireless medium via antenna 903, which further receives data and transmits it to processor 905.

[0204] Processor 905 manages bus 901 and general processing, and also provides various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. Memory 906 can be used to store data used by processor 905 during operation.

[0205] Optionally, the processor 905 can be a CPU, ASIC, FPGA, or CPLD.

[0206] The processor 905 of the electronic device provided in this embodiment can implement each process of the above-described display method embodiments, with one-to-one correspondence of technical features and the same technical effect. To avoid repetition, it will not be described again here.

[0207] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-described display method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0208] This application provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, they implement the various processes of the method described in the embodiment. The technical features are one-to-one and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0209] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0210] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this application.

[0211] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A display method, characterized in that, Applied to a head-mounted display device, wherein the display screen of the head-mounted display device includes a first display screen and a second display screen, the method includes: The outline of the target 3D object is extracted. The outline is formed based on a preset wireframe color and a preset wireframe thickness. The target 3D object is a 3D object that the user is looking at in a virtual reality (VR) scene, determined based on eye-tracking data. The eye-tracking data is the eye-tracking data of the user wearing the head-mounted display device. The outline is displayed on a first display screen and the target three-dimensional object is displayed on a second display screen, the first display screen corresponding to the user's non-affected eyes and the second display screen corresponding to the user's amblyopic eyes.

2. The method according to claim 1, characterized in that, The mesh of the target 3D object is composed of multiple triangles, and the extraction of the contour of the target 3D object includes: The target three-dimensional object is projected onto the first display screen to obtain a two-dimensional image; For each of the plurality of triangles, obtain the coordinates of the three vertices of the triangle projected onto the two-dimensional image; Determine the distance from each of the three vertices in the two-dimensional image to the opposite edge; For each pixel within the triangle in the two-dimensional image, the distances from each pixel to the three sides of the triangle are determined based on the distance from each vertex to its opposite side, thus obtaining the three distances of the pixel. Based on the three distances of the pixel and the preset wireframe thickness, it is determined whether to draw the preset wireframe color on the pixel; wherein, the outline is composed of pixels that are drawn with the preset wireframe color.

3. The method according to claim 2, characterized in that, The step of determining whether to draw the preset wireframe color on the pixel based on the three distances of the pixel and the preset wireframe thickness includes at least one of the following: If the minimum distance among the three distances of the pixel is less than or equal to the preset wireframe thickness, the preset wireframe color is drawn on the pixel. If the minimum distance among the three distances of the pixel is greater than the preset wireframe thickness, a preset fill color is drawn on the pixel, and the preset fill color is different from the preset wireframe color.

4. The method according to claim 1, characterized in that, The method further includes: Receive the user's first selection input for a first 3D model among multiple 3D models; The first 3D model is divided into multiple image fragments; Upon receiving a second selection input from the user for a first image fragment among the plurality of image fragments, the outline of the first image fragment is assembled to the assembly position associated with the first image fragment on the first display screen, and the first image fragment is assembled to the assembly position on the second display screen; or, a prompt is given for the assembly position associated with the first image fragment, and based on the user's operation command for the first image fragment, the outline of the first image fragment is moved to the assembly position on the first display screen, and the first image fragment is moved to the assembly position on the second display screen.

5. The method according to claim 4, characterized in that, The method further includes: When the outline of the first image fragment is successfully assembled to the assembly position on the first display screen, a first prompt message is output and the time when the outline of the first image fragment is successfully assembled is recorded. The first prompt message is used to indicate that the assembly is successful.

6. The method according to claim 4, characterized in that, Assembling the outline of the first image fragment to the assembly position associated with the first image fragment on the first display screen includes: Receive the user's request for help in assembling the outline of the first image fragment; In response to the assembly request, the outline of the first image fragment is assembled to the position to be assembled, and the number of times the user requests assistance is recorded.

7. A VR system, characterized in that, It includes a controller and a head-mounted display device, wherein the display screen of the head-mounted display device includes a first display screen and a second display screen; The controller is used for: The outline of the target 3D object is extracted. The outline is formed based on a preset wireframe color and a preset wireframe thickness. The target 3D object is a 3D object that the user is looking at in a virtual reality (VR) scene, determined based on eye-tracking data. The eye-tracking data is the eye-tracking data of the user wearing the head-mounted display device. The head-mounted display device is used to display the outline on a first display screen and the target three-dimensional object on a second display screen, wherein the first display screen corresponds to the user's non-affected eye and the second display screen corresponds to the user's amblyopic eye.

8. A display device, characterized in that, Applied to a head-mounted display device, the display screen of the head-mounted display device including a first display screen and a second display screen, the device comprising: The first extraction module is used to extract the outline of the target three-dimensional object. The outline is formed based on a preset wireframe color and a preset wireframe thickness. The target three-dimensional object is a three-dimensional object that the user is looking at in a virtual reality (VR) scene, determined based on eye-tracking data. The eye-tracking data is the eye-tracking data of the user wearing the head-mounted display device. A first display module is configured to display the outline on a first display screen and the target three-dimensional object on a second display screen, wherein the first display screen corresponds to the user's non-affected eyes and the second display screen corresponds to the user's amblyopic eyes.

9. An electronic device, characterized in that, The electronic device is a head-mounted display device, the display screen of which includes a first display screen and a second display screen. The electronic device includes a transceiver and a processor. The processor is used for: The outline of the target 3D object is extracted. The outline is formed based on a preset wireframe color and a preset wireframe thickness. The target 3D object is a 3D object that the user is looking at in a virtual reality (VR) scene, determined based on eye-tracking data. The eye-tracking data is the eye-tracking data of the user wearing the head-mounted display device. The outline is displayed on a first display screen and the target three-dimensional object is displayed on a second display screen, the first display screen corresponding to the user's non-affected eyes and the second display screen corresponding to the user's amblyopic eyes.

10. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method as described in any one of claims 1 to 6.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-6.

12. A computer program product, characterized in that, Includes computer instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-6.

Images