Interaction method and device based on eye movement and gesture, equipment and storage medium

Abstract

The invention provides an interaction method and device based on eye movement and gestures, equipment and a storage medium, and belongs to the technical field of computers. The method comprises the steps that a three-dimensional virtual scene is displayed, the three-dimensional virtual scene comprises a plurality of scattered puzzle blocks, and the plurality of puzzle blocks can be spliced into a whole; based on eye actions of an object, displaying a first puzzle block in the plurality of puzzle blocks in an interactive state, the first puzzle block being a puzzle block watched by the object; and under the condition that the first puzzle block is in an interactive state, displaying that the first puzzle block moves in the three-dimensional virtual scene along with a first gesture action on the basis of the first gesture action of the object for the first puzzle block. According to the technical scheme, the puzzle operation can be realized only through the eye action and the gesture action of the object, so that the puzzle operation is more efficient and natural.

Description

Technical Field

[0001] This application relates to the field of computer technology, and particularly to an interaction method, device, equipment and storage medium based on eye movement and gestures. Background Art

[0002] With the development of computer technology, Virtual Reality (VR) technology has emerged. Virtual Reality aims to provide users with an immersive experience. The interaction problem is a key issue in the development of Virtual Reality. The common interaction method is to achieve interaction through handle control. However, the handle also restricts the user's hands, and it is strenuous and easy to break the immersion to operate in the air with the handle for a long time, unable to achieve an immersive effect. Summary of the Invention

[0003] Embodiments of this application provide an interaction method, device, equipment and storage medium based on eye movement and gestures. Only through the eye movements and gesture actions of an object, the puzzle operation can be achieved, making the puzzle operation more efficient and natural. The technical solutions are as follows:

[0004] On the one hand, an interaction method based on eye movement and gestures is provided. The method includes:

[0005] Display a three-dimensional virtual scene, where the three-dimensional virtual scene includes a plurality of scattered puzzle pieces that can be pieced together into a whole;

[0006] Based on the eye movement of the object, display the first puzzle piece among the plurality of puzzle pieces as an interactive state, where the first puzzle piece is the puzzle piece gazed at by the object;

[0007] When the first puzzle piece is in the interactive state, based on the first gesture action of the object on the first puzzle piece, display the first puzzle piece moving in the three-dimensional virtual scene following the first gesture action.

[0008] On the other hand, an interaction device based on eye movement and gestures is provided. The device includes:

[0009] A first display module for displaying a three-dimensional virtual scene, where the three-dimensional virtual scene includes a plurality of scattered puzzle pieces that can be pieced together into a whole;

[0010] A second display module for displaying the first puzzle piece among the plurality of puzzle pieces as an interactive state based on the eye movement of the object, where the first puzzle piece is the puzzle piece gazed at by the object;

[0011] A moving module, configured to, when the first puzzle piece is in an interactive state, display the first puzzle piece moving in the three-dimensional virtual scene following a first gesture action of the object on the first puzzle piece.

[0012] In some embodiments, the second display module is configured to determine a line-of-sight direction of the object based on an eye movement of the object; obtain the first puzzle piece from the plurality of puzzle pieces based on the line-of-sight direction of the object; and display a virtual cursor on the first puzzle piece, where the virtual cursor is used to indicate that the first puzzle piece is in an interactive state.

[0013] In some embodiments, the second display module is configured to display the virtual cursor on the first puzzle piece when a duration for which the line-of-sight direction of the object points to the first puzzle piece satisfies a first duration threshold.

[0014] In some embodiments, the apparatus further includes:

[0015] A splicing module, configured to, when the first puzzle piece stops moving following the first gesture action, display the first puzzle piece spliced together with the puzzle piece where the second boundary is located, when a distance between the first boundary and the second boundary satisfies a first preset condition and the first boundary coincides with the second boundary, where the first boundary is any boundary on the first puzzle piece, and the second boundary is any boundary on any other puzzle piece except the first puzzle piece.

[0016] In some embodiments, the second boundary is any boundary on a second puzzle piece, and at least one of the first puzzle piece and the second puzzle piece is formed by splicing a plurality of sub-puzzle pieces, where the sub-puzzle piece is an indivisible puzzle piece;

[0017] The splicing module is configured to, if a quantity of sub-puzzle pieces included in any one of the first puzzle piece and the second puzzle piece satisfies a second preset condition, when the first puzzle piece stops moving following the first gesture action, display the first puzzle piece spliced together with the second puzzle piece when a distance between the first boundary and the second boundary satisfies a first preset condition and the first boundary coincides with the second boundary.

[0018] In some embodiments, the apparatus further includes:

[0019] A third display module, configured to, when the first puzzle piece stops moving following the first gesture action, display the first puzzle piece retracting to its initial position, where the initial position is the position of the first puzzle piece before it moves following the first gesture action, provided that the distance between the first boundary and the second boundary meets a first preset condition and the first boundary does not coincide with the second boundary.

[0020] In some embodiments, the device further includes:

[0021] A movement module, further configured to, during the movement of the first puzzle piece following the first gesture action, if the first puzzle piece collides with a second puzzle piece and there is no coincident boundary between the first puzzle piece and the second puzzle piece, display the second puzzle piece being pushed by the first puzzle piece.

[0022] In some embodiments, the device further includes:

[0023] A rotation module, configured to, when the first puzzle piece is in an interactive state, display the first puzzle piece rotating around its own center in the direction indicated by the second gesture action based on the second gesture action of the object on the first puzzle piece.

[0024] In some embodiments, the first puzzle piece is a composite puzzle piece formed by splicing a plurality of sub - puzzle pieces, and each sub - puzzle piece is an inseparable puzzle piece; the device further includes:

[0025] A splitting module, configured to, when the first puzzle piece is in an interactive state, display the first puzzle piece being split into the plurality of sub - puzzle pieces based on the third gesture action of the object on the first puzzle piece.

[0026] In some embodiments, the first gesture action is generated by the first hand of the object, and the first puzzle piece is manipulated by the gesture action of the first hand of the object;

[0027] The second display module is further configured to, during the process of manipulating the first puzzle piece, based on a newly generated eye movement of the object, display a third puzzle piece among the plurality of puzzle pieces as being in an interactive state, where the third puzzle piece is the puzzle piece that the object is gazing at;

[0028] The movement module is further configured to, when the third puzzle piece is in an interactive state, display the linkage interaction result between the first puzzle piece and the third puzzle piece based on the gesture action of the second hand of the object.

[0029] In some embodiments, the device further includes:

[0030] A fourth display module, configured to display first prompt information about the first puzzle piece when the duration that the line of sight direction of the object points to the first puzzle piece meets a second duration threshold and no interaction is performed with the first puzzle piece through a gesture action; the first prompt information is used to indicate the splicing information required for splicing the first puzzle piece.

[0031] In some embodiments, the fourth display module is further configured to display second prompt information when the duration that the line of sight direction of the object points to any puzzle piece does not meet a third duration threshold; the second prompt information is used to indicate the splicing information of any puzzle piece that can be spliced.

[0032] In some embodiments, there is a virtual container in the three-dimensional virtual scene, and the virtual container is used to hold puzzle pieces; the apparatus further includes:

[0033] A fourth display module, configured to display an animation of the first puzzle piece entering the virtual container when the first puzzle piece stops moving following the first gesture action and the first puzzle piece collides with the virtual container.

[0034] In some embodiments, the apparatus further includes:

[0035] A fourth display module, configured to display icons of the puzzle pieces included in the virtual container based on a fourth gesture action of the object on the virtual container; in response to a selection operation on any icon, display the puzzle piece corresponding to the icon in the three-dimensional virtual scene.

[0036] On the other hand, a computer device is provided, which includes a processor and a memory. The memory is used to store at least one segment of computer program, and the at least one segment of computer program is loaded and executed by the processor to implement the eye movement and gesture-based interaction method in the embodiments of the present application.

[0037] On the other hand, a computer-readable storage medium is provided, in which at least one segment of computer program is stored, and the at least one segment of computer program is loaded and executed by a processor to implement the eye movement and gesture-based interaction method as in the embodiments of the present application.

[0038] On the other hand, a computer program product is provided, including a computer program. The computer program is stored in a computer-readable storage medium, and a processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the eye movement and gesture-based interaction method provided in the above aspects or various optional implementation manners of the above aspects.

[0039] The embodiment of the present application provides an interaction method based on eye movement and gestures. In the process of assembling multiple scattered puzzle pieces in a three-dimensional virtual scene into a whole, the required puzzle piece for interaction is selected from the multiple puzzle pieces through the eye movement of the object, and the first puzzle piece required for interaction is displayed in an interactive state to notify the object that subsequent interaction operations can be performed. Then, through the gesture movement of the object, the first puzzle piece is controlled to move in the three-dimensional virtual scene. That is, only through the eye movement and gesture movement of the object, the puzzle operation can be realized, making the puzzle operation more efficient and natural, thus enhancing the interaction experience of the object and making the object feel immersive. Moreover, there is no need to configure a handle, reducing the hardware cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0041] Figure 1 is a schematic diagram of the implementation environment of an interaction method based on eye movement and gestures provided by an embodiment of the present application;

[0042] Figure 2 is a flowchart of an interaction method based on eye movement and gestures provided by an embodiment of the present application;

[0043] Figure 3 is a flowchart of another interaction method based on eye movement and gestures provided by an embodiment of the present application;

[0044] Figure 4 is a schematic diagram of a puzzle piece provided by an embodiment of the present application;

[0045] Figure 5 is a schematic diagram of displaying the first puzzle piece in an interactive state provided by an embodiment of the present application;

[0046] Figure 6 is a schematic diagram of a gesture camera provided by an embodiment of the present application;

[0047] Figure 7 is an effect diagram of a human eye viewing an object provided by an embodiment of the present application;

[0048] Figure 8 is a schematic diagram of the structure of a terminal provided by an embodiment of the present application;

[0049] Figure 9 is a schematic diagram of a first gesture movement provided by an embodiment of the present application;

[0050] Figure 10 It is a schematic diagram of a second gesture action provided according to an embodiment of the present application;

[0051] Figure 11 It is a schematic diagram of a third gesture action provided according to an embodiment of the present application;

[0052] Figure 12 It is a framework diagram of interaction based on eye movement and gesture provided according to an embodiment of the present application;

[0053] Figure 13 It is a schematic diagram of interaction based on eye movement and gesture provided according to an embodiment of the present application;

[0054] Figure 14 It is a block diagram of an interaction device based on eye movement and gesture provided according to an embodiment of the present application;

[0055] Figure 15 It is a block diagram of another interaction device based on eye movement and gesture provided according to an embodiment of the present application;

[0056] Figure 16 It is a block diagram of a terminal provided according to an embodiment of the present application. Detailed implementation manners

[0057] To make the objectives, technical solutions and advantages of the present application clearer, the following will further describe the embodiments of the present application in detail with reference to the accompanying drawings.

[0058] In the present application, terms such as "first" and "second" are used to distinguish the same items or similar items with basically the same functions and effects. It should be understood that there is no logical or temporal dependence relationship among "first", "second", and "nth", nor are the quantity and execution order limited.

[0059] In the present application, the term "at least one" means one or more, and the meaning of "multiple" is two or more.

[0060] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.), and signals involved in the present application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions. For example, the eye movements and gesture actions involved in the present application are obtained under full authorization.

[0061] For the convenience of understanding, the following explains the terms involved in the present application.

[0062] Virtual Reality (VR) technology: Also known as virtual reality or immersive technology, it is a brand-new practical technology developed in the 20th century. Virtual reality technology encompasses computer, electronic information, and simulation technologies. Its basic implementation method mainly relies on computer technology and utilizes and integrates the latest development achievements of various high-tech fields such as 3D graphics technology, multimedia technology, simulation technology, display technology, and servo technology. Virtual reality technology uses devices such as computers to generate a virtual world with a realistic three-dimensional visual, tactile, olfactory, and other sensory experiences, enabling people in the virtual world to have a sense of being on the scene. With the continuous development of social productivity and science and technology, the demand for virtual reality technology in all walks of life is increasing day by day. Virtual reality technology has also made great progress and gradually become a new field of science and technology.

[0063] Three-dimensional puzzle: It is mainly different from traditional two-dimensional plane puzzle games. The game design is usually based on photogrammetry to reconstruct three-dimensional models. The game interaction gameplay usually requires wearing a helmet and handle of virtual reality technology for control. The interaction method based on eye movement and gesture provided in this application can be applied to the game of three-dimensional puzzle.

[0064] The interaction method based on eye movement and gesture provided in the embodiments of this application can be executed by a computer device. In some embodiments, the computer device is a terminal or a server. Taking the computer device as a terminal as an example, the implementation environment of the interaction method based on eye movement and gesture provided in the embodiments of this application is introduced below. Figure 1 It is a schematic diagram of the implementation environment of an interaction method based on eye movement and gesture provided in the embodiments of this application. Refer to Figure 1 , this implementation environment includes a terminal 101 and a server 102. The terminal 101 and the server 102 can be directly or indirectly connected through wired or wireless communication methods, and this application does not limit this here.

[0065] In some embodiments, the gesture-based interaction method provided by the embodiments of the present application can be applied to the XR (Extended Reality) field. The terminal 101 is a smart phone, a tablet computer, a laptop computer, a desktop computer, a VR device, an AR (Augmented Reality) device, an MR (Mixed Reality) device, a smart voice interaction device, a vehicle-mounted terminal, etc., but is not limited thereto. The terminal 101 is installed with and runs an application program that supports virtual reality technology. The application program can be a game application program, a sightseeing application program, a live broadcast application program, etc., and the embodiments of the present application do not limit this. Schematically, the terminal 101 is a terminal used by a user. The terminal 101 can display a virtual scene based on virtual reality technology. The user uses the terminal 101 to interact with the puzzle pieces in the virtual scene. Such interaction can be moving the puzzle pieces, rotating the puzzle pieces, or splitting the puzzle pieces, etc., and the embodiments of the present application do not limit this.

[0066] Those skilled in the art can know that the number of the above terminals can be more or less. For example, the above terminal can be only one, or the above terminals can be dozens or hundreds, or a larger number. The embodiments of the present application do not limit the number and device type of the terminals.

[0067] In some embodiments, the server 102 is an independent physical server, and can also be a server cluster or a distributed system composed of multiple physical servers, and can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), big data, and artificial intelligence platforms. The server 102 is used to provide background services for the application program that supports virtual reality technology. In some embodiments, the server 102 undertakes the main computing work, and the terminal 101 undertakes the secondary computing work; or, the server 102 undertakes the secondary computing work, and the terminal 101 undertakes the main computing work; or, the server 102 and the terminal 101 adopt a distributed computing architecture for collaborative computing.

[0068] Figure 2 is a flowchart of an interaction method based on eye movement and gesture provided by the embodiments of the present application. Refer to Figure 2 , and in the embodiments of the present application, it is described by taking the execution by the terminal as an example. The interaction method based on eye movement and gesture includes the following steps:

[0069] 201. The terminal displays a three-dimensional virtual scene, and the three-dimensional virtual scene includes a plurality of scattered puzzle pieces that can be pieced together into a whole.

[0070] In the embodiments of the present application, the three-dimensional virtual scene can be displayed based on virtual reality technology, and the embodiments of the present application do not limit this. The terminal displays a plurality of puzzle pieces in the three-dimensional virtual scene, and the embodiments of the present application do not limit the number of puzzle pieces. The plurality of puzzle pieces are scattered at different positions in the three-dimensional virtual scene. The puzzle pieces can be three-dimensional puzzle pieces or flat puzzle pieces, and the embodiments of the present application do not limit this. The plurality of puzzle pieces in the three-dimensional virtual scene can be pieced together into a whole. The whole can be a three-dimensional model or an image, and the embodiments of the present application do not limit this.

[0071] 202. The terminal displays the first puzzle piece among the plurality of puzzle pieces in an interactive state based on the object's eye movement, and the first puzzle piece is the puzzle piece that the object is gazing at.

[0072] In the embodiments of the present application, the object refers to the user. The object can interact with each puzzle piece in the three-dimensional virtual scene. Among them, the terminal captures the object's eye movement (eye ball movement). The eye movement can be simply referred to as eye movement. Then, the terminal obtains the first puzzle piece that the object's eyes are gazing at from the plurality of puzzle pieces according to the object's eye movement. The first puzzle piece can be any one of the plurality of puzzle pieces, and the embodiments of the present application do not limit this. The terminal sets the first puzzle piece to an interactive state. Correspondingly, the terminal displays the first puzzle piece in an interactive state to notify the object that it can interact with the first puzzle piece subsequently. In other words, step 202 is to select a certain puzzle piece from the plurality of puzzle pieces according to the object's eye movement for subsequent interaction with the selected puzzle piece. Correspondingly, the interactive state of the puzzle piece can be the selected state of the puzzle piece.

[0073] 203. When the first puzzle piece is in an interactive state, the terminal displays the first puzzle piece moving in the three-dimensional virtual scene following the first gesture action of the object with respect to the first puzzle piece.

[0074] In the embodiments of the present application, when the first puzzle piece is in an interactive state, the terminal captures the object's gesture action. The gesture action can be generated by one hand of the object or two hands of the object, and the embodiments of the present application do not limit this. When the object's gesture action is the first gesture action, the terminal displays the first puzzle piece moving in the three-dimensional virtual scene following the first gesture action. That is, when the object controls the first gesture action to move in the real scene, the terminal displays the first puzzle piece moving following the movement of the first gesture action. Equivalently, the object grabs the puzzle piece through the first gesture action and can change the position of the puzzle piece, so as to achieve the purpose of piecing together the puzzle pieces in the three-dimensional virtual scene. The embodiments of the present application do not limit the first gesture action.

[0075] Among them, the moving direction of the first puzzle piece in the three-dimensional virtual scene is the same as the moving direction of the first gesture action. The moving distance of the first puzzle piece in the three-dimensional virtual scene is positively correlated with the moving distance of the first gesture action. The moving distance of the gesture action can be referred to as the action amplitude of the gesture action.

[0076] An embodiment of the present application provides an interaction method based on eye movement and gesture. During the process of piecing together multiple scattered puzzle pieces in a three-dimensional virtual scene into a whole, through the eye movement of the object, the puzzle piece to be interacted with is selected from the multiple puzzle pieces, and the first puzzle piece to be interacted with is displayed in an interactive state to notify the object that subsequent interaction operations can be performed; then, through the gesture action of the object, the first puzzle piece is controlled to move in the three-dimensional virtual scene; that is, only through the eye movement and gesture action of the object, the puzzle operation can be realized, making the puzzle operation more efficient and natural, thus enhancing the interaction experience of the object and making the object feel immersive; and there is no need to configure a handle, reducing the hardware cost.

[0077] Figure 3 is a flowchart of another interaction method based on eye movement and gesture provided by an embodiment of the present application. Refer to Figure 3 , and in the embodiment of the present application, it is described by taking the execution by the terminal as an example. The interaction method based on eye movement and gesture includes the following steps:

[0078] 301. The terminal displays a three-dimensional virtual scene, which includes multiple scattered puzzle pieces that can be pieced together into a whole.

[0079] In the embodiment of the present application, an application program supporting virtual reality technology can run on the terminal. The application program can provide different styles of puzzles, and the embodiment of the present application does not limit the styles of the puzzles. For example, the application program provides 10 three-dimensional puzzles based on scenic spots and historical sites around the world; the scenic spots and historical sites corresponding to different styles of puzzles are different. The puzzle data provided by the application program can be stored in the server or in the terminal, and the embodiment of the present application does not limit this. The puzzle data can be the number of puzzle pieces in the puzzle, the model structure of each puzzle piece, or the position of each puzzle piece in the three-dimensional virtual scene, etc., and the embodiment of the present application does not limit this. For any puzzle, the puzzle is equivalent to a game level. When the terminal starts the puzzle, the terminal displays multiple puzzle pieces participating in the puzzle in the three-dimensional virtual scene.

[0080] Multiple puzzle pieces are scattered throughout a three-dimensional virtual scene. In the three-dimensional virtual scenes corresponding to puzzle tasks of different difficulties, the dispersion of the multiple puzzle pieces can be different. Optionally, in the three-dimensional virtual scene corresponding to a puzzle task with a lower difficulty, for any puzzle piece, the puzzle piece can be exposed in the three-dimensional virtual scene. Without performing any operation, an object can visually see the puzzle piece. In the three-dimensional virtual scene corresponding to a puzzle task with a higher difficulty, for any puzzle piece, the puzzle piece can also be hidden in the three-dimensional virtual scene. The object cannot visually see the puzzle piece and needs to find the puzzle piece before it can be manipulated.

[0081] Among them, the puzzle piece can be hidden among multiple puzzle pieces. That is to say, multiple puzzle pieces are stacked together, and the puzzle piece is among the stacked multiple puzzle pieces. In other words, the puzzle piece is hidden in a position where the object cannot see it due to the occlusion of other puzzle pieces. The object can move other puzzle pieces that occlude the puzzle piece through a gesture action to find the required puzzle piece. For example, the object can move other puzzle pieces that occlude the puzzle piece one by one through a certain gesture action. Or, the object can also control multiple puzzle pieces stacked together to be tiled in the three-dimensional virtual scene through a certain gesture action. That is to say, through one gesture action, multiple puzzle pieces are controlled simultaneously, so that the multiple puzzle pieces are scattered and tiled, so that the object can see all the originally stacked puzzle pieces clearly, so as to find the required puzzle piece faster. The embodiments of the present application do not limit the gesture actions.

[0082] Or, the puzzle piece can also be hidden on other virtual objects in the three-dimensional virtual scene. Optionally, for any puzzle piece, the terminal can display the puzzle piece on a virtual object of the same color as the puzzle piece. This method confuses the object's line of sight through virtual objects of the same color, appropriately increasing the difficulty of finding the puzzle piece, promoting the object to explore the three-dimensional virtual scene, so that the object is more immersed in it, thereby enhancing the game experience. For example, the terminal displays a green puzzle piece in the grass or leaves in the three-dimensional virtual scene; displays a red puzzle piece in the red flowers.

[0083] Among them, the puzzle piece can be colored and contain multiple colors. Correspondingly, for any puzzle piece, the terminal can obtain the area occupied by at least one color on the surface of the puzzle piece. Then, the terminal obtains a target color from the at least one color. The target color is the color with the largest area among the at least one color. Then, the terminal displays the puzzle piece on other virtual objects of the target color. That is to say, the terminal displays the puzzle piece on other virtual objects of the target color to achieve the effect of hiding the puzzle piece. During the process of displaying the puzzle piece on other virtual objects of the target color, the terminal can orient the surface of the puzzle piece that contains the target color towards the object.

[0084] In some embodiments, the layout (dispersion) of multiple puzzle pieces in a three-dimensional virtual scene can be changed. During the process of an object performing a puzzle task, in addition to the manipulation of the puzzle pieces by the object changing the layout of the multiple puzzle pieces, the terminal can also automatically adjust the layout of the multiple puzzle pieces in the three-dimensional virtual scene according to the progress of the puzzle task. Optionally, when the progress of the puzzle task meets the progress condition, the terminal adjusts the positions of the unassembled puzzle pieces. Among them, the progress condition can be that the volume of the assembled puzzle pieces reaches a volume threshold; or, the proportion of the volume of the assembled puzzle pieces in the overall puzzle corresponding to the puzzle task reaches a preset value, and the embodiments of the present application do not limit this. During the process of adjusting the positions of the unassembled puzzle pieces, for any unassembled puzzle piece, the terminal can hide the puzzle piece in the three-dimensional virtual scene. For the specific method of hiding the puzzle piece, please refer to the above, and it will not be elaborated here. The solution provided by the embodiments of the present application can hide the unassembled puzzle pieces when the puzzle task reaches a certain stage, appropriately increase the task difficulty, promote the object to deeply explore the three-dimensional virtual scene, make the object more immersed in it, and thus improve the game experience.

[0085] Among them, before adjusting the positions of the unassembled puzzle pieces, the terminal can also consider the difficulty of the current puzzle task. Optionally, when the difficulty of the puzzle task does not exceed the preset difficulty, the terminal adjusts the positions of the unassembled puzzle pieces. When the difficulty of the puzzle task exceeds the preset difficulty, the terminal keeps the positions of the unassembled puzzle pieces unchanged. The solution provided by the embodiments of the present application adjusts the positions of the unassembled puzzle pieces when the difficulty of the puzzle task is low, appropriately increases the task difficulty, promotes the object to deeply explore the three-dimensional virtual scene, makes the object more immersed in it, and thus improves the game experience; when the difficulty of the puzzle task is high, keeps the positions of the puzzle pieces unchanged, and avoids the object reducing the game experience due to excessive difficulty.

[0086] In some embodiments, the puzzle pieces in the three-dimensional virtual scene can be static or dynamic, and the embodiments of the present application do not limit this. Static means that the puzzle pieces are stationary in the three-dimensional virtual scene. Dynamic means that the puzzle pieces can automatically move in the three-dimensional virtual scene. For dynamic puzzle pieces, the object can grab and manipulate the puzzle pieces through eye movements and hand movements. For the specific manipulation process, please refer to the subsequent steps.

[0087] For any puzzle piece, the puzzle piece can be an irregular three-dimensional puzzle piece or a regular three-dimensional puzzle piece, and the embodiments of the present application do not limit this. For example, Figure 4 is a schematic diagram of a puzzle piece provided according to an embodiment of the present application. Refer to Figure 4 , Figure 4 in which (a) exemplarily shows a puzzle piece that is an irregular hexahedron; Figure 4Among them, (b) exemplarily shows that a puzzle piece is a regular hexahedron. The puzzle difficulty corresponding to puzzle pieces of different shapes is different. Compared with regular puzzle pieces, the puzzle corresponding to irregular puzzle pieces is more difficult. That is to say, the splicing difficulty of the puzzle piece is directly proportional to the regularity of the puzzle piece. Regularity means that the shape of the puzzle piece is a standard three-dimensional shape. For example, a sphere, a cube, a cylinder, a cone, a frustum of a pyramid, a regular tetrahedron, etc. are all objects with regular shapes. Other puzzle pieces that are not standard three-dimensional shapes are collectively referred to as irregular objects.

[0088] For any puzzle (as a whole), the embodiments of the present application do not limit the number of puzzle pieces participating in the puzzle. For example, the number of puzzle pieces participating in the puzzle can be 25, 100, or 400.

[0089] 302. Based on the eye movement of the object, the terminal displays the first puzzle piece among the multiple puzzle pieces as an interactive state, and the first puzzle piece is the puzzle piece gazed at by the object.

[0090] In the embodiments of the present application, an eye movement sensor and a gesture camera are configured in the terminal. The eye movement sensor is used to capture the eye movement of the object. The gesture camera is used to capture the gesture movement of the object. The terminal obtains the eye movement of the object through the eye movement sensor, and thus determines the first puzzle piece locked by the eye movement from among the multiple puzzle pieces. Then, the terminal displays the first puzzle piece as an interactive state. The embodiments of the present application do not limit the display method of the interactive state. The terminal can represent that the first puzzle piece is in an interactive state with a virtual cursor; or, the terminal can also represent that the first puzzle piece is in an interactive state with a preset brightness; or, the terminal can also represent that the first puzzle piece is in an interactive state with a preset color. Among them, the preset brightness is higher than the brightness of other puzzle pieces that are not in an interactive state. The preset color is different from the colors of other puzzle pieces that are not in an interactive state. For example, the color of the puzzle piece that is not in an interactive state is gray, and the color of the puzzle piece that is in an interactive state is colored.

[0091] In some embodiments, the terminal represents that the first puzzle piece is in an interactive state with a virtual cursor. Accordingly, the process by which the terminal displays the first puzzle piece among multiple puzzle pieces as an interactive state based on the eye movement of the object includes: The terminal determines the line of sight direction of the object based on the eye movement of the object. Then, the terminal obtains the first puzzle piece from among the multiple puzzle pieces based on the line of sight direction of the object. Then, the terminal displays a virtual cursor on the first puzzle piece. The virtual cursor is used to indicate that the first puzzle piece is in an interactive state. The virtual cursor can be a virtual arrow or a virtual plus sign, etc. The embodiments of the present application do not limit the form of the virtual cursor. The solution provided by the embodiments of the present application captures the eye movement of the object, determines the first puzzle piece gazed at by the line of sight direction of the object, and thus displays a virtual cursor on the first puzzle piece to notify the object that the first puzzle piece can be interacted with, facilitating subsequent users to interact with the first puzzle piece in a timely manner through gesture actions and improving the interaction efficiency.

[0092] For example, Figure 5 is a schematic diagram showing the first puzzle piece being displayed as an interactive state according to an embodiment of the present application. Refer to Figure 5 , the terminal displays multiple puzzle pieces such as the first puzzle piece 501, the second puzzle piece 502, the third puzzle piece 503, and the fourth puzzle piece 504 in a three-dimensional virtual scene. The terminal determines that the puzzle piece gazed at by the object is the first puzzle piece 501 according to the eye movement of the object. Then, the terminal displays a virtual cursor 505 on the first puzzle piece 501.

[0093] In some embodiments, the terminal can determine whether to set a puzzle piece as an interactive state according to the duration that the object gazes at the puzzle piece. Accordingly, when the duration that the line of sight direction of the object points to the first puzzle piece meets a first duration threshold, the terminal displays the first puzzle piece as an interactive state. (For example, when the duration that the line of sight direction of the object points to the first puzzle piece meets the first duration threshold, the terminal displays a virtual cursor on the first puzzle piece.) Among them, the terminal can obtain the line of sight direction of the object in real time, thereby detecting in real time the puzzle piece pointed to by the line of sight direction, and further determining the duration that the line of sight direction points to the first puzzle piece. The solution provided by the embodiments of the present application will set the first puzzle piece as an interactive state only when the duration that the object gazes at the puzzle piece meets the first duration threshold, which is more in line with the user's intention, avoids frequently switching the state of the puzzle piece due to the unconscious eye movement of the object, and saves running consumption.

[0094] In some embodiments, when the object gazes at a puzzle piece for a long time and does not perform any manipulation, the terminal can provide the object with a first prompt message. Correspondingly, when the duration for which the object's line of sight points to the first puzzle piece meets a second duration threshold and the object does not interact with the first puzzle piece through a gesture, the terminal displays a first prompt message regarding the first puzzle piece. The first prompt message is used to indicate the splicing information required for splicing the first puzzle piece. The splicing information may include at least one puzzle piece that can be spliced with the first puzzle piece, at least one first splicing position, and at least one second splicing position, etc. The first splicing position refers to the position on the first puzzle piece that can be used for splicing. The second splicing position refers to the position on other virtual objects (such as puzzle pieces, bases) where the first puzzle piece can be connected. The first splicing position and the second puzzle position that can be spliced together can be correspondingly displayed by the same color or the same label. Alternatively, the object can also trigger the terminal to display the first prompt message regarding the first puzzle piece through a gesture, and the embodiments of the present application do not limit the gesture.

[0095] In some embodiments, when the object does not gaze at any puzzle piece for a long time, the terminal can provide the object with a second prompt message. The second prompt message is used to indicate the splicing information of any puzzle piece that can be spliced. Correspondingly, when the duration for which the object's line of sight points to any puzzle piece does not meet a third duration threshold, the terminal displays the second prompt message. In the solution provided by the embodiments of the present application, when the duration for which the object's line of sight points to any puzzle piece does not meet the third duration threshold, it indicates that the object's line of sight has been shifting and has not focused on any puzzle piece, that is, the object has not selected the puzzle piece to be manipulated. In the case of not selecting the puzzle piece to be manipulated for a long time, it is very likely that the object does not know how to splice. In this case, the second prompt message is displayed to guide the object's operation, which is beneficial to improving the interaction efficiency and the gaming experience.

[0096] The terminal is equipped with an eye movement sensor. When determining the direction of a subject's gaze, the terminal uses the eye movement sensor to capture the subject's eye movements, thereby determining the subject's gaze direction. The eye movement sensor is equipped with a circle of cameras on each side, each for capturing the subject's left and right eye movements. This embodiment of the present application does not limit the number of cameras in each circle. For example, each circle may contain nine small LED (Light Emitting Diode) cameras. Each of the left and right sides of the eye movement sensor is equipped with an eye movement chip for calculating the corresponding eye's gaze. When determining the direction of a subject's gaze, the terminal may use a pupil-corneal reflection method to calculate the subject's gaze direction, although this embodiment of the present application does not limit this. Specifically, the terminal uses nine LED cameras to simultaneously illuminate the eyeball, thereby obtaining blink information using an infrared light source. The terminal then uses the cameras to capture images of the light reflected from the cornea and pupil to calculate the pupil center. The optical axis of the eyeball is then determined by connecting the corneal and pupil centers. The terminal then uses the angle between the optical axis and the visual axis to calculate the subject's gaze direction.

[0097] In some embodiments, when the first puzzle piece is a dynamic puzzle piece, when the subject's gaze is directed toward the first puzzle piece, the terminal displays the state of the first puzzle piece as switched from dynamic to static. Static is used to indicate that the first puzzle piece is in an interactive state. The solution provided in the embodiments of the present application stops the dynamic puzzle piece from moving when the subject's gaze is directed toward the dynamic puzzle piece, facilitating subsequent manipulation of the puzzle piece by grabbing the puzzle piece through gestures, thereby enriching the interaction methods and improving interaction efficiency.

[0098] When the first puzzle piece is in an interactive state, the terminal can interact with the first puzzle piece through gestures directed at the first puzzle piece by an object. The interaction can include controlling the movement of the first puzzle piece in the three-dimensional virtual scene through gestures. Accordingly, after the terminal completes step 302, it proceeds to steps 303 and 304. Alternatively, the interaction can include controlling the rotation of the first puzzle piece in the three-dimensional virtual scene through gestures. Accordingly, after the terminal completes step 302, it proceeds to step 305. The interaction can include disassembling the first puzzle piece through gestures. Accordingly, after the terminal completes step 302, it proceeds to step 306.

[0099] 303. When the first puzzle piece is in an interactive state, the terminal displays, based on a first gesture action of the object directed to the first puzzle piece, the movement of the first puzzle piece in the three-dimensional virtual scene following the first gesture action.

[0100] In an embodiment of the present application, a gesture camera is configured on a terminal. The terminal captures the gesture actions of an object through the gesture camera. Among them, the terminal not only captures the posture of the gesture action through the gesture camera, but also can capture the position of the gesture action. When moving the first gesture action for the first puzzle piece, the terminal displays the first puzzle piece moving following the first gesture action.

[0101] For example, Figure 6 is a schematic diagram of a gesture camera provided according to an embodiment of the present application. The terminal is a head-mounted display. Refer to Figure 6 , Figure 6 in which (a) exemplarily shows the position of the gesture camera. The gesture camera is located outside the head-mounted display. That is, 4 cameras outside the head-mounted display (at the positions marked by the circles) constitute the gesture camera for capturing the gesture actions of an object. Figure 6 in which (b) exemplarily shows the coverage ranges of the above 4 cameras. Region 601 is a monocular region covered by only one camera. Regions 602, 603, and 604 are stereo regions covered by two or more cameras. 0 degrees represents the human face or the direction pointed by a ray emitted from the nose.

[0102] Figure 7 is an effect diagram of a human eye viewing an object provided according to an embodiment of the present application. When a human eye views a real object, the focusing distance and the convergence distance are always equal. That is, vergence distance (the line-of-sight convergence distance, which is also the focusing distance) = accommodation distance (the accommodation distance, which is also the convergence distance). Refer to Figure 7 A in. Since the projection distance from the screen displaying the three-dimensional virtual scene to the eye is always fixed, that is, the focusing is unchanged. However, the perception of the picture will make the eyes converge at different distances to produce a depth-of-field three-dimensional effect. Refer to Figure 7 B in. Therefore, the above two distances are often inconsistent. In short, in the natural world, when a human eye focuses and converges on an object, the objects at other distances should be blurred. Refer to Figure 7 C in. In the three-dimensional virtual scene, no matter where the human eye focuses, the images of the objects at other distances are clear. Refer to Figure 7D among these do not conform to the laws of the human eye in nature, so the brain will be confused. Therefore, the terminal can calculate the depth of field of the puzzle pieces in the three-dimensional virtual scene through the dual-sided gesture algorithm, so as to map the moving distance of the gesture action in the real scene to the moving distance of the puzzle pieces in the three-dimensional virtual scene. Among them, the terminal sends different images to the two eyes. The two images are very similar and only have slight differences in the horizontal direction. Then, the terminal can obtain the depth of field of the puzzle pieces through triangulation calculation, and then move the puzzle pieces according to the depth of field of the puzzle pieces.

[0103] To introduce the interaction method in the embodiments of the present application more clearly, the following will be further described with reference to the accompanying drawings. Figure 8 is a schematic structural diagram of a terminal provided according to an embodiment of the present application. Refer to Figure 8 , the terminal includes a display module, an eye movement sensor, and a gesture camera. The display module is used to display a three-dimensional virtual scene. The eye movement sensor is used to capture the eye movements of the object. The gesture camera is used to capture the gesture actions of the object. The terminal obtains the eye movements of the object through the eye movement sensor, so as to determine the puzzle piece that the object's line of sight is looking at. The terminal obtains the gesture actions of the object through the gesture camera, and then interacts with the puzzle piece according to the gesture actions. The terminal displays the interaction result between the object and the puzzle piece through the display module. The terminal can also be configured with a handle to interact with the puzzle pieces in the three-dimensional virtual scene. Correspondingly, a handle camera can be installed on the terminal. The handle camera is used to capture the position of the handle. Then, the terminal displays that the puzzle pieces move in the three-dimensional virtual scene following the movement of the handle.

[0104] In some embodiments, during the movement of the first puzzle piece following the first gesture action, if the first puzzle piece collides with the second puzzle piece and there is no matching boundary between the first puzzle piece and the second puzzle piece, the terminal displays that the second puzzle piece is pushed by the first puzzle piece. The solution provided by the embodiments of the present application enables the stationary puzzle piece to be pushed by the moving puzzle piece, so that the object can achieve the purpose of pushing multiple puzzle pieces at one time by controlling one puzzle piece, without separately controlling the movement of each puzzle piece, thereby improving the interaction efficiency. For example, move the first puzzle piece and the second puzzle piece to the same area at one time.

[0105] Or, during the movement of the first puzzle piece following the first gesture action, if the first puzzle piece collides with the second puzzle piece and there is no matching boundary between the first puzzle piece and the second puzzle piece, the terminal displays that the second puzzle piece blocks the movement of the first puzzle piece. That is, the second puzzle piece is not pushed by the first puzzle piece and remains in its original position without moving.

[0106] Among them, whether the second puzzle piece will be pushed by the first puzzle piece can depend on the volumes of the second puzzle piece and the first puzzle piece. Optionally, if the volume of the second puzzle piece reaches a preset multiple of the volume of the first puzzle piece, when the first puzzle piece collides with the second puzzle piece and there is no matching boundary between the first puzzle piece and the second puzzle piece, the terminal displays that the second puzzle piece is not pushed by the first puzzle piece. If the volume of the second puzzle piece does not reach the preset multiple of the volume of the first puzzle piece, when the first puzzle piece collides with the second puzzle piece and there is no matching boundary between the first puzzle piece and the second puzzle piece, the terminal displays that the second puzzle piece is pushed by the first puzzle piece. The embodiments of the present application do not limit the preset multiple. Or, if the volume of the second puzzle piece is greater than the volume of the first puzzle piece, when the first puzzle piece collides with the second puzzle piece and there is no matching boundary between the first puzzle piece and the second puzzle piece, the terminal displays that the second puzzle piece is not pushed by the first puzzle piece. If the volume of the second puzzle piece is not greater than the volume of the first puzzle piece, when the first puzzle piece collides with the second puzzle piece and there is no matching boundary between the first puzzle piece and the second puzzle piece, the terminal displays that the second puzzle piece is pushed by the first puzzle piece.

[0107] When there is a matching boundary between the first puzzle piece and the second puzzle piece, the terminal detects the distance between the matching boundaries. When the distance between the matching boundaries meets the first preset condition, the terminal displays the first puzzle piece and the second puzzle piece spliced together based on the matching boundary. When the distance between the matching boundaries does not meet the first preset condition, the terminal displays that the first puzzle piece passes by the second puzzle piece and continues to move.

[0108] In some embodiments, the object can manipulate a puzzle piece with each hand respectively. The first gesture action is generated by the object's first hand. The first puzzle piece is manipulated by the gesture action of the object's first hand. Correspondingly, during the process of manipulating the first puzzle piece, the terminal displays the third puzzle piece among the multiple puzzle pieces as an interactive state based on the newly generated eye movement of the object. The third puzzle piece is the puzzle piece that the object is gazing at. When the third puzzle piece is in an interactive state, the terminal displays the linkage interaction result between the first puzzle piece and the third puzzle piece based on the gesture action of the object's second hand. That is to say, the object can control different puzzle pieces to move respectively through the gesture actions of different hands. The linkage interaction result can be that the first puzzle piece and the third puzzle piece move simultaneously, and the moving directions can be the same or different; or, the first puzzle piece moves and the third puzzle piece rotates; or, the first puzzle piece moves and the third puzzle piece is split; or, the first puzzle piece and the third puzzle piece are joined together, etc. The embodiments of the present application do not limit this. The solution provided by the embodiments of the present application can control two puzzle pieces simultaneously, which can not only improve the interaction efficiency, but also ensure the accuracy of the interaction because each puzzle piece is controlled by a corresponding gesture action.

[0109] The embodiments of the present application do not limit the first gesture action. For example, Figure 9 is a schematic diagram of a first gesture action provided according to an embodiment of the present application. Refer to Figure 9 , a first puzzle piece 901 is displayed in a three-dimensional virtual scene. The first gesture action 902 is a gesture action of pinching with the thumb and index finger. When the first puzzle piece 901 is in an interactive state, the terminal displays the first puzzle piece 901 moving in the three-dimensional virtual scene following the first gesture action 902 based on the object's first gesture action 902 for the first puzzle piece 901. That is to say, through the first gesture action, the object can move and rummage through the required puzzle pieces.

[0110] 304. When the first puzzle piece stops moving following the first gesture action, when the distance between the first boundary and the second boundary meets the first preset condition and the first boundary coincides with the second boundary, the terminal displays that the first puzzle piece is joined together with the puzzle piece where the second boundary is located. The first boundary is any boundary on the first puzzle piece, and the second boundary is any boundary on any other puzzle piece except the first puzzle piece.

[0111] In the embodiments of the present application, the boundary can be a boundary line or a boundary surface, and the embodiments of the present application do not limit this. When the puzzle piece is a three-dimensional puzzle piece, the terminal mainly detects whether the boundary surfaces between the puzzle pieces to be joined match. When the puzzle piece is a flat puzzle piece, the terminal mainly detects whether the boundary lines between the puzzle pieces to be joined match. The first preset condition can be that the maximum distance between the first boundary and the second boundary does not exceed the first distance threshold; alternatively, the first preset condition can also be that the proportion of the part of the first boundary that is no more than the second distance threshold from the second boundary in the whole of the first boundary reaches a preset value, etc., and the embodiments of the present application do not limit this. For example, the first preset condition is that 80% of the boundary parts of the first boundary are no more than the second distance threshold from the second boundary. When the first gesture action stops moving, the first puzzle piece also stops moving accordingly. The terminal detects whether there are other puzzle pieces that can be joined around the first puzzle piece. In the case where there are other puzzle pieces that can be joined, the terminal displays the first puzzle piece and the other puzzle piece joined together. The solution provided by the embodiments of the present application can automatically join the two puzzle pieces where the boundaries are located together when the distance between two joinable boundaries meets the first preset condition, achieving an automatic adsorption effect and improving the interaction efficiency.

[0112] In some embodiments, the second boundary is any boundary on the second puzzle piece. At least one of the first puzzle piece and the second puzzle piece is composed of a plurality of sub-puzzle pieces joined together. The sub-puzzle piece is an indivisible puzzle piece. Accordingly, the process by which the terminal displays the first puzzle piece and the puzzle piece where the second boundary is located joined together includes: if the number of sub-puzzle pieces included in any one of the first puzzle piece and the second puzzle piece meets the second preset condition, when the first puzzle piece stops moving following the first gesture action, and when the distance between the first boundary and the second boundary meets the first preset condition and the first boundary and the second boundary match, the terminal displays the first puzzle piece and the second puzzle piece joined together. The second preset condition can be that the number of sub-puzzle pieces included in the puzzle piece reaches a quantity threshold; or, the proportion of the number of sub-puzzle pieces included in the puzzle piece in the total number of sub-puzzle pieces reaches a preset value, and the embodiments of the present application do not limit this.

[0113] Alternatively, if the volume of either the first puzzle piece or the second puzzle piece satisfies a third preset condition, when the first puzzle piece stops moving following a first gesture action, if the distance between the first boundary and the second boundary satisfies a first preset condition and the first boundary coincides with the second boundary, the terminal displays that the first puzzle piece and the second puzzle piece are joined together. The third preset condition may be that the volume of the puzzle piece reaches a volume threshold; or the proportion of the volume of the puzzle piece in the overall puzzle reaches a preset value, which is not limited in the embodiments of the present application. In the solution provided by the embodiments of the present application, when the number of sub-puzzle pieces joined together satisfies a second preset condition, or the volume of the joined puzzle piece satisfies a third condition, once two puzzle pieces that can be joined together are close, the two puzzle pieces can be automatically joined together.

[0114] In the case where the puzzle pieces cannot be automatically joined together, the terminal can also control the first puzzle piece to be joined with other puzzle pieces based on a new gesture action, which is not limited in the embodiments of the present application.

[0115] In some embodiments, when the first puzzle piece cannot be joined to other puzzle pieces, the terminal can restore the first puzzle piece to its initial position. Correspondingly, when the first puzzle piece stops moving following a first gesture action, if the distance between the first boundary and the second boundary satisfies a first preset condition and the first boundary does not coincide with the second boundary, the terminal displays that the first puzzle piece retreats to its initial position. The initial position is the position where the first puzzle piece was located before moving following the first gesture action. The solution provided by the embodiments of the present application restores the first puzzle piece to its initial position when it cannot be joined to other puzzle pieces, enriching the interaction result and attracting users to actively participate in the interaction.

[0116] 305. When the first puzzle piece is in an interactive state, the terminal displays that the first puzzle piece rotates around its own center in the direction indicated by a second gesture action of an object with respect to the first puzzle piece.

[0117] In the embodiments of the present application, when the second gesture action of an object with respect to the first puzzle piece moves when the first puzzle piece is in an interactive state, the terminal displays that the first puzzle piece rotates around its own center in the moving direction of the second gesture action. The solution provided by the embodiments of the present application controls the rotation of the puzzle piece through the second gesture action, enabling the object to see the various surfaces of the puzzle piece, facilitating the joining of the puzzle pieces in the three-dimensional virtual scene, and thus enhancing the user's interaction experience.

[0118] The embodiments of the present application do not limit the second gesture action. For example, Figure 10 is a schematic diagram of a second gesture action provided according to an embodiment of the present application. Refer to Figure 10, a first puzzle piece 1001 is displayed in a three-dimensional virtual scene. A second gesture action moves to the right. When the first puzzle piece 1001 is in an interactive state, based on the second gesture action 1002 of the object on the first puzzle piece 1001, the terminal displays the first puzzle piece 1001 rotating to the right around its own center in the moving direction of the second gesture action 1002.

[0119] 306. When the first puzzle piece is in an interactive state, based on the third gesture action of the object on the first puzzle piece, the terminal displays the first puzzle piece being split into multiple sub-puzzle pieces. [[ID=G4]]

[0120] In the embodiment of the present application, the first puzzle piece is a composite puzzle piece formed by splicing multiple sub-puzzle pieces. Each sub-puzzle piece is an unsplittable puzzle piece. When the first puzzle piece is in an interactive state, when the terminal captures that the gesture action is the third gesture action, the terminal displays the first puzzle piece being split into multiple sub-puzzle pieces. The multiple sub-puzzle pieces obtained by splitting can be scattered nearby irregularly or tiled on the ground of the three-dimensional virtual scene. The embodiment of the present application does not limit this.

[0121] The embodiment of the present application does not limit the third gesture action. For example, Figure 11 is a schematic diagram of a third gesture action provided according to the embodiment of the present application. Refer to Figure 11 , a first puzzle piece 1101 is displayed in a three-dimensional virtual scene. The third gesture action 1102 is a gesture action with five fingers spread apart. When the first puzzle piece 1101 is in an interactive state, based on the third gesture action 1102 of the object on the first puzzle piece 1101, the terminal displays the first puzzle piece 1101 being split into multiple sub-puzzle pieces 1103. That is, through the third gesture action, the object can split (break up) the puzzle piece. Through the third gesture action, the object can also tile the sub-puzzle pieces obtained by splitting on the ground in the three-dimensional virtual scene.

[0122] To more clearly describe the interaction method based on eye movement and gesture provided by the embodiment of the present application, the following further introduces this interaction method with reference to the accompanying drawings. Figure 12 is a framework diagram of interaction based on eye movement and gesture provided according to the embodiment of the present application. Refer to Figure 12, after the user wears the terminal, they enter the jigsaw puzzle game. There are two fisheye cameras, two eye movement sensors, and one gesture camera installed on the terminal. The fisheye cameras are used to obtain the user's position information. The eye movement sensors are used to capture the user's eye movements. The gesture camera is used to capture the user's gesture movements. Then, the terminal sends data such as the user's position information, eye movements, and gesture movements to the server, and the server calculates the interaction results of the puzzle pieces. Then, the terminal receives the interaction results returned by the server and displays them. Among them, the terminal can display the puzzle pieces in the three-dimensional virtual scene at a preset distance in front of the user according to the user's position information.

[0123] In the embodiment of the present application, the terminal can be only used to display the interaction results, and the server is used to process the data, which can reduce the running consumption of the terminal. Figure 13 It is a schematic diagram of interaction based on eye movement and gesture according to the embodiment of the present application. Refer to Figure 13 , the puzzle data provided by the application (jigsaw puzzle game) is stored in the server. After the user wears the terminal, they enter the jigsaw puzzle game. The terminal captures real-time motion data such as the user's eye movements and gesture movements. Among them, the frequency at which the terminal obtains the motion data is 90 Hz per second, and this is not limited in the embodiment of the present application. Then, the terminal sorts out the data format of the motion data. During the process of sorting out the data format, the terminal can remove the bias data in the motion data. Then, the terminal sends the motion data with the sorted format to the server. After receiving the motion data, the server can perform calculations on the motion data and the currently stored puzzle data to obtain new puzzle data. Then, the server encodes the new puzzle data and sends it to the terminal. After receiving the new puzzle data, the terminal parses it and displays the interaction results of the puzzle pieces in the three-dimensional virtual scene according to the parsed puzzle data.

[0124] Among them, different difficulty levels (jigsaw puzzles) can be provided in the jigsaw puzzle game. The user can select the level to challenge through the first gesture action. That is, the terminal displays the start button of the target level among multiple levels as an interactive state based on the user's eye movements. The target level is the level that the user is looking at. When the start button is in an interactive state, the terminal displays multiple puzzle pieces corresponding to the target level in the three-dimensional virtual scene based on the user's first gesture action on the start button. The principle of selecting the level is similar to the principle of manipulating the puzzle pieces in steps 302 to 303, and will not be elaborated here. For any button displayed by the terminal, the above method can be used to trigger it. For example, a hint button is provided in the jigsaw puzzle game. The user can trigger the hint button through the first gesture action to view the connection relationship information between at least two puzzle pieces in the three-dimensional virtual scene, which is conducive to quickly piecing together the puzzle pieces and improving the interaction efficiency.

[0125] During the process of piecing together puzzle pieces, an object can divide the current puzzle task into multiple subtasks. Correspondingly, in response to the division operation of the puzzle task, the terminal determines the multiple subtasks into which the puzzle task is divided. Different subtasks correspond to different scene areas in the three-dimensional virtual scene. Then, for any one of the multiple puzzle pieces, the terminal displays the puzzle piece in the scene area corresponding to the subtask to which the puzzle piece belongs. That is, the terminal divides the multiple puzzle pieces into different scene areas according to the multiple subtasks, so that the object can complete the subtasks one by one. When any subtask is completed, the terminal can save the piecing result corresponding to the subtask, so as to subsequently piece together the piecing results of the multiple subtasks, reducing the piecing difficulty.

[0126] Among them, the terminal can provide multiple division methods for the object. The multiple division methods can include division methods according to position, division methods according to structure, etc., and the embodiments of the present application do not limit this. The terminal can display the above multiple division methods. The division operation can be a selection operation by the object for any one of the division methods.

[0127] Dividing according to position means dividing according to the position area where the puzzle piece is located in the overall puzzle corresponding to the piecing task. For example, the object can divide the overall puzzle corresponding to the puzzle task into 4 parts. The piecing process of each part can be regarded as a subtask. Then, the terminal can display the puzzle pieces corresponding to the 4 subtasks in different scene areas such as the northeast, southeast, northwest, and southwest areas of the three-dimensional virtual scene. The object can piece together (build) the subtasks separately and finally form a complete puzzle.

[0128] Dividing according to structure means dividing according to the structures included in the overall puzzle. Structure refers to the virtual objects presented in the overall puzzle. For example, the overall puzzle includes structures such as the ground, multiple buildings, and cars. The piecing of any one structure can be regarded as a subtask. The terminal displays the puzzle pieces that make up the ground among the multiple puzzle pieces in the same scene area, displays the puzzle pieces that make up the same building in the same scene area, and displays the puzzle pieces that make up the car in the same scene area. The puzzle pieces corresponding to different structures are in different scene areas. The object can piece together different structures separately and finally form a complete puzzle.

[0129] In some embodiments, there are virtual containers in the three-dimensional virtual scene. The virtual containers are used to hold puzzle pieces. If the user finds that the multiple puzzle pieces scattered in the three-dimensional virtual scene are too chaotic, the user can place the multiple puzzle pieces in the virtual containers and then obtain the puzzle pieces from the virtual containers for piecing during the piecing process. Accordingly, when the first puzzle piece stops moving following the first gesture action, in the case where the first puzzle piece collides with the virtual container, the terminal displays an animation of the first puzzle piece entering the virtual container. Then, based on the fourth gesture action of the user with respect to the virtual container, the terminal displays the icons of the puzzle pieces contained in the virtual container. In response to the selection operation on any icon, the terminal displays the puzzle piece corresponding to the icon in the three-dimensional virtual scene. The user can also place the piecing result during the piecing process in the virtual container. The virtual container is equivalent to a "transfer station" that holds the piecing result during the piecing process and prepares for the subsequent piecing process.

[0130] The terminal can also display a leaderboard for the puzzle game. In the leaderboard, the terminal sequentially displays the corresponding users in ascending order of the duration from the first completion of the current puzzle task (level). The leaderboard can be a friend leaderboard or a regional leaderboard, etc., and the embodiments of the present application do not limit this.

[0131] The embodiments of the present application provide an interaction method based on eye movement and gesture. During the process of piecing multiple puzzle pieces scattered in a three-dimensional virtual scene into a whole, through the user's eye movement, the puzzle piece to be interacted with is selected from the multiple puzzle pieces, and the first puzzle piece to be interacted with is displayed in an interactive state to notify the user that subsequent interaction operations can be performed. Then, through the user's gesture action, the first puzzle piece is controlled to move in the three-dimensional virtual scene. That is, only through the user's eye movement and gesture action, the puzzle operation can be realized, making the puzzle operation more efficient and natural, thereby enhancing the user's interaction experience and making the user feel immersive. Moreover, there is no need to configure a gamepad, reducing the hardware cost.

[0132] Figure 14 is a block diagram of an interaction device based on eye movement and gesture according to an embodiment of the present application. The interaction device based on eye movement and gesture is used to execute the steps when the above-mentioned interaction method based on eye movement and gesture is executed. Refer to Figure 14 The interaction device based on eye movement and gesture includes: a first display module 1401, a second display module 1402, and a movement module 1403.

[0133] The first display module 1401 is used to display a three-dimensional virtual scene, which includes multiple scattered puzzle pieces that can be pieced into a whole;

[0134] The second display module 1402 is configured to display the first puzzle piece among a plurality of puzzle pieces in an interactive state based on the eye movement of the object, where the first puzzle piece is the puzzle piece gazed at by the object;

[0135] The movement module 1403 is configured to, when the first puzzle piece is in an interactive state, display the first puzzle piece moving in the three-dimensional virtual scene following the first gesture action based on the first gesture action of the object on the first puzzle piece.

[0136] In some embodiments, Figure 15 is a block diagram of another interactive device based on eye movement and gesture according to an embodiment of the present application. Refer to Figure 15 the second display module 1402 is configured to determine the line-of-sight direction of the object based on the eye movement of the object; obtain the first puzzle piece from the plurality of puzzle pieces based on the line-of-sight direction of the object; and display a virtual cursor on the first puzzle piece, where the virtual cursor is used to indicate that the first puzzle piece is in an interactive state.

[0137] In some embodiments, the second display module 1402 is configured to display a virtual cursor on the first puzzle piece when the duration for which the line-of-sight direction of the object points to the first puzzle piece satisfies a first duration threshold.

[0138] In some embodiments, continue to refer to Figure 15 the device further includes:

[0139] The splicing module 1404 is configured to, when the first puzzle piece stops moving following the first gesture action, display the first puzzle piece spliced together with the puzzle piece where the second boundary is located when the distance between the first boundary and the second boundary satisfies a first preset condition and the first boundary coincides with the second boundary, where the first boundary is any boundary on the first puzzle piece, and the second boundary is any boundary on any other puzzle piece except the first puzzle piece.

[0140] In some embodiments, the second boundary is any boundary on the second puzzle piece, and at least one of the first puzzle piece and the second puzzle piece is composed of a plurality of sub-puzzle pieces, where the sub-puzzle piece is an indivisible puzzle piece;

[0141] Continue to refer to Figure 15 the splicing module 1404 is configured to, if the number of sub-puzzle pieces included in any one of the first puzzle piece and the second puzzle piece satisfies a second preset condition, display the first puzzle piece spliced together with the second puzzle piece when the first puzzle piece stops moving following the first gesture action and the distance between the first boundary and the second boundary satisfies the first preset condition and the first boundary coincides with the second boundary.

[0142] In some embodiments, continue to refer to Figure 15 the device further includes:

[0143] The third display module 1405 is configured to, when the first puzzle piece stops moving following the first gesture action, and when the distance between the first boundary and the second boundary meets a first preset condition and the first boundary does not coincide with the second boundary, display the first puzzle piece retracting to its initial position, where the initial position is the position where the first puzzle piece was located before moving following the first gesture action.

[0144] In some embodiments, referring further to Figure 15 , the moving module is further configured to, during the movement of the first puzzle piece following the first gesture action, if the first puzzle piece collides with the second puzzle piece and there is no coincident boundary between the first puzzle piece and the second puzzle piece, display the second puzzle piece being pushed by the first puzzle piece.

[0145] In some embodiments, referring further to Figure 15 , the apparatus further includes:

[0146] The rotation module 1406 is configured to, when the first puzzle piece is in an interactive state, based on a second gesture action of an object on the first puzzle piece, display the first puzzle piece rotating around its own center in the direction indicated by the second gesture action.

[0147] In some embodiments, the first puzzle piece is a composite puzzle piece formed by splicing a plurality of sub - puzzle pieces, and each sub - puzzle piece is an indivisible puzzle piece; referring further to Figure 15 , the apparatus further includes:

[0148] The splitting module 1407 is configured to, when the first puzzle piece is in an interactive state, based on a third gesture action of an object on the first puzzle piece, display the first puzzle piece being split into a plurality of sub - puzzle pieces.

[0149] In some embodiments, the first gesture action is generated by the first hand of the object, and the first puzzle piece is controlled by the gesture action of the first hand of the object;

[0150] Referring further to Figure 15 , the second display module 1402 is further configured to, during the process of controlling the first puzzle piece, based on a newly generated eye movement of the object, display a third puzzle piece among the plurality of puzzle pieces as an interactive state, where the third puzzle piece is the puzzle piece being gazed at by the object;

[0151] The moving module 1403 is further configured to, when the third puzzle piece is in an interactive state, based on the gesture action of the second hand of the object, display the linkage interaction result between the first puzzle piece and the third puzzle piece.

[0152] In some embodiments, referring further to Figure 15 , the apparatus further includes:

[0153] A fourth display module 1408, configured to display a first prompt message regarding a first puzzle piece when the duration for which the line of sight direction of an object points to the first puzzle piece meets a second duration threshold and no interaction is performed with the first puzzle piece through a gesture action; the first prompt message is used to indicate the splicing information required for splicing the first puzzle piece. [[ID=②]] [[ID=③]]

[0154] [[ID=④]]In some embodiments, the fourth display module 1408 is further configured to display a second prompt message when the duration for which the line of sight direction of the object points to any puzzle piece does not meet a third duration threshold; the second prompt message is used to indicate the splicing information of any puzzle piece that can be spliced. [[ID=⑤]] [[ID=⑥]]

[0155] [[ID=⑦]]In some embodiments, there is a virtual container in the three-dimensional virtual scene, and the virtual container is used to hold puzzle pieces; the apparatus further includes: [[ID=⑧]] [[ID=⑨]]

[0156] [[ID=⑩]]A fourth display module 1408, configured to display an animation of the first puzzle piece entering the virtual container when the first puzzle piece stops moving following a first gesture action and collides with the virtual container. [[ID=⑪]] [[ID=⑫]]

[0157] [[ID=⑬]]In some embodiments, the apparatus further includes: [[ID=⑭]] [[ID=⑮]]

[0158] [[ID=⑯]]A fourth display module 1408, configured to display icons of the puzzle pieces included in the virtual container based on a fourth gesture action of the object on the virtual container; in response to a selection operation on any icon, display the puzzle piece corresponding to the icon in the three-dimensional virtual scene. [[ID=⑰]] [[ID=⑱]]

[0159] [[ID=⑲]]An interaction apparatus based on eye movement and gesture provided by an embodiment of the present application, during the process of splicing a plurality of scattered puzzle pieces in a three-dimensional virtual scene into a whole, selects the puzzle piece to be interacted with from the plurality of puzzle pieces through the eye movement of the object, and displays the first puzzle piece to be interacted with as an interactive state to notify the object that subsequent interaction operations can be performed; then, controls the movement of the first puzzle piece in the three-dimensional virtual scene through the gesture action of the object; that is, only through the eye movement and gesture action of the object, the puzzle operation can be realized, making the puzzle operation more efficient and natural, thereby enhancing the interaction experience of the object and making the object feel immersive; and there is no need to configure a handle, reducing the hardware cost. [[ID=⑳]] [[ID=㉑]]

[0160] It should be noted that when the interactive device based on eye movement and gesture provided in the above embodiments runs an application program, only the division of the above functional modules is used for illustration. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the interactive device based on eye movement and gesture provided in the above embodiments and the embodiments of the interactive method based on eye movement and gesture belong to the same concept. For the specific implementation process, please refer to the method embodiments and will not be elaborated here.

[0161] Figure 16 It is a structural block diagram of a terminal 1600 according to an embodiment of the present application. Generally, the terminal 1600 includes a processor 1601 and a memory 1602.

[0162] The processor 1601 may include one or more processing cores, such as a 4-core processor or an 8-core processor. The processor 1601 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 1601 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1601 may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1601 may further include an AI (Artificial Intelligence) processor, and the AI processor is used to process computational operations related to machine learning.

[0163] The memory 1602 may include one or more computer-readable storage media, and the computer-readable storage media may be non-transitory. The memory 1602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1602 is used to store at least one computer program, and the at least one computer program is used to be executed by the processor 1601 to implement the interactive method based on eye movement and gesture provided in the method embodiments of the present application.

[0164] In some embodiments, the terminal 1600 may further optionally include: a peripheral device interface 1603 and at least one peripheral device. The processor 1601, the memory 1602, and the peripheral device interface 1603 may be connected by a bus or signal lines. Each peripheral device may be connected to the peripheral device interface 1603 through a bus, signal lines, or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 1604, a display screen 1605, a camera assembly 1606, an audio circuit 1607, and a power supply 1608.

[0165] The peripheral device interface 1603 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1601 and the memory 1602. In some embodiments, the processor 1601, the memory 1602, and the peripheral device interface 1603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1601, the memory 1602, and the peripheral device interface 1603 can be implemented on a separate chip or circuit board, and this embodiment does not limit this.

[0166] The radio frequency circuit 1604 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1604 communicates with a communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1604 converts an electrical signal into an electromagnetic signal for transmission, or converts a received electromagnetic signal into an electrical signal. In some embodiments, the radio frequency circuit 1604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and so on. The radio frequency circuit 1604 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: the World Wide Web, a metropolitan area network, an intranet, each generation of mobile communication networks (2G, 3G, 4G, and 5G), a wireless local area network, and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1604 may further include a circuit related to NFC (Near Field Communication), and this application does not limit this.

[0167] The display screen 1605 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1605 is a touch display screen, the display screen 1605 also has the ability to collect touch signals on or above the surface of the display screen 1605. The touch signals can be input as control signals to the processor 1601 for processing. At this time, the display screen 1605 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1605, which is disposed on the front panel of the terminal 1600; in other embodiments, there may be at least two display screens 1605, which are respectively disposed on different surfaces of the terminal 1600 or are in a folding design; in other embodiments, the display screen 1605 may be a flexible display screen, which is disposed on a curved surface or a folding surface of the terminal 1600. Even further, the display screen 1605 can also be set to an irregular non-rectangular shape, that is, an irregular-shaped screen. The display screen 1605 can be prepared using materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0168] The camera module 1606 is used to collect images or videos. In some embodiments, the camera module 1606 includes a front camera and a rear camera. Generally, the front camera is disposed on the front panel of the terminal, and the rear camera is disposed on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth camera, a wide-angle camera, and a telephoto camera respectively, to implement functions such as background blurring by fusing the main camera and the depth camera, panoramic shooting by fusing the main camera and the wide-angle camera, and VR (Virtual Reality) shooting functions or other fused shooting functions. In some embodiments, the camera module 1606 may further include a flash. The flash can be a single-color temperature flash or a two-color temperature flash. A two-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.

[0169] The audio circuit 1607 may include a microphone and a speaker. The microphone is used to collect sound waves of the user and the environment, and convert the sound waves into electrical signals for input to the processor 1601 for processing, or input to the radio frequency circuit 1604 to achieve voice communication. For the purpose of stereo collection or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 1600. The microphone may also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert the electrical signals from the processor 1601 or the radio frequency circuit 1604 into sound waves. The speaker may be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for uses such as ranging. In some embodiments, the audio circuit 1607 may further include a headphone jack.

[0170] The power supply 1608 is used to supply power to each component in the terminal 1600. The power supply 1608 may be alternating current, direct current, a primary battery or a rechargeable battery. When the power supply 1608 includes a rechargeable battery, the rechargeable battery may be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery charged through a wired line, and a wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0171] In some embodiments, the terminal 1600 further includes one or more sensors 1609. The one or more sensors 1609 include but are not limited to: an acceleration sensor 1610, a gyroscope sensor 1611, a pressure sensor 1612, an optical sensor 1613, and a proximity sensor 1614.

[0172] The acceleration sensor 1610 can detect the magnitudes of accelerations on the three coordinate axes of the coordinate system established with the terminal 1600. For example, the acceleration sensor 1610 can be used to detect the components of the gravitational acceleration on the three coordinate axes. The processor 1601 can control the display screen 1605 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 1610. The acceleration sensor 1610 can also be used for collecting game or user's motion data.

[0173] The gyroscope sensor 1611 can detect the body direction and rotation angle of the terminal 1600. The gyroscope sensor 1611 can cooperate with the acceleration sensor 1610 to collect the 3D actions of the user on the terminal 1600. According to the data collected by the gyroscope sensor 1611, the processor 1601 can implement the following functions: motion sensing (such as changing the UI according to the user's tilting operation), image stabilization during shooting, game control, and inertial navigation.

[0174] The pressure sensor 1612 can be disposed on the side frame of the terminal 1600 and / or the lower layer of the display screen 1605. When the pressure sensor 1612 is disposed on the side frame of the terminal 1600, it can detect the holding signal of the user on the terminal 1600, and the processor 1601 can perform left / right hand recognition or quick operation according to the holding signal collected by the pressure sensor 1612. When the pressure sensor 1612 is disposed on the lower layer of the display screen 1605, the processor 1601 can control the operable controls on the UI interface according to the pressure operation of the user on the display screen 1605. The operable controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0175] The optical sensor 1613 is used to collect the ambient light intensity. In one embodiment, the processor 1601 can control the display brightness of the display screen 1605 according to the ambient light intensity collected by the optical sensor 1613. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1605 is increased; when the ambient light intensity is low, the display brightness of the display screen 1605 is decreased. In another embodiment, the processor 1601 can also dynamically adjust the shooting parameters of the camera module 1606 according to the ambient light intensity collected by the optical sensor 1613.

[0176] The proximity sensor 1614, also known as the distance sensor, is usually disposed on the front panel of the terminal 1600. The proximity sensor 1614 is used to collect the distance between the user and the front of the terminal 1600. In one embodiment, when the proximity sensor 1614 detects that the distance between the user and the front of the terminal 1600 is gradually decreasing, the processor 1601 controls the display screen 1605 to switch from the lit state to the off state; when the proximity sensor .1614 detects that the distance between the user and the front of the terminal 1600 is gradually increasing, the processor 1601 controls the display screen 1605 to switch from the off state to the lit state.

[0177] Those skilled in the art can understand that Figure 16 the structure shown in does not constitute a limitation on the terminal 1600, and it may include more or fewer components than shown in the figure, or combine some components, or adopt different component arrangements.

[0178] The embodiments of the present application further provide a computer-readable storage medium, in which at least one segment of computer program is stored. The at least one segment of computer program is loaded and executed by a processor of a computer device to implement the operations performed by the computer device in the above-described interaction method based on eye movement and gestures. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0179] The embodiments of the present application further provide a computer program product, including a computer program stored in a computer-readable storage medium. A processor of a terminal reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the interaction method based on eye movement and gestures provided in the above various alternative implementations.

[0180] Those of ordinary skill in the art can understand that all or part of the steps to implement the above embodiments can be completed by hardware, or can be completed by a program instructing relevant hardware. The program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a magnetic disk, or an optical disc, etc.

[0181] The above are only alternative embodiments of the present application and are not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the protection scope of the present application.

Claims

1. An interaction method based on eye movement and gesture, characterized in that, The method includes: Displaying a three-dimensional virtual scene, which includes a plurality of scattered puzzle pieces that can be assembled into a whole; Based on the eye movement of the object, displaying a first puzzle piece among the plurality of puzzle pieces in an interactive state, where the first puzzle piece is the puzzle piece gazed at by the object; When the first puzzle piece is in an interactive state, based on the first gesture action of the object on the first puzzle piece, displaying the first puzzle piece moving in the three-dimensional virtual scene following the first gesture action.

2. The method according to claim 1, characterized in that, The step of displaying a first puzzle piece among the plurality of puzzle pieces in an interactive state based on the eye movement of the object includes: Determining the line-of-sight direction of the object based on the eye movement of the object; Obtaining the first puzzle piece from the plurality of puzzle pieces based on the line-of-sight direction of the object; Displaying a virtual cursor on the first puzzle piece, where the virtual cursor is used to indicate that the first puzzle piece is in an interactive state.

3. The method according to claim 2, wherein The step of displaying a virtual cursor on the first puzzle piece includes: When the duration for which the line-of-sight direction of the object points to the first puzzle piece meets a first duration threshold, displaying the virtual cursor on the first puzzle piece.

4. The method according to claim 1, characterized in that The method further includes: When the first puzzle piece stops moving following the first gesture action, and when the distance between the first boundary and the second boundary meets a first preset condition and the first boundary coincides with the second boundary, displaying the first puzzle piece and the puzzle piece where the second boundary is located being assembled together, where the first boundary is any boundary of the first puzzle piece, and the second boundary is any boundary of any other puzzle piece except the first puzzle piece.

5. The method according to claim 4, characterized in that, The second boundary is any boundary of a second puzzle piece, and at least one of the first puzzle piece and the second puzzle piece is composed of a plurality of sub-puzzle pieces, where the sub-puzzle piece is an indivisible puzzle piece; The step of, when the first puzzle piece stops moving following the first gesture action, and when the distance between the first boundary and the second boundary meets a first preset condition and the first boundary coincides with the second boundary, displaying the first puzzle piece and the puzzle piece where the second boundary is located being assembled together includes: If the number of sub-puzzle pieces included in any one of the first puzzle piece and the second puzzle piece meets a second preset condition, when the first puzzle piece stops moving following the first gesture action, and when the distance between the first boundary and the second boundary meets a first preset condition and the first boundary coincides with the second boundary, displaying the first puzzle piece and the second puzzle piece being assembled together.

6. The method according to claim 4, characterized in that, The method further includes: When the first puzzle piece stops moving following the first gesture action, and when the distance between the first boundary and the second boundary meets a first preset condition and the first boundary does not coincide with the second boundary, displaying the first puzzle piece retreating to the initial position, where the initial position is the position where the first puzzle piece was located before moving following the first gesture action.

7. The method according to claim 1, characterized in that, The method further includes: During the movement of the first puzzle piece following the first gesture action, if the first puzzle piece collides with the second puzzle piece and there is no matching boundary between the first puzzle piece and the second puzzle piece, it is displayed that the second puzzle piece is pushed by the first puzzle piece.

8. The method according to claim 1, wherein The method further includes: When the first puzzle piece is in an interactive state, based on a second gesture action of the object on the first puzzle piece, it is displayed that the first puzzle piece rotates around its own center in the direction indicated by the second gesture action.

9. The method according to claim 1, characterized in that The first puzzle piece is a composite puzzle piece formed by splicing multiple sub-puzzle pieces, and each sub-puzzle piece is an indivisible puzzle piece; the method further includes: When the first puzzle piece is in an interactive state, based on a third gesture action of the object on the first puzzle piece, it is displayed that the first puzzle piece is split into the multiple sub-puzzle pieces.

10. The method according to claim 1, characterized in that, The first gesture action is generated by the first hand of the object, and the first puzzle piece is controlled by the gesture action of the first hand of the object; The method further includes: During the process of manipulating the first puzzle piece, based on a newly generated eye movement of the object, a third puzzle piece among the multiple puzzle pieces is displayed as an interactive state, and the third puzzle piece is the puzzle piece gazed at by the object; When the third puzzle piece is in an interactive state, based on the gesture action of the second hand of the object, a linkage interaction result between the first puzzle piece and the third puzzle piece is displayed.

11. The method according to claim 1, characterized in that The method further includes: When the duration of the object's line of sight pointing to the first puzzle piece meets a second duration threshold and there is no interaction with the first puzzle piece through a gesture action, a first prompt message regarding the first puzzle piece is displayed; the first prompt message is used to indicate the splicing information required for splicing the first puzzle piece.

12. The method according to claim 1, characterized in that, The method further includes: When the duration of the object's line of sight pointing to any puzzle piece does not meet a third duration threshold, a second prompt message is displayed; the second prompt message is used to indicate the splicing information of any puzzle piece that can be spliced.

13. The method according to claim 1, characterized in that, There is a virtual container in the three-dimensional virtual scene, and the virtual container is used to hold puzzle pieces; The method further includes: When the first puzzle piece stops moving following the first gesture action, if the first puzzle piece collides with the virtual container, an animation of the first puzzle piece entering the virtual container is displayed.

14. The method according to claim 13, wherein The method further includes: Based on a fourth gesture action of the object on the virtual container, the icons of the puzzle pieces contained in the virtual container are displayed; In response to a selection operation on any icon, the puzzle piece corresponding to the icon is displayed in the three-dimensional virtual scene.

15. An interaction device based on eye movement and gestures, characterized in that, The device includes: A first display module for displaying a three-dimensional virtual scene, which includes multiple scattered puzzle pieces that can be spliced into a whole; A second display module for, based on the eye movement of the object, displaying a first puzzle piece among the multiple puzzle pieces as an interactive state, and the first puzzle piece is the puzzle piece gazed at by the object; A movement module, configured to, when the first puzzle piece is in an interactive state, display the first puzzle piece moving in the three-dimensional virtual scene following a first gesture action of the object on the first puzzle piece.

16. A computer device, characterized in that, The computer device includes a processor and a memory. The memory is configured to store at least one segment of computer program, and the at least one segment of computer program is loaded and executed by the processor to perform the eye movement and gesture based interaction method according to any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium is configured to store at least one segment of computer program, and the at least one segment of computer program is used to perform the eye movement and gesture based interaction method according to any one of claims 1 to 14.

18. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the eye movement and gesture based interaction method according to any one of claims 1 to 14.

Images