Operation interface generating method, control method and device

Abstract

The present disclosure proposes a method for generating an operation interface, a control method and an apparatus, and relates to the technical field of virtual reality. The method for generating an operation interface includes obtaining movement information of a user's operating hand, determining a three-dimensional movement route of the operating hand based on the movement information of the operating hand, mapping the three-dimensional movement route of the operating hand to a preset arc surface mesh model to generate an operation interface, the preset arc surface mesh model being a model determined based on a preset wiring method and a preset angle. The generation method allows the user to accurately know the specific movement route of the operating hand, making it easy for the user to establish a unique operation method, allowing the user to accurately control a virtual object through a three-dimensional operation interface, improving the accuracy of the user's operation of a distant virtual object, reducing the difficulty of the user's operation of the virtual object, and improving the efficiency of the user's operation of the virtual object.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This disclosure is based on and claims priority to Chinese patent application CN202210450377.1, entitled "Operation interface generating method, control method and device," filed on April 24, 2022, the disclosure of which is incorporated herein by reference in its entirety.

[0002] [Technical field] The present disclosure relates to the technical field of virtual reality, and in particular to a method for generating, a control method and an apparatus for an operation interface. [Background technology]

[0003] In the field of human-machine interaction, users generally adopt a screen-based touch interaction method to interact with machines and information. With the development of mixed reality (MR) technology, users can also adopt gesture-based interaction methods to interact with machines and information. The gesture-based interaction method allows users to break free from the limitations of the screen and provides users with a natural and human-like interaction method, allowing users to directly manipulate and control virtual objects in the virtual world.

[0004] Currently, commonly used gesture interaction methods include an operation method based on gesture extension and a method using gesture interaction directly. However, when using the operation method based on gesture extension, if a user increases the operation angle of his / her hand, the selection range of a distant object doubles, which reduces the operation accuracy of the distant object. When using the method using gesture interaction directly, the difficulty of a user to operate a virtual object at different positions in the virtual world is different, which reduces the operation efficiency of the virtual object. Summary of the Invention [Problem to be solved by the invention]

[0005] The present disclosure provides a method and device for generating and controlling an operation interface. [Means for solving the problem]

[0006] An embodiment of the present disclosure provides a method for generating an operation interface, the method including acquiring movement information of a user's operating hand, determining a three-dimensional movement route of the operating hand based on the movement information of the operating hand, mapping the three-dimensional movement route of the operating hand to a predetermined arc surface mesh model to generate an operation interface, wherein the predetermined arc surface mesh model is a model determined based on a predetermined wiring method and a predetermined angle.

[0007] An embodiment of the present disclosure provides a method for controlling an operation interface, generating an operation interface based on any of the operation interface generation methods in the embodiments of the present disclosure, the method including: acquiring operation information of a user in the operation interface; and determining a control method for a virtual object by the user based on the operation information.

[0008] An embodiment of the present disclosure provides an operation interface generating device including a first acquisition module configured to acquire movement information of a user's operating hand, a route determination module configured to determine a three-dimensional movement route of the operating hand based on the movement information of the operating hand, and a generation module configured to map the three-dimensional movement route of the operating hand to a preset arc surface mesh model and generate an operation interface, where the preset arc surface mesh model is a model determined based on a preset wiring method and a preset angle.

[0009] An embodiment of the present disclosure provides a control device for an operation interface, generating an operation interface based on any of the operation interface generation methods in the embodiments of the present disclosure, the device including a second acquisition module configured to acquire operation information in the operation interface by a user, and a control module configured to determine a control method for a virtual object by the user based on the operation information.

[0010] An embodiment of the present disclosure provides an electronic device including one or more processors and a memory in which one or more programs are stored, the one or more programs being executed by the one or more processors, causing the one or more processors to realize any of the methods for generating an operational interface in an embodiment of the present disclosure or any of the methods for controlling an operational interface in an embodiment of the present disclosure.

[0011] An embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored therein, the computer-readable storage medium realizing, when executed by a processor, any of the methods for generating an operational interface in the embodiments of the present disclosure or any of the methods for controlling an operational interface in the embodiments of the present disclosure.

[0012] The above examples and other aspects and embodiments of the present disclosure are provided in greater detail in the Brief Description of the Drawings, Detailed Description and Claims. [Brief description of the drawings]

[0013] [Figure 1] 1 shows a flowchart of a method for generating an operation interface according to an embodiment of the present disclosure. [Diagram 2] 1 shows a schematic diagram of an operation interface according to an embodiment of the present disclosure. [Diagram 3] 1 illustrates a schematic diagram of a preset arc surface mesh model according to an embodiment of the present disclosure; [Figure 4]1 shows a schematic diagram of a construction form of a preset arc surface mesh model according to an embodiment of the present disclosure; [Diagram 5] FIG. 13 is a schematic diagram of a user operation method in the process of constructing a preset arc surface mesh model according to an embodiment of the present disclosure; [Figure 6] FIG. 13 is a schematic diagram of a user operation method in the process of constructing a preset arc surface mesh model according to another embodiment of the present disclosure; [Figure 7] 1 shows a schematic diagram of a three-dimensional manipulation interface according to one embodiment of the present disclosure. [Figure 8] 1 shows a flowchart of a method for controlling an operation interface according to an embodiment of the present disclosure. [Figure 9] FIG. 1 shows a configuration block diagram of an operation interface generating device according to an embodiment of the present disclosure. [Figure 10] FIG. 2 shows a configuration block diagram of a control device of an operation interface according to an embodiment of the present disclosure. [Figure 11] 1 shows a structural schematic diagram of an electronic device according to an embodiment of the present disclosure. [Figure 12] 1 shows a flowchart of a method for operating an electronic device according to an embodiment of the present disclosure. [Figure 13] 1 shows a schematic diagram of a three-dimensional manipulation interface according to another embodiment of the present disclosure. [Figure 14] 1 shows a schematic diagram of a three-dimensional manipulation interface according to another embodiment of the present disclosure. [Figure 15] FIG. 2 illustrates a structural diagram of an exemplary hardware architecture of a computing device capable of implementing a method for generating an operation interface or a method for controlling an operation interface according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] In order to make the objectives, technical solutions and advantages of the present disclosure clearer, the embodiments of the present disclosure are described in detail below with reference to the drawings. However, unless contradictory, the embodiments and features of the embodiments in the present disclosure may be arbitrarily combined with each other.

[0015] Mixed reality (MR) technology is a technology that presents virtual scenario information in a real scenario. MR technology can build an information circuit of interaction feedback between the real world, the virtual world, and the user, and enhance the realism of the user experience. MR technology may include augmented reality (AR) technology and virtual reality (VR) technology. VR technology is an interactive three-dimensional dynamic view and entity behavior system simulation that fuses multi-source information, and immerses the user in a virtual environment. AR technology is a technology that calculates the position and angle of the camera in real time and adds corresponding images, videos, and three-dimensional models, and AR technology can skillfully fuse virtual information with the real world.

[0016] Currently, commonly used gesture interaction methods include 1) and 2). 1) The selection and operation of distant objects are realized based on gesture extension lines, but the operation method based on gesture extension lines has very low operation accuracy for distant objects, for example, a slight angle change of the hand will cause a large change in the location of the distant object selection. In addition, this method also causes cases where the user's operation command cannot be reached, reducing the user's satisfaction with the usage experience. 2) The gesture interaction is directly used to operate objects in three-dimensional space, but the display method of the operation interface in the three-dimensional space of the conventional AR device is still a planar display method, so the display area displayed to the user and the display position of the user's operation area do not match, reducing the reachability of the user's gesture in space, and the user cannot easily and quickly operate the virtual object.

[0017] 1 shows a flowchart of a method for generating an operation interface according to an embodiment of the present disclosure. The method for generating an operation interface can be applied to an operation interface generating device. As shown in FIG. 1, the method for generating an operation interface in an embodiment of the present disclosure may include the following steps:

[0018] In step S101, movement information of the user's operating hand is acquired. Step S102: A three-dimensional movement route of the operating hand is determined based on the movement information of the operating hand.

[0019] Step S103: A three-dimensional movement route of the operating hand is mapped onto a preset arc surface mesh model to generate an operation interface.

[0020] The preset arc surface mesh model is a model that is determined based on a preset wiring method and a preset angle.

[0021] In this embodiment, the three-dimensional movement route of the operating hand is determined based on the acquired movement information of the operating hand of the user, so that the specific movement route of the operating hand of the user can be accurately known, and the user can easily establish a personalized operation method. The three-dimensional movement route of the operating hand is mapped to a preset arc surface mesh model to generate an operation interface, and the preset arc surface mesh model is a model determined based on a preset wiring method and a preset angle, so that the user can accurately control the virtual object through the three-dimensional operation interface, improve the operation accuracy of the virtual object far away by the user, reduce the difficulty of the operation of the virtual object by the user, improve the operation efficiency of the virtual object by the user, and allow the user to obtain the highest satisfaction in the use experience.

[0022] The operation interface includes at least one operation control area, and the operation control area is used to display a plurality of operation control elements that can be operated and controlled by a user. The operation control elements include at least one of an operation button, an operation keyboard, an alphabet button, and an expression button.

[0023] By providing the user with at least one operation area, the user can operate and control a virtual object using a plurality of different operation control elements in the operation area, making it easier for the user to process the virtual object, and further increasing the accuracy with which the user can operate and control the virtual object, thereby providing the user with a more satisfactory user experience.

[0024] For example, Fig. 2 shows a schematic diagram of an operation interface according to an embodiment of the present disclosure. As shown in Fig. 2, the operation interface includes, but is not limited to, an operation control area of ​​a keyboard 201 and / or an operation button 202. A user operates the keyboard 201 and / or the operation button 202 to realize operation and control of an object in a virtual world.

[0025]

number

[0026] The operation buttons 202 include four alphabet buttons (e.g., buttons "A", "B", "C", and "D") and one black button, which may be used to operate and control the direction of a virtual object, thereby improving the control accuracy of the direction of the virtual object.

[0027] Note that the operation interface shown in FIG. 2 is a set of different operation control elements that are presented only on a plane (i.e., two-dimensional space). When a user uses an AR device, a VR device, or an MR device, the user needs to operate and control virtual objects in three-dimensional space. Each operation control element that is presented only in two-dimensional space is disadvantageous to the user's manual operation. To facilitate the user's operation, the operation interface needs to be mapped to three-dimensional space.

[0028] Before executing step S103, a preset arc surface mesh model is determined based on a preset wiring method and a preset angle, and then the three-dimensional movement route of the operating hand in step S103 is mapped onto the preset arc surface mesh model to generate an operation interface, so that the operation interface in Figure 2 can be three-dimensionally displayed on the preset arc surface mesh model.

[0029] The preset wiring method includes a method of dividing the wiring of the sphere based on the latitude and longitude lines, or a method of dividing the wiring of the sphere based on a polygon. A polygon is a shape displayed on a plane that is constructed by connecting the end points of three or more line segments in order. For example, the polygon may include a triangle, a quadrangle, ... an N-sided polygon, etc., where N is an integer equal to or greater than 3.

[0030] It should be noted that the above preset wiring method is merely described as an example, and can be specifically set according to actual needs. Other preset wiring methods not described are also within the scope of protection of the present disclosure, and will not be described further here.

[0031] 3 shows a schematic diagram of a preset circular arc surface mesh model according to an embodiment of the present disclosure. As shown in FIG. 3, a sphere 3-1 represents a longitude and latitude sphere for performing division wiring on the sphere based on a longitude and latitude line, a sphere 3-2 represents a triangular sphere for performing division wiring on the sphere based on a triangle, and a sphere 3-3 represents a polygonal sphere for performing division wiring on the sphere based on a quadrangle. The preset circular arc surface mesh model may be the entire sphere or a part of the sphere, and it is preferable to facilitate the user's spatial operation.

[0032] In addition, the number of wires in the mesh spheres having different wiring methods is directly proportional to the user operation fitting, and the user operation fitting is the degree of fitting between the three-dimensional movement route of the user's operating hand and the operation interface.

[0033] For example, the greater the number of wirings in the preset arc surface mesh model, i.e., the finer the division of the mesh sphere, the higher the user operation fitting, i.e., the higher the degree of fit between the three-dimensional movement route of the user's operating hand and the operation interface.

[0034] In some specific implementations, before performing the obtaining of the movement information of the user's operating hand in step S101, the method further includes obtaining attribute information of the user's operating hand, and establishing a preset arc surface mesh model based on the attribute information of the operating hand.

[0035] The attribute information of the user's operating hand includes at least one of the length of the user's arm, attribute information of the palm, and attribute information of a plurality of fingers.

[0036] In addition, since the attribute information of the operating hand of different users is different (for example, the length of the user's arm is different, the attribute information of the palm is different, and the corresponding finger lengths of different users are also different), the corresponding pre-configured arc surface network model also differs from person to person.

[0037] By constructing preset arc surface mesh models suitable for different users based on different dimensional attribute information of the operating hands of different users, the operation information of the users in three-dimensional space can be more personalized, the constructed preset arc surface mesh models can be more suitable for the operating habits of different users, and the users can easily operate in three-dimensional space, thereby enabling the users to accurately control objects in the virtual world and improving the satisfaction of the users' usage experience.

[0038] In some specific implementations, the attribute information of the user's operating hand includes: arm attribute information of the user and position information of a shoulder joint of the user. Establishing a preset arc surface mesh model based on the attribute information of the operating hand includes: constructing a preset arc surface mesh model based on the position information of the shoulder joint of the user and the arm attribute information of the user.

[0039] For example, Fig. 4 shows a schematic diagram of a construction form of a preset circular arc surface mesh model according to an embodiment of the present disclosure. The left half of Fig. 4 shows the position information of the user's shoulder joint (e.g., coordinate values ​​(x, y, z) in a three-dimensional space coordinate system, where x, y, and z are all real numbers), and the length L1 of the user's upper arm and the length L2 of the user's lower arm, where L1 and L2 are all real numbers greater than 1. The attribute information can concretely express the attribute information of the user's operating hand (e.g., right hand or left hand), and it is easy to subsequently use the attribute information of the operating hand to construct a preset circular arc surface mesh model.

[0040] The right half of Fig. 4 shows a mesh sphere constructed with the position coordinate point (x, y, z) of the user's shoulder joint as the center and the sum of the lengths of the upper and lower arms (i.e., L1 + L2) as the radius, and the wiring method of the mesh sphere may be any of the preset wiring methods. Furthermore, all or a part of the arc surface mesh of the mesh sphere can be used as a preset arc surface mesh model.

[0041] By constructing a preset arc surface mesh model based on the position information of the user's shoulder joint and the user's arm attribute information, the individual usage needs of different users can be embodied, and the preset arc surface mesh model can better meet the usage needs of the user, thereby improving the satisfaction of the user's usage experience.

[0042] For example, before acquiring the movement information of the user's operating hand in step S101, the method further includes acquiring attribute information of the user's operating hand and information of the user's visual range, and establishing a preset arc surface mesh model based on the attribute information of the operating hand and the user's visual range information.

[0043] The user's visual range information includes the visibility range of the user's eyes. According to the visible range of the user's eyes, the first mesh sphere constructed according to the attribute information of the user's operating hand is restricted and adapted, so as to obtain a part of the three-dimensional arc surface in the first mesh sphere, and the part of the three-dimensional arc surface is made into a preset arc surface mesh model, so that the spatial range in which the user can operate can be more accurately planned, and the preset arc surface mesh model can better meet the operating needs of the user, facilitating the user's operation and use.

[0044] In some specific implementations, the attribute information of the user's operating hand includes the user's arm attribute information and the user's shoulder joint position information, and establishing a predetermined arc surface mesh model based on the attribute information of the operating hand and the user's visual range information includes: constructing a mesh sphere based on the user's shoulder joint position information and the user's arm attribute information, where the mesh sphere is a sphere determined based on a predetermined wiring manner; determining a predetermined angle based on the user's visual range information; and determining a predetermined arc surface mesh model based on the mesh sphere and the predetermined angle.

[0045] The preset angle can be used to characterize the user's viewing angle, for example, when the user keeps his / her head still, the user's eyes are set as the center of the circle, the radial line extending in front of the user is set as 0 degrees, and the user's viewable viewing angle range is 50 degrees up to 70 degrees down. Also, for example, when the user keeps his / her head still, the user's eyes can be set as the center of the circle, the user's line of sight can be extended to the left and right, respectively, and the user's viewable angle can be 90 degrees left to 90 degrees right, etc. By characterizing the user's viewing angle three-dimensionally, the visual range information that the user can view can be clarified, and the user's operation can be facilitated.

[0046] By constructing a mesh sphere with the position information of the user's shoulder joint as the center of the circle and the length of the user's upper arm and / or the length of the user's lower arm as the radius, the user can easily operate the operation element in the mesh sphere based on the length of the arm. Since the user's visible range is limited, a preset angle is determined based on the user's visual range information (e.g., information such as the user's head coordinates and eye coordinates), and a part of the arc surface mesh in the mesh sphere is cut out based on the preset angle to obtain an optimal operation control area model, i.e., a preset arc surface mesh model, so that the preset arc surface mesh model is more suitable for the user's operation controllable range, and the user's operation accuracy can be improved.

[0047] In some specific implementations, determining a three-dimensional movement route of the operating hand based on the movement information of the operating hand in step S102 includes determining edge displacement information and edge angle information based on the movement information of the operating hand, and determining a three-dimensional movement route of the operating hand based on the edge displacement information and edge angle information.

[0048] For example, at least one movement range (e.g., the at least one movement range includes a first movement range determined by displacement information based on a closed path of the palm, and / or a second movement range determined by displacement information based on a preset number of swings of the palm) can be determined based on movement information of the operating hand, and by determining edge displacement information and edge angle information based on the at least one movement range, the farthest spatial position at which the user can operate (e.g., the farthest displacement information and / or the maximum angle information at which the operating hand can rotate, etc.) can be known, the movement route of the user's operating hand can be easily determined, and a three-dimensional movement route of the operating hand can be determined based on the edge displacement information and edge angle information, thereby improving the user's operation accuracy in three-dimensional space.

[0049] The movement information of the operating hand includes displacement information based on a closed path of the palm and / or displacement information based on a preset number of swings of the palm (eg, at least two swings in opposite directions).

[0050] For example, Fig. 5 shows a schematic diagram of a user operation manner in the process of constructing a preset circular arc surface mesh model according to an embodiment of the present disclosure. As shown in Fig. 5, the user's operating hand swings in a circular motion in space, thereby forming a spatially closed path, which can reflect the displacement information based on the closed path of the user's palm, thereby obtaining a first movement range determined by the displacement information based on the closed path of the palm.

[0051] For example, Fig. 6 shows a schematic diagram of a user operation mode in the process of constructing a preset arc surface mesh model according to another embodiment of the present disclosure. As shown in Fig. 6, the user's operating hand is waved at least twice in a spatially opposite direction, thereby obtaining displacement information in the process of the user's palm wave, and further obtaining a second movement range determined by the displacement information based on the palm waved at least twice in the opposite direction.

[0052] In addition, in order to ensure the accuracy of the second movement range, the preset number of swings during the user's swing process can be set according to actual detection needs (e.g., set to 3 times, 5 times, etc.), and FIG. 6 merely illustrates an example of the swing direction and number of swings of the user's palm, and other preset swing numbers not shown are also within the scope of protection of the present disclosure and will not be described further here.

[0053] By obtaining the first movement range and / or the second movement range through the above operation, edge displacement information and edge angle information in the user's three-dimensional space can be clarified, and optimization processes such as selection and / or complementation can be performed on the meshes through which the three-dimensional movement route of the operator's hand passes based on the edge displacement information and edge angle information, thereby obtaining a more accurate user operation range and improving the accuracy of determining the user's controllable operation range.

[0054] Furthermore, the size of the keyboard 201 shown in Fig. 2 is adjusted so that the generated three-dimensional operation interface can fully present the keyboard 201 at its maximum size. Fig. 7 shows a schematic diagram of a three-dimensional operation interface according to an embodiment of the present disclosure. A user can view the three-dimensional operation interface by using a VR device, an AR device, or an XR device, and each operation element in the three-dimensional operation interface supports the user to perform touch operations by direct gestures.

[0055] As shown in Figure 7, by displaying the operation keyboard using a three-dimensional operation interface, the user can easily perform close-range operation control of different buttons on the keyboard in the optimal operation control space, thereby improving the user's user experience and satisfaction.

[0056] In addition, because the preset wiring methods and the number of wirings in the preset circular arc surface mesh models are different, when the user operates the keyboard in the three-dimensional operation interface, the corresponding fitting degree is also different. When the number of wirings is smaller than the preset number threshold, the user's operation precision is reduced, while when the number of wirings is larger than the preset number threshold, the user's operation sensitivity is enhanced. The number of wirings can be set according to the actual needs of the user, and can be set according to the user's usage habits, just like the user sets the operation control precision of a mouse, so as to meet the user's individual needs.

[0057] When a user directly uses gesture operation to perform a touch operation on a three-dimensional keyboard in a three-dimensional operation interface, all operation control elements of the three-dimensional keyboard are within the optimal achievable range for the user, which obviously improves the user's operation experience and touch efficiency on the three-dimensional keyboard.

[0058] In some specific implementations, after mapping the three-dimensional movement route of the operating hand to a predetermined arc surface mesh model and generating the operation interface, when it is determined that a predetermined part of the user has moved, the method further includes acquiring movement information of the predetermined part, updating movement information of the operating hand based on the movement information of the predetermined part, updating the three-dimensional movement route of the operating hand based on the updated movement information of the operating hand, and updating the operation interface based on the updated three-dimensional movement route of the operating hand.

[0059] The preset parts include any one of a shoulder joint, a head, a nose, a mouth, and an eye. In the process of a user using an operation control element in a three-dimensional operation interface to operate and control a virtual object, the user's preset part may be displaced or rotated (for example, the shoulder joint is displaced horizontally for a short distance, and the head is rotated by a certain angle, etc.). At this time, the operation interface generating device needs to obtain the movement information of the changed preset part, and thereby update the movement information of the operating hand according to the movement information of the preset part (for example, when the shoulder joint is displaced horizontally for a short distance, the corresponding preset circular arc surface mesh model may change spatially), fine-tune related parameters for constructing the preset circular arc surface mesh model (for example, update the position information of the shoulder joint, which is the center of the circle, and change the original position coordinates (x, y, z) of the shoulder joint to the position information of the shoulder joint after the change). The shoulder joint position coordinates are updated to (x0, y0, z0), where x0, y0 and z0 are all real numbers), and the three-dimensional movement route of the operating hand is updated based on the updated movement information of the operating hand (for example, based on the conversion relationship between the shoulder joint position coordinates (x, y, z) and (x0, y0, z0), a corresponding conversion is performed on the movement information of the operating hand (for example, the displacement distance or rotation angle of the operating hand) to obtain the updated three-dimensional movement route of the operating hand), thereby making the updated three-dimensional movement route of the operating hand more compatible with the updated preset arc surface mesh model, so that the updated operation interface can be more compatible with the user's range of use, achieving the tracking effect of the three-dimensional operation interface in the process of changes in the user's preset parts, and improving the user's satisfaction with the usage experience.

[0060] For example, the updated three-dimensional operation interface is obtained by performing a tracking movement or rotation operation on the original three-dimensional operation interface, so that the relative position of the updated three-dimensional operation interface and the user's shoulder joint point match, and the user's operation needs can be met.

[0061] In some specific implementations, the method further includes mapping the three-dimensional movement route of the operator's hand onto a preset arc surface mesh model; and after generating the operation interface, when it is determined that a reconstruction request fed back by a user is obtained, reconstructing the three-dimensional operation interface based on the obtained updated movement information of the operator's hand.

[0062] In addition, when the displacement or angle of the user's preset part is significantly changed (for example, the change displacement of the preset part is larger than the preset displacement threshold (for example, the change displacement is larger than 1 meter) or the change angle of the preset part is larger than the preset angle threshold (for example, the change angle is larger than 90 degrees), the user cannot easily operate the device, so that the device for generating the operation interface can reconstruct the three-dimensional operation interface based on the updated movement information of the operating hand by sending a reconstruction request to the device for generating the operation interface. The reconstruction method is the same as the generation method of any of the operation interfaces in the embodiments of the present disclosure, and will not be described further here.

[0063] When it is decided that a reconstruction request fed back by the user has been obtained, the three-dimensional operation interface is reconstructed based on the obtained updated movement information of the operating hand, thereby allowing the user to obtain a three-dimensional operation interface that more satisfies the user's current position, and based on the updated three-dimensional operation interface, the user can more accurately operate virtual objects, thereby improving the user's user experience satisfaction.

[0064] 8 shows a flowchart of a method for controlling an operation interface according to an embodiment of the present disclosure. The method for generating an operation interface can be applied to an apparatus for controlling an operation interface. The operation interface is an interface generated according to any of the methods for generating an operation interface in the embodiments of the present disclosure.

[0065] As shown in FIG. 8, the method for controlling an operation interface in an embodiment of the present disclosure may include the following steps.

[0066] In step S801, operation information on a user's operation interface is obtained. The operation information may include control information for different operation elements in the operation interface by the user, such as alphabet information corresponding to an alphabet button pressed by the user, movement direction information corresponding to a directional button operated by the user, and so on.

[0067] In step S802, a control method for the virtual object by the user is determined based on the operation information.

[0068] For example, when the operation information includes that the user controls the directional button upward and the input displacement information is one meter, it can be made clear that the user wants to control the virtual object to move one meter upward. Also, when the operation information includes that the input alphabet information is consecutive M, O, V, E, it can be known that the user needs to "move" the virtual object (i.e., move the virtual object).

[0069] Different operation information can clarify different control methods for the user to the virtual object, allowing the user to precisely control the virtual object and improve the interaction efficiency between the user and the machine device.

[0070] In this embodiment, by adopting any of the operation interface generation methods in the embodiments of the present disclosure, an operation interface is generated, allowing a user to accurately control a virtual object through a three-dimensional operation interface, and by obtaining operation information in the operation interface by the user, the accuracy of the user's operation of a distant virtual object is improved, and a control method for the user to control the virtual object is determined based on the operation information, thereby reducing the difficulty of the user's operation of the virtual object, allowing the user to more precisely control the operation of the virtual object, and improving the efficiency of interaction between the user and a machine device.

[0071] Hereinafter, the operation interface generating device according to an embodiment of the present disclosure will be described in detail with reference to the drawings. Figure 9 shows a structural schematic diagram of the operation interface generating device according to an embodiment of the present disclosure. As shown in Figure 9, the operation interface generating device 900 may include the following modules:

[0072] The present invention includes a first acquisition module 901 configured to acquire movement information of a user's operating hand, a route determination module 902 configured to determine a three-dimensional movement route of the operating hand based on the movement information of the operating hand, and a generation module 903 configured to map the three-dimensional movement route of the operating hand to a preset arc surface mesh model and generate an operation interface, where the preset arc surface mesh model is a model determined based on a preset wiring method and a preset angle.

[0073] In the operation interface generating device according to the embodiment of the present disclosure, the route determining module determines a three-dimensional movement route of the operating hand based on the acquired movement information of the operating hand of the user, so that the specific movement route of the operating hand of the user can be accurately known, and the user can easily establish a personalized operation method. The generating module maps the three-dimensional movement route of the operating hand to a preset arc surface mesh model to generate an operation interface, and the preset arc surface mesh model is a model determined based on a preset wiring method and a preset angle, so that the user can accurately control the virtual object through the three-dimensional operation interface, improve the operation accuracy of the virtual object by the user at a distance, reduce the difficulty of the operation of the virtual object by the user, increase the operation efficiency of the virtual object by the user, and allow the user to obtain the highest user experience satisfaction.

[0074] FIG. 10 shows a structural schematic diagram of an operation interface control device according to an embodiment of the present disclosure. The operation interface is generated according to any operation interface generation method in the embodiment of the present disclosure. As shown in FIG. 10, the operation interface control device 1000 includes: A second acquisition module 1001 configured to acquire operation information in an operation interface by a user; and a control module 1002 configured to determine a control method for a virtual object by a user based on the operation information.

[0075] In an operation interface generating device according to an embodiment of the present disclosure, by adopting any of the operation interface generating methods in the embodiments of the present disclosure to generate an operation interface, a user can accurately control a virtual object through a three-dimensional operation interface, and the second acquisition module improves the accuracy of the user's operation of a distant virtual object by acquiring operation information in the user's operation interface, and the control module determines a control method for the user to control the virtual object based on the operation information, thereby reducing the difficulty of the user's operation of the virtual object, allowing the user to operate and control the virtual object more precisely, and improving the efficiency of interaction between the user and the machine device.

[0076] 11 shows a structural schematic diagram of an electronic device according to an embodiment of the present disclosure. For example, the electronic device may be any one of a VR device, an AR device, and an MR device. A user uses the electronic device to perform gesture-based information interaction with a machine device, thereby realizing the user's operation in a three-dimensional space.

[0077] As shown in FIG. 11, the electronic device includes, but is not limited to, an acquisition module 1101, a display module 1102, a control module 1103, a judgment module 1104, and a calculation and storage module 1105.

[0078] The acquisition module 1101 is configured to acquire user data. For example, the acquisition module 1101 acquires attribute information of the user's operating hand and movement information of the operating hand by using a sensor in the VR device, a sensor in the AR device, or a sensor in the XR device. The sensor may include at least one of an electromyogram sensor, a posture sensor, and a voice receiving sensor. The VR device may further be equipped with a three-dimensional depth camera and / or a binocular camera. The attribute information of the operating hand includes at least one of position information of the user's shoulder joint in three-dimensional space, the length of the upper arm, and the length of the lower arm. The movement information of the operating hand includes position information of the user's palm passing through during the movement, and data of the user's gesture interaction with the display interface. The above user data may be stored in the calculation and storage module 1105.

[0079] The display module 1102 is configured to display the information stored in the calculation and storage module 1105 in a three-dimensional space for easy user review.

[0080] The control module 1103 is configured to, when receiving the control command sent by the judgment module 1104, control the acquisition module 1101 to acquire user data and control the display module 1102 to display information.

[0081] The determination module 1104 is configured to determine whether to send a control command to the control module 1103 (e.g., determine whether to generate a control command or the like according to a user's usage demand) based on the user data stored in the calculation and storage module 1105. When it is determined that a control command needs to be sent, the determination module 1104 causes the control model 1103 to perform a corresponding operation based on the control command by sending the control command to the control module 1103. The control command may include controlling the display module 1102 to display a plurality of operation control elements in at least one operation control area, so as to allow the user to control these operation control elements.

[0082] The calculation and storage module 1105 is configured to calculate and store the user data acquired by the acquisition module 1101. For example, the calculation and storage module 1105 determines a three-dimensional movement route of the operating hand based on the movement information of the operating hand, maps the three-dimensional movement route of the operating hand to a preset arc surface mesh model, and generates an operation interface, which can be displayed in a three-dimensional space. In addition, in order to easily confirm the control method of the user for the virtual object based on the operation information, the calculation and storage module 1105 also stores operation information of the user in the operation interface.

[0083] 12 shows a flow chart of an operation method of an electronic device according to an embodiment of the present disclosure. As shown in FIG. 12, the operation method of the electronic device includes, but is not limited to, the following steps:

[0084] In step S1201, characteristics of a plurality of operation control elements displayed on a plane are obtained, and a plane operation interface is determined based on the characteristics of the plurality of operation control elements.

[0085] The operation control elements may include multiple different elements such as alphabet buttons, directional buttons, facial expression buttons, alphabet buttons, function keys, etc., and multiple operation control elements can be arranged and combined according to the user's usage habits, thereby determining a planar operation interface.

[0086] For example, the planar operation interface may be displayed in the form of an operation keyboard and / or an operation button control area, etc., to facilitate user use.

[0087] In step S1202, attribute information of the user's operating hand and movement information of the operating hand are acquired.

[0088] The attribute information of the operating hand includes at least one of position information of the user's shoulder joint in three-dimensional space, the length of the upper arm, and the length of the lower arm.

[0089] The movement information of the operating hand includes displacement information based on a closed path of the palm and / or displacement information based on a preset number of swings of the palm, and data on gesture interaction between the user and the display interface.

[0090] Step S1203: Establish a preset arc surface mesh model based on the attribute information of the operating hand.

[0091] The preset arc surface network model is a model determined based on a preset wiring method and a preset angle, and the preset wiring method includes a method of dividing a sphere based on a latitude and longitude line, or a method of dividing a sphere based on a polygon, and the polygon is a planar graphic formed by connecting the end points of three or more line segments in order.

[0092] In step S1204, a three-dimensional movement route of the operating hand is determined based on the movement information of the operating hand.

[0093] The movement information of the operating hand includes a movement trajectory in a three-dimensional space formed by the user shaking the palm at least twice, which can determine the three-dimensional movement route of the operating hand and further clarify edge position information and edge angle information on which the user's palm can move.

[0094] In step S1205, the three-dimensional movement route of the operating hand is mapped onto a preset arc surface mesh model, and a three-dimensional operation interface that can be displayed in three-dimensional space is generated.

[0095] For example, Fig. 13 shows a schematic diagram of a three-dimensional operation interface according to another embodiment of the present disclosure. As shown in Fig. 13, the operation button 202 in Fig. 2 is mapped to a preset arc surface mesh model, so that the operation button 202 can be presented to the user in a three-dimensional manner.

[0096] The display range of the generated three-dimensional operation interface is an optimal display range determined based on edge position information and edge angle information within which the user's palm can move, and the operation buttons 202 can be presented sufficiently on the three-dimensional operation interface to facilitate the user's operation. For the user, each operation button displayed on the three-dimensional operation interface is within the user's optimal achievable range, which can improve the user's touch operation efficiency and enhance the user's operation experience.

[0097] In addition, for example, when there are too many operation elements, the present invention can further support the user to simultaneously establish multiple three-dimensional operation interfaces, for example, by constructing multiple three-dimensional operation interfaces according to the three-dimensional movement routes of the user's left and right hands.

[0098] Fig. 14 is a schematic diagram of a three-dimensional operation interface according to another embodiment of the present disclosure. As shown in Fig. 14, the keyboard 201 and the operation buttons 202 in Fig. 2 are each mapped to a preset arc surface mesh model, and multiple three-dimensional operation interfaces are presented to the user in a stereoscopic manner, thereby facilitating the user's operation.

[0099] Step S1206: A three-dimensional operation interface is displayed to the user, so that the user can operate and control the virtual object in the three-dimensional operation interface.

[0100] For example, a user can operate each operation control element on the keyboard 201 in the three-dimensional operation interface with his / her right hand, and can also operate each button among the operation buttons 202 in the three-dimensional operation interface with his / her left hand, thereby satisfying the user's individual needs and improving the user's operation efficiency.

[0101] In this embodiment, the three-dimensional movement route of the operating hand is determined based on the acquired movement information of the operating hand of the user, so that the specific movement route of the operating hand of the user can be accurately known, and the user can easily establish a personalized operation method. The three-dimensional movement route of the operating hand is mapped to a preset circular arc surface mesh model to generate a three-dimensional operation interface, so that the user can accurately control the virtual object through the stereoscopic three-dimensional operation interface, thereby improving the operation accuracy of the virtual object far away by the user, reducing the difficulty of the user to operate the virtual object, improving the operation efficiency of the virtual object by the user, allowing the user to intuitively control the virtual object through gesture operation, and obtaining the highest user experience satisfaction.

[0102] In particular, the present disclosure is not limited to the specific configurations and processes described and illustrated in the above embodiments. For convenience and brevity, detailed descriptions of known methods are omitted herein, and specific operation procedures of the above-mentioned systems, modules, and units may refer to the corresponding procedures in the above-mentioned method embodiments and are not described herein.

[0103] FIG. 15 illustrates an exemplary hardware architecture diagram of a computing device capable of implementing the method for generating an operation interface or the method for controlling an operation interface according to an embodiment of the present disclosure.

[0104] 15, the computing device 1500 includes an input device 1501, an input interface 1502, a central processor 1503, a memory 1504, an output interface 1505, and an output device 1506. The input interface 1502, the central processor 1503, the memory 1504, and the output interface 1505 are connected to each other via a bus 1507, and the input device 1501 and the output device 1506 are connected to the bus 1507 via the input interface 1502 and the output interface 1505, respectively, and are further connected to other components of the computing device 1500.

[0105] Specifically, the input device 1501 receives input information from the outside and transmits the input information to the central processor 1503 via the input interface 1502, the central processor 1503 processes the input information based on computer-executable instructions stored in the memory 1504 to generate output information, stores the output information temporarily or permanently in the memory 1504, and transmits the output information to the output device 1506 via the output interface 1505, and the output device 1506 outputs the output information outside the computing device 1500 for use by the user.

[0106] In one embodiment, the computing device shown in FIG. 15 may be realized as an electronic device, which may include a memory configured to store a program, and a processor configured to execute the program stored in the memory to perform any of the methods for generating an operational interface described in the above embodiments, or any of the methods for controlling an operational interface in the embodiments of the present disclosure.

[0107] In one embodiment, the computing device shown in FIG. 15 can be implemented as an operational interface generation system, which may include a memory configured to store a program, and a processor configured to execute the program stored in the memory to perform any of the operational interface generation methods described in the above embodiments.

[0108] In one embodiment, the computing device shown in FIG. 15 may be implemented as a control system for an operational interface, which may include a memory configured to store a program and a processor configured to execute the program stored in the memory to perform any of the operational interface control methods described in the above embodiments.

[0109] The above are merely exemplary embodiments of the present disclosure, and are not intended to limit the scope of protection of the present disclosure. In general, various embodiments of the present disclosure can be realized in hardware, or dedicated circuits, software, logic, or any combination thereof. For example, some aspects may be realized in hardware, and other aspects may be realized in firmware or software that may be executed by a controller, microprocessor, or other computing device, and the present disclosure is not limited thereto.

[0110] Embodiments of the present disclosure may be implemented by hardware, for example in a processor entity, or by a combination of software and hardware, by a data processor of a mobile device executing computer program instructions, which may be assembly instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or target code written in any combination of one or more programming languages.

[0111] Any logic flow block diagrams in the drawings of this disclosure may represent program steps, or may represent interconnected logic circuits, modules and functions, or may represent a combination of program steps, logic circuits, modules and functions. The computer program may be stored in a memory. The memory may have any type suitable for the local technology environment and may be implemented using any suitable data storage technology, such as, but not limited to, read-only memory (ROM), random access memory (RAM), optical memory devices and systems (digital versatile disks DVD or CD optical disks). The computer readable medium may include a non-transitory storage medium. The data processor may be any type suitable for the local technology environment, such as, but not limited to, a general purpose computer, an application specific computer, a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (FGPA), and a processor based on a multi-core processor architecture.

[0112] Detailed descriptions of exemplary embodiments of the present disclosure have been given above by way of illustrative and non-limiting examples. However, various modifications and adjustments to the above-described embodiments will be apparent to those skilled in the art in view of the drawings and claims, but do not depart from the scope of the present disclosure. Thus, the appropriate scope of the present disclosure is determined based on the claims.

Claims

1. A method for generating an operation interface, comprising the steps of: Acquiring movement information of a user's operating hand; determining a three-dimensional movement route of the operating hand based on the movement information of the operating hand; A method for generating an operation interface, comprising: mapping a three-dimensional movement route of the operating hand onto a preset arc surface mesh model to generate an operation interface, the preset arc surface mesh model being a model determined based on a preset wiring method and a preset angle.

2. Determining a three-dimensional movement route of the operating hand based on the movement information of the operating hand, determining edge displacement information and edge angle information based on the movement information of the operating hand; and determining a three-dimensional movement route of the hand based on the edge displacement information and the edge angle information.

3. The method according to claim 2 , wherein the movement information of the operating hand includes displacement information based on a closed path of the palm and / or displacement information based on a preset number of swings of the palm.

4. A three-dimensional movement route of the operator's hand is mapped onto a preset arc surface mesh model to generate an operation interface; When it is determined that a preset body part of the user has moved, acquiring movement information of the preset body part; updating movement information of the operating hand based on movement information of the preset part; updating a three-dimensional movement route of the operating hand based on the updated movement information of the operating hand; The method according to claim 1 , further comprising: updating the operation interface based on the updated three-dimensional movement route of the operating hand.

5. Before acquiring the movement information of the user's operating hand, acquiring attribute information of the user's operating hand including arm attribute information and position information of the user's shoulder joint; The method according to claim 1 , further comprising: constructing the preset arc surface mesh model based on position information of the user's shoulder joint and arm attribute information of the user.

6. Before acquiring the movement information of the user's operating hand, acquiring attribute information of the user's operating hand and visual range information of the user, the attribute information of the user's operating hand including arm attribute information of the user and position information of a shoulder joint of the user; constructing a mesh sphere based on position information of the shoulder joint of the user and arm attribute information of the user, the mesh sphere being a sphere determined based on the preset wiring method; determining the preset angle based on visual range information of the user; The method of claim 1 , further comprising: determining the preset arc surface mesh model based on the mesh sphere and the preset angle.

7. The method according to claim 6 , wherein the preset wiring method includes a method of dividing the wiring for the sphere based on a latitude and longitude line, or a method of dividing the wiring for the sphere based on a polygon.

8. The method according to any one of claims 1 to 7, wherein the operational interface includes at least one operational control area for displaying a plurality of operational control elements which the user can operationally control.

9. A method for controlling an operation interface for generating an operation interface based on the method for generating an operation interface according to any one of claims 1 to 8, comprising the steps of: acquiring operation information of the operation interface by a user; and determining a control method for the user with respect to a virtual object based on the operation information.

10. An operation interface generating device, comprising: A first acquisition module configured to acquire movement information of a user's operating hand; A route determination module configured to determine a three-dimensional movement route of the operator hand based on the movement information of the operator hand; and a generation module configured to map a three-dimensional movement route of the operating hand onto a preset arc surface mesh model and generate an operation interface, the preset arc surface mesh model being a model determined based on a preset wiring method and a preset angle.

11. A control device for an operation interface that generates an operation interface based on the method for generating an operation interface according to any one of claims 1 to 8, comprising: A second acquisition module configured to acquire operation information of a user on the operation interface; and a control module configured to determine a control method for a virtual object by the user based on the operation information.

12. An electronic device, one or more processors; An electronic device comprising: a memory in which one or more programs are stored, the one or more programs being executed by the one or more processors, and causing the one or more processors to realize the method for generating an operational interface described in any one of claims 1 to 8, or the method for controlling an operational interface described in claim 9.

13. A computer-readable storage medium having a computer program stored therein, the computer-readable storage medium realizing, when executed by a processor, the method for generating an operation interface according to any one of claims 1 to 8, or the method for controlling an operation interface according to claim 9.

Images