User interface generation method and method for controlling avatar motion by means of user interface

Abstract

The present application provides a user interface generation method and a method for controlling avatar motion by means of a user interface. A user interface is formed by means of a generated aerial view, navigation grid map, local environment map, and front view, and an avatar is controlled by means of the user interface, so that a user can immersively and visually control the avatar, and at the same time, an operator can control the degree of freedom of an avatar body movement.

Description

Method for generating user interface and method for controlling avatar movement through user interface

[0001] Cross-references of related documents

[0002] This application claims priority to the Chinese patent application No. 2024106739947, filed on May 28, 2024, and entitled “Method for generating user interface and method for controlling avatar movement through user interface”, the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0003] The present application belongs to the technical field of controlling avatar movement in virtual space, and particularly relates to a method for generating user interface and a method for controlling avatar movement through user interface. BACKGROUND

[0004] In the prior art, the avatar in the virtual space is controlled in the following ways: Scheme one: a 3D keyboard or virtual keyboard can be created in combination with a control interaction component and a control component. The keyboard keys are recognized as clickable buttons and correspond to physical keyboard keys or mouse clicks. The trigger event script of the button executes the action animation that the avatar should perform after clicking the button, and such animation is a preset animation made by an animator in advance. For example, game events such as opening a door and moving forward, the user thus controls the avatar to move in the virtual space by manipulating the interaction component.

[0005] Scheme two: limb animation driving mode. The limb posture in the full-body video captured by the vision sensor is recognized by artificial intelligence technology and mapped to the virtual skeleton of the avatar, thereby driving the movement of the avatar.

[0006] The existing scheme one is a common user and game character interaction control method in games, and this method is also applied to the current mainstream metaverse space. The significant disadvantage of this scheme is that the user needs to constantly press the keys or click the mouse, or click the virtual keys to control the local movement of the avatar. For most users, it is difficult to remember the terrain and destination distribution of a large range of metaverse space, so the control of avatar movement is tedious and boring.

[0007] The second existing scheme is commonly used for virtual human limb movement control in a small virtual scene, such as a live application of a real person driving a virtual person to simulate the limb movement of the real person. This scheme can also be applied to user control of the movement of an avatar in a metaverse space. However, the main disadvantages are as follows: 1. It is not suitable for mobile terminals, because the built-in visual sensor needs the user to stand at a certain distance to capture the full-body video image in order to recognize the limb posture. Mobile terminals, especially smart phones, are the most popular hardware devices for logging into a metaverse space, so this scheme will limit the popularity of such applications. 2. Although the movement freedom of the avatar is very high (can completely simulate the real limb movement of the user), it brings trouble to the operators of the metaverse space, such as operators who cannot limit certain limb movements of the avatar.

[0008] In order to not only provide a way for users to immerseively and intuitively control the avatar, but also give the operator the freedom to control the limb movement of the avatar, a new solution needs to be proposed. Technical solutions

[0009] In order to solve or alleviate the problems in the prior art, the technical scheme provided by the present application can enable users to immerseively and intuitively control the avatar, and also give the operator the freedom to control the limb movement of the user avatar.

[0010] In a first aspect, an embodiment of the present application provides a method for generating a user interface, the user interface comprising a main view, an overhead view, a navigation grid view and a local environment view, the method comprising:

[0011] adding a first virtual camera in the virtual space, scaling all virtual objects in the virtual space until the virtual space is displayed in the camera view of the first virtual camera, determining an overhead view according to the view obtained by the first virtual camera;

[0012] voxelizing the entire virtual space, calculating a plurality of walking areas and a plurality of non-walking areas of all avatars, regarding each non-walking area as an obstacle object, determining the edges of the walking areas and collecting edge points on the edges, connecting all the edge points to generate a polygon grid, determining a grid view according to the polygon grid, adding the positions of all avatars in the virtual space on the grid view and displaying them as first circular points, and obtaining a navigation grid view;

[0013] adding a second virtual camera in the virtual space, binding the second virtual camera with each avatar so that the second virtual camera moves with the walking of the avatar, wherein the second virtual camera is placed behind the avatar, determining a local environment view according to the view of the second virtual camera;

[0014] According to the change of the position of the avatar in the virtual space, the bird's-eye view, the navigation grid map and the local environment map are updated.

[0015] According to the change of the position of the avatar in the virtual space, the bird's-eye view, the navigation grid map and the local environment map are updated.

[0016] A third virtual camera is added in the virtual space, and the virtual space is rendered in real time under the view of the third virtual camera, and the view of the first virtual camera is updated according to the change of the virtual space, and then the bird's-eye view is updated.

[0017] According to the update of the bird's-eye view, the navigation grid map is updated.

[0018] According to the change of the position of the avatar in the virtual space, the bird's-eye view, the navigation grid map and the local environment map are updated.

[0019] When the position of the avatar in the virtual space changes, the view of the avatar in the first virtual camera is determined.

[0020] According to the perspective relationship of the first virtual camera, the position change of the avatar is projected to the navigation grid map, and a corresponding second circle point is superimposed on the navigation grid map, and the second circle point represents the updated position of the avatar.

[0021] According to the change of the position of the avatar in the virtual space, the bird's-eye view, the navigation grid map and the local environment map are updated.

[0022] In the bird's-eye view, a region is selected for zooming in, the content of the region is synchronized to the local environment map, and the region is highlighted in the navigation grid map.

[0023] Compared with the prior art, the embodiment of the application provides a generation method of a user interface for controlling the movement of an avatar in a meta-universe virtual space. Through the user interface composed of the generated bird's-eye view, navigation grid map and local environment map and main view, the user can intuitively control the avatar in an immersive manner, and the operator can also control the freedom degree of the user's avatar's body movement.

[0024] In a second aspect, the embodiment of the application also provides a method for controlling the movement of an avatar through a user interface, wherein the user interface is generated by the generation method of the first aspect, and specifically includes:

[0025] The destination of the avatar determined by the user on the bird's-eye view or the navigation grid map is acquired.

[0026] Determine a route for the avatar to walk according to a path finding algorithm of the navigation mesh map, and control the avatar to move to the destination;

[0027] Obtain an instruction of a user clicking any one of the bird's eye view, the global navigation map and the local environment map;

[0028] Set a virtual camera corresponding to the map clicked by the user as an active camera of the main view so as to display a view corresponding to a position of the avatar displayed in the any one of the maps clicked in the main view.

[0029] As a preferred embodiment of the present application, the method further comprises:

[0030] In the local environment map, obtain a selected avatar of interest of a user, and enter a chat mode.

[0031] As a preferred embodiment of the present application, the method further comprises:

[0032] Add a fourth virtual camera in the virtual space, and place the fourth virtual camera in front of a face of the avatar and display the face of the avatar in an isometric view;

[0033] Add a floating view window to render an image of the face of the avatar in real time in a view of the fourth virtual camera, wherein the floating view window is located below the main view.

[0034] As a preferred embodiment of the present application, the method further comprises:

[0035] In the chat mode, obtain an instruction of a user driving a facial expression of the avatar by a facial expression driving method;

[0036] Control the avatar to make a corresponding facial expression according to the instruction of driving the facial expression.

[0037] As a preferred embodiment of the present application, the method further comprises:

[0038] Obtain chat content of a user and the avatar;

[0039] Determine a body action of the avatar according to the chat content;

[0040] Control the avatar to make a corresponding body action according to the determined body action of the avatar. Advantages

[0041] Compared with the prior art, the embodiments of this application provide a method for controlling avatars through a user interface, which can enable avatars to enter a chat dialogue mode, a user interface mode for avatars to move under global path planning, and a user interface mode for users to control avatars when switching to the movement of avatars in a local environment. Therefore, the user interface provided by this application can enable users to control avatars in an immersive and intuitive way, while also giving operators the freedom to control the avatar's body movements. Attached Figure Description

[0042] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. Some specific embodiments of this application will be described in detail below with reference to the accompanying drawings in an exemplary and non-limiting manner. The same reference numerals in the drawings designate the same or similar parts or components. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:

[0043] Figure 1 is a flowchart illustrating a method for generating a user interface for controlling the movement of an avatar in a metaverse virtual space, according to an embodiment of this application.

[0044] Figure 2 is a schematic diagram of the layout of the user interface provided in an embodiment of this application;

[0045] Figure 3 is a schematic diagram of the user interface layout for communicating in a local environment map according to an embodiment of this application;

[0046] Figure 4 is a schematic diagram of the user interface layout for magnifying the selected area in a local environment map, provided in an embodiment of this application. The best embodiment of the present invention

[0047] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some, not all, of the embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort should fall within the scope of protection of the present application.

[0048] In a first aspect, as shown in Figure 1, this application provides a method for generating a user interface for controlling the movement of an avatar in a metaverse virtual space. The user interface includes a main view, a bird's-eye view, a navigation grid map, and a local environment map. The method includes:

[0049] Step S01: Add a first virtual camera to the virtual space, scale all virtual objects in the virtual space until the virtual space is displayed in the camera view of the first virtual camera, and determine the bird's-eye view based on the view obtained by the first virtual camera.

[0050] Specifically, the user interface provided in this application embodiment includes: a main view, a bird's-eye view, a navigation grid map, and a partial environment map. The main view is a metaverse space, and three layered mini-maps are laid out in the sidebar container and can be hidden and shown by clicking a button. Users can switch to different display modes of the metaverse space in the main view by clicking the mini-map, while users can still easily view the display of the metaverse space in other views in the sidebar.

[0051] In this step, the generated bird's-eye view is mainly used for route following. First, a first virtual camera is added to the virtual space, usually set directly above the virtual space, looking down and displaying the virtual space scene content in an isometric view. The length and width of the first virtual camera view are the same as the length and width of the image displayed in the main view window. Since the virtual space is large, in order to display the entire scene in the first virtual camera view, all objects in the entire virtual space are first scaled. Specifically, an empty object is created and the empty object is used as the parent object of all other objects. The empty object is scaled until the entire virtual space is displayed in the view of the first virtual camera. Finally, the parent-child relationship between all objects and the empty object is canceled.

[0052] Step S02: Voxelize the entire virtual space and calculate multiple walking areas and multiple non-walking areas for all avatars. Treat each non-walking area as an obstacle object, determine the edge of the walking area and collect the edge points on the edge, connect all the edge points to generate a polygonal mesh, determine the navigation mesh map based on the polygonal mesh, add the positions of all avatars in the virtual space to the navigation mesh map and display them as first dots to obtain the navigation mesh map;

[0053] In this step, a navigation mesh map is generated for global planning. This can be achieved by generating a navigation mesh, which is a polygonal mesh used to mark walkable areas in the virtual space. The mesh can be generated using a reshaping method in navigation, a method not covered by this application and therefore not described in detail. This method primarily involves first voxelizing the virtual space (i.e., generating small cubes), then calculating a flat area sufficient to support avatar movement in the voxel space, and generating the edges of each flat area. Non-walkable areas are treated as obstacles. Edge points of the flat areas are sampled and connected to generate a polygonal mesh. All avatars in the virtual space are used as proxy objects. For ease of display, the geometry of the proxy objects is represented by dots, thus displaying the positions of all avatars in the virtual space as dots on the navigation mesh map. Finally, the navigation mesh and proxy objects are displayed in the bird's-eye view camera view described in step S01.

[0054] Step S03: Add a second virtual camera to the virtual space and bind the second virtual camera to each avatar so that the second virtual camera moves as the avatar walks. The second virtual camera is placed behind the avatar, and a local environment map is determined based on the view of the second virtual camera.

[0055] In this step, a local environment map is generated for local motion and animation. A second virtual camera is added and bound to the avatar object, so the camera moves with the avatar. An offset is set between the second virtual camera and the avatar so that the second virtual camera is positioned behind the avatar. The virtual space content is displayed in perspective view, with the length and width of the second virtual camera view matching the length and width of the image displayed in the main view window. The translation, rotation, and scaling of the second virtual camera can be adjusted manually or via code scripts to ensure that the second virtual camera view at least covers the entire avatar.

[0056] Step S03: Update the bird's-eye view, navigation grid map, and local environment map according to the changes in the position of the avatar in the virtual space.

[0057] In this step, updating the navigation grid map based on the avatar's position change in the virtual space includes:

[0058] A third virtual camera is added to the virtual space, and the virtual space is rendered in real time under the view of the third virtual camera. The view of the first virtual camera is updated according to the changes in the virtual space, thereby updating the bird's-eye view.

[0059] The navigation grid map is then updated based on the updated bird's-eye view.

[0060] Specifically, this application mainly involves dynamically adding virtual cameras to render virtual scenes in real time from the perspective of the newly added virtual cameras, thereby displaying images in the local environment map, and dynamically overlaying game objects and projecting them onto the first virtual camera to update the navigation mesh map.

[0061] In this embodiment of the application, updating the navigation grid map based on the updated bird's-eye view includes:

[0062] When the avatar's position changes in the virtual space, determine the avatar's view in the first virtual camera;

[0063] The change in the avatar's position is projected onto the navigation grid map according to the perspective relationship of the first virtual camera, and a corresponding second dot is superimposed on the navigation grid map. The second dot represents the updated position of the avatar.

[0064] Specifically, changes generated by user interactions on the local environment map, such as controlling the avatar's movement, need to be synchronized to the navigation grid map; for avatars with updated positions, they are projected onto the navigation grid map according to the perspective relationship of the first virtual camera, and a dot is superimposed to highlight the avatar.

[0065] In this embodiment of the application, the step of updating the navigation grid map based on the update of the bird's-eye view further includes:

[0066] Select an area in the bird's-eye view and zoom in. The content of the selected area needs to be synchronized to the local environment map and highlighted in the navigation grid map.

[0067] Specifically, in the bird's-eye view, an area is selected and zoomed in. The content of the zoomed-in area needs to be synchronized to the local environment map and highlighted in the navigation grid map. Dynamically regenerating the minimap according to the frame rate would greatly increase the computational and network load. The specific steps are as follows: Add a third virtual camera and place it directly above the selected area, with its view dimensions consistent with the main view. Assuming the virtual space coordinate system is Y-axis pointing upwards, and the ratio of the size of the selected area to the size of the newly added third virtual camera view is m, then the "directly above" position is the center point of the selected area in the XY plane, and the Z-axis position is the first virtual camera's Z-axis position multiplied by m. Because the size of the selected area is smaller than the size of the newly added third camera view, the Z-axis position of the newly added third virtual camera is smaller than the Z-axis position of the first virtual camera. Therefore, the newly added third virtual camera is closer to the scene content to be displayed, thus achieving the zoomed-in display effect.

[0068] Compared with the prior art, the embodiments of this application provide a method for generating a user interface for controlling the movement of an avatar in the metaverse virtual space. The user interface, which consists of a bird's-eye view, a navigation grid map, a local environment map, and a main view, allows users to control the avatar in an immersive and intuitive way, while also giving operators the freedom to control the avatar's limb movements.

[0069] Secondly, embodiments of this application also provide a user interface, including: a front view, a bird's-eye view, a global navigation map, and a local environment map;

[0070] The main view is used to display the virtual space shown by the bird's-eye view, global navigation map, or local environment map.

[0071] Compared with the prior art, the user interface provided in this application has the same beneficial effects as the first aspect, and will not be repeated here.

[0072] Thirdly, embodiments of this application also provide a method for controlling an avatar's movement via a user interface, including:

[0073] Obtain the destination that the avatar is heading to, as determined by the user on the bird's-eye view or the navigation grid map;

[0074] The avatar's route is determined based on the pathfinding algorithm of the navigation grid, and the avatar is controlled to move towards the destination;

[0075] Get the user's command to click on any of the three maps: the bird's-eye view, the global navigation map, and the local environment map;

[0076] Set the virtual camera corresponding to the map clicked by the user as the active camera of the main view so that the view corresponding to the location of the avatar displayed on any of the clicked maps can be displayed in the main view.

[0077] As a preferred embodiment of this application, the method further includes:

[0078] In the local environment map, the user's selected avatar of interest is obtained, and the chat mode is entered.

[0079] As a preferred embodiment of this application, the method further includes:

[0080] A fourth virtual camera is added to the virtual space, and the fourth virtual camera is placed directly in front of the avatar's face and the avatar's face is displayed in an isometric view.

[0081] A new floating view window is added to render the avatar's facial image in real time in the view of the fourth virtual camera, wherein the floating view window is located below the main view.

[0082] Specifically, as shown in Figure 3, when switching to the local environment map, avatars can communicate and talk. A small dialogue window is displayed in the lower right corner of the local environment map, and the avatar's face is displayed on the small dialogue window.

[0083] As a preferred embodiment of this application, the method further includes:

[0084] In the chat mode, the user's instructions to drive the avatar's facial expressions using a facial expression-driven method are obtained;

[0085] The avatar is controlled to make corresponding facial expressions according to the instructions of the facial expression driver.

[0086] Specifically, as shown in Figure 4, if a user selects a local area in the selected space on the interface of the local environment map, the content displayed in the selected window will be enlarged and displayed in a pop-up view window.

[0087] As a preferred embodiment of this application, the method further includes:

[0088] Obtain the chat content between the user and the avatar;

[0089] The body language of the avatar is determined based on the chat content;

[0090] The avatar is controlled to perform corresponding limb movements based on the determined limb movements of the avatar.

[0091] The user interface provided in this application embodiment, and the user interaction process are described in detail below:

[0092] After entering the main view, the user clicks a button on the interface to expand the sidebar container, displaying three small maps (a bird's-eye view, a navigation grid map, and a partial environment map). The user can click a button to hide the container. Clicking on one of the small maps sets the main view's active camera to the virtual camera corresponding to that small map, and the main view window is displayed in a virtual space under the updated active camera. If it is a navigation small map, it will display the navigation grid map.

[0093] Users can choose their avatar's destination in two ways. One is by specifying the destination on the navigation grid map (usually visualized by a specific color); the other is by selecting an area on the bird's-eye view and zooming in to see if it's a destination of interest, such as a local environment or whether the avatar in that area is of interest. The first method generally occurs when the user is familiar with the virtual space and knows or has a known destination; the second method is generally used when the user is still exploring the virtual space and doesn't have a specific destination in mind. After selecting a destination, the avatar enters route-following mode and moves towards the destination according to the walking animation and the path-finding algorithm based on the navigation grid map.

[0094] When a user clicks on a partial environment map, the main view's active camera is updated to the virtual camera of that partial environment map, thus displaying the virtual space from the new active camera's perspective in the main view window. In this view, the user can finely control the avatar's movement, including but not limited to keyboard commands. Other multimodal AI algorithms, such as voice-driven avatar movements, can also be applied. The avatar can therefore perform a variety of personalized walking or other body animations under user control.

[0095] In the local environment view, users can chat with nearby avatars within the view area. Users can select an avatar of interest by clicking on other avatars, but are not limited to this; they can also enter chat mode through other interactive methods such as specific gestures or specific voice commands. A new virtual camera has been added, placed directly in front of the avatar's face and displayed in an isometric view. A new floating view window displays a real-time rendered image of the avatar's face from the new camera's perspective; the floating window is located in the lower right corner of the main view.

[0096] In the chat mode of the user interface provided in this application embodiment, the user is not limited to controlling the avatar's body movements. Users can enhance the immersive experience of chatting by using facial expression-driven methods such as video-driven facial expressions and voice-driven facial expressions. Based on the chat content, the avatar performs corresponding body animations. The animations can be preset animations, but are not limited to them; they can also be voice-driven body animations implemented using artificial intelligence algorithms.

[0097] Compared with the prior art, the embodiments of this application provide a method for controlling avatars through a user interface, which can enable avatars to enter a chat dialogue mode, a user interface mode for avatars to move under global path planning, and a user interface mode for users to control avatars when switching to the movement of avatars in a local environment. Therefore, the user interface provided by this application can enable users to control avatars in an immersive and intuitive way, while also giving operators the freedom to control the avatar's body movements.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for generating a user interface, the user interface comprising a front view, a bird's-eye view, a navigation grid map, and a partial environment map, characterized in that, The method includes: A first virtual camera is added to the virtual space, and all virtual objects in the virtual space are scaled until the virtual space is displayed in the camera view of the first virtual camera. A bird's-eye view is determined based on the view obtained by the first virtual camera. The entire virtual space is voxelized, and multiple walking areas and multiple non-walking areas of all avatars are calculated. Each non-walking area is treated as an obstacle object. The edge of the walking area is determined and edge points on the edge are collected. All edge points are connected to generate a polygonal mesh. A mesh map is determined based on the polygonal mesh. The positions of all avatars in the virtual space are added to the mesh map and displayed as first dots to obtain a navigation mesh map. A second virtual camera is added to the virtual space, and the second virtual camera is bound to each of the avatars so that the second virtual camera moves as the avatars walk, wherein the second virtual camera is placed behind the avatars, and a local environment map is determined based on the view of the second virtual camera; The bird's-eye view, navigation grid map, and local environment map are updated based on the changes in the avatar's position in the virtual space.

2. The method for generating a user interface as described in claim 1, characterized in that, The step of updating the navigation grid map based on the changes in the avatar's position in the virtual space includes: A third virtual camera is added to the virtual space, and the virtual space is rendered in real time under the view of the third virtual camera. The view of the first virtual camera is updated according to the changes in the virtual space, thereby updating the bird's-eye view. The navigation grid map is then updated based on the updated bird's-eye view.

3. The method for generating a user interface as described in claim 2, characterized in that, The step of updating the navigation grid map based on the updated bird's-eye view includes: When the avatar's position changes in the virtual space, determine the avatar's view in the first virtual camera; The change in the avatar's position is projected onto the navigation grid map according to the perspective relationship of the first virtual camera, and a corresponding second dot is superimposed on the navigation grid map. The second dot represents the updated position of the avatar.

4. The method for generating a user interface as described in claim 2, characterized in that, The step of updating the navigation grid map based on the updated bird's-eye view further includes: Select an area in the bird's-eye view, zoom in, synchronize the content of the area to the local environment map, and highlight the area in the navigation grid map.

5. A method for controlling avatar movement via a user interface, characterized in that, The user interface is generated by the generation method according to any one of claims 1 to 4, specifically including: Obtain the destination that the avatar is heading to, as determined by the user on the bird's-eye view or the navigation grid map; The avatar's route is determined based on the pathfinding algorithm of the navigation grid, and the avatar is controlled to move towards the destination; Get the user's command to click on any of the three maps: the bird's-eye view, the global navigation map, and the local environment map; Set the virtual camera corresponding to the map clicked by the user as the active camera of the main view so that the view corresponding to the location of the avatar displayed on any of the clicked maps can be displayed in the main view.

6. The method for controlling avatar movement via a user interface as described in claim 5, characterized in that, The method further includes: In the local environment map, the user's selected avatar of interest is obtained, and the chat mode is entered.

7. The method for controlling avatar movement via a user interface as described in claim 6, characterized in that, The method further includes: A fourth virtual camera is added to the virtual space, and the fourth virtual camera is placed directly in front of the avatar's face and the avatar's face is displayed in an isometric view. A new floating view window is added to render the avatar's facial image in real time in the view of the fourth virtual camera, wherein the floating view window is located below the main view.

8. The method for controlling avatar movement via a user interface as described in claim 6, characterized in that, The method further includes: In the chat mode, the user's instructions to drive the avatar's facial expressions using a facial expression-driven method are obtained; The avatar is controlled to make corresponding facial expressions according to the instructions of the facial expression driver.

9. The method for controlling avatar movement via a user interface as described in claim 8, characterized in that, The method further includes: Obtain the chat content between the user and the avatar; The body language of the avatar is determined based on the chat content; The avatar is controlled to perform corresponding limb movements based on the determined limb movements of the avatar.

Images