Entity determination method, apparatus, device and medium in virtual environment

Abstract

The application discloses a method and device for determining an entity in a virtual environment, an apparatus, and a medium, and relates to the field of human-computer interaction. The method comprises the following steps: displaying a master virtual character and a non-player character in a virtual environment; receiving a natural language command, the natural language command being used for controlling the non-player character; and controlling the non-player character to move in the virtual environment according to a target entity indicated by the natural language command, wherein the target entity is determined from the virtual environment in combination with environment perception information, and the environment perception information comprises information perceived by at least one of the master virtual character and the non-player character from the virtual environment. The method can realize intelligent control of the non-player character.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of human-computer interaction, and in particular to a method, apparatus, device, and medium for determining an entity in a virtual environment. Background Art

[0002] In the field of game design and development, AI (Artificial Intelligence) controlling NPC (Non-Player Character) has become a key technology for enhancing game immersion and interactivity.

[0003] In related technologies, players can control the actions of NPCs by issuing simple commands, such as "attack" and "follow." AI can also control the activities of non-player characters based on simple user-generated commands.

[0004] However, simple commands cannot accurately direct non-player characters to perform complex actions. For example, when a user wants to direct a non-player character to move to a specific location, they can only repeatedly issue simple commands such as "forward," "backward," "left," and "right" to direct the non-player character to move closer to the specified location, but cannot accurately control the non-player character to reach the specified location. Summary of the Invention

[0005] The present invention provides a method, apparatus, device, and medium for determining an entity in a virtual environment. The technical solution is as follows:

[0006] In one aspect, a method for determining an entity in a virtual environment is provided, the method comprising:

[0007] displaying at least one of a host virtual character and a non-player character in a virtual environment;

[0008] receiving a natural language command for controlling the non-player character;

[0009] controlling the non-player character to move in the virtual environment according to the target entity indicated by the natural language command;

[0010] The target entity is determined from the virtual environment in combination with environmental perception information, and the environmental perception information includes information perceived by at least one of the main virtual character and the non-player character from the virtual environment.

[0011] In another aspect, a method for determining an entity in a virtual environment is provided, the method comprising:

[0012] Acquire an entity set, the entity set including entity information of at least one entity in the virtual environment;

[0013] Acquiring environmental perception information, wherein the environmental perception information controls information perceived by at least one of the virtual character and the non-player character from the virtual environment;

[0014] The target entity is determined from the entity set according to the target entity information of the target entity and the environment perception information.

[0015] In another aspect, a device for determining an entity in a virtual environment is provided, the device comprising:

[0016] A display module for displaying at least one of a main control virtual character and a non-player character in a virtual environment;

[0017] a receiving module, configured to receive a natural language command, wherein the natural language command is used to control the non-player character;

[0018] a control module, configured to control the non-player character to move in the virtual environment according to a target entity indicated by the natural language command;

[0019] The target entity is determined from the virtual environment in combination with environmental perception information, and the environmental perception information includes information perceived by at least one of the main virtual character and the non-player character from the virtual environment.

[0020] The environmental perception information includes at least one of the following:

[0021] Visual information perceived within the visual field of the master virtual character;

[0022] Auditory information perceived within the auditory range of the master virtual character;

[0023] The information perceived by the master virtual character's perception skills or virtual props;

[0024] Visual information perceived within the field of view of the non-player-controlled character;

[0025] Auditory information perceived within the hearing range of the non-player-controlled character;

[0026] The non-player-controlled character has perception skills or information perceived by the perception virtual props.

[0027] In an optional embodiment, the virtual environment includes multiple candidate entities that match the natural language command, and the target entity is an entity screened from the multiple candidate entities based on the environmental perception information of the main virtual character or the non-player-controlled character.

[0028] In an optional embodiment, the device further includes:

[0029] A first query module is configured to query and obtain the target entity from the entity set of the virtual environment according to the target entity information indicated by the natural language command and the environment perception information;

[0030] The control module is configured to control the non-player-controlled character to perform an activity associated with the target entity according to the intention of the natural language command.

[0031] In an optional embodiment, the first query module is used to parse the natural language command to obtain the target entity information; the target entity information includes at least one of the following: entity type, entity name, entity location, and entity characteristics;

[0032] The first query module is used to calculate the similarity between the target entity information and each entity information in the entity set;

[0033] The first query module is configured to determine the target entity from the entity set according to the similarity and the environment perception information.

[0034] In an optional embodiment, the similarity includes text similarity; the entity set includes entity information of the first entity;

[0035] The first query module is configured to perform word segmentation processing on the target entity information to obtain at least one target entity label;

[0036] The first query module is configured to convert the at least one target entity label into at least one target embedding vector;

[0037] The first query module is configured to obtain entity information of the first entity, wherein the entity information includes a text embedding vector, and the text embedding vector is an embedding vector obtained by converting a text label of the first entity;

[0038] The first query module is used to respectively calculate the text parent similarity between the at least one target embedding vector and the text embedding vector, and obtain at least one text parent similarity corresponding to the at least one target embedding vector;

[0039] The first query module is configured to determine the sum of the at least one text parent similarity as the text similarity between the target entity information and the entity information of the first entity.

[0040] In an optional embodiment, the entity information of the first entity includes at least one text embedding vector; the at least one target embedding vector includes a first target embedding vector;

[0041] The first query module is configured to respectively calculate a text sub-similarity between the first target embedding vector and the at least one text embedding vector to obtain at least one text sub-similarity;

[0042] The first query module is configured to determine a highest value among the at least one text sub-similarity as the text parent similarity corresponding to the first target embedding vector.

[0043] In an optional embodiment, the similarity includes image similarity; the entity set includes entity information of the first entity;

[0044] The first query module is configured to perform word segmentation processing on the target entity information to obtain at least one target entity label;

[0045] The first query module is configured to convert the at least one target entity label into at least one target embedding vector;

[0046] The first query module is configured to obtain entity information of the first entity, wherein the entity information includes an image embedding vector, and the image embedding vector is an embedding vector extracted based on the image of the first entity;

[0047] The first query module is used to respectively calculate the image parent similarity between the at least one target embedding vector and the text embedding vector, and obtain at least one image parent similarity corresponding to the at least one target embedding vector;

[0048] The first query module is configured to determine the sum of the at least one image parent similarity as the image similarity between the target entity information and the entity information of the first entity.

[0049] In an optional embodiment, the entity information of the first entity includes at least one image embedding vector; the at least one target embedding vector includes a first target embedding vector;

[0050] The first query module is configured to respectively calculate image sub-similarity between the first target embedding vector and the at least one image embedding vector to obtain at least one image sub-similarity;

[0051] The first query module is configured to determine a highest value among the at least one image sub-similarity as the image parent similarity corresponding to the first target embedding vector.

[0052] In an optional embodiment, the device further includes:

[0053] A first preprocessing module, configured to obtain at least one perspective image of the first entity;

[0054] A first preprocessing module, configured to obtain at least one text label of the first entity;

[0055] A first preprocessing module is used to call a multimodal model to extract visual features of the first entity based on the at least one perspective image and the at least one text label of the first entity, and obtain the image embedding vector of the first entity.

[0056] In an optional embodiment, the entity set includes a first entity, and the apparatus further includes:

[0057] The first query module is configured to, when the first entity and the target entity have corresponding text similarity and image similarity, determine an average value of the text similarity and the image similarity as the similarity between the first entity and the target entity.

[0058] In an optional embodiment, the first query module is used to determine a search range according to the target entity information and the environment perception information;

[0059] The first query module is configured to filter the target entity from the entities sensed within the search range according to the similarity.

[0060] In an optional embodiment, the first query module is configured to determine the entity with the highest similarity in the search range as the target entity;

[0061] The first query module is configured to determine, when there are at least two entities with the highest similarity in the search range, the entity with the highest similarity in the search range and closest to the master virtual character as the target entity.

[0062] In another aspect, a device for determining an entity in a virtual environment is provided, the device comprising:

[0063] An acquisition module, configured to acquire an entity set, wherein the entity set includes entity information of at least one entity in the virtual environment;

[0064] The acquisition module is used to acquire environmental perception information, wherein the environmental perception information mainly controls information perceived by at least one of the virtual character and the non-player character from the virtual environment;

[0065] The second query module is configured to determine the target entity from the entity set according to the target entity information of the target entity and the environment perception information.

[0066] The environmental perception information includes at least one of the following:

[0067] Visual information perceived within the visual field of the master virtual character;

[0068] Auditory information perceived within the auditory range of the master virtual character;

[0069] The information perceived by the master virtual character's perception skills or virtual props;

[0070] Visual information perceived within the field of view of the non-player-controlled character;

[0071] Auditory information perceived within the hearing range of the non-player-controlled character;

[0072] The non-player-controlled character has perception skills or information perceived by the perception virtual props.

[0073] In an optional embodiment, the virtual environment includes multiple candidate entities that match the natural language command, and the target entity is an entity screened from the multiple candidate entities based on the environmental perception information of the main virtual character or the non-player-controlled character.

[0074] In an optional embodiment, the second query module is used to calculate the similarity between the target entity information of the target entity and each entity information in the entity set;

[0075] The second query module is configured to determine the target entity from the entity set according to the similarity and the environment perception information.

[0076] In an optional embodiment, the similarity includes text similarity; the entity set includes entity information of the first entity;

[0077] The second query module is configured to perform word segmentation processing on the target entity information to obtain at least one target entity label;

[0078] The second query module is configured to convert the at least one target entity label into at least one target embedding vector;

[0079] The second query module is configured to obtain entity information of the first entity, where the entity information includes a text embedding vector, where the text embedding vector is an embedding vector obtained by converting a text label of the first entity;

[0080] The second query module is used to respectively calculate the text parent similarity between the at least one target embedding vector and the text embedding vector, and obtain at least one text parent similarity corresponding to the at least one target embedding vector;

[0081] The second query module is configured to determine the sum of the at least one text parent similarity as the text similarity between the target entity information and the entity information of the first entity.

[0082] In an optional embodiment, the entity information of the first entity includes at least one text embedding vector; the at least one target embedding vector includes a first target embedding vector;

[0083] The second query module is used to calculate the text sub-similarity between the first target embedding vector and the at least one text embedding vector respectively to obtain at least one text sub-similarity;

[0084] The second query module is configured to determine a highest value among the at least one text sub-similarity as the text parent similarity corresponding to the first target embedding vector.

[0085] In an optional embodiment, the similarity includes image similarity; the entity set includes entity information of the first entity;

[0086] The second query module is configured to perform word segmentation processing on the target entity information to obtain at least one target entity label;

[0087] The second query module is configured to convert the at least one target entity label into at least one target embedding vector;

[0088] The second query module is configured to obtain entity information of the first entity, where the entity information includes an image embedding vector, where the image embedding vector is an embedding vector extracted based on the image of the first entity;

[0089] The second query module is used to respectively calculate the image parent similarity between the at least one target embedding vector and the text embedding vector to obtain at least one image parent similarity corresponding to the at least one target embedding vector;

[0090] The second query module is configured to determine the sum of the at least one image parent similarity as the image similarity between the target entity information and the entity information of the first entity.

[0091] In an optional embodiment, the entity information of the first entity includes at least one image embedding vector; the at least one target embedding vector includes a first target embedding vector;

[0092] The second query module is configured to respectively calculate the image sub-similarity between the first target embedding vector and the at least one image embedding vector to obtain at least one image sub-similarity;

[0093] The second query module is configured to determine a highest value among the at least one image sub-similarity as the image parent similarity corresponding to the first target embedding vector.

[0094] In an optional embodiment, the device further includes:

[0095] a second preprocessing module, configured to obtain at least one perspective image of the first entity;

[0096] The second preprocessing module is used to obtain at least one text label of the first entity;

[0097] The second preprocessing module is used to call the multimodal model to extract the visual features of the first entity based on the at least one perspective image and the at least one text label of the first entity, and obtain the image embedding vector of the first entity.

[0098] In an optional embodiment, the entity set includes a first entity;

[0099] The second query module is configured to, when the first entity and the target entity have corresponding text similarity and image similarity, determine an average value of the text similarity and the image similarity as the similarity between the first entity and the target entity.

[0100] In an optional embodiment, the second query module is used to determine a search range according to the target entity information and the environment perception information;

[0101] The second query module is configured to filter the target entity from the entities sensed within the search range according to the similarity.

[0102] In an optional embodiment, the second query module is configured to determine the entity with the highest similarity in the search range as the target entity;

[0103] The second query module is configured to determine the entity with the highest similarity in the search range and the closest distance to the master virtual character as the target entity when the number of entities with the highest similarity in the search range is at least two.

[0104] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the entity determination method in a virtual environment as described above.

[0105] On the other hand, a computer-readable storage medium is provided, in which at least one instruction, at least one program, code set or instruction set is stored. The at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the entity determination method in a virtual environment as described above.

[0106] In another aspect, an embodiment of the present application provides a computer program product or computer program, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method for determining an entity in a virtual environment provided in the aforementioned optional implementation.

[0107] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0108] Based on the information perceived by the controlling virtual character and / or the non-player character from the virtual environment, the target entity indicated in the natural language command is identified. For example, if the natural language command is "move behind the truck," then based on the field of view of the controlling virtual character when issuing the natural language command, the truck that the controlling virtual character can see within the field of view is determined to be the "truck" referred to in the natural language command, and the non-player character is controlled to move accurately to the rear of the truck. Alternatively, if the natural language command is "Is there a stream nearby?", based on the stream sound effect heard by the non-player character in the virtual environment, the non-player character is controlled to explore in the direction of the source of the stream sound effect, and based on the non-player character's visual image, it is identified whether the non-player character has found the stream, and feedback on the natural language command is generated based on the exploration results. By adopting this method, the target entity indicated in the natural language command can be accurately determined from the many entities in the virtual environment based on the environmental perception information. For the natural language commands received with vague descriptions, the target entity indicated by the natural language command can be inferred based on the information perceived by the main virtual character from the virtual environment when the command is issued, and the non-player virtual character can be controlled to perform the behavioral activities related to the target entity indicated in the natural language command, thereby realizing flexible control of the non-player character using natural language. BRIEF DESCRIPTION OF THE DRAWINGS

[0109] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0110] Figure 1is a structural block diagram of a computer system provided by an exemplary embodiment of the present application;

[0111] Figure 2 is a flowchart of a method for determining an entity in a virtual environment provided by another exemplary embodiment of the present application;

[0112] Figure 3 is a schematic diagram of a method for determining an entity in a virtual environment provided by another exemplary embodiment of the present application;

[0113] Figure 4 is a flowchart of a method for determining an entity in a virtual environment provided by another exemplary embodiment of the present application;

[0114] Figure 5 is a flowchart of a method for determining an entity in a virtual environment provided by another exemplary embodiment of the present application;

[0115] Figure 6 is a flowchart of a method for determining an entity in a virtual environment provided by another exemplary embodiment of the present application;

[0116] Figure 7 is a flowchart of a method for determining an entity in a virtual environment provided by another exemplary embodiment of the present application;

[0117] Figure 8 is a schematic diagram of a method for determining an entity in a virtual environment provided by another exemplary embodiment of the present application;

[0118] Figure 9 is a flowchart of a method for determining an entity in a virtual environment provided by another exemplary embodiment of the present application;

[0119] Figure 10 is a flowchart of a method for determining an entity in a virtual environment provided by another exemplary embodiment of the present application;

[0120] Figure 11 is a flowchart of a method for determining an entity in a virtual environment provided by another exemplary embodiment of the present application;

[0121] Figure 12 is a flowchart of a method for determining an entity in a virtual environment provided by another exemplary embodiment of the present application;

[0122] Figure 13 is a flowchart of a method for determining an entity in a virtual environment provided by another exemplary embodiment of the present application;

[0123] Figure 14 is a schematic diagram of a method for determining an entity in a virtual environment provided by another exemplary embodiment of the present application;

[0124] Figure 15 is a flowchart of a method for determining an entity in a virtual environment provided by another exemplary embodiment of the present application;

[0125] Figure 16 is a flowchart of a method for determining an entity in a virtual environment provided by another exemplary embodiment of the present application;

[0126] Figure 17 is a schematic diagram of a method for determining an entity in a virtual environment provided by another exemplary embodiment of the present application;

[0127] Figure 18 is a flowchart of a method for determining an entity in a virtual environment provided by another exemplary embodiment of the present application;

[0128] Figure 19 is a device block diagram of a device for determining an entity in a virtual environment provided by another exemplary embodiment of the present application;

[0129] Figure 20 is a device block diagram of a device for determining an entity in a virtual environment provided by another exemplary embodiment of the present application;

[0130] Figure 21 is a block diagram of a terminal provided by another exemplary embodiment of the present application. DETAILED DESCRIPTION

[0131] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0132] First, a brief introduction to the terms involved in the embodiments of this application is given:

[0133] A virtual environment is a virtual environment displayed (or provided) when an application is running on a terminal. This virtual environment can be a simulation of the real world, a semi-simulated and semi-imaginary environment, or a purely imaginary environment. The virtual environment can be any of two-dimensional, 2.5-dimensional, and three-dimensional virtual environments, and is not limited in this embodiment of the present application.

[0134] Avatar: A vatar refers to an animated object in a virtual environment. This can be a virtual person, virtual animal, or anime character, such as a person displayed in a virtual environment. Alternatively, a vatar is a three-dimensional model created using animation skeletal technology. Each vatar has its own unique shape and volume within the three-dimensional virtual world, occupying a portion of the space within the world.

[0135] User Interface (UI) controls: Any visual control or element visible on an application's user interface. Examples include images, input boxes, text boxes, buttons, labels, and other controls. Some UI controls respond to user actions.

[0136] 3D (three-dimensional) model: In computer graphics and video game design, a 3D model is a three-dimensional object defined by three-dimensional geometric data. This data can be created manually or generated by a 3D scanner. A complete 3D model allows the user to view the object from any angle and may include additional information such as texture, color, and surface material properties. In video games, this type of model is often used to create the image of the player or NPC (non-player character).

[0137] First-person Shooting (FPS) refers to a shooting game that users can play from a first-person perspective. The screen of the virtual environment in the game is a screen of observing the virtual environment from the perspective of the virtual character controlled by the user. In the game, virtual characters of at least two camps play a single-game battle mode in the virtual environment. The virtual characters survive in the virtual environment by avoiding attacks launched by other virtual characters and dangers in the virtual environment (such as gas circles, swamps, etc.). When the health value of the virtual character in the virtual environment is zero, the life of the virtual character in the virtual environment ends, and the virtual character that survives the virtual environment is the winner. Optionally, the competitive mode of the battle (game) can include a single-player battle mode, a two-player team battle mode, or a multiplayer group battle mode. The battle mode in the embodiment of the present application is not limited.

[0138] Figure 1 A block diagram of a computer system provided by an exemplary embodiment of the present application is given. The computer system includes: a terminal 110 and a server 120.

[0139] Terminal 110 has a client 111 installed and running that supports a virtual environment. This client 111 is a client for an application. When the terminal runs client 111, the user interface of client 111 is displayed on the screen of terminal 110. The application can be any of a battle royale shooting game, a virtual reality (VR) application, an augmented reality (AR) program, a three-dimensional map program, a virtual reality game, an augmented reality game, a first-person shooter (FPS), a third-person shooter (TPS), a multiplayer online battle arena (MOBA), or a simulation game (SLG).

[0140] In this embodiment, the client is used as an example for a MOBA game. Terminal 110 is used by a first user 112. During a game session, first user 112 uses terminal 110 to control a first virtual character in a virtual environment. This first virtual character can be referred to as the primary virtual character of first user 112 in the game session. The first virtual character's activities include, but are not limited to, at least one of adjusting body posture, crawling, walking, running, riding, flying, jumping, driving, picking up, shooting, attacking, and throwing.

[0141] Figure 1 Only one terminal is shown, but in different embodiments, multiple other terminals can access the server 120. Optionally, there are one or more terminals corresponding to developers, and a development and editing platform for a client that supports a virtual environment is installed on the terminal. The developer can edit and update the client on the terminal and transmit the updated client installation package to the server 120 via a wired or wireless network. The terminal 110 can download the client installation package from the server 120 to update the client.

[0142] The terminal 110 and other terminals are connected to the server 120 via a wireless network or a wired network.

[0143] Server 120 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Server 120 provides backend services for clients supporting virtual environments. Optionally, server 120 performs primary computing tasks, while terminals perform secondary computing tasks; alternatively, server 120 performs secondary computing tasks, while terminals perform primary computing tasks; alternatively, server 120 and terminals utilize a distributed computing architecture for collaborative computing.

[0144] In an illustrative example, server 120 includes a processor, a user account database, a battle service module, and a user-facing input / output interface (I / O interface). The processor is used to load instructions stored in server 120 and process data in the user account database and the battle service module. The user account database is used to store user account data used by terminal 110 and other terminals, such as user account avatars, user account nicknames, user account combat power indexes, and the service areas where user accounts are located. The battle service module is used to provide multiple battle rooms for users to battle, such as 1v1 battles, 3v3 battles, 5v5 battles, etc. The user-facing I / O interface is used to establish communication and exchange data with terminal 110 via a wireless network or a wired network.

[0145] The embodiments of the present application provide a solution for users to control non-player characters in a virtual environment using natural language voice commands. In this solution, in addition to the main virtual character controlled by the user, the user usually also has one or more NPCs as teammates. The user can use a relatively casual rather than mechanical human conversation method to command the NPC to give the desired feedback to the first entity in the virtual environment, thereby commanding the NPC to complete the task in collaboration with the main virtual character. Figure 1 , the user controls the main virtual character 10 to play the game, and the main virtual character 10 has an NPC teammate 20. There is a car 30 within the field of view of the main virtual character 10. The user speaks a natural language voice command: "No. 2, you go to ambush behind car 30", and then the NPC teammate 20 will automatically move to the back of car 30 to ambush. It should be noted that Figure 1 There may be multiple cars in the scene shown, and the NPC teammate 20 will accurately understand that the car the user is talking about is the car 30 located within the user's field of view. In other words, the NPC teammate 20 has a relatively intelligent natural language understanding ability.

[0146] Combined with reference Figure 1 The program includes at least one of the following five stages:

[0147] Phase 1: Preprocessing of spatial data;

[0148] The server pre-processes the first entity in the virtual environment and constructs spatial data of the virtual environment.

[0149] The spatial data of the virtual environment includes a matching tag of the first entity and a spatial position of the first entity. In some examples, the spatial data also includes a perspective image of the first entity. The first entity is any object that appears in the virtual environment, such as Figure 1Cars, walls, boxes, etc.

[0150] Matching tags for the first entity: first, obtain attribute information of the first entity in the virtual environment, such as at least one of the name, size, and other information of the first entity; and obtain a perspective image of the first entity, where the perspective image includes images obtained by observing the first entity from at least two perspectives. Observing the first entity from multiple perspectives can ensure that the perspective image carries comprehensive appearance information of the first entity.

[0151] The server invokes the multimodal model to predict the attribute information and viewpoint image of the first entity, obtaining an object label for the first entity. The object label describes the first entity in at least one dimension, such as material, transparency, color, and shape. The server then invokes the large language model to optimize the object label and obtain matching labels for the scene objects. The large language model, with its text generation capabilities, rewrites the object labels input to the large language model in a manner consistent with spoken natural language, achieving label optimization and obtaining matching labels for the scene objects.

[0152] Regarding the spatial position of the first entity: the spatial position includes at least one of the following of the first entity: coordinate position (Location), such as the coordinate information of the center point or preset point of the first entity in the virtual environment; orientation (Rotation), such as the direction facing the front of the first entity in the virtual environment; bounding box (Bounding Box), used to indicate the size of the first entity in the virtual environment; cover points (Cover points), used to indicate the recommended virtual character position points when the virtual character approaches the first entity, so that the first entity can cover the virtual character.

[0153] Regarding the perspective image of the first entity, the perspective image is obtained in the process of predicting the matching label of the first entity, and the perspective image includes images obtained by observing the first entity from at least two perspectives.

[0154] Phase 2: Speech recognition and intent recognition;

[0155] When the user controls the main virtual character, the terminal receives the natural language command input by the user in the form of voice; the server performs voice recognition and intent recognition on the natural language command to analyze the user's input instruction;

[0156] Speech recognition is to perform text conversion on the natural language instructions input by the user through voice, obtain the instruction text corresponding to the natural language instructions, and convert the voice information into text information. Call the encoding network to perform feature encoding on the instruction text to obtain the feature representation of the natural language instructions in the latent space; call the text segmentation network to perform segmentation on the feature representation to obtain multiple clauses in the instruction text, and then perform intent recognition on each of the multiple clauses separately. In the case that the natural language instructions input by the user through voice are long sentences, the instruction text corresponding to the natural language instructions can be segmented to meet the needs of analyzing the user's input instructions in the long sentence scenario.

[0157] The following introduces the intent recognition of each clause; the user's input instruction is obtained through intent recognition, and the user's input instruction includes the entity, semantic type, subject type and intent type in the clause.

[0158] Phase 3: Query of spatial data;

[0159] Taking the user's input as input, and the spatial data of the virtual environment and the runtime information corresponding to the master virtual character as reference, the AI ​​model is used to determine the target entity in the virtual space and the control instructions for the NPC;

[0160] For example, the text input command is first parsed to identify the entity within the text command. Then, using the virtual environment's spatial data as the query scope and the controlling virtual character's real-time position and orientation as query reference conditions, the target entity location in the virtual environment corresponding to the input command is retrieved. This allows the NPC to be subsequently controlled to move to the vicinity of the target entity location, or to perform the action indicated by the input command on the target entity location.

[0161] Stage 4: NPC voice feedback;

[0162] The NPC will provide voice feedback to the user. The types of voice feedback include: immediate feedback, command execution feedback, and dynamic feedback. Among them, immediate feedback means that after the user issues a natural language command to the NPC, the NPC immediately provides voice feedback on the natural language command, such as "received", "start execution", "OK", etc.; execution command feedback means that after the NPC executes the control command indicated by the intention of the natural language command, the NPC provides voice feedback on the execution result of the control command, such as "successfully executed", "failed to execute successfully", etc.; dynamic feedback means that when the user does not issue a natural language command to the NPC, the NPC provides voice feedback based on real-time environmental perception.

[0163] The voice feedback is generated by converting text content into speech. This text content is inferred using a large language model combined with the aforementioned spatial data and dynamic runtime information. Using a large language model to dynamically generate text content for voice feedback can overcome the monotony, mechanical, and repetitive nature of fixed, templated voice feedback. It also eliminates the need to pre-store numerous pre-created template voices, thus reducing the overall client data size.

[0164] Stage 5: NPC behavior control;

[0165] These natural language commands typically include control instructions, which are instructions that can be executed by the NPC's behavior tree. These instructions can indicate the subsequent actions the NPC needs to perform, either a single action or a sequence of actions. In the case of a multi-action sequence instruction, the multi-action sequence instruction is used to instruct the NPC to perform a sequence of actions. The data structure storing these multi-action sequence instructions is in list form, caching the pending instructions in the sequence instruction. When the NPC's behavior tree completes the current instruction, the cache is called back and the pending instructions in the cache are sent for execution.

[0166] Hot word update function:

[0167] For phase 2, this solution also introduces a hot word system to improve the accuracy of speech recognition.

[0168] The hot word system is used to provide scene hot words related to the scenes involved in natural language instructions during the speech recognition process, so that when a natural language instruction input by voice is received, it can prioritize searching for words from the hot word library as the instruction analysis result based on the currently running virtual environment, so that the matching degree between the instruction analysis result and the virtual environment is higher.

[0169] The virtual environment includes at least one of a plurality of virtual environments such as a virtual battle scene, a virtual transaction scene, a virtual office scene, and a virtual kitchen scene. Frequently used words in a plurality of virtual environments, specific entities existing in a virtual environment, etc. (such as virtual buildings and virtual characters that only exist in a certain type of scene) are pre-analyzed as scene hot words, and a scene hot word library consisting of a plurality of scene hot words is obtained, and the plurality of scene hot words respectively correspond to a virtual environment. In addition, different scene language models are pre-trained for different virtual environments. For example, the scene language model in the battle scene focuses more on words related to battle, and the scene language model in the transaction scene focuses more on words related to transaction. The scene language model can perform a more targeted analysis of the natural language instructions received in the virtual environment.

[0170] For example: when a user controls a non-player character, a natural language instruction is received. Based on the game state, object position, and current task completion status, the first virtual environment in which the target virtual character (at least one of the main virtual character and the non-player character) is currently located is determined to be a virtual battle scene, and a battle scene language model corresponding to the virtual battle scene is obtained. In addition, based on the field of view of the target virtual character, the first scene hot words in the virtual battle scene are obtained, including: "truck", "virtual grass", "virtual house", "enter", "open truck", "defense", "desert", "attack", "oak tree", "stable", etc. Under the constraints of the first scene hot words, the natural language instruction is decoded and processed by a pre-trained natural language analysis model (including acoustic network, language network, preset dictionary, etc.), and the instruction analysis result is output to accurately control the non-player character. The instruction analysis result is illustrated as follows.

[0171] (1) The command analysis result is "No. 1, you go to the front Defense The command analysis result can be used to specifically control No. 1 to move forward and take a defensive posture, preventing the enemy virtual character from attacking the main virtual character first; if there is no constraint of the hot words in the first scene, it is easy to recognize the natural language command as "No. 1, go to the front and make a sound", thereby affecting the player's virtual combat plan.

[0172] (2) The result of the command analysis is "No. 2, you go desert The command analysis result can be used to specifically control No. 2 to move to the desert to explore whether there are enemy virtual characters or virtual treasure chests and other entities; if there is no constraint of the hot words in the first scene, it is easy to identify the natural language command as "No. 2, you go Mountains "Explore it", which causes No. 2 to move to the wrong position, and the human-computer interaction efficiency is low.

[0173] (3) The command analysis result is "No. 1, No. 2, give me attack ", the command analysis result can be used to specifically control No. 1 and No. 2 to attack nearby attackable virtual characters; if there is no constraint of the hot words in the first scene, it is easy to identify the natural language command as "No. 1 No. 2, give me supply ”, causing No. 1 and No. 2 to make incorrect behaviors that do not meet the players’ expectations, which can neither protect the main virtual character nor easily allow the enemy virtual character to win the virtual game.

[0174] (4) The command analysis result is “No. 1, go find a nearby oak tree”. This command analysis result can be used to specifically control No. 1 to find an oak tree in the nearby area so that he can complete the game task or find an oak tree that can avoid attacks. If there is no constraint of the hot words in the first scene, it is easy to identify the natural language command as “No. 1, go find a nearby oak tree”. Project Book”, causing No. 1 to search nearby for entities that do not meet the player’s expectations and fail to meet the player’s virtual combat needs.

[0175] (5) The command analysis result is "No. 1, No. 2, go over there stable OK", the command analysis result can be used to control No. 1 and No. 2 to search for the nearby stables and move to the location of the stables in a targeted manner; if there is no constraint of the hot words in the first scene, it is easy to recognize the natural language command as "No. 1 No. 2, go over there , right away OK", thus making No. 1 and No. 2 mistakenly believe that the main virtual character currently wants to complete the virtual game on its own, and therefore unable to provide better game assistance to the main virtual character.

[0176] (6) The command analysis result is “No. 2, pick up the props in front”. This command analysis result can specifically control No. 2 to search for props from the front and pick up the props. If there is no constraint of the hot words in the first scene, it is easy to identify the natural language command as “No. 2, pick up the props in front”. Pickled Crab ", which makes it impossible for No. 2 to accurately find the entity that the main virtual character in front wants to pick up. It may prompt the player that "the crab cannot be found", or it may cause the player to pick up the wrong entity "crab", interfering with the player's virtual combat process.

[0177] Ambient sound effect function:

[0178] In addition, the solution also provides a spatial audio enhancement solution for the ambient sound effects of the entire virtual environment.

[0179] The audio played by the terminal includes ambient audio and NPC audio. Ambient audio is generated based on the characteristics of the virtual environment in which the main virtual object is currently located, giving the user an immersive experience. NPC audio is generated based on the characteristics of the NPC, allowing the user to intuitively experience the character's emotions and physical state through the sounds they hear.

[0180] Ambient Audio: This system identifies scene elements within the virtual environment of the controlled virtual object, generates appropriate element sound effects in real time based on these scene elements, or selects appropriate element sound effects from a sound effects library. These element sound effects are then synthesized to create the ambient audio. For example, if the virtual environment is a forest at night, scene elements include trees, owls, and insects, and the ambient audio includes the rustling of leaves, the calls of owls, and the chirping of insects.

[0181] NPC Audio: Identify the NPCs in the virtual environment and generate corresponding voiceovers based on their character type, current emotion, and current behavior. For example, if the NPC is a middle-aged man running, the generated voiceover will have a deep male voice with the sound of running breaths.

[0182] When playing ambient audio and / or NPC audio to the user, the terminal performs audio enhancement processing on the ambient audio and / or NPC audio to improve the realism of the audio.

[0183] In combination with the above description of the implementation environment, the entity determination method in the virtual environment provided by the embodiment of the present application is described.

[0184] Exemplarily, the method is executed by an application program. When the application program is a stand-alone game application program, the method is executed by a terminal.

[0185] When the application is a multi-person online application, the method is executed by a client of the application installed on the terminal, or is executed collaboratively by the terminal and the server.

[0186] Among them, when the method is executed collaboratively by the terminal and the server, the terminal is used to display the picture, receive natural language commands, and report the natural language commands to the server; the server is used to perform target entity recognition according to the natural language commands, and generate control instructions for non-player characters, and send the control instructions to the terminal; the terminal controls the non-player characters to move in the virtual environment according to the control instructions, and renders and displays the corresponding pictures.

[0187] Alternatively, when the method is executed collaboratively by a terminal and a server, the terminal is only used to display images and receive trigger operations and natural language commands; the terminal reports the received trigger operations and natural language commands to the server, and the server performs target entity identification and generates control instructions; the server renders images of non-player characters moving in a virtual environment according to the control instructions, and sends the images to the terminal; the terminal receives and displays the images.

[0188] Figure 2 A flowchart of a method for determining an entity in a virtual environment provided by an exemplary embodiment of the present application is shown. Figure 1 The method includes:

[0189] Step 210 : Display at least one of the main virtual character and the non-player character in the virtual environment.

[0190] Exemplarily, the method can be executed by a client installed on the terminal, where the client is a client of an application program, and the application program is an application program that supports a virtual environment, for example, an application program for a shooting game.

[0191] It should be noted that the calculations in the embodiments of the present application can be performed entirely by the client. For example, the game application is a stand-alone game application, and all calculations are performed by the client. It can also be performed entirely by the server. For example, in a cloud gaming scenario, all calculations of the application are performed by the server, and the client is only responsible for displaying and collecting the user's operation behavior. It can also be completed collaboratively by the client and the server. For example, the client is responsible for part of the calculation and the server is responsible for another part of the calculation.

[0192] Optionally, the terminal displays a game interface, which includes a virtual environment screen, in which the controlling virtual character and non-player characters are displayed. The virtual environment screen is a screen obtained by observing the virtual environment from the perspective of the controlling virtual character.

[0193] The master avatar is a avatar directly controlled by the terminal. Based on pre-set control logic and received control commands, the terminal can directly control the master avatar's behavior. For example, a forward command can cause the master avatar to move forward; a jump command can cause the master avatar to jump upward.

[0194] For example, the terminal displays control widgets corresponding to the main virtual character (e.g., movement widgets, shooting widgets, throwing widgets, jumping widgets, etc.), and the user triggers the control widgets to control the main virtual character's activities in the virtual environment. Alternatively, the terminal is connected to an input device (e.g., a mouse, keyboard, camera, sensor, etc.), and the user issues control commands through the input device, and controls the main virtual character's activities in the virtual environment according to the control commands.

[0195] A non-player character (NPC) is a virtual character controlled by a game application, either by a server or by an artificial intelligence program in a client. The NPC can be directed by a user, who directs the NPC's activities through natural language commands. In embodiments of the present application, the game application can control the behavior of the NPC based on received natural language commands. The game application can understand the intent of the natural language commands and generate control instructions based on the intent to control the NPC to execute the intent indicated by the natural language commands.

[0196] It's important to note that "command" and "control" are different modes of manipulation. Player command of a non-player character (NPC) means that the player uses natural language commands to communicate their intentions (e.g., control intentions or behavioral expectations) to the NPC. The AI ​​will then understand the intent in the natural language commands and generate control instructions based on the NPC's autonomous behavioral capabilities. The AI ​​then uses these control instructions to control the NPC to perform the actions indicated by the intentions.

[0197] When the player controls the main virtual character, the player directly gives control operations. Each control operation corresponds to a control instruction, and the main virtual character directly performs activities according to the control instructions.

[0198] In an optional embodiment, the controlling virtual character is a virtual character controlled by control instructions generated by triggering UI controls; the non-player characters are virtual characters controlled by AI according to natural language commands. When no natural language commands are received, the AI ​​can control the non-player characters' movements in the virtual environment; when natural language commands are received, the AI ​​can control the non-player characters' movements in the virtual environment according to the intent of the natural language commands.

[0199] Exemplarily, the controlling virtual character and the non-player character have an association relationship. For example, the non-player character is a teammate of the controlling virtual character, or the non-player character is a pet of the controlling virtual character, or the non-player character is a servant of the controlling virtual character, or the non-player character is an intelligent robot controlled by the controlling virtual character, or the non-player character is an avatar of the controlling virtual character.

[0200] In an optional embodiment, the game application may provide multiple types of non-player virtual characters, and different types of non-player virtual characters may have different personalities, different behavioral habits, or different autonomous behavioral capabilities.

[0201] For example, a gaming application might feature a sniper NPC with 99% shooting accuracy. Alternatively, a gaming application might feature a forager NPC. Once a forager NPC finds a virtual resource to be collected in the virtual environment, it prioritizes collecting that resource, ignoring the player's natural language commands. Alternatively, a gaming application might feature a directionally challenged NPC. This NPC is required to always follow the controlling NPC. Once the controlling NPC leaves the directionally challenged NPC's field of view, the NPC loses its target and becomes unable to move.

[0202] For example, in a team-based game, a player must team up with at least one other player. If a player lacks teammates, the game application can provide a non-player character as the player's teammate to play alongside the player. During the game, the player can direct the non-player character's actions by issuing voice commands (natural language commands), ensuring that the non-player character cooperates with the main control character to complete the game.

[0203] For example, during a game, players can bring their pets with them to explore the virtual environment. Players can use voice or text commands to direct their pets to move around the virtual environment, exploring unknown areas and facilitating strategic decision-making and deployment for the main virtual character.

[0204] Alternatively, in a MOBA game, a main avatar teams up with four other players to battle against each other. If one of the teammates is offline and unable to control their avatar, for example, the player of the first avatar is offline and unable to control the first avatar, the main avatar can use voice commands to direct the first avatar during the teammate's offline period, allowing the first avatar to assist the other teammates in team combat or to direct the first avatar to take precise actions in response to changes in the battle situation.

[0205] Step 220: Receive a natural language command, where the natural language command is used to control a non-player character.

[0206] Natural language commands are instructions conveyed using natural human language. The terminal receives natural language commands from the player and controls non-player characters based on the intent corresponding to the natural language commands. For example, the terminal receives the player's voice audio and converts the audio into text to generate the natural language commands. Alternatively, the terminal receives the text of the natural language commands input by the player.

[0207] Players can issue natural language commands using spoken or written language. The game application uses a large language model to understand the natural language commands, extract the intent expressed in the natural language commands, and generate control instructions based on the intent to control the activities of non-player characters.

[0208] For example, the natural language command may be "Come to me", then the game application can parse out that the intention of the natural language command is "control the non-player virtual character to move to the location of the master virtual character", then the game application obtains the location of the master virtual character, generates a navigation route for the non-player virtual character to move to the location of the master virtual character, and controls the non-player virtual character to move to the location of the master virtual character according to the navigation route.

[0209] For another example, the natural language command may be "go pick up the treasure chest", then the game application can parse out that the intention of the natural language command is "control the non-player virtual character to move to the location of the treasure chest and pick up the treasure chest", then the game application can obtain the location of the treasure chest, control the non-player virtual character to move to the location of the treasure chest, and perform the operation of picking up the treasure chest after arriving at the location of the treasure chest.

[0210] Optionally, the natural language commands are used to control the non-player character to perform an action related to a target entity. For example, the target entity may be a destination for the non-player character to move to, an object that the non-player character needs to observe, a target that the non-player character needs to attack, or an object that the non-player character needs to interact with.

[0211] A target entity is a virtual object that exists in a virtual environment. A target entity is the entity described in a natural language command. An entity can refer to a virtual object that is stationary in a virtual environment. For example, a target entity can be a virtual building, virtual terrain, virtual vehicle, virtual plant, virtual prop, virtual item, etc. An entity can also refer to a virtual object in a virtual environment that can interact with a virtual character. For example, a target entity can be an interaction point in the virtual environment (e.g., a door, window, cabinet, cellar, etc.), a virtual prop, a virtual character, a virtual light source, etc.

[0212] For example, a natural language command might be "Please open the door for me," in which case the target entity might be "door," and the non-player character might need to perform a door-related action, "Open the door." Alternatively, a natural language command might be "Is the kitchen safe?" in which case the target entity might be "kitchen," and the non-player character might need to perform a kitchen-related action, "Check the kitchen for hostile avatars or other dangerous situations."

[0213] Optionally, the natural language command includes target entity information of the target entity. The target entity information may be text describing the target entity. For example, the target entity information may include at least one of the following: the name, type, location, and characteristics of the target entity. The game application can use the target entity information in the natural language to locate the target entity from among multiple entities in the virtual environment.

[0214] Step 230 , controlling the non-player character to move in the virtual environment according to the target entity indicated by the natural language command.

[0215] The game application parses the natural language command, extracts the target entity information therein, determines the target entity from the entity set of the virtual environment based on the target entity information, and then controls the non-player character to perform activities related to the target entity according to the intention of the natural language command.

[0216] The target entity is determined from the virtual environment in combination with environmental perception information, and the environmental perception information includes information perceived by at least one of the main virtual character and the non-player character from the virtual environment.

[0217] Since the target entity information in natural language commands is usually vague, for example, a natural language command can be "move to the red truck", where the target entity information is "red truck", and there may be many red trucks in the virtual environment. It is impossible to accurately determine the target entity in the virtual environment based solely on the target entity information in the natural language command.

[0218] Therefore, embodiments of the present application provide a method for determining a target entity by combining target entity information and environmental perception information. For the previous example, the "red truck" expressed by the player in the natural language command should be a red truck that the player can see. Therefore, based on the field of view of the controlling virtual character, the red truck within the field of view of the controlling virtual character can be selected from multiple red trucks in the virtual environment. This red truck is the target entity indicated by the player in the natural language command.

[0219] Because natural language commands are issued based on the player's perception of the virtual environment, the game application uses the player's environmental perception information at the time of the command to accurately identify the target entity. Based on the player's perception of the virtual environment, the game application infers the entity in the virtual environment that is closest to the target entity and is considered the target entity.

[0220] Exemplarily, the virtual environment includes multiple candidate entities that match the natural language command, and the target entity is an entity selected from the multiple candidate entities based on the environmental perception information of the controlling virtual character or the non-player character. For example, the game application first selects multiple candidate entities from the entity set that match the target entity information in the natural language command, and then selects an entity that can be perceived by the controlling virtual character or the non-player character from the multiple candidate entities as the target entity based on the environmental perception information.

[0221] Exemplarily, the environmental perception information includes at least one of the following:

[0222] Visual information perceived within the field of view of the controlling virtual character;

[0223] Auditory information perceived within the hearing range of the controlling avatar;

[0224] The perception skills possessed by the controlling virtual character or the information perceived by the virtual props;

[0225] Visual information perceived within the field of view of non-player characters;

[0226] Auditory information perceived within the hearing range of non-player characters;

[0227] Information perceived by non-player characters through their perception skills or virtual items.

[0228] For example, the environmental perception information of the main virtual character may include: the pictures or entities that the main virtual character can see when observing the virtual environment, the environmental sound effects that the main virtual character can hear, the source direction of the environmental sound effects, the type and volume of the environmental sound effects, the perception information obtained by the main virtual character through the use of perception skills (for example, visual information, auditory information, sound wave information, light reflection information, etc.), and the perception information obtained by the main virtual character through the use of perception virtual props (for example, the sensor signal of the sensor, the positioning signal of the positioning props, etc.).

[0229] The environmental perception information of the non-player character may include: the images or entities that the non-player character can see when observing the virtual environment, the environmental sounds that the non-player character can hear, the source direction of the environmental sounds, the type and volume of the environmental sounds, the perception information obtained by the non-player character through the use of perception skills (for example, visual information, auditory information, sound wave information, light reflection information, etc.), and the perception information obtained by the non-player character through the use of perception virtual props (for example, sensor signals from sensors, positioning signals from positioning props, etc.).

[0230] For example, Figure 3 As shown, when the natural language command is "Come to the stairs in front of me", the game application can determine that the target entity is the stairs 301 located in front of the main virtual character based on the field of view of the main virtual character.

[0231] It should be noted that environmental perception information can include both the real-time perception of the controlling avatar and non-player characters upon receiving a natural language command, and can also include historical perception information of the controlling avatar and non-player characters prior to receiving the natural language command. In other words, the target entity may be an entity currently visible to the player, or an entity the player has previously seen. Therefore, the game application needs to combine the real-time and historical environmental perception information of the controlling avatar and / or non-player characters to identify the target entity.

[0232] For example, when the natural language command is "Let's go back to the hotel we just passed by", the game application needs to query the hotels that the main virtual character has passed by based on the historical environmental perception information of the main virtual character.

[0233] In summary, the method provided in this embodiment identifies the target entity indicated in a natural language command based on information perceived by the controlling virtual character and / or non-player characters from the virtual environment. For example, if the natural language command is "Move behind the truck," then based on the controlling virtual character's field of view at the time the natural language command is issued, the truck visible to the controlling virtual character within that field of view is determined to be the "truck" referred to in the natural language command, and the non-player character is then controlled to move precisely behind the truck. Alternatively, if the natural language command is "Is there a stream nearby?", based on the stream sound the non-player character hears in the virtual environment, the non-player character is controlled to explore in the direction of the stream sound. Based on the non-player character's visual image, it is determined whether the non-player character has discovered the stream, and feedback for the natural language command is generated based on the exploration results. By adopting this method, the target entity indicated in the natural language command can be accurately determined from the many entities in the virtual environment based on the environmental perception information. For the natural language commands received with vague descriptions, the target entity indicated by the natural language command can be inferred based on the information perceived by the main virtual character from the virtual environment when the command is issued, and the non-player virtual character can be controlled to perform the behavioral activities related to the target entity indicated in the natural language command, thereby realizing flexible control of the non-player character using natural language.

[0234] Figure 4 A flowchart of a method for determining an entity in a virtual environment provided by an exemplary embodiment of the present application is shown. Figure 1 The method includes:

[0235] Step 240: Acquire an entity set, where the entity set includes entity information of at least one entity in the virtual environment.

[0236] Exemplarily, the identification of the target entity may be performed by the client or by the server.

[0237] For example, the client reports the received natural language command to the server. The server parses the natural language command and obtains the target entity information. Then, the execution Figure 4 The method shown determines a target entity indicated by target entity information from a set of entities in a virtual environment based on environmental perception information.

[0238] Step 250: Acquire environmental perception information, where the environmental perception information includes information perceived by at least one of the controlling virtual character and the non-player character from the virtual environment.

[0239] Step 260 : determining a target entity from the entity set according to the target entity information and the environment perception information of the target entity.

[0240] Exemplarily, the method for the server to determine the target entity from the entity set is the same as Figure 2 The embodiments shown are the same except that Figure 2 The embodiment shown is executed by the client, Figure 4 The illustrated embodiment is executed by a server, and the specific method will not be described in detail here.

[0241] Exemplarily, the server can determine a list of candidate entities that match the target entity information from the entity set based on the target entity information of the target entity; determine the perception range of the main virtual character and / or non-player character based on the environmental perception information, and use the perception range as the search range; and determine the entity in the candidate entity list that is within the search range as the target entity.

[0242] Alternatively, the server determines the perception range of the master virtual character and / or non-player character based on the environmental perception information, uses the perception range as the search range, and determines the entities within the search range as candidate entities; the candidate entities are matched with the target entity information of the target entity in turn for similarity, and the candidate entity with the highest similarity is determined as the target entity.

[0243] In summary, the method provided in this embodiment identifies the target entity indicated in a natural language command based on information perceived by the controlling virtual character and / or non-player characters from the virtual environment. For example, if the natural language command is "Move behind the truck," then based on the controlling virtual character's field of view at the time the natural language command is issued, the truck visible to the controlling virtual character within that field of view is determined to be the "truck" referred to in the natural language command, and the non-player character is then controlled to move precisely behind the truck. Alternatively, if the natural language command is "Is there a stream nearby?", based on the stream sound the non-player character hears in the virtual environment, the non-player character is controlled to explore in the direction of the stream sound. Based on the non-player character's visual image, it is determined whether the non-player character has discovered the stream, and feedback for the natural language command is generated based on the exploration results. By adopting this method, the target entity indicated in the natural language command can be accurately determined from the many entities in the virtual environment based on the environmental perception information. For the natural language commands received with vague descriptions, the target entity indicated by the natural language command can be inferred based on the information perceived by the main virtual character from the virtual environment when the command is issued, and the non-player virtual character can be controlled to perform the behavioral activities related to the target entity indicated in the natural language command, thereby realizing flexible control of the non-player character using natural language.

[0244] Exemplarily, an exemplary embodiment of determining a target entity from multiple entities in a virtual environment is given.

[0245] Figure 5A flowchart of a method for determining an entity in a virtual environment provided by an exemplary embodiment of the present application is shown. Figure 1 Based on the terminal in Figure 2 In the illustrated embodiment, step 230 may include step 231 and step 232 .

[0246] Step 210 : Display at least one of the main virtual character and the non-player character in the virtual environment.

[0247] Optionally, the terminal may receive control operations to control the activities of the master virtual character, and receive natural language commands to control the activities of non-player virtual characters.

[0248] For example, in a shooting game, a player can be assigned an AI teammate (non-player character). The player can give the AI ​​teammate any natural language commands to instruct the AI ​​teammate to scout, fight, or cooperate with him to complete tactical cooperation tasks.

[0249] Step 220: Receive a natural language command, where the natural language command is used to control a non-player character.

[0250] For example, a game application displays a game interface, which includes an AI communication control. In response to an operation that triggers the AI ​​communication control, voice audio recording begins; in response to triggering the AI ​​communication control again, voice audio recording stops, and voice audio is obtained; the voice audio is converted into text to obtain natural language commands.

[0251] Step 231 : According to the target entity information indicated by the natural language command and the environment perception information, a target entity is queried from the entity set of the virtual environment.

[0252] In one optional embodiment, the client queries and obtains the target entity from a set of entities in the virtual environment based on the target entity information indicated by the natural language command and the environment perception information. In another optional embodiment, the client reports the natural language command to the server, and the server queries and obtains the target entity from a set of entities in the virtual environment based on the target entity information indicated by the natural language command and the environment perception information.

[0253] Optionally, the game application infers the target entity based on the target entity information, the environment perception information, and the pre-processed virtual environment data, wherein the pre-processed virtual environment data includes a set of entities in the virtual environment.

[0254] The entity set includes entity information of each entity in the virtual environment. The entity information includes at least one of the following: name, type, location, features, text label, text label embedding vector, image, and image embedding vector.

[0255] The text label may be at least one word obtained after segmenting a feature (for example, a descriptive text of an entity appearance), and the embedding model is called to obtain an embedding vector corresponding to each text label.

[0256] The image may include an image of a three-dimensional model of the entity observed from at least one direction. For example, the image may include three views of the entity. The image embedding vector is the embedding vector of the visual feature label. The visual feature label is obtained by using a multimodal model to perform feature recognition based on the image and text label of the entity.

[0257] For example, Figure 6 As shown, to control the non-player character to move to any position in the virtual environment, it is necessary to search for all entities in the virtual environment based on natural language commands. Therefore, it is necessary to pre-process the virtual environment data first to generate a searchable entity set. Optionally, the entities in the entity set can include two categories: area / building 403, and other entity objects 404. The entity information of the area / building can be manually calibrated. Since the area / building 403 is usually large in scope and there may be subspaces inside, such as there may be rooms in a building, it is necessary to manually calibrate the position, orientation, etc. of the area, building and its subspace. The other entity objects 404 can automatically export their entity information such as position, orientation, bounding box, cover point, etc. through the game engine. This entity information exists in the game engine assets themselves and can be directly recorded.

[0258] For example, the entity set includes first-category entities and second-category entities. The first-category entities include at least one of regions and buildings, and the second-category entities include physical objects. The text labels for the first-category entities are manually labeled. For example, the text labels for a room in a building might be: region, building, floor, room. The text labels for the second-category entities are automatically generated by invoking a large language model.

[0259] A method for generating text labels for the second type of entity may include: obtaining at least one image of the entity, inputting the at least one image into a large language model to obtain a description text of the entity; segmenting the description text to obtain at least one text label for the entity, where each text label includes a word obtained after the segmentation process; and performing a vectorization operation on the text label to obtain an embedding vector corresponding to the text label.

[0260] For example, Figure 7 As shown, the text label generation process for an oil barrel includes: shooting Figure 8The oil barrel 501 is shown with nine images from different perspectives. A large language model is then called to output a descriptive text 405 for the oil barrel, "a metal oil barrel," based on the nine images. The descriptive text is segmented 406 to obtain at least one text label. For example, "a metal oil barrel" is segmented into "metal" and "oil barrel," representing all the characteristics of the oil barrel. The segmentation can be processed using a large language model, and an embedding model 407 is called to perform a vectorization operation on each word in the segmentation result, generating an embedding vector and recording the embedding vector index corresponding to each text label of the oil barrel.

[0261] Embedding vectors are high-dimensional vector data of text labels. Pre-generating embedding vectors for text labels eliminates the need to repeatedly generate vectors for each entity's text label when matching target entities from an entity set based on target entity information, thereby improving the matching efficiency between target entity information and entity information.

[0262] It's important to note that this method doesn't use the description text directly as the text label. Instead, it uses the word segmentation results of the description text as the text label. This is because description texts often vary in length, and the embedding vectors of longer description texts are less effective when performing actual searches and comparing similarities. For example, when searching for "a small blue truck with rusty body" using the word "truck," "a small blue truck with rusty body" will likely be ranked after "a car" in the recall results. However, in reality, the search for "truck" should prioritize searching for "truck" regardless of the complexity of its additional description. Therefore, when performing similarity searches, the comparison should be based on individual feature words, not the description text.

[0263] In addition, in order to further extract the visual features of the entity, the method provided in the embodiment of the present application can also further extract the hidden visual features in the image of the entity based on the text label and image of the entity. Taking the first entity as an example, Figure 9 As shown, the method includes: obtaining at least one perspective image of the first entity 408; obtaining at least one text label of the first entity 409; calling a multimodal model 410, extracting visual features of the first entity based on at least one perspective image and at least one text label of the first entity, and obtaining an image embedding vector 411 of the first entity.

[0264] Since text labels are manually annotated or generated by a large language model based on human language characteristics, and text labels extracted based on human language habits may ignore some features of the entity. For example, when manually annotating a text label for an oil drum, only more eye-catching text labels such as "metal" and "rusty" may be annotated, while ignoring detailed features such as "rusty", "blue", "yellow", and "paint peeling in the lower right corner". Therefore, the method provided in the embodiment of the present application also uses a multimodal model based on the entity's text label and entity image, so that the multimodal model extracts more visual features from it; based on the visual features, it performs image similarity matching with the target entity to improve the recognition accuracy of the target entity.

[0265] The multimodal model training method can include: inputting sample entity images and sample labels into a pre-trained multimodal model, and fine-tuning the pre-trained multimodal model based on the loss of the predicted labels and sample labels output by the pre-trained multimodal model. This results in a multimodal model that can output visual feature labels based on the input text labels and images. The visual feature labels include more detailed descriptive text extracted from the image. For example, the text description of "a metal oil drum" can only be associated with the two text labels "metal" and "oil drum." However, the Clip (multimodal) model can also identify hidden visual information in the image of the oil drum, such as the visual feature labels "rusty," "blue," and "yellow." These visual features are not present in the text labels. Therefore, combining the Clip model with visual feature search can further improve the accuracy of entity search.

[0266] In an optional embodiment, after obtaining the entity set, the target entity can be matched from the entity set according to the target entity information in the natural language command. Figure 10 As shown, the game application parses the input natural language command 401 to obtain target entity information; performs target entity query 402 based on the preprocessed virtual environment data and the real-time position and orientation of the main virtual character; finds the target entity that is within the field of view of the main virtual character and matches the target entity information from the preprocessed virtual environment data, and then determines the position of the target entity, so as to determine the target movement position of the non-player character.

[0267] In an optional embodiment, as Figure 11 As shown, step 231 may include steps 2311 to 2313. The method may be executed by a terminal or a server.

[0268] Step 2311, parse the natural language command to obtain target entity information; the target entity information includes at least one of the following: entity type, entity name, entity location, and entity characteristics.

[0269] Optionally, the game application calls a large language model to parse the natural language command to obtain target entity information of the target entity, and calls an embedding model to vectorize the target entity information to obtain a target embedding vector of the target entity information.

[0270] For example, if the natural language command is "Come to the blue truck in front", the target entity information that can be parsed by the large language model includes: the direction information "front" and the entity name "blue truck".

[0271] For example, named entity recognition technology in the field of natural language processing can also be used to extract target entity information from natural language commands. Named entity recognition technology is used to identify entities with specific meanings in text, such as names of people, places, locative words, adjectives, etc.

[0272] For example, Figure 12 As shown, natural language command 412 is "Go to the motel on the first floor and find a cardboard box behind the red sofa." Using named entity recognition technology, we can identify the following: the architectural noun "motel first floor"; the object nouns "sofa" and "cardboard box"; the locative word "behind"; and the adjective "red." After logical construction, a hierarchical scene query call form 413 is formed, which contains search type, content, locative words, and constraint information. The search type is used to narrow the data retrieval scope based on similarity matching, the search content is a specific entity description, and the adjectives are incorporated into the description text for querying together. The locative words are used to constrain the search scope. The floor constraint is determined by the player's location. In indoor scenes, the range of up and down should be narrowed to avoid finding entities that are not visible across floors.

[0273] Step 2312: Calculate the similarity between the target entity information and each entity information in the entity set.

[0274] For example, the entity set includes entity information for at least one entity, and the entity information for each entity may include at least one of a text embedding vector for a text label and an embedding vector for a visual feature. The target entity information may then be similarity-calculated with the text embedding vector and the image embedding vector, respectively, and the entity with the highest similarity may be determined as the target entity.

[0275] The following are methods for similarity matching with text embedding vectors and similarity matching with image embedding vectors.

[0276] 1. Similarity includes the text similarity between the target entity information and the text embedding vector.

[0277] Taking the first entity in the entity set as an example, the game application (client or server) segments the target entity information to obtain at least one target entity tag; converts the at least one target entity tag into at least one target embedding vector; obtains the entity information of the first entity, the entity information includes a text embedding vector, and the text embedding vector is an embedding vector obtained based on the text tag conversion of the first entity; calculates the text parent similarity between the at least one target embedding vector and the text embedding vector respectively, and obtains at least one text parent similarity corresponding to the at least one target embedding vector; and determines the sum of the at least one text parent similarity as the text similarity between the target entity information and the entity information of the first entity.

[0278] For example, there are two target entity labels, target entity label 1 and target entity label 2, and there are also two entities in the entity set, entity 1 and entity 2. Entity 1 corresponds to text embedding vector 1, and entity 2 corresponds to text embedding vector 2. Similarity 1 is calculated between target entity label 1 and text embedding vector 1, and similarity 2 is calculated between target entity label 2 and text embedding vector 1. The sum of similarity 1 and similarity 2 is determined as the text similarity between the target entity information and entity 1. Similarity 3 is calculated between target entity label 1 and text embedding vector 2, and similarity 4 is calculated between target entity label 2 and text embedding vector 2. The sum of similarity 3 and similarity 4 is determined as the text similarity between the target entity information and entity 2.

[0279] Exemplarily, the entity information of the first entity includes at least one text embedding vector; at least one target embedding vector includes a first target embedding vector; the game application calculates the text sub-similarity between the first target embedding vector and the at least one text embedding vector respectively to obtain at least one text sub-similarity; and the highest value of the at least one text sub-similarity is determined as the text parent similarity corresponding to the first target embedding vector.

[0280] For example, the number of target entity tags is 1: target entity tag 1. The number of entities in the entity set is also 1: entity 1. Entity 1 corresponds to text embedding vector 1 and text embedding vector 3. Similarity 1 is calculated between target entity tag 1 and text embedding vector 1, and similarity 5 is calculated between target entity tag 1 and text embedding vector 3. The larger value of similarity 1 or similarity 5 is determined as the similarity between the target entity information (target entity tag 1) and entity 1.

[0281] For example, for the input target entity information "blue car," we first perform word segmentation on the target entity information, breaking it down into the target entity labels "blue" and "car," indicating that the query target entity has two features. We then calculate similarity between each feature and each entity in the entity set. The maximum similarity score for each feature is taken, and finally, the sum of the similarity scores corresponding to all query features is taken, representing the textual similarity between the target entity information and the query entity.

[0282] like Figure 13 As shown in the figure, the target entity information includes two target entity labels, "blue" and "car", which are converted into two target embedding vectors. Then, the entity information in the entity set is obtained. For example, the entity set includes entity 1 and entity 2. The text labels of entity 1 include "metal", "old", "car", "truck", "blue", and "damaged", and the text labels of entity 2 include "metal", "scratched", "car", "damaged", "armored truck", and "black".

[0283] Calculate the text similarity between the target entity information and entity 1: first calculate the similarity between the target entity label "blue" and each text label of entity 1, and take the maximum value. For example, the similarity between the target entity label "blue" and the text label "blue" of entity 1 is 1 (1 is the maximum value); then calculate the similarity between the target entity label "car" and each text label of entity 1, and take the maximum value. For example, the similarity between the target entity label "car" and the text label "car" of entity 1 is 1. Then sum the two similarities of the target entity labels "car" and "blue" to obtain the final similarity between the target entity information and entity 1, which is 2.

[0284] Similarly, calculate the text similarity between the target entity information and entity 2: first calculate the similarity between the target entity label "blue" and each text label of entity 2, and take the maximum value. For example, the similarity between the target entity label "blue" and the text label "black" of entity 1 is 0.91; then calculate the similarity between the target entity label "car" and each text label of entity 1, and take the maximum value. For example, the similarity between the target entity label "car" and the text label "car" of entity 2 is 1 (1 is the maximum value). Then sum the two similarities of the target entity labels "car" and "blue" to obtain the final similarity between the target entity information and entity 2, which is 1.91.

[0285] It can be seen that the similarity between the target entity information and entity 1 is 2, which is higher than the similarity between the target entity information and entity 2, which is 1.91.

[0286] 2. Similarity includes the image similarity between the target entity information and the image embedding vector.

[0287] Taking the first entity in the entity set as an example, the game application (client or server) segments the target entity information to obtain at least one target entity tag; converts the at least one target entity tag into at least one target embedding vector; obtains the entity information of the first entity, the entity information includes an image embedding vector, and the image embedding vector is an embedding vector obtained based on the image extraction of the first entity; calculates the image parent similarity of the at least one target embedding vector and the text embedding vector respectively, and obtains at least one image parent similarity corresponding to the at least one target embedding vector; and determines the sum of the at least one image parent similarity as the image similarity between the target entity information and the entity information of the first entity.

[0288] For example, there are two target entity labels, target entity label 1 and target entity label 2, and there are also two entities in the entity set, entity 1 and entity 2. Entity 1 corresponds to image embedding vector 1, and entity 2 corresponds to image embedding vector 2. Similarity 1 is calculated between target entity label 1 and image embedding vector 1, and similarity 2 is calculated between target entity label 2 and image embedding vector 1. The sum of similarity 1 and similarity 2 is determined as the image similarity between the target entity information and entity 1. Similarity 3 is calculated between target entity label 1 and image embedding vector 2, and similarity 4 is calculated between target entity label 2 and image embedding vector 2. The sum of similarity 3 and similarity 4 is determined as the image similarity between the target entity information and entity 2.

[0289] Exemplarily, the entity information of the first entity includes at least one image embedding vector; the at least one target embedding vector includes a first target embedding vector; the game application calculates the image sub-similarity between the first target embedding vector and the at least one image embedding vector respectively to obtain at least one image sub-similarity; and the highest value of the at least one image sub-similarity is determined as the image parent similarity corresponding to the first target embedding vector.

[0290] For example, the number of target entity tags is 1: target entity tag 1. The number of entities in the entity set is also 1: entity 1. Entity 1 corresponds to image embedding vector 1 and image embedding vector 3. Similarity 1 is calculated between target entity tag 1 and image embedding vector 1, and similarity 5 is calculated between target entity tag 1 and image embedding vector 3. The larger value of similarity 1 or similarity 5 is determined as the similarity between the target entity information (target entity tag 1) and entity 1.

[0291] In an optional embodiment, the target image embedding vector can be used to calculate image similarity, that is, the target image embedding vector is used instead of the target embedding vector in the above method. The target image embedding vector can be obtained by: segmenting the target entity information to obtain at least one target entity label; and inputting the at least one target entity label into a multimodal model to obtain the target image embedding vector.

[0292] like Figure 14 As shown in the figure, based on the target entity information "red door", through image similarity search, many features that do not exist in the text labels can be found. For example, the feature "red" does not exist in the text labels of many "red door" entities, but through image similarity search, entities with red door features can still be searched and images of these entities can be returned. For example, Figure 14 (1) is a wooden door, and the image similarity is 0.4898; Figure 14 (2) is a door made of wood, and the image similarity is 0.4895; Figure 14 (3) in the image is a metal door, and the image similarity is 0.4539; Figure 14 (4) in the image is a rusty metal door, and the image similarity is 0.4494; Figure 14 (5) in the image is a rusty metal door, and the image similarity is 0.4605; Figure 14 (6) is a door frame made of wood, with a crossbeam on the top of the door, a column on each side of the door, and a crossbeam on the top of the column. The image similarity is 0.4581; Figure 14 (7) in the image is a rusty metal door with a rectangular window in the middle. The image similarity is 0.4439. Figure 14 (8) is a rusty metal door, and the image similarity is 0.4439.

[0293] 3. Similarity includes text similarity and image similarity.

[0294] When the first entity and the target entity have corresponding text similarity and image similarity, the game application determines an average value of the text similarity and the image similarity as the similarity between the first entity and the target entity.

[0295] Alternatively, when the first entity and the target entity have corresponding text similarity and image similarity, the larger value of the text similarity and the image similarity is determined as the similarity between the first entity and the target entity.

[0296] Alternatively, when the first entity and the target entity have corresponding text similarity and image similarity, the sum of the text similarity and the image similarity is determined as the similarity between the first entity and the target entity.

[0297] Alternatively, the weighting coefficients of text similarity and image similarity are pre-set. When the first entity and the target entity have corresponding text similarity and image similarity, the weighted average of the text similarity and the image similarity is determined as the similarity between the first entity and the target entity. In addition, different weighting coefficients can be configured for different types of entities. For example, when there are entities in the virtual environment that are easily confused in appearance but have large differences in text expression (such as microwave ovens and safes), players may confuse the two and misidentify them. When calculating the similarity of such entities, the weight of image similarity is configured higher, which makes it easier to accurately judge the user's true intentions.

[0298] Step 2313: Determine the target entity from the entity set based on the similarity and the environment perception information.

[0299] In an optional embodiment, as Figure 15 As shown, the game application first performs a similarity search 414 based on the target entity information, obtains at least one candidate entity with a high similarity, and sorts them according to the similarity; then, based on the position, position, direction and other information of the main virtual character and / or the non-player character, the game application performs perceptual filtering and sorting 415, and finally obtains the target entity.

[0300] In an optional embodiment, the game application determines a search range based on environmental perception information; and filters the entities perceived within the search range based on similarity to obtain a target entity.

[0301] Determining the search range based on the environmental perception information may include at least one of the following:

[0302] According to the position and orientation of the master virtual character, the visual range of the master virtual character is determined as the search range;

[0303] According to the position of the master virtual character, the hearing range of the master virtual character is determined as the search range;

[0304] Determine the detection range of the perception virtual props used by the master virtual character as the search range;

[0305] According to the position and orientation of the non-player character, the field of view of the non-player character is determined as the search range;

[0306] According to the position of the non-player character, the hearing range of the non-player character is determined as the search range;

[0307] Determine the detection range of the perception virtual props used by the non-player character as the search range;

[0308] According to the position and orientation of the controlling virtual character, the position and orientation of the non-player characters, the overlapping portion of the visual ranges of the controlling virtual character and the non-player characters is determined as the search range;

[0309] According to the position of the controlling virtual character and the position of the non-player character, the overlapping part of the hearing range of the controlling virtual character and the non-player character is determined as the search range.

[0310] In another optional embodiment, the game application determines a search range based on target entity information and environmental perception information; and filters the entities perceived within the search range based on similarity to obtain the target entity.

[0311] For example, when the target entity information includes the target entity's location information, a search range is determined based on the target entity's location information and at least one of the control virtual character's and / or non-player character's field of view, hearing range, and virtual item search range indicated by the environmental perception information. The search range is an estimated range within which the target entity is perceived by the target virtual character and / or non-player character. Exemplarily, the game application determines a perception range based on the environmental perception information. The perception range includes at least one of the following: the control virtual character's field of view, the control virtual character's hearing range, the non-player character's field of view, the non-player character's hearing range, and the detection range of perceived virtual items. The area within the perception range that matches the location information is determined as the search range.

[0312] Alternatively, the game application determines x entities with the highest similarity from the entity set based on similarity to form a candidate entity list; determines a search range based on target entity information and environmental perception information, and determines entities within the search range among the x entities as target entities, where x is a positive integer.

[0313] Exemplarily, the game application determines the entity with the highest similarity in the search range as the target entity.

[0314] It should be noted that when there are multiple entities with equal similarity within the search range, the multiple entities can be sorted according to their distance from the controlling virtual character. For example, if there are at least two entities with the highest similarity within the search range, the entity with the highest similarity within the search range and the closest distance to the controlling virtual character will be determined as the target entity.

[0315] For example, when a natural language command includes a fuzzy positional description of a target entity, the game application can perform text matching based on the entity location recorded in the entity set and the position described in the natural language command, and query to obtain a target entity whose location better matches the natural language command. When the target entity information includes the target entity's location information, the game application can call upon a large language model to generate a real-time relative position description of each entity based on the location of each entity recorded in the entity set and the current location of the controlling virtual character and / or non-player character. The real-time relative position description is used to describe the relative positional relationship between the entity and the controlling virtual character / non-player character. The text similarity calculated by the game application includes the similarity between the location information and the real-time relative position description.

[0316] For example, Figure 16 As shown, the search range is determined according to the position and direction 416 of the master virtual character. For example, if the target entity information indicates that the target entity is located in front of the master virtual character, the following can be constructed: Figure 16 The search range shown is a fan-shaped area in front of the controlling avatar with a certain visual range. Entities outside this fan-shaped area are filtered out. Entities are then sorted by similarity. If two entities within the search range have the highest and equal similarity scores—that is, Entity 2 and Entity 3 both have a similarity score of 2—Entity 2, which is closer to the controlling avatar, is selected as the target entity.

[0317] In an optional embodiment, the game application determines the entity with the highest similarity within the search range and above a threshold as the target entity. Exemplarily, the number of search ranges is at least one. For example, the search range includes at least one of the following: the control virtual character's field of view, the control virtual character's hearing range, the control virtual character's perceived virtual item search range, the non-player character's field of view, the non-player character's hearing range, and the non-player character's perceived virtual item search range.

[0318] For example, the game application may sequentially traverse at least one of the aforementioned search ranges, or the game application may sequentially traverse at least one search range determined based on a natural language command. For example, if the natural language command is "Can you see the truck in front? Move to the truck?", the game application may prioritize traversing the control virtual character's field of view, followed by the field of view of non-player characters, and match entities with the highest similarity that exceeds a threshold.

[0319] For example, when there are at least two search ranges, the order in which the at least two search ranges are traversed may be predetermined, or the order in which the at least two search ranges are traversed may be determined based on the intent indicated in the natural language command. For example, when the intent is to search for an object, the visual range is prioritized. When the intent is to search for combat scenes, the auditory range is prioritized.

[0320] If there are multiple levels of nested instructions in the natural language command, the first target entity will be searched first, and then the next target entity will be searched based on the position of the previous target entity. In the case where the natural language command includes a second target entity determined based on the position of the first target entity, the first target entity is queried from the entity set of the virtual environment based on the first target entity information and environmental perception information of the first target entity indicated by the natural language command; then, the second target entity is queried from the entity set of the virtual environment based on the second target entity information, the first target entity, and the environmental perception information.

[0321] The multi-level nested instruction refers to a natural language command including at least two entities, at least one of which serves as a reference entity, and the reference entity is used to determine the target entity. In other words, the reference entity is used to define the target entity.

[0322] For example, the natural language command describes the positional relationship between the reference entity and the target entity. For example, if the natural language command is "move to the blue car behind the red car", then the red car is the reference entity, the blue car is the target entity, and the target entity is located behind the reference entity.

[0323] Alternatively, the natural language command describes the similarity between the reference entity and the target entity. For example, if the natural language command is "Find out if there are any other red treasure chests nearby", then the reference entity is the red treasure chest that the controlling virtual character can currently see, and the target entity is a nearby red treasure chest that has not been discovered by the controlling virtual character. The target entity and the reference entity have the same appearance.

[0324] Exemplarily, the target entity information includes reference entity information of a reference entity, and the reference entity is used to determine the target entity by reference; the game application queries and obtains the reference entity from the entity set based on the reference entity information and environmental perception information; and queries and obtains the target entity from the entity set based on the reference entity, the target entity information and the environmental perception information.

[0325] For example, the target entity information includes: reference entity information, the positional relationship between the reference entity and the target entity, and the feature text of the target entity; the game application calculates the similarity between the feature text and each entity information in the entity set; determines the search range based on the environmental perception information; and based on the similarity and positional relationship, filters the target entity from the entities within the search range in the entity set.

[0326] Optionally, the target entity information may include reference entity information of n reference entities, where the i-th reference entity among the n reference entities is used to determine / limit the i+1-th reference entity, where n is an integer greater than 1 and i is a positive integer less than n. The game application then queries and obtains the i-th reference entity from the entity set based on the reference entity information and environmental perception information of the i-th reference entity; queries and obtains the i+1-th reference entity from the entity set based on the reference entity information and environmental perception information of the i-th reference entity and the i+1-th reference entity, and so on until the n-th reference entity is obtained; and queries and obtains the target entity from the entity set based on the n-th reference entity, the target entity information, and the environmental perception information.

[0327] For example, Figure 17 As shown, when the natural language command is "go to the broken car on the left of the red truck in front", the game application will find the nearest red truck 503 based on the position and orientation of the main control virtual character, and then find the nearest car wreck 504 on the left of the red truck 503.

[0328] In step 232 , the non-player character is controlled to perform an activity associated with the target entity according to the intention of the natural language command.

[0329] Exemplarily, a large language model is called to identify the intent of a natural language command, and a control instruction or a control instruction sequence is generated according to the intent. According to the control instruction or the control instruction sequence, a non-player character is controlled to complete an activity based on a target entity.

[0330] In summary, the method provided in this embodiment proposes a deep learning-based game scene perception and query method that can initiate spatial query positioning of the game scene in the form of natural language. Combining pre-built static game scene data with real-time game data, a method for controlling non-player characters is provided. For example, if a player inputs "surround the house in front" or "hide behind this rock," the method can parse the physical locations of the "house" or "rock" based on the player's current position and move the non-player character to a reasonable location next to the entity. Furthermore, the method can also adapt to complex and ambiguous instructions, such as "go to the red truck inside the house" or "go upstairs."

[0331] The method provided in this embodiment, applied to a product that uses voice commands for non-player characters in shooting games, can dispatch thousands of target locations within the game scene. It can adapt to any text command input by the player and parse the non-player character's action pattern and target location. The action pattern corresponds to the fixed command of traditional solutions, while the target location represents the precise control of the non-player character, ultimately achieving accurate and complex control of the non-player character.

[0332] The method provided in this embodiment can greatly enhance the gaming experience of players controlling non-player characters. Players can issue arbitrary commands and direct teammates to complete the dispatch of thousands of target locations within the game scene, with a high degree of freedom. At the same time, compared with related technical solutions that require the design of complex comparative game logic to complete the control of non-player characters, this solution can save a lot of game design and development costs.

[0333] An exemplary embodiment of the present application provides a method for controlling a virtual character. The method includes:

[0334] Displaying at least one of a main control virtual character and an NPC in a virtual environment;

[0335] For example, the controlling avatar is the avatar that the user directly controls in the virtual environment. The virtual environment also contains one or more non-player characters (NPCs). NPCs and the controlling avatar belong to the same virtual faction and are teammates of the controlling avatar, following the controlling avatar in the virtual environment. NPCs can also be followers or pets controlled by the controlling avatar. Furthermore, NPCs can be neutral characters, acting in concert with the controlling avatar only under certain conditions.

[0336] Obtain natural language commands;

[0337] Exemplarily, natural language commands include natural semantics for commanding NPCs; commanding NPCs in virtual scenes through natural language commands, natural language commands can be directly input by users or extracted from voice information input by users. This application does not limit the method of obtaining natural language commands.

[0338] A natural language command includes at least one of a behavioral intent and a scene entity. The behavioral intent is used to indicate the type of virtual activity to be performed by the NPC, such as indicating which type of virtual activity to be performed in the virtual environment. The scene entity includes a first scene object, such as indicating which scene object in the virtual environment the virtual activity is performed on. The behavioral intent and scene entity included in the natural language command are partial words in the natural language command.

[0339] In one example, the natural language command controls the NPC in terms of what type of virtual activity to perform and for which scene objects in the virtual environment the virtual activity is performed; it is a complex instruction for controlling the NPC.

[0340] Controlling NPCs to perform virtual activities in response to natural language commands;

[0341] Exemplarily, controlling an NPC to perform a virtual activity is determined based on environmental perception information from the controlling virtual character and / or NPC. In some examples, the controlling virtual character and / or NPC obtains environmental perception information within the search range of perception methods such as vision, hearing, and movement trajectory. Exemplarily, the virtual activity closely matches the controlling virtual character and / or NPC's current perception of the virtual environment, and the NPC is controlled based on the virtual character's and / or NPC's current perception of the virtual environment. Exemplarily, the NPC has autonomous behavioral capabilities, and the user merely directs the NPC. The user gives commands, and the NPC understands and executes the commands based on its own autonomous behavioral capabilities.

[0342] To sum up, the method provided in this embodiment determines the virtual activities that need to be performed by referring to the environmental perception information of the master virtual character and / or NPC in response to complex instructions for controlling the NPC in terms of what type of virtual activities to perform and for what scene objects in the virtual environment to perform the virtual activities; and uses the environmental perception information of the master virtual character and / or NPC as a reference to ensure that the NPC and the master virtual character perform virtual activities in collaboration.

[0343] Figure 18 A flowchart of a method for controlling a virtual character provided by an exemplary embodiment of the present application is shown. The method includes:

[0344] Step 201: Acquire a spatial dataset;

[0345] The spatial dataset of the virtual environment includes visual labels of scene objects, which are used to describe the visual characteristics of scene objects in at least one dimension, such as describing scene objects in terms of material, transparency, color, shape, etc.

[0346] Step 202: Display at least one of the main virtual character and the NPC in the virtual environment;

[0347] For example, the main virtual character is a virtual character that the user directly controls in the virtual environment. The virtual environment also contains one or more NPCs; the NPCs and the main virtual character belong to the same virtual camp and are teammates of the main virtual character.

[0348] Step 203: Acquire a natural language command in voice form;

[0349] Natural language commands include natural semantics for directing NPCs, and command information includes behavioral intentions and scene entities. Behavioral intentions are used to indicate the type of virtual activity to be performed by the NPC, and scene entities are used to indicate the target entity of the virtual activity.

[0350] Step 204: converting the natural language command in voice form into a natural language command in text form;

[0351] For example, Automatic Speech Recognition (ASR) processing is performed on the natural language command in the form of speech to determine the natural language command in the form of text. Speech-to-text processing generally involves invoking components such as acoustic models and language models to recognize the pronunciation, vocabulary, and grammatical structure of the natural language command in the form of speech and convert it into the natural language command in the form of text.

[0352] Step 205: Perform intent recognition on the natural language command to obtain a first classification label;

[0353] Exemplarily, the first intention indication corresponding to the first classification label includes a command intention directed at a non-player character on an intention dimension; for example, at least one of the following: indicating which of the multiple non-player characters the non-player character is commanded by the command text, indicating whether the virtual activity performed by the non-player character is related to launching a virtual attack, and indicating what kind of virtual activity the non-player character performs.

[0354] Step 206: Perform entity recognition on the natural language command to obtain a target entity;

[0355] The target entity is determined from the virtual environment using environmental perception information, which includes information perceived by at least one of the controlling avatar and non-player characters. The target entity can be the entity currently visible to the player or an entity the player has seen previously.

[0356] Step 207: In response to the natural language command, controlling the NPC to perform a virtual activity according to the first intention corresponding to the first classification label, or controlling the NPC to perform a virtual activity associated with the target entity, or controlling the NPC to perform a virtual activity associated with the target entity according to the first intention corresponding to the first classification label;

[0357] Exemplarily, the NPC is controlled to perform a virtual activity according to the first classification tag and / or the instruction of the target entity.

[0358] Step 208: reporting the NPC's feedback information;

[0359] For example, the game application can generate corresponding feedback information in real time based on the environmental perception information of the non-player character and broadcast it. For example, when the environmental perception information of the non-player character triggers a broadcast condition, the game application can generate corresponding feedback information in real time based on the environmental perception information that triggered the broadcast condition; or when the non-player character or the master virtual character triggers a broadcast condition, the game application can generate corresponding feedback information in real time based on the environmental perception information of the non-player character.

[0360] The preprocessing stage of step 201 can be implemented as follows:

[0361] Sub-step 1: Obtain attribute text of scene objects in the virtual scene;

[0362] For example, attribute text is used to describe the inherent properties of scene objects in a virtual scene. On the one hand, attribute text describes scene objects in a textual manner, providing semantic information for predicting their visual labels. On the other hand, attribute text describes scene objects in a virtual scene. Given the large number of objects in a virtual scene and the presence of reused object models, it can more accurately describe the inherent properties of scene objects in the virtual scene.

[0363] In an optional implementation, the attribute text includes at least one of a name of the scene object in the virtual scene and a size of the scene object in the virtual scene.

[0364] Sub-step 2: obtaining an appearance image of a scene object in the virtual scene;

[0365] Exemplarily, an appearance image is used to describe the style of a scene object. An appearance image, in a picture modality, carries the scene object's appearance style, such as color, texture, and shape, as well as the relative positional relationships between its sub-components, enabling a comprehensive description of the scene object from a picture modality (or visual modality). In one optional implementation, the appearance image of a scene object includes images obtained by observing the scene object from at least two perspectives.

[0366] Sub-step 3: Call the multimodal model to perform prediction on the attribute text and appearance image of the scene object to obtain the visual label of the scene object;

[0367] Exemplarily, the multimodal model has the ability to perform model predictions on text and image information in different modalities. In this embodiment, the input parameters of the multimodal model are the attribute text and appearance image of the scene object; the multimodal model predicts the visual label of the scene object from both the text modality and the image modality. The visual label is used to describe the visual characteristics of the scene object in at least one dimension.

[0368] Optionally, the multimodal model includes a visual question answering model; the question statement carries attribute text of the scene object; on the one hand, the question statement is used to guide the visual question answering model to convert the appearance image into the visual label of the scene object; on the other hand, while guiding the conversion of the visual label of the scene object, the question statement provides supplementary information of the scene object in text form.

[0369] Get the expected information of scene objects;

[0370] Exemplarily, the desired information is used to indicate desired description dimensions of scene objects in the visual tag and / or the desired format of the visual tag. In one example, the desired information is used to indicate that the desired description dimensions of scene objects in the visual tag include, but are not limited to, at least one of the following: type description, material, transparency, color, surface features, and shape of the scene object. In another example, the desired information is used to indicate that the desired format of the visual tag is at least one of the following: Comma Separated Values ​​(CSV), JavaScript Object Notation (JSON), or Extensible Markup Language (XML).

[0371] Construct a question statement of the appearance image based on the expected information and attribute text;

[0372] Exemplarily, the first subpart of the question statement is supplementary introduction information of the scene object, carrying the attribute text of the scene object; the second subpart of the question statement is an answer guidance statement for the visual question answering model, carrying expected information.

[0373] Optionally, the spatial dataset includes matching labels of scene objects in the virtual scene; accordingly: performing pseudo-colloquial rewriting on the visual labels of the scene objects to obtain matching labels that conform to spoken expressions of natural language.

[0374] For example, as described above, visual labels are used to describe the visual features of scene objects in at least one dimension, and the appearance image of the scene object presents rich visual features of the scene object. However, in the spoken expression of natural language, the description of the scene object cannot cover the visual features of the scene object in all dimensions. The purpose of performing a pseudo-colloquial rewriting on the visual label of the scene object is to obtain a matching label that is closer to the spoken expression; it can be understood that performing a pseudo-colloquial rewriting can be to delete part of the content in the visual label, or to change the visual label to a label with the same semantics but different textual expression.

[0375] In one optional implementation, the pseudo-colloquial rewriting is performed by invoking a natural language model. The visual labels of scene objects are input into the natural language model, and matching labels that match the spoken expressions of natural language are predicted. The pseudo-colloquial rewriting of the visual labels is performed based on invoking the natural language model. Exemplarily, the natural language model carries prior knowledge of spoken natural language expressions. For example, the natural language model is implemented as a Large Language Model (LLM).

[0376] Optionally, the spatial dataset also includes spatial information of scene objects, and accordingly, also includes:

[0377] The spatial position of the scene object in the virtual scene is obtained, and the spatial position of the scene object is determined as auxiliary information of the visual label of the scene object.

[0378] Exemplarily, the spatial position of the scene object in the virtual scene is used to indicate the deployment of the scene object in the virtual scene, and the spatial information is used to indicate the position, size, and other information of the scene object in the virtual scene after the scene object is deployed in the virtual scene.

[0379] Further, obtaining at least one of the coordinate position, orientation information, bounding box information, and cover point information of the scene object in the virtual scene;

[0380] Among them, the coordinate position (Location) is used to indicate the position of the scene object in the virtual scene, such as the coordinate information of the center point or preset point of the scene object in the virtual scene; the orientation (Rotation) is used to indicate the direction facing the scene object in the virtual scene, such as the direction facing the front of the scene object in the virtual scene; the bounding box (BoundingBox) is used to indicate the size of the scene object in the virtual scene; the cover points (Cover points) are used to indicate the recommended virtual character position points when the virtual character approaches the scene object, so that the scene object can mask the virtual character.

[0381] The intention recognition phase of step 205 can be implemented as follows:

[0382] Sub-step 4: Call at least one hierarchical prediction network to perform intent recognition on the natural language command to obtain a first classification label of the natural language command intent to command the NPC;

[0383] Exemplarily, the hierarchical prediction network has the ability to predict the classification label corresponding to the command text. Exemplarily, the hierarchical prediction network includes at least two layers of sub-networks, with the upper and lower layers being cascaded, and the lower layer further performing classification label prediction based on the prediction results output by the upper layer. At least two layers of sub-networks are constructed based on a tree structure of multiple classification labels. The hierarchical prediction network constructs at least two layers of sub-networks corresponding to the tree structure of the classification labels, splitting the prediction task of a wide variety of classification labels into prediction sub-tasks performed by at least two layers of sub-networks, thereby reducing the complexity of classification prediction in each layer of sub-network.

[0384] In an optional implementation, each hierarchical prediction network in at least one hierarchical prediction network is used to predict the command intention of the command text for a non-player character on an intention dimension; the intention dimension includes at least one of the subject dimension, the semantic dimension, and the behavior dimension. Exemplarily, in at least one hierarchical prediction network, the i-th layer sub-network in each hierarchical prediction network is used to predict the first-level behavior label of the command text on an intention dimension, and the i+1-th layer sub-network in the hierarchical prediction network is used to predict the second-level behavior label corresponding to the command text in the first-level behavior label, where i is a positive integer; the high-level behavior label (such as the first-level behavior label) predicted by the high-level sub-network (such as the i-th layer sub-network) in the hierarchical prediction network includes multiple sub-labels, or subordinate lower-level labels; the corresponding low-level sub-network (such as the i+1-th layer sub-network) needs to be called to further perform classification prediction (such as predicting the second-level behavior label). The prediction task of a wide variety of classification labels is split into prediction sub-tasks performed by at least two layers of sub-networks to reduce the complexity of classification prediction of each layer of sub-network.

[0385] Furthermore, the intent dimension includes a subject dimension, and the hierarchical prediction network includes a hierarchical subject prediction network, which has the ability to predict the subject type in a natural language command; the subject type is used to indicate the identity of the NPC commanded by the natural language command; exemplarily, the subject prediction network is used to predict which of the many non-player characters the non-player character commanded by the command text is, and the number of non-player characters commanded by the command text can be one or more.

[0386] That is, the game application inputs the natural language command into the subject prediction network and obtains the subject type output by the network. The subject type is used to determine the subject that performs the action indicated in the natural language command. Optionally, the game application can also determine the NPC directed by the natural language command, that is, the subject that performs the action, based on the subject type output by the subject prediction network and the environmental perception information. For example, based on the environmental perception information, the NPC of the subject type closest to the main virtual character is determined as the subject that performs the action; or, based on the environmental perception information, the NPC of the subject type within the field of view of the main virtual character is determined as the subject that performs the action; or, based on the environmental perception information, the NPC of the subject type that the main virtual character's visual focus is directed at is determined as the subject that performs the action.

[0387] And / or, the intent dimension includes a semantic dimension, the hierarchical prediction network includes a hierarchical semantic prediction network, and the semantic prediction network has the ability to predict the semantic type in natural language commands; the semantic type is used to indicate the NPC's control method for initiating a virtual attack; exemplarily, the subject prediction network is used to predict whether the virtual activity performed by the non-player character commanded by the command text is related to initiating a virtual attack.

[0388] Alternatively, the intent dimension includes a behavior dimension, the hierarchical prediction network includes a hierarchical behavior prediction network, and the behavior prediction network is capable of predicting the behavior intent type in a natural language command; the behavior intent type is used to indicate how the NPC will behave in performing a virtual activity. Exemplarily, the behavior prediction network is used to predict what virtual activity the non-player character will perform.

[0389] Accordingly, in step 207 , in response to the natural language command, the NPC is controlled to perform a virtual activity according to the first intention corresponding to the first classification label;

[0390] For example, the first intent corresponding to the first category tag indicates a command intent directed at a non-player character on an intent dimension, such as at least one of the following: indicating which non-player character is being commanded by the command text among a plurality of non-player characters, indicating whether the virtual action performed by the non-player character is related to launching a virtual attack, and indicating what virtual action the non-player character is to perform. According to the instruction of the first category tag, the NPC is controlled to perform the virtual action.

[0391] The target entity identification process of step 206 can be implemented as follows:

[0392] Sub-step 5: querying and obtaining the target entity from the entity set of the virtual environment according to the target entity information of the target entity indicated by the natural language command and the environment perception information;

[0393] The target entity is determined from the virtual environment in combination with environmental perception information, and the environmental perception information includes information perceived by the controlling virtual character and / or the non-player character from the virtual environment.

[0394] Since the target entity information in natural language commands is usually vague, for example, a natural language command can be "move to the red truck", where the target entity information is "red truck", and there may be many red trucks in the virtual environment. It is impossible to accurately determine the target entity in the virtual environment based solely on the target entity information in the natural language command.

[0395] Therefore, embodiments of the present application provide a method for determining a target entity by combining target entity information and environmental perception information. For the previous example, the "red truck" expressed by the player in the natural language command should be a red truck that the player can see. Therefore, based on the field of view of the controlling virtual character, the red truck within the field of view of the controlling virtual character can be selected from multiple red trucks in the virtual environment. This red truck is the target entity indicated by the player in the natural language command.

[0396] Because natural language commands are issued based on the player's perception of the virtual environment, the game application uses the player's environmental perception information at the time of the command to accurately identify the target entity. Based on the player's perception of the virtual environment, the game application infers the entity in the virtual environment that is closest to the target entity and is considered the target entity.

[0397] Exemplarily, the virtual environment includes multiple candidate entities that match the natural language command, and the target entity is an entity selected from the multiple candidate entities based on the environmental perception information of the controlling virtual character or the non-player character. For example, the game application first selects multiple candidate entities from the entity set that match the target entity information in the natural language command, and then selects an entity that can be perceived by the controlling virtual character or the non-player character from the multiple candidate entities as the target entity based on the environmental perception information.

[0398] For example, the environmental perception information of the main virtual character may include: the pictures or entities that the main virtual character can see when observing the virtual environment, the environmental sound effects that the main virtual character can hear, the source direction of the environmental sound effects, the type and volume of the environmental sound effects, the perception information obtained by the main virtual character through the use of perception skills (for example, visual information, auditory information, sound wave information, light reflection information, etc.), and the perception information obtained by the main virtual character through the use of perception virtual props (for example, the sensor signal of the sensor, the positioning signal of the positioning props, etc.).

[0399] The environmental perception information of the non-player character may include: the images or entities that the non-player character can see when observing the virtual environment, the environmental sounds that the non-player character can hear, the source direction of the environmental sounds, the type and volume of the environmental sounds, the perception information obtained by the non-player character through the use of perception skills (for example, visual information, auditory information, sound wave information, light reflection information, etc.), and the perception information obtained by the non-player character through the use of perception virtual props (for example, sensor signals from sensors, positioning signals from positioning props, etc.).

[0400] It should be noted that environmental perception information can include both the real-time perception of the controlling avatar and non-player characters upon receiving a natural language command, and can also include historical perception information of the controlling avatar and non-player characters prior to receiving the natural language command. In other words, the target entity may be an entity currently visible to the player, or an entity the player has previously seen. Therefore, the game application needs to combine the real-time and historical environmental perception information of the controlling avatar and / or non-player characters to identify the target entity.

[0401] For example, when the natural language command is "Let's go back to the hotel we just passed by", the game application needs to query the hotels that the main virtual character has passed by based on the historical environmental perception information of the main virtual character.

[0402] In one optional embodiment, the client queries and obtains the target entity from a set of entities in the virtual environment based on the target entity information indicated by the natural language command and the environment perception information. In another optional embodiment, the client reports the natural language command to the server, and the server queries and obtains the target entity from a set of entities in the virtual environment based on the target entity information indicated by the natural language command and the environment perception information.

[0403] Optionally, the game application infers the target entity based on the target entity information, the environment perception information, and the pre-processed virtual environment data, wherein the pre-processed virtual environment data includes a set of entities in the virtual environment.

[0404] The entity set includes entity information of each entity in the virtual environment. The entity information includes at least one of the following: name, type, location, features, text label, text label embedding vector, image, and image embedding vector.

[0405] The text label may be at least one word obtained after segmenting a feature (for example, a descriptive text of an entity appearance), and the embedding model is called to obtain an embedding vector corresponding to each text label.

[0406] The image may include an image of a three-dimensional model of the entity observed from at least one direction. For example, the image may include three views of the entity. The image embedding vector is the embedding vector of the visual feature label. The visual feature label is obtained by using a multimodal model to perform feature recognition based on the image and text label of the entity.

[0407] For example, the entity set includes first-category entities and second-category entities. The first-category entities include at least one of regions and buildings, and the second-category entities include physical objects. The text labels for the first-category entities are manually labeled. For example, the text labels for a room in a building might be: region, building, floor, room. The text labels for the second-category entities are automatically generated by invoking a large language model.

[0408] A method for generating text labels for the second type of entity may include: obtaining at least one image of the entity, inputting the at least one image into a large language model to obtain a description text of the entity; segmenting the description text to obtain at least one text label for the entity, where each text label includes a word obtained after the segmentation process; and performing a vectorization operation on the text label to obtain an embedding vector corresponding to the text label.

[0409] Embedding vectors are high-dimensional vector data of text labels. Pre-generating embedding vectors for text labels eliminates the need to repeatedly generate vectors for each entity's text label when matching target entities from an entity set based on target entity information, thereby improving the matching efficiency between target entity information and entity information.

[0410] It's important to note that this method doesn't use the description text directly as the text label. Instead, it uses the word segmentation results of the description text as the text label. This is because description texts often vary in length, and the embedding vectors of longer description texts are less effective when performing actual searches and comparing similarities. For example, when searching for "a small blue truck with rusty body" using the word "truck," "a small blue truck with rusty body" will likely be ranked after "a car" in the recall results. However, in reality, the search for "truck" should prioritize searching for "truck" regardless of the complexity of its additional description. Therefore, when performing similarity searches, the comparison should be based on individual feature words, not the description text.

[0411] In addition, in order to further extract the visual features of the entity, the method provided in the embodiment of the present application can also further extract hidden visual features in the image of the entity based on the text label and image of the entity. Taking the first entity as an example, the method includes: obtaining at least one perspective image of the first entity; obtaining at least one text label of the first entity; calling the multimodal model to extract the visual features of the first entity based on at least one perspective image and at least one text label of the first entity, and obtain the image embedding vector of the first entity.

[0412] Since text labels are manually annotated or generated by a large language model based on human language characteristics, and text labels extracted based on human language habits may ignore some features of the entity. For example, when manually annotating a text label for an oil drum, only more eye-catching text labels such as "metal" and "rusty" may be annotated, while ignoring detailed features such as "rusty", "blue", "yellow", and "paint peeling in the lower right corner". Therefore, the method provided in the embodiment of the present application also uses a multimodal model based on the entity's text label and entity image, so that the multimodal model extracts more visual features from it; based on the visual features, it performs image similarity matching with the target entity to improve the recognition accuracy of the target entity.

[0413] The multimodal model training method can include: inputting sample entity images and sample labels into a pre-trained multimodal model, and fine-tuning the pre-trained multimodal model based on the loss of the predicted labels and sample labels output by the pre-trained multimodal model. This results in a multimodal model that can output visual feature labels based on the input text labels and images. The visual feature labels include more detailed descriptive text extracted from the image. For example, the text description of "a metal oil drum" can only be associated with the two text labels "metal" and "oil drum." However, the Clip (multimodal) model can also identify hidden visual information in the image of the oil drum, such as the visual feature labels "rusty," "blue," and "yellow." These visual features are not present in the text labels. Therefore, combining the Clip model with visual feature search can further improve the accuracy of entity search.

[0414] In an optional embodiment, the game application may query the target entity using the following method.

[0415] (1) Parsing the natural language command to obtain target entity information; the target entity information includes at least one of the following: entity type, entity name, entity location, and entity features.

[0416] Optionally, the game application calls a large language model to parse the natural language command to obtain target entity information of the target entity, and calls an embedding model to vectorize the target entity information to obtain a target embedding vector of the target entity information.

[0417] For example, if the natural language command is "Come to the blue truck in front", the target entity information that can be parsed by the large language model includes: the direction information "front" and the entity name "blue truck".

[0418] For example, named entity recognition technology in the field of natural language processing can also be used to extract target entity information from natural language commands. Named entity recognition technology is used to identify entities with specific meanings in text, such as names of people, places, locative words, adjectives, etc.

[0419] For example, if the natural language command is "Go to the motel on the first floor and find a cardboard box behind the red sofa," named entity recognition technology can identify the following: the architectural noun "motel first floor"; the object nouns "sofa" and "cardboard box"; the locative word "behind"; and the adjective "red." After logical construction, a hierarchical scenario query call is formed, complete with search type, content, locative words, and constraint information. The search type narrows the data retrieval scope based on similarity matching, the search content is a specific entity description, and adjectives are incorporated into the description text for querying. The locative words constrain the search scope. The floor constraint is determined by the player's location. In indoor scenes, the range must be narrowed to avoid searching for entities that are not visible across floors.

[0420] (2) Calculate the similarity between the target entity information and each entity information in the entity set.

[0421] For example, the entity set includes entity information for at least one entity, and the entity information for each entity may include at least one of a text embedding vector for a text label and an embedding vector for a visual feature. The target entity information may then be similarity-calculated with the text embedding vector and the image embedding vector, respectively, and the entity with the highest similarity may be determined as the target entity.

[0422] The following are methods for similarity matching with text embedding vectors and similarity matching with image embedding vectors.

[0423] 1) Similarity includes the text similarity between the target entity information and the text embedding vector.

[0424] Taking the first entity in the entity set as an example, the game application (client or server) segments the target entity information to obtain at least one target entity tag; converts the at least one target entity tag into at least one target embedding vector; obtains the entity information of the first entity, the entity information includes a text embedding vector, and the text embedding vector is an embedding vector obtained based on the text tag conversion of the first entity; calculates the text parent similarity between the at least one target embedding vector and the text embedding vector respectively, and obtains at least one text parent similarity corresponding to the at least one target embedding vector; and determines the sum of the at least one text parent similarity as the text similarity between the target entity information and the entity information of the first entity.

[0425] For example, there are two target entity labels, target entity label 1 and target entity label 2, and there are also two entities in the entity set, entity 1 and entity 2. Entity 1 corresponds to text embedding vector 1, and entity 2 corresponds to text embedding vector 2. Similarity 1 is calculated between target entity label 1 and text embedding vector 1, and similarity 2 is calculated between target entity label 2 and text embedding vector 1. The sum of similarity 1 and similarity 2 is determined as the text similarity between the target entity information and entity 1. Similarity 3 is calculated between target entity label 1 and text embedding vector 2, and similarity 4 is calculated between target entity label 2 and text embedding vector 2. The sum of similarity 3 and similarity 4 is determined as the text similarity between the target entity information and entity 2.

[0426] Exemplarily, the entity information of the first entity includes at least one text embedding vector; at least one target embedding vector includes a first target embedding vector; the game application calculates the text sub-similarity between the first target embedding vector and the at least one text embedding vector respectively to obtain at least one text sub-similarity; and the highest value of the at least one text sub-similarity is determined as the text parent similarity corresponding to the first target embedding vector.

[0427] For example, the number of target entity tags is 1: target entity tag 1. The number of entities in the entity set is also 1: entity 1. Entity 1 corresponds to text embedding vector 1 and text embedding vector 3. Similarity 1 is calculated between target entity tag 1 and text embedding vector 1, and similarity 5 is calculated between target entity tag 1 and text embedding vector 3. The larger value of similarity 1 or similarity 5 is determined as the similarity between the target entity information (target entity tag 1) and entity 1.

[0428] For example, for the input target entity information "blue car," we first perform word segmentation on the target entity information, breaking it down into the target entity labels "blue" and "car," indicating that the query target entity has two features. We then calculate similarity between each feature and each entity in the entity set. The maximum similarity score for each feature is taken, and finally, the sum of the similarity scores corresponding to all query features is taken, representing the textual similarity between the target entity information and the query entity.

[0429] For example, the target entity information contains two target entity labels, "blue" and "car." The two target entity labels are converted into two target embedding vectors. Then, the entity information in the entity set is obtained. For example, the entity set includes entity 1 and entity 2. The text labels of entity 1 include "metal," "old," "car," "truck," "blue," and "damaged." The text labels of entity 2 include "metal," "scratches," "car," "damaged," "armored truck," and "black."

[0430] Calculate the text similarity between the target entity information and entity 1: first calculate the similarity between the target entity label "blue" and each text label of entity 1, and take the maximum value. For example, the similarity between the target entity label "blue" and the text label "blue" of entity 1 is 1 (1 is the maximum value); then calculate the similarity between the target entity label "car" and each text label of entity 1, and take the maximum value. For example, the similarity between the target entity label "car" and the text label "car" of entity 1 is 1. Then sum the two similarities of the target entity labels "car" and "blue" to obtain the final similarity between the target entity information and entity 1, which is 2.

[0431] Similarly, calculate the text similarity between the target entity information and entity 2: first calculate the similarity between the target entity label "blue" and each text label of entity 2, and take the maximum value. For example, the similarity between the target entity label "blue" and the text label "black" of entity 1 is 0.91; then calculate the similarity between the target entity label "car" and each text label of entity 1, and take the maximum value. For example, the similarity between the target entity label "car" and the text label "car" of entity 2 is 1 (1 is the maximum value). Then sum the two similarities of the target entity labels "car" and "blue" to obtain the final similarity between the target entity information and entity 2, which is 1.91.

[0432] It can be seen that the similarity between the target entity information and entity 1 is 2, which is higher than the similarity between the target entity information and entity 2, which is 1.91.

[0433] 2) Similarity includes the image similarity between the target entity information and the image embedding vector.

[0434] Taking the first entity in the entity set as an example, the game application (client or server) segments the target entity information to obtain at least one target entity tag; converts the at least one target entity tag into at least one target embedding vector; obtains the entity information of the first entity, the entity information includes an image embedding vector, and the image embedding vector is an embedding vector obtained based on the image extraction of the first entity; calculates the image parent similarity of the at least one target embedding vector and the text embedding vector respectively, and obtains at least one image parent similarity corresponding to the at least one target embedding vector; and determines the sum of the at least one image parent similarity as the image similarity between the target entity information and the entity information of the first entity.

[0435] For example, there are two target entity labels, target entity label 1 and target entity label 2, and there are also two entities in the entity set, entity 1 and entity 2. Entity 1 corresponds to image embedding vector 1, and entity 2 corresponds to image embedding vector 2. Similarity 1 is calculated between target entity label 1 and image embedding vector 1, and similarity 2 is calculated between target entity label 2 and image embedding vector 1. The sum of similarity 1 and similarity 2 is determined as the image similarity between the target entity information and entity 1. Similarity 3 is calculated between target entity label 1 and image embedding vector 2, and similarity 4 is calculated between target entity label 2 and image embedding vector 2. The sum of similarity 3 and similarity 4 is determined as the image similarity between the target entity information and entity 2.

[0436] Exemplarily, the entity information of the first entity includes at least one image embedding vector; the at least one target embedding vector includes a first target embedding vector; the game application calculates the image sub-similarity between the first target embedding vector and the at least one image embedding vector respectively to obtain at least one image sub-similarity; and the highest value of the at least one image sub-similarity is determined as the image parent similarity corresponding to the first target embedding vector.

[0437] For example, the number of target entity tags is 1: target entity tag 1. The number of entities in the entity set is also 1: entity 1. Entity 1 corresponds to image embedding vector 1 and image embedding vector 3. Similarity 1 is calculated between target entity tag 1 and image embedding vector 1, and similarity 5 is calculated between target entity tag 1 and image embedding vector 3. The larger value of similarity 1 or similarity 5 is determined as the similarity between the target entity information (target entity tag 1) and entity 1.

[0438] In an optional embodiment, the target image embedding vector can be used to calculate image similarity, that is, the target image embedding vector is used instead of the target embedding vector in the above method. The target image embedding vector can be obtained by: segmenting the target entity information to obtain at least one target entity label; and inputting the at least one target entity label into a multimodal model to obtain the target image embedding vector.

[0439] 3) Similarity includes text similarity and image similarity.

[0440] When the first entity and the target entity have corresponding text similarity and image similarity, the game application determines an average value of the text similarity and the image similarity as the similarity between the first entity and the target entity.

[0441] Alternatively, when the first entity and the target entity have corresponding text similarity and image similarity, the larger value of the text similarity and the image similarity is determined as the similarity between the first entity and the target entity.

[0442] Alternatively, when the first entity and the target entity have corresponding text similarity and image similarity, the sum of the text similarity and the image similarity is determined as the similarity between the first entity and the target entity.

[0443] (3) Determine the target entity from the entity set based on similarity and environmental perception information.

[0444] In an optional embodiment, the game application first performs a similarity search based on the target entity information, obtains at least one candidate entity with a high similarity, and sorts them according to the similarity; then, based on the position, position, orientation and other information of the main virtual character and / or the non-player character, the game application performs perceptual filtering and sorting, and finally filters out the target entity.

[0445] Exemplarily, the game application determines a target search range based on target entity information and environmental perception information; and obtains the target entity by filtering entities perceived within the target search range based on similarity.

[0446] Alternatively, the game application determines x entities with the highest similarity from the entity set based on similarity to form a candidate entity list; determines a target search range based on target entity information and environmental perception information, and determines entities within the target search range among the x entities as target entities, where x is a positive integer.

[0447] Exemplarily, the game application determines the entity with the highest similarity in the search range as the target entity.

[0448] It should be noted that when there are multiple entities with equal similarity within the target search range, the multiple entities can be sorted according to their distance from the controlling virtual character. For example, if there are at least two entities with the highest similarity within the search range, the entity with the highest similarity within the search range and the closest distance to the controlling virtual character will be determined as the target entity.

[0449] In an optional embodiment, the game application determines the entity with the highest similarity within the search range and above a threshold as the target entity. Exemplarily, the number of search ranges is at least one. For example, the search range includes at least one of the following: the control virtual character's field of view, the control virtual character's hearing range, the control virtual character's perceived virtual item search range, the non-player character's field of view, the non-player character's hearing range, and the non-player character's perceived virtual item search range.

[0450] For example, the game application may sequentially traverse at least one of the aforementioned search ranges, or the game application may sequentially traverse at least one search range determined based on a natural language command. For example, if the natural language command is "Can you see the truck in front? Move to the truck?", the game application may prioritize traversing the control virtual character's field of view, followed by the field of view of non-player characters, and match entities with the highest similarity that exceeds a threshold.

[0451] For example, when there are at least two search ranges, the order in which the at least two search ranges are traversed may be predetermined, or the order in which the at least two search ranges are traversed may be determined based on the intent indicated in the natural language command. For example, when the intent is to search for an object, the visual range is prioritized. When the intent is to search for combat scenes, the auditory range is prioritized.

[0452] If there are multiple levels of nested instructions in the natural language command, the first target entity will be searched first, and then the next target entity will be searched based on the position of the previous target entity. In the case where the natural language command includes a second target entity determined based on the position of the first target entity, the first target entity is queried from the entity set of the virtual environment based on the first target entity information and environmental perception information of the first target entity indicated by the natural language command; then, the second target entity is queried from the entity set of the virtual environment based on the second target entity information, the first target entity, and the environmental perception information.

[0453] In response to natural language commands, the NPC is controlled to perform a virtual activity associated with a target entity.

[0454] Exemplarily, a large language model is called to identify the behavioral intention of natural language commands, and a control instruction or a control instruction sequence is generated according to the behavioral intention. According to the control instruction or the control instruction sequence, a non-player character is controlled to complete activities based on the target entity.

[0455] The broadcasting of the NPC feedback information in step 208 can be implemented as follows:

[0456] Sub-step 6: reporting feedback information of the non-player character, wherein the feedback text corresponding to the feedback information is a non-fixed text generated based on the environmental perception information of the non-player character;

[0457] For example, the game application can generate corresponding feedback information in real time based on the environmental perception information of the non-player character and broadcast it. For example, when the environmental perception information of the non-player character triggers a broadcast condition, the game application can generate corresponding feedback information in real time based on the environmental perception information that triggered the broadcast condition; or when the non-player character or the master virtual character triggers a broadcast condition, the game application can generate corresponding feedback information in real time based on the environmental perception information of the non-player character.

[0458] For example, the feedback text is derived by inferring the large language model based on the static entity data of the virtual environment and the dynamic environmental perception information of the non-player characters. The large language model generates different feedback texts based on the different perception conditions of the non-player characters.

[0459] For example, the feedback text can be derived by a large language model based on static entity data of the virtual environment and the dynamic environmental perception information of the non-player character, and can be tailored to the personality traits of the non-player character. For example, if the non-player character is a strong and handsome man, the feedback text can adopt a heroic tone; if the non-player character is a reporter, the feedback text can adopt a news report tone.

[0460] It should be noted that due to the randomness of the results generated by the large language model, in the same scene, when the environmental perception information of the non-player character is the same, the generated feedback text may also be different.

[0461] The environmental perception information may include information perceived by the non-player character in real time, or may include information perceived by the non-player character in history.

[0462] For example, the feedback information is derived by the game application based on the non-player character's environmental perception information and static entity data of the virtual environment. The feedback information can be broadcast in the form of voice or text. Static entity data includes data on relatively unchanging entities in the virtual environment, such as model information of various buildings, terrain, and vehicles in the virtual environment.

[0463] For example, the feedback information may include at least one of the following: immediate feedback, execution feedback, and dynamic feedback. Instant feedback and execution feedback are both feedback information generated in response to natural language commands, while dynamic feedback is feedback information spontaneously generated by non-player characters.

[0464] 1. Instant feedback is the feedback information generated immediately upon receiving a natural language command. Instant feedback can include reply announcements and instant announcements. Reply announcements are used to reply to the inquiry in the natural language command; instant announcements are used to provide feedback on the reception of the natural language command.

[0465] For example, a client can direct a non-player character to move in a virtual environment based on the intent of a natural language command. Upon receiving a natural language command, the client can generate immediate feedback corresponding to the natural language command. The immediate feedback indicates receipt of the natural language command, or provides an immediate response to the natural language command.

[0466] In an optional embodiment, the feedback information includes a reply broadcast. The reply broadcast includes a response to a natural language query command issued by the player. For example, if the natural language query command includes a query for the location of a target entity, the corresponding reply broadcast should include the query result for the location of the target entity.

[0467] When the behavioral intention of the received natural language command is an inquiry, the terminal broadcasts a reply message of the non-player character; the reply message includes a reply content for the natural language command generated based on the environmental perception information of the non-player character about the virtual environment.

[0468] The natural language command is used to request information from a non-player character. This natural language command can be referred to as a query natural language command. A query natural language command is a natural language command intended to be a query. The query natural language command contains at least one question.

[0469] Natural language commands are instructions conveyed using natural human language. The terminal receives natural language commands from players and directs non-player characters based on the intent of the natural language commands. For example, the terminal receives the player's voice audio and converts it into text to generate the natural language commands. Alternatively, the terminal receives the text of natural language commands input by the player.

[0470] Players can issue natural language commands using spoken or written language. The game application uses a large language model to understand the natural language commands, extract the intent expressed in the natural language commands, and generate control instructions based on the intent to direct the activities of non-player characters.

[0471] For example, the natural language command may be "Come to me", then the game application can parse out that the intention of the natural language command is "control the non-player character to move to the location of the controlling virtual character", then the game application obtains the location of the controlling virtual character, generates a navigation route for the non-player character to move to the location of the controlling virtual character, and controls the non-player character to move to the location of the controlling virtual character according to the navigation route.

[0472] For another example, the natural language command may be "go pick up the treasure chest", then the game application can parse out that the intention of the natural language command is "control the non-player character to move to the location of the treasure chest and pick up the treasure chest", then the game application can obtain the location of the treasure chest, control the non-player character to move to the location of the treasure chest, and perform the operation of picking up the treasure chest after arriving at the location of the treasure chest.

[0473] For example, when the natural language command includes an intelligence inquiry about the target entity, a first reply report is broadcast based on the non-player character's perception of the target entity; the first reply report includes the non-player character's intelligence perception result about the target entity.

[0474] Alternatively, in a case where the natural language command includes an inquiry about the location of the target entity, a second reply broadcast is broadcast based on the non-player character's perception of the target entity, and the second reply broadcast includes a description of the location of the target entity.

[0475] Exemplarily, a large language model is called to parse the received natural language command to obtain the intent of the natural language command. When the intent is of the inquiry type, the natural language command can be determined to be an inquiry natural language command. When the natural language command is an inquiry natural language command, the large language model can infer the intent of the natural language command based on the static entity data and environmental perception information to obtain the reply text. That is, the large language model is called to parse the received natural language command, and the intent of the natural language command is output as an inquiry, as well as the reply text of the natural language command. The game application calls the text-to-speech service according to the processing logic of the inquiry intent, converts the reply text into a reply broadcast, and the reply broadcast is an audio broadcast, and the reply audio is sent to the client for broadcast.

[0476] For example, when the large language model recognizes that the intention of the natural language command is to ask a question, since the user may ask hundreds of questions and it is impossible to store the answer to each question locally on the client, the game application will call the text-to-speech service to generate a reply broadcast in real time based on the reply text returned by the large language model, so that the game application can instantly generate a corresponding reply broadcast to respond to the player's question.

[0477] For example, a game application (client or server) calls a large language model to parse natural language commands, and infers a reply text based on the static environmental data and environmental perception information of the virtual environment; and generates human voice audio based on the reply text to obtain a reply broadcast.

[0478] For example, the client reports the received natural language query command to the server, the server calls the large language model to parse the natural language query command, and infers the reply text based on the static environmental data and environmental perception information of the virtual environment; generates human voice audio based on the reply text to obtain a reply broadcast; the server sends the reply broadcast to the client; and the client broadcasts the reply broadcast.

[0479] 2. Execution feedback is the feedback generated when a task is executed according to the intent indicated by the natural language command after receiving it. Execution feedback can also be called execution broadcast.

[0480] For example, when a natural language command is used to instruct a non-player character to perform a target task, the client can generate execution feedback during the non-player character's performance of the target task, and the execution feedback is used to indicate the execution status of the target task; or, the client can generate execution feedback after the non-player character completes the target task, and the execution feedback is used to indicate the execution result of the target task.

[0481] In an optional embodiment, the feedback information includes a feedback report (also referred to as execution feedback). The feedback report includes a report on the task execution status and results generated in response to the task natural language command issued by the player. For example, if the task natural language command includes moving to a target entity, the corresponding feedback report may include: moving to the target entity, or having reached the target entity.

[0482] When the behavioral intention of the received natural language command is to execute a task, the terminal broadcasts a feedback broadcast; the feedback broadcast includes the execution status after the natural language command is executed according to the environmental perception information of the virtual environment by the non-player character.

[0483] The natural language command is used to instruct the non-player character to perform a task. This natural language command may be referred to as a task natural language command. The task natural language command includes a description of the task. For example, the description may include at least one of the following: the task name, the action to be performed by the non-player character, the method to be used to perform the task, the task objective, and the location of the task objective.

[0484] For example, when the intention of the task natural language command includes searching for a target entity, a first feedback announcement is broadcast, and the first feedback announcement includes the search result of the non-player character for the target entity.

[0485] Alternatively, when the intention of the task natural language command includes using a virtual prop, a second feedback announcement is broadcast, and the second feedback announcement is used to indicate that the virtual prop is ready.

[0486] Alternatively, when the intention of the task natural language command includes the use of virtual props, a third feedback announcement is broadcast, and the third feedback announcement is used to indicate the result of the use of the virtual props.

[0487] Alternatively, when the intention of the task natural language command includes controlling the movement of the non-player character, a fourth feedback announcement is announced, and the fourth feedback announcement is used to indicate the movement result of the movement.

[0488] Alternatively, when the intention of the task natural language command includes performing an interaction with a target entity, a fifth feedback announcement is broadcast, and the fifth feedback announcement is used to indicate at least one of the search result of the non-player character for the target entity and the interaction result between the non-player character and the target entity.

[0489] Alternatively, when the intention of the task natural language command includes item interaction, a sixth feedback announcement is broadcast, and the sixth feedback announcement is used to indicate the item interaction result of the non-player character performing the item interaction.

[0490] Exemplarily, a large language model is called to parse the received natural language command to obtain the intent of the natural language command. When the intent is a task, the natural language command can be determined to be a task natural language command. When the natural language command is a task natural language command, the large language model can reason according to the intent of the natural language command, based on static entity data and environmental perception information, to obtain a task instruction sequence. The task instruction sequence is used to control the non-player character to perform the task. When the game application receives the task instruction sequence, it controls the non-player character's activities to perform the task according to the task instructions in the task instruction sequence. While controlling the non-player character according to the task instructions, the corresponding execution status can be obtained and feedback broadcasted according to the feedback broadcast logic corresponding to different task instructions.

[0491] For example, since the commands that non-player characters can execute in the virtual environment are finite and traversable, the client can locally store the feedback broadcast corresponding to each character's command. When the non-player character executes the corresponding command, the client can read the local feedback broadcast and perform voice broadcast.

[0492] Alternatively, when there is variable content in the feedback broadcast corresponding to a certain task instruction, for example, the target entity in the feedback broadcast is variable, or the location information in the feedback broadcast is variable; the client can also generate a broadcast voice with variable content based on the target entity or location information returned by the large language model, and splice it with the broadcast voice with immutable content stored locally to obtain the final feedback broadcast.

[0493] For example, if the task natural language command is "Find a nearby treasure chest," parsing it with the large language model reveals the task intent, the target entity, and the treasure chest. Reasoning based on static entity data and environmental perception information reveals the presence of a treasure chest to the left and front of the main control virtual character. If the target entity can be found, the corresponding feedback announcement for this task objective is "Found aaa at xxx," where xxx is the target entity's location and aaa is its name. The game application can then use online text-to-speech technology to generate a location audio based on the inference result, "front left." Based on the target entity's name, "treasure chest," the application can also use online text-to-speech technology to generate a name audio. The location audio and name audio are then concatenated into the feedback announcement for the final feedback announcement.

[0494] Exemplarily, a game application (client or server) calls a large language model to parse natural language commands for a task, and infers task execution instructions based on static environmental data and environmental perception information of the virtual environment; controls a non-player character to perform the task according to the task execution instructions; generates feedback text based on the execution status of the non-player character performing the task; and generates human voice audio based on the feedback text to obtain a feedback broadcast.

[0495] For example, the client reports the task natural language command to the server; the server parses the task natural language command and infers the task execution instructions based on the static environmental data and environmental perception information of the virtual environment; the non-player character is controlled to perform the task according to the task execution instructions; feedback text is generated according to the execution status of the non-player character performing the task; human voice audio is generated based on the feedback text to obtain a feedback broadcast; the server sends the feedback broadcast to the client; the client receives and broadcasts the feedback broadcast.

[0496] 3. Dynamic feedback is spontaneously generated based on the non-player character's environmental perception information. Dynamic feedback can also be called dynamic broadcast.

[0497] For example, the client can spontaneously generate dynamic feedback based on its perception of the virtual environment without receiving natural language commands. Dynamic feedback is used to indicate abnormal conditions detected by the non-player character in the virtual environment. For example, abnormal conditions may include: detecting hostile virtual characters, detecting changes in the status of friendly virtual characters, detecting dangerous situations, and detecting traces of combat or looting.

[0498] In an optional embodiment, the feedback information includes dynamic notifications (also referred to as "dynamic feedback"). Dynamic notifications include notifications of abnormal situations perceived by non-player characters. For example, when a non-player character perceives being attacked, a notification of "being attacked" is given; when a non-player character perceives a dangerous situation ahead, a notification of "danger ahead" is given.

[0499] The terminal broadcasts a dynamic broadcast when the environmental perception information of the non-player character on the virtual environment meets the dynamic broadcast conditions.

[0500] Dynamic notifications are generated by the game application based on the environmental perception information of non-player characters and when it determines that an abnormal situation requires notification. Dynamic notifications include alerts about abnormal situations. For example, abnormal situations may include at least one of the following: the discovery of a hostile virtual character, the discovery of a change in the status of a friendly virtual character, or the discovery of new traces.

[0501] For example, when a non-player character perceives a hostile virtual character, a first dynamic broadcast is broadcast; the first dynamic broadcast includes the position of the hostile virtual character perceived by the non-player character.

[0502] Alternatively, when the non-player character senses a dangerous situation, a second dynamic broadcast is broadcast; the second dynamic broadcast is used to prompt the dangerous situation.

[0503] Alternatively, when the non-player character senses a change in the state of the friendly virtual character, a third dynamic broadcast is broadcast; the third dynamic broadcast is used to prompt that the state of the friendly virtual character has changed.

[0504] Alternatively, when the non-player character discovers new traces in the virtual environment, a fourth dynamic broadcast is broadcasted; the fourth dynamic broadcast is used to indicate the new traces, wherein the new traces may be combat traces and / or looting traces.

[0505] For example, since the number of dynamic announcements that non-player characters can trigger in the virtual environment is limited and traversable, the client can locally store dynamic announcement voices. When the corresponding dynamic announcement voice is triggered, the client can read the dynamic announcement voice from the local storage and announce it. Similarly, some dynamic announcement voices can contain variable content. The client can generate variable content in real time using text-to-speech technology and splice it with the dynamic announcement template to produce the final dynamic announcement.

[0506] In an optional embodiment, when generating feedback information, it may be necessary to query the location of a target entity in the virtual environment, or to determine the target entity indicated in a natural language command from the virtual environment. In this case, an entity query method is required to query the target entity.

[0507] Before executing a target entity query, static entity data for the virtual environment must be pre-built. This static entity data serves both the inference of the large language model and the real-time feedback text generation on the game application side after the inference results are returned.

[0508] In an optional embodiment, entities in the virtual environment are traversed, and entity information of each entity is exported from the game engine to obtain static entity data.

[0509] For example, for in-game scenes, editor tools have been developed to perform StaticMesh traversal of the entire scene, special Actor traversal (such as interactive doors, which do not fall into the StaticMesh category), vegetation traversal, and export the entity's position, orientation, and bounding box size as static entity data for subsequent labeling and inference by the large language model.

[0510] StaticMesh is a static geometry resource type in the Unreal Engine 4 (UE4) game engine, used to represent immutable 3D models such as buildings and props. Actor is a basic object class in the Unreal Engine 4 (UE4) game engine, representing an entity in the game world, such as a character, object, or light source. Actors can contain multiple components to implement different functions.

[0511] For example, static entity data can also include spatial data, which is obtained by manual labeling. Spatial data is more complex than static entities because there are multiple floors in the space and multiple layers of nesting within the floors (such as the motel area contains the second floor, and the second floor area contains the guest rooms). For this type of data, manual calibration is used to construct it. Add Volume to the scene in the editor, divide the area required for the entire image and label it accordingly. Volume is a special Actor class in the Unreal Engine 4 (UE4) game engine, which represents a three-dimensional area with specific functions, such as a trigger area, audio area, etc.

[0512] For example, if there are three-dimensional spatial areas in the virtual environment that need to be identified, these three-dimensional spatial areas can be identified by creating spatial entities. That is, the entity set includes spatial entities, and the static entity data corresponding to the spatial entities includes text labels for describing spatial relationships; the spatial entities are used to identify three-dimensional spatial areas in the virtual environment; the game application creates spatial entities in the virtual environment, and the spatial entities are used to surround the three-dimensional spatial areas to be identified; the text labels of the spatial entities are obtained, and the text labels include descriptive text of the spatial relationships of the three-dimensional spatial areas; wherein the descriptive text of the spatial relationships includes at least one of the following: the scene to which the three-dimensional spatial area belongs, the building to which the three-dimensional spatial area belongs, the floor of the three-dimensional spatial area in the building to which it belongs, the spatial orientation of the three-dimensional spatial area on the floor to which it belongs, and the spatial name of the three-dimensional spatial area.

[0513] For example, if there is a second-floor area of ​​a recycling station tower in the virtual environment, a spatial entity is created in the second-floor area. The spatial entity is used to surround the second-floor area, determine the spatial location of the second-floor area, and set a description text of the spatial relationship "Farm Map-Recycling Station Tower-Second Floor Area" for the spatial entity.

[0514] After obtaining the static entity data, you can query entities or spaces based on the static entity data during game operation.

[0515] For example, when a player issues a natural language command, it is accompanied by pre-built static entity data and real-time runtime data (including environmental perception information of non-player characters and / or the controlling avatar) to make an inference request to the large language model. Furthermore, when an NPC needs to provide dynamic feedback based on runtime data, an entity query is also performed to generate feedback text based on the current state.

[0516] When a natural language command contains a target entity, the game application will query the target entity based on the position and orientation of the current controlling virtual character, the position and orientation of teammates or enemies, and based on the target entity information of the target entity described in the natural language command and the environmental perception information of the controlling virtual character and / or non-player character, filter out the target entity that matches the target entity information from at least one candidate entity within the perception range of the controlling virtual character and / or non-player character.

[0517] Optionally, natural language commands are used to instruct the non-player character to perform actions related to a target entity. For example, the target entity can be a destination for the non-player character to move to, an object that the non-player character needs to observe, a target that the non-player character needs to attack, or an object that the non-player character needs to interact with.

[0518] A target entity is a virtual object that exists in a 3D virtual environment. A target entity is an entity described in a natural language command. An entity can refer to a virtual object that is stationary in a 3D virtual environment. For example, a target entity can be a virtual building, virtual terrain, virtual vehicle, virtual plant, virtual prop, virtual item, etc. An entity can also refer to a virtual object in a 3D virtual environment that can interact with a virtual character. For example, a target entity can be an interaction point in a 3D virtual environment (e.g., a door, window, cabinet, cellar, etc.), a virtual prop, a virtual character, a virtual light source, etc.

[0519] For example, a natural language command could be "Please open the door for me," then the target entity could be "door," and the non-player character would need to perform a door-related action, "Open the door." Alternatively, a natural language command could be "Is the kitchen safe?" then the target entity could be "kitchen," and the non-player character would need to perform a kitchen-related action, "Check the kitchen for enemy avatars or other dangerous situations."

[0520] Optionally, the natural language command includes target entity information of the target entity. The target entity information may be text describing the target entity. For example, the target entity information may include at least one of the following: the name, type, location, and characteristics of the target entity. Based on the target entity information in the natural language, the game application can locate the target entity from among multiple entities in the three-dimensional virtual environment.

[0521] The game application parses the natural language commands, extracts the target entity information therein, determines the target entity from the entity set of the three-dimensional virtual environment based on the target entity information, and then controls the non-player character to perform activities related to the target entity according to the intention of the natural language commands.

[0522] The target entity is determined from the three-dimensional virtual environment in combination with environmental perception information, and the environmental perception information includes information perceived by at least one of the main virtual character and the non-player character from the three-dimensional virtual environment.

[0523] Since the target entity information in natural language commands is usually vague, for example, a natural language command can be "move to the red truck", where the target entity information is "red truck", and there may be many red trucks in the three-dimensional virtual environment. It is impossible to accurately determine the target entity in the three-dimensional virtual environment based solely on the target entity information in the natural language command.

[0524] Therefore, embodiments of the present application provide a method for determining a target entity by combining target entity information and environmental perception information. For the previous example, the "red truck" expressed by the player in the natural language command should be a red truck that the player can see. Therefore, based on the field of view of the controlling virtual character, the red truck within the field of view of the controlling virtual character can be selected from multiple red trucks in the three-dimensional virtual environment. This red truck is the target entity indicated by the player in the natural language command.

[0525] Because natural language commands are issued based on the player's perception of the 3D virtual environment, the game application uses the player's environmental perception information at the time the command is issued to accurately identify the target entity. Based on the player's perception of the 3D virtual environment, the game application infers the entity in the 3D virtual environment that is closest to the target entity's information. This entity is then identified as the target entity.

[0526] Exemplarily, a three-dimensional virtual environment includes multiple candidate entities that match a natural language command, and a target entity is an entity selected from the multiple candidate entities based on environmental perception information of a controlling virtual character or a non-player character. For example, a game application first selects multiple candidate entities from a set of entities that match the target entity information in the natural language command, and then selects an entity that can be perceived by the controlling virtual character or the non-player character from the multiple candidate entities as the target entity based on the environmental perception information.

[0527] For example, based on the position and orientation of the controlling virtual character, the controlling virtual character's perception range is determined, and this perception range is used as the search range for the target entity. For example, the perception range is a sector-shaped area of ​​a certain visual field in front of the controlling virtual character; entities outside the sector-shaped area are filtered out. The target entity information and the candidate entity information are then sorted based on their similarity, and the entity with the highest similarity is selected as the target entity. If two entities within the perception range have the highest and equal similarity scores, for example, entity 2 and entity 3 both have a similarity score of 2, entity 2, which is closer to the controlling virtual character, is selected as the target entity.

[0528] When feedback information needs to be generated based on the position information of the target entity, the game application can perform a spatial query to generate a position description based on the positional relationship between the master virtual character and the target entity.

[0529] Spatial queries are primarily used to obtain the locations of the main virtual character, non-player characters, hostile virtual characters, gunshots, and more. For example, a report containing spatial information can be broadcast, such as "Enemy spotted on the second floor of a motel."

[0530] Spatial query logic defines the concepts of parent regions and child regions. A parent region refers to a large area within a 3D virtual environment that encompasses multiple buildings or isolated spaces, such as a motel with front and back yards, multiple rooms, and a basement. A child region refers to an independent space within a parent region. These can be enclosed spaces or specialized spaces, such as a motel guest room or kitchen.

[0531] In order to make the location description returned by the spatial query closer to human expression, when the controlling virtual character and the queried location are in different parent areas, the query result returns the complete text information (for example, if the controlling virtual character is outside the motel and the queried location is on the second floor of the motel, then the query result returns "The enemy is on the second floor of the motel"). When the controlling virtual character and the queried location are in the same parent area but different child areas, the query result only returns the child area information (for example, if the controlling virtual character is on the first floor of the motel and the queried location is on the second floor, then the query result returns "The enemy is on the second floor"). When the controlling virtual character and the queried location are in the same parent area and the same child area, the query result only returns the direction relative to the main perspective player (for example, if the controlling virtual character and the queried location are both on the second floor of the motel, then the query result returns "The enemy is in the right front position").

[0532] Exemplarily, when the controlling virtual character and the target entity are located in different parent spaces, the position description includes the parent space and child space where the target entity is located. When the controlling virtual character and the target entity are located in different child spaces of the same parent space, the position description includes the child space where the target entity is located. When the controlling virtual character and the target entity are located in the same child space of the same parent space, the position description includes the relative position of the target entity and the controlling virtual character; wherein, a parent space includes at least one child space.

[0533] Regarding the broadcasting of the NPC feedback information in step 208, the feedback information is implemented as spatial human voice audio processed with spatial sound effects. The spatial sound effect processing process can be implemented as follows:

[0534] Sub-step 7: Obtain the human voice audio corresponding to the feedback text;

[0535] In the embodiment of the present application, the process of generating the feedback text is as shown in the above-mentioned sub-step 6, which will not be repeated here.

[0536] In some embodiments, the process of obtaining corresponding human voice audio based on feedback text can be implemented as follows: obtaining feedback text and character feature information corresponding to a non-player character; and generating human voice audio matching the character feature information based on the feedback text.

[0537] Illustratively, character feature information is used to describe the attributes of a non-player character. Specifically, character feature information is information used to describe the current state and characteristics of a non-player character. In some embodiments, the attributes of a non-player character include at least one of basic attributes and situational attributes. Basic attributes are pre-configured attributes for a non-player character, i.e., attributes that do not change due to the 3D virtual environment or the current situation. Examples include the non-player character's age, gender, or martial arts school. Scenario-specific attributes are attributes determined in real time within the context of the 3D virtual environment. These attributes are associated with the current situation, such as the non-player character's emotions and behavior in the current situation.

[0538] Optionally, the character characteristic information includes at least one of basic character information, character emotion information, and character behavior information. The basic character information is used to indicate the basic characteristics of the non-player character, and is information about the basic attributes of the non-player character, such as the non-player character's age, gender, or martial arts background. The character emotion information is used to indicate the emotional state of the non-player character in the dialogue context corresponding to the feedback text, and is a situational attribute of the non-player character, such as indicating whether the non-player character is excited, angry, or sad. The character behavior information is used to indicate the actions performed by the non-player character in the dialogue context, and is a situational attribute of the non-player character, such as whether the non-player character is running, attacking, or receiving injury treatment.

[0539] In some embodiments, human voice audio is generated using a pre-trained speech generation model. The speech generation model is used to generate speech that matches the character characteristics of the non-player character. Illustratively, feedback text and character characteristics are input into the pre-trained speech generation model to generate human voice audio, which is then used as the human voice audio.

[0540] Optionally, the above-mentioned speech generation model can be implemented by a neural network model such as a convolutional neural network, a feedforward neural network, a residual network, a Transformer, a multimodal large language model (MLLM), etc., which is not specifically limited here.

[0541] In some embodiments, the speech generation model includes a text encoder, a role encoder, and a decoder. The text encoder is configured to perform text encoding on the input feedback text, i.e., the feedback text is input into the text encoder to obtain a text encoding representation. The role encoder is configured to perform feature encoding on the input role feature information, i.e., the role feature information is input into the role encoder to obtain a role encoding representation.

[0542] In an illustrative manner, after obtaining the text coding representation and the role coding representation, the text coding representation and the role coding representation are fused to obtain the coding representation input to the decoder, that is, the text coding representation and the role coding representation are fused to obtain a joint coding representation, and the joint coding representation is input into the decoder to generate human voice audio.

[0543] In some embodiments, the character encoder includes at least one of a first sub-encoder, a second sub-encoder, and a third sub-encoder. The first sub-encoder is configured to perform feature encoding based on the basic attributes of the non-player character. Specifically, if the character feature information includes basic character information, the basic character information is input into the first sub-encoder to obtain a first character encoding representation. The second sub-encoder is configured to perform feature encoding based on the emotional state of the non-player character. Specifically, if the character feature information includes character emotional information, the character emotional information is input into the second sub-encoder to obtain a second character encoding representation. The third sub-encoder is configured to perform feature encoding based on the behavioral state of the non-player character. Specifically, if the character feature information includes character behavioral information, the character behavioral information is input into the third sub-encoder to obtain a third character encoding representation.

[0544] Sub-step 8: obtaining sound effect parameters corresponding to the non-player character based on the relative position relationship between the non-player character and the controlling virtual character in the three-dimensional virtual environment;

[0545] Illustratively, the relative position relationship between the non-player character and the controlling virtual character in the 3D virtual environment is used to indicate the relative relationship between the first position corresponding to the non-player character and the second position corresponding to the controlling virtual character in the 3D virtual environment.

[0546] Optionally, the above-mentioned relative orientation relationship includes at least one of position distance, position direction, and spatial encirclement between the non-player character and the main virtual character, wherein the position distance is used to indicate the distance relationship between the non-player character and the main virtual character, the position direction is used to indicate the angular relationship between the first position of the non-player character and the orientation direction of the main virtual character, and the spatial encirclement is used to indicate the surrounding situation of the main virtual character by the spatial elements forming the virtual space in the three-dimensional virtual environment, and the propagation effect of the space on the audio when the non-player character emits audio in the virtual space.

[0547] In some embodiments, the determination of the relative orientation relationship between the non-player character and the controlling virtual character can be implemented as follows: obtaining first position information of the non-player character in the three-dimensional virtual environment, obtaining second position information of the controlling virtual character in the three-dimensional virtual environment, and determining the relative orientation relationship between the non-player character and the controlling virtual character based on the first position information and the second position information.

[0548] In some embodiments, the first position information of the non-player character and the second position information of the controlling virtual character are position information determined based on the same preset coordinate system. Optionally, the preset coordinate system can be implemented as a world coordinate system corresponding to the three-dimensional virtual environment, or as a coordinate system established with the controlling virtual character as the origin.

[0549] In some embodiments, when the sound effect parameters corresponding to the non-player characters are pre-generated and stored in a database, the sound effect parameters corresponding to the non-player characters are retrieved from the database. In other embodiments, the sound effect parameters corresponding to the non-player characters may also be generated in real time, i.e., based on relative position relationships.

[0550] Optionally, the sound effect parameters corresponding to the non-player character may include at least one of the following types:

[0551] The first type is distance type

[0552] Illustratively, a distance-based sound effect parameter is used to indicate adjustments to the sound effects of the human voice audio based on the distance between the non-player character and the controlling virtual character. In some embodiments, the distance-based sound effect parameter can adjust the volume of the human voice audio to simulate the effect of distance on the volume of the sound; and / or the distance-based sound effect parameter can adjust the audio delay of the human voice audio to simulate the effect of distance on the sound propagation time.

[0553] Second, direction type

[0554] Illustratively, directional sound effect parameters are used to indicate adjustments to the sound effects of the human voice audio based on the direction of the non-player character relative to the controlling avatar. In some embodiments, directional sound effect parameters can adjust the volume of the human voice audio to simulate whether the controlling avatar is facing a scene audio element; and / or directional sound effect parameters can adjust the volume parameters of the human voice audio in different channels to simulate the direction of the audio emitted by a scene sound source element.

[0555] The third type is spatial special effects

[0556] Illustratively, spatial effect-type sound effect parameters are used to indicate adjustments to the sound effects within the virtual space based on the virtual space formed by spatial elements. Specifically, spatial effect-type sound effect parameters indicate the impact of the virtual space on the audio performance of audio emitted by non-player characters. In some embodiments, spatial effect-type sound effect parameters can include echo and reverberation effects to simulate the effect of sound within a space.

[0557] Optionally, the spatial effect type sound effect parameters include at least one of a reverberation time parameter, a pre-delay parameter, a wet / dry mix parameter, a space width parameter, and a distance effect parameter. The reverberation time parameter indicates the rate at which audio decays in the virtual space; the pre-delay parameter indicates the time difference between the audio directly reaching the controlling virtual character and the audio reaching the controlling virtual character after the first reflection; the wet / dry mix parameter indicates the ratio of directly transmitted audio to reflected audio; the space width parameter indicates the degree of audio diffusion on the horizontal plane in the virtual space; and the distance effect parameter indicates the attenuation of audio as it propagates over distance.

[0558] Fourth, custom type

[0559] Schematically, the custom type of sound effect parameters are user-defined parameters for adjusting sound effects. Optionally, the user can customize the overall volume of the scene audio, whether to add background music, customize the volume of different types of audio, etc.

[0560] In some embodiments, the client provides a user with a customization interface for sound effect parameters, and the user configures the customized sound effect parameters through the customization interface.

[0561] In some embodiments, a parameter prediction model for personalized learning of user-defined sound effect parameters is configured in the server. Illustratively, the customized sound effect parameters of multiple candidate accounts are obtained, the multiple customized sound effect parameters are input into the parameter prediction model to be trained, the parameter prediction model to be trained is iteratively trained to obtain a parameter prediction model, and the parameter prediction model is used to optimize the sound effect parameters generated by the system (e.g., the sound effect parameters of the aforementioned distance-type, direction-type, and spatial-type).

[0562] Sub-step 9: Adjusting the vocal audio based on the sound effect parameters, generating and broadcasting spatial vocal audio corresponding to the non-player character;

[0563] Among them, spatial human voice audio is used to express the perceptual effect of the audio produced by the main virtual character to the non-player character under the relative orientation relationship.

[0564] In an embodiment of the present application, human voice audio is adjusted based on the sound effect parameters corresponding to the non-player character, thereby generating spatial human voice audio corresponding to the non-player character. Specifically, spatial human voice audio is audio data corresponding to the non-player character obtained by adjusting the human voice audio using the sound effect parameters. In some embodiments, the spatial human voice audio includes the audio after the sound effect parameters have been adjusted, as well as at least one of the following data: audio duration information, audio playback start timestamp, audio playback end timestamp, audio playback condition information, and element identification information corresponding to the audio.

[0565] Optionally, when the sound effect parameters indicate an adjustment to the volume of the non-player character's vocal audio, the volume of the non-player character's vocal audio on at least one channel is adjusted based on the sound effect parameters. Optionally, when the sound effect parameters indicate an adjustment to the playback delay of the non-player character's vocal audio, the start time of the audio playback corresponding to the non-player character's vocal audio is adjusted based on the sound effect parameters. Optionally, when the sound effect parameters indicate an adjustment to the speed of the non-player character's vocal audio, the playback speed of the non-player character's vocal audio is adjusted based on the sound effect parameters. Optionally, when the sound effect parameters indicate a pitch of the non-player character's vocal audio, the frequencies in the spectrum of the non-player character's vocal audio are adjusted based on the sound effect parameters. When the sound effect parameters indicate a timbre of the non-player character's vocal audio, a filter corresponding to the non-player character is determined based on the sound effect parameters, and the timbre of the vocal audio is adjusted using the filter.

[0566] Schematically, the client plays the spatial human voice audio to realize the broadcast of non-player character feedback information.

[0567] The speech-to-text conversion phase of step 204 can be implemented as follows:

[0568] Sub-step 10: obtaining a plurality of first scene hot words corresponding to the first virtual scene where the target virtual character is located;

[0569] The target virtual character includes at least one of an NPC and a master virtual character.

[0570] For illustration, the controlling avatar is the avatar commanded by the player, while the NPC is the avatar that assists the controlling avatar in virtual games. Typically, a virtual game involves one controlling avatar and at least one NPC. There are certain differences between the controlling avatar and the NPC.

[0571] Optionally, the main virtual character is the virtual character mainly controlled by the player, and the NPC is the virtual character chosen by the player to command. For example, the main virtual character is the virtual character manually controlled by the player during the game, and the player's manual operations on the terminal interface are used to control the main virtual character; the NPC is the virtual character commanded by the player in voice form, and the player occasionally issues natural language commands in voice form to command the NPC.

[0572] In some embodiments, the first virtual scene is determined based on the target virtual character, and the target virtual character is at least one of an NPC and a master virtual character, which means that the process of determining the first virtual scene is implemented as at least one of the following.

[0573] (1) If the master virtual character is selected as the target virtual character, the virtual scene where the master virtual character is located is used as the first virtual scene;

[0574] (2) If an NPC is selected as the target virtual character, the virtual scene where the NPC is located is the first virtual scene; if there are multiple NPCs, the virtual scene including the most NPCs can be used as the first virtual scene. Alternatively, the NPC closest to the main virtual character can be selected as the target virtual character, and the virtual scene where the NPC is located can be used as the first virtual scene. Alternatively, the NPC with the largest character attribute value (such as at least one of the attribute values ​​such as virtual health, virtual mana, and virtual defense) can be selected as the target virtual character, and the virtual scene where the NPC is located can be used as the first virtual scene.

[0575] (3) If the main virtual character and the NPC are selected as the target virtual characters, the virtual scene in which the main virtual character and the NPC are located can be used as the first virtual scene, etc.

[0576] In some embodiments, the target virtual character is determined based on the player's selection; alternatively, the target virtual character is a system default setting.

[0577] In some embodiments, a plurality of first scene hot words are determined based on the first virtual scene.

[0578] The plurality of first scene hot words are scene-related words of the first virtual scene, that is, there is an association relationship between the first scene hot words and the first virtual scene, and the first scene hot words are words that describe the first virtual scene.

[0579] Optionally, the first scene hot words include the scene status of the first virtual scene. For example, if the first virtual scene is a virtual restaurant, the first scene hot words include business status, rest status, closed status, etc.

[0580] Optionally, the first scene hot words include element names of entities in the first virtual scene, such as: the first virtual scene is a virtual restaurant, and the first scene hot words include virtual tables, virtual chairs, virtual kitchen utensils, virtual plates, etc.

[0581] Optionally, the first scene hot words include interactive vocabulary for interacting with entities in the first virtual scene. For example, if the first virtual scene is a virtual battlefield, the first scene hot words include attack, attack virtual character, defense, building barriers, etc.

[0582] In some embodiments, after receiving a natural language command, multiple first scene hot words are collected based on the first virtual scene; or, after receiving a natural language command, multiple first scene hot words corresponding to the first virtual scene are filtered from multiple scene hot words acquired in advance based on the first virtual scene, etc.

[0583] In an optional embodiment, the perception range of the target virtual character in the first virtual scene is obtained.

[0584] Among them, the perception range is the three-dimensional spatial range in which the target virtual character perceives the existence of other scene elements.

[0585] Optionally, when the target virtual character is the master virtual character, the perception range of the master virtual character in the first virtual scene is obtained; when the target virtual character is an NPC, the perception range of the NPC in the first virtual scene is obtained; or, when the target virtual character is the master virtual character and the NPC, the perception range of the master virtual character in the first virtual scene and the perception range of the NPC in the first virtual scene are obtained.

[0586] In some embodiments, the perception range includes at least one of the following.

[0587] (1) The visual perception range (field of view) of the target virtual character.

[0588] For illustration, visual perception range refers to the spatial range that an observer can visually perceive or understand. It is often used to describe the area that an organism or technological device can visually cover or perceive. The visual perception range of a target avatar is the three-dimensional spatial range perceived by the target avatar as the observer.

[0589] Optionally, based on the position and visual observation ability of the target virtual character in the first virtual scene, the visual perception range of the target virtual character is acquired.

[0590] (2) The auditory perception range of the target virtual character (auditory range).

[0591] For illustration, auditory perception range refers to the spatial range that a listener can perceive or understand. It is typically used to describe the range of sounds that an organism or technological device can hear or perceive. The auditory perception range of a target avatar is the three-dimensional spatial range perceived by the target avatar through its auditory perception capabilities.

[0592] Optionally, based on the position and auditory perception ability of the target virtual character in the first virtual scene, the auditory perception range of the target virtual character is acquired.

[0593] (3) The olfactory perception range of the target virtual character (olfactory range).

[0594] Schematically, the olfactory perception range refers to the spatial range of the odor perceived or recognized by the sense of smell. The olfactory perception range of the target virtual character is the three-dimensional spatial range that the target virtual character can perceive through the sense of smell.

[0595] Optionally, the olfactory perception range of the target virtual character is acquired based on the position of the target virtual character in the first virtual scene and the olfactory perception ability.

[0596] (4) The detection range of the target virtual character’s perception skills or the perception virtual props.

[0597] Illustratively, the perception skill is the target virtual character's ability to perceive the first virtual scene, and the perception skill includes at least one of a perception acquisition form and a perception enhancement form.

[0598] Optionally, the "perception acquisition" mode is used to represent the process of acquiring a previously unpossessed perceptual skill. Once a target virtual character acquires a perceptual skill through the "perception acquisition" mode, they can use the perceptual skill to conduct targeted scene perception of the first virtual scene. Alternatively, the "perception enhancement" mode is used to represent the process of enhancing a certain perceptual ability through the acquisition of a perceptual skill.

[0599] Illustratively, the perception virtual prop is a virtual prop used to perceive the first virtual scene, and the perception virtual prop includes at least one of a perception acquisition form and a perception enhancement form.

[0600] Optionally, different perception skills or perception virtual props each correspond to a detection range, which represents the range interval that can be perceived when the perception skill or perception virtual prop is applied, and the detection range represents the perception range.

[0601] In some embodiments, the sensing range is a spherical range, an annular range, an irregular three-dimensional space range, etc. The shape of the sensing range is not limited here.

[0602] In an optional embodiment, a plurality of first scene hot words are acquired based on environmental perception information within the perception range.

[0603] Schematically, the environmental perception information is used to represent the environmental information perceived and acquired by the target virtual character within the perception range. Optionally, the environmental perception information includes entities determined by raycasting, and may also include regional structure information determined by analyzing the geometric structure and layout information of the virtual environment.

[0604] Schematically, after determining the perception range corresponding to the target virtual character, multiple first scene hot words representing the state of the target virtual character are obtained based on the range of the perception range in the first virtual scene and the environmental perception information determined based on perception of the target virtual character within the perception range.

[0605] In an optional embodiment, the environmental perception information includes entities; and entities within the perception range in the first virtual scene are determined.

[0606] Schematically, the perception range is a part of the three-dimensional space range in the first virtual scene; the first virtual scene includes a large number of entities, and the entities are elements that constitute the virtual scene, such as virtual ground, virtual buildings, virtual trees, virtual characters, virtual stones, etc.

[0607] In some embodiments, when the perception range is determined, entities within the perception range are determined from multiple entities corresponding to the first virtual scene.

[0608] In some embodiments, the entity name of the entity is used as the first scene hot word; or, the action name of the interactive action corresponding to the entity is used as the first scene hot word.

[0609] Sub-step 11: converting the natural language command in voice form into text form based on the multiple first scenario hot words;

[0610] Indicatively, the hot words based on the first scene are words that are highly associated with the first virtual scene. Therefore, when hoping to command the NPC to better cooperate with the activities of the main virtual character through natural language commands, the hot words of the first scene can be used as constraints on the basis of the first virtual scene. In this way, in the process of converting the natural language commands into text form, the text content of the command analysis results is more consistent with the first virtual scene, avoiding obtaining command analysis results that are incompatible with the first virtual scene.

[0611] In an optional embodiment, when a natural language command is analyzed by a natural language analysis model, multiple speech units corresponding to the natural language command are acquired through an acoustic network.

[0612] Among them, phonetic units are the basic building blocks of vocabulary pronunciation.

[0613] In some embodiments, a command feature representation corresponding to a natural language command is extracted; and the command feature representation is analyzed through an acoustic network.

[0614] For example, using a command feature representation representing multiple acoustic features, the acoustic network aims to map a continuous sequence of acoustic features to a sequence of speech units, such as phonemes or segments. It learns to predict the likely sequence of speech units (phonemes or segments) given the acoustic features.

[0615] Optionally, the acoustic network outputs one or more possible speech unit sequences, each of which includes multiple speech units. Different speech unit sequences may contain different speech units, or may simply differ in the order of the speech units. The multiple speech unit sequences represent the most likely arrangements of speech units considered by the language network when understanding the input natural language command.

[0616] The natural language analysis model includes an acoustic network, a preset dictionary, and language sub-networks corresponding to at least two virtual scenes.

[0617] Schematically, the virtual environment includes at least two virtual scenes. For example, the virtual environment is a large virtual world, which includes a virtual kitchen 1, a virtual kitchen 2, a virtual office, etc., and each of the virtual kitchen 1, the virtual kitchen 2, the virtual office, etc. can be regarded as a virtual scene.

[0618] In an optional embodiment, based on a first virtual scene in which the target virtual character is located, a first language sub-network corresponding to the first virtual scene is determined from at least two language sub-networks.

[0619] Schematically, multiple virtual scenes each correspond to a language subnetwork, and different language subnetworks are used to analyze the corresponding virtual scenes in a targeted manner. For example, virtual scene A corresponds to language subnetwork 1, and virtual scene B corresponds to language subnetwork 2. When analysis is required based on virtual scene A, language subnetwork 1 is used for language analysis; when analysis is required based on virtual scene B, language subnetwork 2 is used for language analysis, and so on.

[0620] Schematically, the first virtual scene is a virtual scene among at least two virtual scenes. In addition to determining the first scene hot words based on the first virtual scene in which the target virtual character is located, the first language subnetwork corresponding to the first virtual scene is also determined from at least two language subnetworks based on the first virtual scene. The first language subnetwork is a subnetwork layer that performs semantic analysis on natural language commands based on the first virtual scene.

[0621] In an optional embodiment, the first language sub-network analyzes the sequence matching relationship between multiple speech units and selected words in a preset dictionary to convert the natural language command in speech form into text form.

[0622] Optionally, different language sub-networks are language sub-networks pre-trained based on corresponding virtual scenes, wherein the first language sub-network is a language sub-network pre-trained based on the first virtual scene.

[0623] In some embodiments, matching relationships between multiple speech units and selected words in a preset dictionary are analyzed to obtain multiple candidate word sequences.

[0624] The selected words include at least a plurality of first-scene hot words and common words.

[0625] Illustratively, the process of analyzing matching relationships represents the process of analyzing the matching between selected vocabulary and speech units. Illustratively, each selected vocabulary corresponds to at least one speech unit. By analyzing multiple speech units and the selected vocabulary, the conversion probability of converting a speech unit to the selected vocabulary is determined. The conversion probability represents the probability of matching between the selected vocabulary and the speech unit. A greater conversion probability indicates that the speech unit is closer to the selected vocabulary, i.e., a greater likelihood that the speech unit can express the selected vocabulary, and thus a stronger matching relationship between the selected vocabulary and the speech unit.

[0626] Optionally, multiple speech units form at least one speech unit sequence, and based on a matching analysis process between the speech units and the selected vocabulary, at least one candidate vocabulary sequence corresponding to the at least one speech unit sequence is determined, and multiple candidate vocabulary sequences are obtained.

[0627] In some embodiments, the sequence semantics of multiple candidate vocabulary sequences are analyzed by the first language sub-network, and at least one candidate vocabulary sequence is obtained from the multiple candidate vocabulary sequences as a text form for natural language command conversion.

[0628] Schematically, sequence semantics is used to represent the semantic information expressed by the candidate vocabulary sequence, which can not only reflect the semantic changes of at least one word in the candidate vocabulary sequence, but also reflect the sentence semantics of the entire candidate vocabulary sequence.

[0629] Schematically, the natural language command in text form is called the command analysis result; multiple candidate vocabulary sequences are evaluated separately by the first language sub-network, so that at least one candidate vocabulary sequence that is more in line with the current semantic situation is taken as the command analysis result, that is, the natural language command in speech form is converted into text form.

[0630] It should be noted that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0631] The following are device embodiments of the present application. For details not described in detail in the device embodiments, reference may be made to the above method embodiments.

[0632] Figure 19 This is a block diagram of an entity determination device in a virtual environment provided by an exemplary embodiment of the present application. The device is used to implement a terminal. The device includes:

[0633] A display module 1003 is configured to display at least one of a main virtual character and non-player characters in a virtual environment;

[0634] A receiving module 1001 is configured to receive a natural language command, wherein the natural language command is used to control the non-player character;

[0635] A control module 1002 is configured to control the non-player character to move in the virtual environment according to the target entity indicated by the natural language command;

[0636] The target entity is determined from the virtual environment in combination with environmental perception information, and the environmental perception information includes information perceived by at least one of the main virtual character and the non-player character from the virtual environment.

[0637] In an optional embodiment, the environmental perception information includes at least one of the following:

[0638] Visual information perceived within the visual field of the master virtual character;

[0639] Auditory information perceived within the auditory range of the master virtual character;

[0640] The information perceived by the master virtual character's perception skills or virtual props;

[0641] visual information perceived within the field of view of the non-player character;

[0642] Auditory information perceived within the hearing range of the non-player character;

[0643] The non-player character's perception skills or information perceived by the perception virtual props.

[0644] In an optional embodiment, the virtual environment includes multiple candidate entities matching the natural language command, and the target entity is an entity screened from the multiple candidate entities based on environmental perception information of the master virtual character or the non-player character.

[0645] In an optional embodiment, the device further includes:

[0646] A first query module 1005 is configured to query and obtain the target entity from the entity set of the virtual environment according to the target entity information indicated by the natural language command and the environment perception information;

[0647] The control module 1002 is configured to control the non-player character to perform an activity associated with the target entity according to the intention of the natural language command.

[0648] In an optional embodiment, the first query module 1005 is configured to parse the natural language command to obtain the target entity information; the target entity information includes at least one of the following: entity type, entity name, entity location, and entity features;

[0649] The first query module 1005 is configured to calculate the similarity between the target entity information and each entity information in the entity set;

[0650] The first query module 1005 is configured to determine the target entity from the entity set according to the similarity and the environment perception information.

[0651] In an optional embodiment, the similarity includes text similarity; the entity set includes entity information of the first entity;

[0652] The first query module 1005 is configured to segment the target entity information to obtain at least one target entity tag;

[0653] The first query module 1005 is configured to convert the at least one target entity label into at least one target embedding vector;

[0654] The first query module 1005 is configured to obtain entity information of the first entity, where the entity information includes a text embedding vector, where the text embedding vector is an embedding vector obtained by converting a text label of the first entity;

[0655] The first query module 1005 is used to respectively calculate the text parent similarity between the at least one target embedding vector and the text embedding vector, and obtain at least one text parent similarity corresponding to the at least one target embedding vector;

[0656] The first query module 1005 is configured to determine the sum of the at least one text parent similarity as the text similarity between the target entity information and the entity information of the first entity.

[0657] In an optional embodiment, the entity information of the first entity includes at least one text embedding vector; the at least one target embedding vector includes a first target embedding vector;

[0658] The first query module 1005 is configured to respectively calculate the text sub-similarity between the first target embedding vector and the at least one text embedding vector to obtain at least one text sub-similarity;

[0659] The first query module 1005 is configured to determine the highest value among the at least one text sub-similarity as the text parent similarity corresponding to the first target embedding vector.

[0660] In an optional embodiment, the similarity includes image similarity; the entity set includes entity information of the first entity;

[0661] The first query module 1005 is configured to segment the target entity information to obtain at least one target entity tag;

[0662] The first query module 1005 is configured to convert the at least one target entity label into at least one target embedding vector;

[0663] The first query module 1005 is configured to obtain entity information of the first entity, where the entity information includes an image embedding vector, where the image embedding vector is an embedding vector extracted based on the image of the first entity;

[0664] The first query module 1005 is configured to respectively calculate the image parent similarity between the at least one target embedding vector and the text embedding vector, to obtain at least one image parent similarity corresponding to each of the at least one target embedding vectors;

[0665] The first query module 1005 is configured to determine the sum of the at least one image parent similarity as the image similarity between the target entity information and the entity information of the first entity.

[0666] In an optional embodiment, the entity information of the first entity includes at least one image embedding vector; the at least one target embedding vector includes a first target embedding vector;

[0667] The first query module 1005 is configured to respectively calculate the image sub-similarity between the first target embedding vector and the at least one image embedding vector to obtain at least one image sub-similarity;

[0668] The first query module 1005 is configured to determine the highest value of the at least one image sub-similarity as the image parent similarity corresponding to the first target embedding vector.

[0669] In an optional embodiment, the device further includes:

[0670] A first pre-processing module 1004 is configured to obtain at least one perspective image of the first entity; and

[0671] A first pre-processing module 1004 is configured to obtain at least one text label of the first entity;

[0672] The first preprocessing module 1004 is used to call the multimodal model to extract the visual features of the first entity based on the at least one perspective image and the at least one text label of the first entity, and obtain the image embedding vector of the first entity.

[0673] In an optional embodiment, the entity set includes a first entity, and the apparatus further includes:

[0674] The first query module 1005 is configured to, when the first entity and the target entity have corresponding text similarity and image similarity, determine an average value of the text similarity and image similarity as the similarity between the first entity and the target entity.

[0675] In an optional embodiment, the first query module 1005 is configured to determine a search range based on the target entity information and the environment perception information;

[0676] The first query module 1005 is configured to filter the target entity from the entities sensed within the search range according to the similarity.

[0677] In an optional embodiment, the first query module 1005 is configured to determine the entity with the highest similarity in the search range as the target entity;

[0678] The first query module 1005 is configured to determine the entity with the highest similarity in the search range and closest to the controlling virtual character as the target entity when there are at least two entities with the highest similarity in the search range.

[0679] Figure 20 This is a block diagram of an entity determination device in a virtual environment provided by an exemplary embodiment of the present application. The device is used to implement a server. The device includes:

[0680] An acquisition module 1006 is configured to acquire an entity set, wherein the entity set includes entity information of at least one entity in the virtual environment;

[0681] The acquisition module 1006 is configured to acquire environmental perception information, wherein the environmental perception information is information perceived by at least one of the virtual character and the non-player character from the virtual environment;

[0682] The second query module 1008 is configured to determine the target entity from the entity set according to the entity information of the target entity and the environment perception information.

[0683] In an optional embodiment, the environmental perception information includes at least one of the following:

[0684] Visual information perceived within the visual field of the master virtual character;

[0685] Auditory information perceived within the auditory range of the master virtual character;

[0686] The information perceived by the master virtual character's perception skills or virtual props;

[0687] visual information perceived within the field of view of the non-player character;

[0688] auditory information perceived within the hearing range of the non-player character;

[0689] The non-player character's perception skills or information perceived by the perception virtual props.

[0690] In an optional embodiment, the virtual environment includes multiple candidate entities matching the natural language command, and the target entity is an entity screened from the multiple candidate entities based on environmental perception information of the master virtual character or the non-player character.

[0691] In an optional embodiment, the second query module 1008 is configured to calculate the similarity between the target entity information of the target entity and each entity information in the entity set;

[0692] The second query module 1008 is configured to determine the target entity from the entity set according to the similarity and the environment perception information.

[0693] In an optional embodiment, the similarity includes text similarity; the entity set includes entity information of the first entity;

[0694] The second query module 1008 is configured to segment the target entity information to obtain at least one target entity tag;

[0695] The second query module 1008 is configured to convert the at least one target entity label into at least one target embedding vector;

[0696] The second query module 1008 is configured to obtain entity information of the first entity, where the entity information includes a text embedding vector, where the text embedding vector is an embedding vector obtained by converting a text label of the first entity;

[0697] The second query module 1008 is used to respectively calculate the text parent similarity between the at least one target embedding vector and the text embedding vector, and obtain at least one text parent similarity corresponding to the at least one target embedding vector;

[0698] The second query module 1008 is configured to determine the sum of the at least one text parent similarity as the text similarity between the target entity information and the entity information of the first entity.

[0699] In an optional embodiment, the entity information of the first entity includes at least one text embedding vector; the at least one target embedding vector includes a first target embedding vector;

[0700] The second query module 1008 is configured to respectively calculate the text sub-similarity between the first target embedding vector and the at least one text embedding vector to obtain at least one text sub-similarity;

[0701] The second query module 1008 is configured to determine the highest value among the at least one text sub-similarity as the text parent similarity corresponding to the first target embedding vector.

[0702] In an optional embodiment, the similarity includes image similarity; the entity set includes entity information of the first entity;

[0703] The second query module 1008 is configured to segment the target entity information to obtain at least one target entity tag;

[0704] The second query module 1008 is configured to convert the at least one target entity label into at least one target embedding vector;

[0705] The second query module 1008 is configured to obtain entity information of the first entity, where the entity information includes an image embedding vector, where the image embedding vector is an embedding vector extracted based on the image of the first entity;

[0706] The second query module 1008 is configured to respectively calculate the image parent similarity between the at least one target embedding vector and the text embedding vector, and obtain at least one image parent similarity corresponding to the at least one target embedding vector;

[0707] The second query module 1008 is configured to determine the sum of the at least one image parent similarity as the image similarity between the target entity information and the entity information of the first entity.

[0708] In an optional embodiment, the entity information of the first entity includes at least one image embedding vector; the at least one target embedding vector includes a first target embedding vector;

[0709] The second query module 1008 is configured to respectively calculate the image sub-similarity between the first target embedding vector and the at least one image embedding vector to obtain at least one image sub-similarity;

[0710] The second query module 1008 is configured to determine the highest value of the at least one image sub-similarity as the image parent similarity corresponding to the first target embedding vector.

[0711] In an optional embodiment, the device further includes:

[0712] A second pre-processing module 1007 is configured to obtain at least one perspective image of the first entity; and

[0713] The second pre-processing module 1007 is configured to obtain at least one text label of the first entity;

[0714] The second preprocessing module 1007 is used to call the multimodal model to extract the visual features of the first entity based on the at least one perspective image and the at least one text label of the first entity to obtain the image embedding vector of the first entity.

[0715] In an optional embodiment, the entity set includes a first entity;

[0716] The second query module 1008 is configured to, when the first entity and the target entity have corresponding text similarity and image similarity, determine an average value of the text similarity and image similarity as the similarity between the first entity and the target entity.

[0717] In an optional embodiment, the second query module 1008 is configured to determine a search range based on the target entity information and the environment perception information;

[0718] The second query module 1008 is configured to filter the target entity from the entities sensed within the search range according to the similarity.

[0719] In an optional embodiment, the second query module 1008 is configured to determine the entity with the highest similarity in the search range as the target entity;

[0720] The second query module 1008 is configured to determine the entity with the highest similarity in the search range and closest to the controlling virtual character as the target entity when there are at least two entities with the highest similarity in the search range.

[0721] It should be noted that the above-described embodiments of the apparatus for determining entities in a virtual environment are merely illustrative of the division of the aforementioned functional modules. In actual applications, the aforementioned functions can be assigned to different functional modules as needed, i.e., the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the apparatus for determining entities in a virtual environment provided in the above-described embodiments and the embodiments of the method for determining entities in a virtual environment are based on the same concept. The specific implementation process is detailed in the method embodiments and will not be further elaborated here.

[0722] The present application also provides a terminal, which includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the entity determination method in the virtual environment provided by the above-mentioned various method embodiments.

[0723] Figure 21 The following is a block diagram of a terminal 900 according to an exemplary embodiment of the present application. Terminal 900 may be a smartphone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, or a desktop computer. Terminal 900 may also be referred to as user equipment, a portable terminal, a laptop terminal, a desktop terminal, or other similar names.

[0724] Typically, the terminal 900 includes a processor 901 and a memory 902 .

[0725] The processor 901 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 901 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 901 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 901 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 901 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0726] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 902 is used to store at least one instruction, which is executed by the processor 901 to implement the information prompt method for the turn-based chess game provided in the method embodiment of the present application.

[0727] In some embodiments, terminal 900 may optionally include a peripheral device interface 903 and at least one peripheral device. Processor 901, memory 902, and peripheral device interface 903 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 903 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 904, a display screen 905, a camera assembly 906, an audio circuit 907, a location assembly 908, and a power supply 909.

[0728] The peripheral device interface 903 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 901 and the memory 902. In some embodiments, the processor 901, the memory 902, and the peripheral device interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 901, the memory 902, and the peripheral device interface 903 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0729] The RF circuit 904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 904 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the RF circuit 904 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The RF circuit 904 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 904 may also include circuits related to Near Field Communication (NFC), which is not limited in this application.

[0730] Display screen 905 is used to display a user interface (UI). This UI may include graphics, text, icons, videos, or any combination thereof. When display screen 905 is a touchscreen display, it is also capable of collecting touch signals on or above the surface of display screen 905. These touch signals can be input as control signals to processor 901 for processing. Display screen 905 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there can be a single display screen 905, located on the front panel of terminal 900. In other embodiments, there can be at least two display screens 905, located on different surfaces of terminal 900 or in a foldable design. In still other embodiments, display screen 905 can be a flexible display, located on a curved or foldable surface of terminal 900. Display screen 905 can also be configured as a non-rectangular, irregular shape, also known as a special-shaped screen. Display screen 905 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0731] The camera assembly 906 is used to capture images or videos. Optionally, the camera assembly 906 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 906 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0732] The audio circuit 907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals to be input into the processor 901 for processing, or input into the radio frequency circuit 904 to achieve voice communication. For the purpose of stereo sound collection or noise reduction, there may be multiple microphones, each located in different parts of the terminal 900. The microphone may also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert electrical signals from the processor 901 or the radio frequency circuit 904 into sound waves. The speaker may be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 907 may also include a headphone jack.

[0733] The location component 908 is used to locate the current geographic location of the terminal 900 to implement navigation or LBS (Location Based Service). The location component 908 can be a location component based on the GPS (Global Positioning System), Beidou system or Galileo system.

[0734] Power supply 909 is used to power various components in terminal 900. Power supply 909 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 909 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0735] In some embodiments, the terminal 900 further includes one or more sensors 910 , including but not limited to: an acceleration sensor 911 , a gyroscope sensor 912 , a pressure sensor 913 , an optical sensor 915 , and a proximity sensor 916 .

[0736] The accelerometer 911 can detect the magnitude of acceleration along the three coordinate axes of the coordinate system established by the terminal 900. For example, the accelerometer 911 can be used to detect the components of gravity acceleration along the three coordinate axes. The processor 901 can control the display screen 905 to display the user interface in a landscape or portrait view based on the gravity acceleration signal collected by the accelerometer 911. The accelerometer 911 can also be used to collect game or user motion data.

[0737] The gyroscope sensor 912 can detect the orientation and rotation angle of the terminal 900. It can work with the accelerometer 911 to collect the user's 3D movements on the terminal 900. Based on the data collected by the gyroscope sensor 912, the processor 901 can implement the following functions: motion sensing (for example, changing the UI based on the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0738] The pressure sensor 913 can be set on the side frame of the terminal 900 and / or the lower layer of the display screen 905. When the pressure sensor 913 is set on the side frame of the terminal 900, it can detect the user's grip signal of the terminal 900, and the processor 901 performs left and right hand recognition or shortcut operations based on the grip signal collected by the pressure sensor 913. When the pressure sensor 913 is set on the lower layer of the display screen 905, the processor 901 controls the operable controls on the UI interface based on the user's pressure operation on the display screen 905. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0739] The optical sensor 915 is used to detect ambient light intensity. In one embodiment, the processor 901 can control the display brightness of the display screen 905 based on the ambient light intensity detected by the optical sensor 915. Specifically, when the ambient light intensity is high, the display brightness of the display screen 905 is increased; when the ambient light intensity is low, the display brightness of the display screen 905 is decreased. In another embodiment, the processor 901 can also dynamically adjust the shooting parameters of the camera assembly 906 based on the ambient light intensity detected by the optical sensor 915.

[0740] Proximity sensor 916, also known as a distance sensor, is typically located on the front panel of terminal 900. Proximity sensor 916 is used to detect the distance between the user and the front of terminal 900. In one embodiment, when proximity sensor 916 detects that the distance between the user and the front of terminal 900 is gradually decreasing, processor 901 controls display screen 905 to switch from the screen-on state to the screen-off state. When proximity sensor 916 detects that the distance between the user and the front of terminal 900 is gradually increasing, processor 901 controls display screen 905 to switch from the screen-off state to the screen-on state.

[0741] Those skilled in the art will understand that Figure 21 The structure shown in the figure does not constitute a limitation on the terminal 900, and the terminal 900 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0742] The memory also includes one or more programs, which are stored in the memory and include a method for performing entity determination in a virtual environment provided by an embodiment of the present application.

[0743] The present application provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the entity determination method in a virtual environment provided by each of the above method embodiments.

[0744] The present application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method for determining an entity in a virtual environment provided in the above-mentioned optional implementation.

[0745] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0746] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0747] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for determining an entity in a virtual environment, characterized in that: The method comprises: displaying at least one of a host virtual character and a non-player character in a virtual environment; receiving a natural language command for directing the non-player character; controlling the non-player character to move in the virtual environment according to the target entity indicated by the natural language command; The target entity is determined from the virtual environment in combination with environmental perception information, wherein the environmental perception information includes information perceived by the controlling virtual character and at least one of the non-player characters from the virtual environment; The target entity is selected from a plurality of candidate entities based on the environmental perception information of the controlling virtual character or the non-player character, the plurality of candidate entities being entities that match the target entity information in the natural language command; In the case where the natural language command includes a second target entity determined based on the position of the first target entity, the second target entity is obtained by querying the first target entity from the entity set of the virtual environment based on the first target entity information of the first target entity indicated by the natural language command and the environmental perception information; and then obtained by querying from the entity set of the virtual environment based on the second target entity information of the second target entity, the first target entity and the environmental perception information.

2. The method according to claim 1, characterized in that The environmental perception information includes at least one of the following: Visual information perceived within the visual field of the master virtual character; Auditory information perceived within the auditory range of the master virtual character; The information perceived by the master virtual character's perception skills or virtual props; visual information perceived within the field of view of the non-player character; auditory information perceived within the hearing range of the non-player character; The non-player character's perception skills or information perceived by the perception virtual props.

3. The method according to claim 1, characterized in that The virtual environment includes a plurality of candidate entities that match the natural language command, and the target entity is an entity screened from the plurality of candidate entities based on environmental perception information of the master virtual character or the non-player character.

4. The method according to any one of claims 1 to 3, characterized in that: The controlling the non-player character to move in the virtual environment according to the physical object indicated by the natural language command comprises: According to the target entity information indicated by the natural language command and the environment perception information, query and obtain the target entity from the entity set of the virtual environment; According to the intention of the natural language command, the non-player character is controlled to perform an action associated with the target entity.

5. The method according to claim 4, characterized in that The querying and obtaining the target entity from the entity set of the virtual environment according to the target entity information indicated by the natural language command and the environment perception information includes: Parsing the natural language command to obtain the target entity information; the target entity information includes at least one of the following: entity type, entity name, entity location, and entity characteristics; Calculating the similarity between the target entity information and each entity information in the entity set; The target entity is determined from the entity set according to the similarity and the environment perception information.

6. The method according to claim 5, characterized in that The similarity includes text similarity; the entity set includes entity information of the first entity; The calculating the similarity between the target entity information and each entity information in the entity set includes: Segmenting the target entity information to obtain at least one target entity label; converting the at least one target entity label into at least one target embedding vector; Acquire entity information of the first entity, where the entity information includes a text embedding vector, where the text embedding vector is an embedding vector obtained by converting a text label of the first entity; Calculating the text parent similarity between the at least one target embedding vector and the text embedding vector respectively, to obtain at least one text parent similarity corresponding to the at least one target embedding vector respectively; The sum of the at least one text parent similarity is determined as the text similarity between the target entity information and the entity information of the first entity.

7. The method according to claim 6, characterized in that The entity information of the first entity includes at least one text embedding vector; the at least one target embedding vector includes a first target embedding vector; The calculating the text parent similarity between the at least one target embedding vector and the text embedding vector includes: respectively calculating text sub-similarity between the first target embedding vector and the at least one text embedding vector to obtain at least one text sub-similarity; The highest value of the at least one text sub-similarity is determined as the text parent similarity corresponding to the first target embedding vector.

8. The method according to claim 5, characterized in that The similarity includes image similarity; the entity set includes entity information of the first entity; The calculating the similarity between the target entity information and each entity information in the entity set includes: Segmenting the target entity information to obtain at least one target entity label; converting the at least one target entity label into at least one target embedding vector; Acquire entity information of the first entity, the entity information including an image embedding vector, the image embedding vector being an embedding vector extracted based on the image of the first entity; Calculating the image parent similarity between the at least one target embedding vector and the image embedding vector respectively, to obtain at least one image parent similarity corresponding to the at least one target embedding vector; The sum of the at least one image parent similarity is determined as the image similarity between the target entity information and the entity information of the first entity.

9. The method according to claim 6, characterized in that The entity information of the first entity includes at least one image embedding vector; the at least one object embedding vector includes a first object embedding vector; The calculating the image parent similarity between the at least one target embedding vector and the image embedding vector includes: respectively calculating image sub-similarity between the first target embedding vector and the at least one image embedding vector to obtain at least one image sub-similarity; The highest value of the at least one image sub-similarity is determined as the image parent similarity corresponding to the first target embedding vector.

10. The method according to claim 8, characterized in that The method further comprises: Acquire at least one perspective image of the first entity; and acquire at least one text label of the first entity; the text label is used to introduce the inherent attributes of the first entity in the virtual environment; The multimodal model is called to extract visual features of the first entity based on the at least one perspective image and the at least one text label of the first entity to obtain the image embedding vector of the first entity.

11. The method according to any one of claims 5 to 9, characterized in that: The entity set includes a first entity, and the method further includes: In the case that the first entity and the target entity have corresponding text similarity and image similarity, an average value of the text similarity and the image similarity is determined as the similarity between the first entity and the target entity.

12. The method according to any one of claims 5 to 9, characterized in that: Determining the target entity from the entity set according to the similarity and the environmental perception information includes: Determine a search range based on the target entity information and the environmental perception information; The target entity is obtained by screening the entities sensed within the search range according to the similarity.

13. The method according to claim 12, characterized in that The step of filtering the target entity from the entities sensed within the search range according to the similarity includes one of the following: Determine the entity with the highest similarity in the search range as the target entity; When the number of entities with the highest similarity in the search range is at least two, the entity with the highest similarity in the search range and the closest distance to the master virtual character is determined as the target entity.

14. A method for determining an entity in a virtual environment, characterized in that: The method comprises: Acquire an entity set, the entity set including entity information of at least one entity in the virtual environment; Acquiring environmental perception information, the environmental perception information including information perceived by at least one of the controlling virtual character and the non-player character from the virtual environment; determining the target entity from the entity set according to the target entity information of the target entity and the environment perception information; The target entity is selected from a plurality of candidate entities based on the environmental perception information of the controlling virtual character or the non-player character, the plurality of candidate entities being entities that match the target entity information in the natural language command, the natural language command being used to command the non-player character; In the case where the natural language command includes a second target entity determined based on the position of the first target entity, the second target entity is obtained by querying the first target entity from the entity set of the virtual environment based on the first target entity information of the first target entity indicated by the natural language command and the environmental perception information; and then obtained by querying from the entity set of the virtual environment based on the second target entity information of the second target entity, the first target entity and the environmental perception information.

15. The method according to claim 14, characterized in that The environmental perception information includes at least one of the following: Visual information perceived within the visual field of the master virtual character; Auditory information perceived within the auditory range of the master virtual character; The information perceived by the master virtual character's perception skills or virtual props; visual information perceived within the field of view of the non-player character; Auditory information perceived within the hearing range of the non-player character; The non-player character's perception skills or information perceived by the perception virtual props.

16. The method according to claim 14, characterized in that The virtual environment includes a plurality of candidate entities that match the natural language command, and the target entity is an entity screened from the plurality of candidate entities based on environmental perception information of the master virtual character or the non-player character.

17. The method according to any one of claims 14 to 16, characterized in that: The determining the target entity from the entity set according to the target entity information and the environment perception information of the target entity includes: Calculate the similarity between the target entity information and each entity information in the entity set; The target entity is determined from the entity set according to the similarity and the environmental perception information.

18. The method according to claim 17, characterized in that The similarity includes text similarity; the entity set includes entity information of the first entity; The calculating the similarity between the target entity information and each entity information in the entity set includes: Segmenting the target entity information to obtain at least one target entity label; converting the at least one target entity label into at least one target embedding vector; Acquire entity information of the first entity, where the entity information includes a text embedding vector, where the text embedding vector is an embedding vector obtained by converting a text label of the first entity; Calculating the text parent similarity between the at least one target embedding vector and the text embedding vector respectively, to obtain at least one text parent similarity corresponding to the at least one target embedding vector respectively; The sum of the at least one text parent similarity is determined as the text similarity between the target entity information and the entity information of the first entity.

19. The method according to claim 18, characterized in that The entity information of the first entity includes at least one text embedding vector; the at least one target embedding vector includes a first target embedding vector; The calculating the text parent similarity between the at least one target embedding vector and the text embedding vector includes: respectively calculating text sub-similarity between the first target embedding vector and the at least one text embedding vector to obtain at least one text sub-similarity; The highest value of the at least one text sub-similarity is determined as the text parent similarity corresponding to the first target embedding vector.

20. The method according to claim 17, wherein The similarity includes image similarity; the entity set includes entity information of the first entity; The calculating the similarity between the target entity information and each entity information in the entity set includes: Segmenting the target entity information to obtain at least one target entity label; converting the at least one target entity label into at least one target embedding vector; Acquire entity information of the first entity, the entity information including an image embedding vector, the image embedding vector being an embedding vector extracted based on the image of the first entity; Calculating the image parent similarity between the at least one target embedding vector and the image embedding vector respectively, to obtain at least one image parent similarity corresponding to the at least one target embedding vector; The sum of the at least one image parent similarity is determined as the image similarity between the target entity information and the entity information of the first entity.

21. The method according to claim 20, characterized in that The entity information of the first entity includes at least one image embedding vector; the at least one object embedding vector includes a first object embedding vector; The calculating the image parent similarity between the at least one target embedding vector and the image embedding vector includes: respectively calculating image sub-similarity between the first target embedding vector and the at least one image embedding vector to obtain at least one image sub-similarity; The highest value of the at least one image sub-similarity is determined as the image parent similarity corresponding to the first target embedding vector.

22. The method according to claim 20, characterized in that The method further comprises: Acquiring at least one perspective image of the first entity; and, Obtaining at least one text label of the first entity; The multimodal model is called to extract visual features of the first entity based on the at least one perspective image and the at least one text label of the first entity to obtain the image embedding vector of the first entity.

23. The method according to claim 17, wherein The entity set includes a first entity, and the method further includes: In the case that the first entity and the target entity have corresponding text similarity and image similarity, an average value of the text similarity and the image similarity is determined as the similarity between the first entity and the target entity.

24. The method according to claim 17, wherein Determining the target entity from the entity set according to the similarity and the environment perception information includes: Determine a search range based on the target entity information and the environmental perception information; The target entity is obtained by screening the entities sensed within the search range according to the similarity.

25. The method according to claim 24, characterized in that The step of filtering the target entity from the entities sensed within the search range according to the similarity includes one of the following: Determine the entity with the highest similarity in the search range as the target entity; When the number of entities with the highest similarity in the search range is at least two, the entity with the highest similarity in the search range and the closest distance to the master virtual character is determined as the target entity.

26. An entity determination device in a virtual environment, characterized in that: The device comprises: A display module for displaying at least one of a main control virtual character and a non-player character in a virtual environment; a receiving module, configured to receive a natural language command, wherein the natural language command is used to control the non-player character; a control module, configured to control the non-player character to move in the virtual environment according to a target entity indicated by the natural language command; The target entity is determined from the virtual environment in combination with environmental perception information, wherein the environmental perception information includes information perceived by the controlling virtual character and at least one of the non-player characters from the virtual environment; The target entity is selected from a plurality of candidate entities based on the environmental perception information of the controlling virtual character or the non-player character, the plurality of candidate entities being entities that match the target entity information in the natural language command; In the case where the natural language command includes a second target entity determined based on the position of the first target entity, the second target entity is obtained by querying the first target entity from the entity set of the virtual environment based on the first target entity information of the first target entity indicated by the natural language command and the environmental perception information; and then obtained by querying from the entity set of the virtual environment based on the second target entity information of the second target entity, the first target entity and the environmental perception information.

27. An entity determination device in a virtual environment, characterized in that: The device comprises: An acquisition module, configured to acquire an entity set, wherein the entity set includes entity information of at least one entity in the virtual environment; The acquisition module is used to acquire environmental perception information, wherein the environmental perception information includes information perceived by at least one of the main virtual character and the non-player character from the virtual environment; a second query module, configured to determine the target entity from the entity set based on the target entity information and the environment perception information of the target entity; The target entity is selected from a plurality of candidate entities based on the environmental perception information of the controlling virtual character or the non-player character, the plurality of candidate entities being entities that match the target entity information in the natural language command, the natural language command being used to command the non-player character; In the case where the natural language command includes a second target entity determined based on the position of the first target entity, the second target entity is obtained by querying the first target entity from the entity set of the virtual environment based on the first target entity information of the first target entity indicated by the natural language command and the environmental perception information; and then obtained by querying from the entity set of the virtual environment based on the second target entity information of the second target entity, the first target entity and the environmental perception information.

28. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the entity determination method in a virtual environment as described in any one of claims 1 to 25.

29. A computer-readable storage medium, characterized in that The readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to implement the entity determination method in a virtual environment as described in any one of claims 1 to 25.

30. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes to implement the entity determination method in a virtual environment as described in any one of claims 1 to 25.

Images