Virtual character control method and device, equipment, medium and program product

Abstract

The invention discloses a virtual character control method and device, equipment, a medium and a program product, and relates to the field of human-computer interaction. The method comprises the following steps: displaying at least one of a master control virtual character and an NPC in a virtual environment; acquiring a natural language command in a voice form; in response to the behavior intention indicated by the natural language command, controlling the NPC to execute a virtual activity; wherein the behavior intention is identified from the natural language command according to the environment perception information, and the environment perception information comprises information perceived by at least one of the master control virtual character and the NPC for the virtual environment. By means of the mode, the natural language command can be better understood by means of the environment perception information, the accuracy of the behavior intention obtained through recognition is improved, the behavior intention is better matched with the virtual environment, and the control accuracy of the NPC is improved. The method and the device can be applied to various scenes such as terminal game scenes, virtual reality scenes and augmented reality scenes.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of human-computer interaction, and in particular to a method, device, equipment, medium and program product for controlling a virtual character. Background Art

[0002] With the improvement of cultural and entertainment living standards, people's life experience and requirements for virtual scenes are getting higher and higher. Games, as a way of expressing virtual scenes, have become a channel for many people to relieve stress. In current game applications, players can control the main virtual character to play virtual games individually or in teams.

[0003] In the related art, in order to increase the fun of the game, when the player controls the main virtual character to participate in a virtual game, the player can select an NPC to cooperate with the main virtual character to play the virtual game together; if the player uses natural language commands in the form of voice to control the NPC, a pre-trained speech recognition system is usually used to perform text analysis on the natural language commands to control the NPC through the text content obtained by analysis.

[0004] However, the speech recognition system is trained with a large amount of text vocabulary, so it is easy to be unable to flexibly distinguish text words with similar pronunciations during the text analysis process, which makes the output text content usually have a broad meaning and low accuracy, which in turn makes the control accuracy of the NPC low, affecting the efficiency of human-computer interaction. Summary of the invention

[0005] The embodiments of the present application provide a method, device, equipment, medium and program product for controlling a virtual character, which can better understand natural language commands with the help of environmental perception information, improve the accuracy of recognized behavioral intentions, make the behavioral intentions more compatible with the virtual environment, and improve the accuracy of controlling the NPC. The technical solution is as follows.

[0006] In one aspect, a method for controlling a virtual character is provided, the method comprising:

[0007] Displaying at least one of a main control virtual character and an NPC in a virtual environment;

[0008] Acquire a natural language command in speech form, wherein the natural language command includes natural semantics for instructing the NPC;

[0009] In response to the behavioral intention indicated by the natural language command, controlling the NPC to perform the virtual activity;

[0010] The behavioral intention is identified from the natural language command based on environmental perception information, and the environmental perception information includes information perceived by at least one of the main virtual character and the NPC about the virtual environment.

[0011] On the other hand, a method for controlling a virtual character is provided, the method comprising:

[0012] Acquire a natural language command in speech form, wherein the natural language command includes natural semantics for instructing the NPC;

[0013] Convert the natural language command into a command analysis result in text form, wherein the command analysis result is a result of converting the natural language command voice into text based on environmental perception information, wherein the environmental perception information includes information perceived by at least one of the main control virtual character and the NPC about the virtual environment;

[0014] Behavior intention recognition is performed on the command analysis result to obtain behavior intention, and the behavior intention is used to control the NPC to perform virtual activities.

[0015] In another aspect, a control device for a virtual character is provided, the device comprising:

[0016] A display module, used to display at least one of a main control virtual character and an NPC in a virtual environment;

[0017] An acquisition module, used for acquiring a natural language command in a voice form, wherein the natural language command includes natural semantics for indicating the NPC;

[0018] A control module is used to control the NPC to perform the virtual activity in response to the behavioral intention indicated by the natural language command; wherein the behavioral intention is identified from the natural language command based on environmental perception information, and the environmental perception information includes information perceived by at least one of the main virtual character and the NPC about the virtual environment.

[0019] In another aspect, a control device for a virtual character is provided, the device comprising:

[0020] An acquisition module, used for acquiring a natural language command in a voice form, wherein the natural language command includes natural semantics for instructing the NPC;

[0021] A conversion module, configured to convert the natural language command into a command analysis result in text form, wherein the command analysis result is a result of converting the natural language command voice into text based on environment perception information, wherein the environment perception information includes information perceived by at least one of the main control virtual character and the NPC about the virtual environment;

[0022] The recognition module is used to perform behavior intention recognition on the command analysis result to obtain the behavior intention, and the behavior intention is used to control the NPC to perform the virtual activity.

[0023] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a method for controlling a virtual character as described in any of the above-mentioned embodiments of the present application.

[0024] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, a code set or an instruction set is stored in the storage medium, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a method for controlling a virtual character as described in any of the above-mentioned embodiments of the present application.

[0025] On the other hand, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the control method of the virtual character described in any of the above embodiments.

[0026] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0027] After obtaining the natural language command in the form of voice for commanding the NPC, the environmental perception information obtained by at least one of the main virtual character and the NPC from the virtual environment is used to identify the behavioral intention from the natural language command, so as to control the NPC to perform virtual activities through the behavioral intention. The environmental perception information of the main virtual character and / or the NPC helps to better understand the natural language command, improve the accuracy of the analysis of the natural language command, and then improve the accuracy of the recognized behavioral intention, so that the behavioral intention is more matched with the virtual environment, greatly avoiding invalid or wrong commands to the NPC, improving the accuracy of the control of the NPC, and providing players with a more efficient and comfortable operating environment, thereby improving the efficiency of human-computer interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1It is a flowchart of using a natural language voice command (natural language instruction) to command an NPC in a virtual environment provided by an exemplary embodiment of the present application;

[0030] Figure 2 is a structural block diagram of a control system provided by an exemplary embodiment of the present application;

[0031] Figure 3 is a flow chart of a method for controlling a virtual character provided by an exemplary embodiment of the present application;

[0032] Figure 4 is a flow chart of a method for controlling a virtual character provided by another exemplary embodiment of the present application;

[0033] Figure 5 is a flowchart of a method for controlling a virtual character provided by another exemplary embodiment of the present application;

[0034] Figure 6 is a schematic diagram of an interface for determining a perception range of an object provided by an exemplary embodiment of the present application;

[0035] Figure 7 is a flow chart of behavioral intention recognition provided by an exemplary embodiment of the present application;

[0036] Figure 8 is a schematic diagram of performing behavioral intention recognition on command analysis results provided by an exemplary embodiment of the present application;

[0037] Fig. 9 is a schematic diagram of a primary candidate tag provided by an exemplary embodiment of the present application;

[0038] Fig.10 is a schematic diagram of a primary candidate tag provided by another exemplary embodiment of the present application;

[0039] Fig.11 is an analysis flow chart of behavior intention recognition provided by an exemplary embodiment of the present application;

[0040] Fig.12 is a schematic diagram of a command recognition result obtained by segmentation provided by an exemplary embodiment of the present application;

[0041] Fig.13 is a schematic diagram of a prediction network training method provided by an exemplary embodiment of the present application;

[0042] Fig.14 is a flow chart of a server obtaining a behavior intention based on natural language command analysis provided by an exemplary embodiment of the present application;

[0043] Fig.15is a technical flow chart of a method for controlling a virtual character provided by an exemplary embodiment of the present application;

[0044] Fig.16 is a schematic diagram of an interface of a method for controlling a virtual character provided by an exemplary embodiment of the present application;

[0045] Fig.17 is a flow chart of a method for controlling a virtual character provided by an exemplary embodiment of the present application;

[0046] Fig.18 is a structural block diagram of a control device for a virtual character provided by an exemplary embodiment of the present application;

[0047] Fig.19 is a structural block diagram of a control device for a virtual character provided by another exemplary embodiment of the present application;

[0048] Fig. 20 It is a schematic diagram of the structure of a terminal provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0050] First, a brief introduction is given to the terms involved in the embodiments of the present application.

[0051] Virtual scene: a virtual scene displayed (or provided) when an application is running on a terminal. The virtual scene can be a simulation of a real scene, a semi-simulation and semi-fictitious scene, or a purely fictitious scene. The virtual scene can be any one of a two-dimensional virtual scene, a 2.5-dimensional virtual scene, and a three-dimensional virtual scene, and this application does not limit this. The following embodiments are illustrated by taking a three-dimensional virtual scene as an example.

[0052] Virtual model: a model used to simulate a real scene in a virtual scene. Exemplarily, a virtual model occupies a certain volume in a virtual scene. Exemplarily, virtual models include: terrain models, building models, animal and plant models, virtual prop models, virtual vehicle models, and virtual object models. For example, terrain models include: ground, mountains, water, stones, steps, etc.; building models include: houses, walls, containers, and fixed facilities inside buildings: tables, chairs, cabinets, beds, etc.; animal and plant models include: trees, flowers, birds, etc.; virtual prop models include: virtual attack props, medicine boxes, airdrops, etc.; virtual vehicle models include: cars, ships, helicopters, etc.; virtual object models include: people, animals, cartoon characters, etc.

[0053] Virtual character / virtual object: refers to an active object in a virtual scene. The active object can be a virtual object, a virtual animal, an animated character, etc., such as a person, an animal, a plant, an oil drum, a wall, a stone, etc. displayed in a three-dimensional virtual scene. Optionally, the virtual object is a three-dimensional stereo model created based on animation skeleton technology. Each virtual object has its own shape and volume in a three-dimensional virtual scene and occupies a part of the space in the three-dimensional virtual scene.

[0054] In the related art, in order to improve the fun of the game, when the player controls the main virtual character to participate in the virtual game, the player can choose an NPC to cooperate with the main virtual character to play the virtual game together; if the player uses natural language commands in the form of voice to control the NPC, a pre-trained speech recognition system is usually used to perform text analysis on the natural language commands to control the NPC through the text content obtained by analysis. However, the speech recognition system is trained with a large number of text words, so it is easy to be unable to flexibly distinguish text words with similar pronunciations during the text analysis process, so that the meaning of the output text content is usually relatively broad and the accuracy is low, which in turn makes the control accuracy of the NPC low, affecting the efficiency of human-computer interaction.

[0055] In an embodiment of the present application, a method for controlling a virtual character is introduced, which can better understand natural language commands in the form of voice with the help of environmental perception information in a virtual scene, thereby obtaining a more accurate command analysis result in the form of text, thereby improving the accuracy of the behavioral intention obtained based on the command analysis result, making the behavioral intention more matched with the virtual environment, and improving the accuracy of controlling NPCs in the virtual environment. The method for controlling a virtual character provided in an embodiment of the present application can be applied to virtual scenes such as terminal game scenes, virtual reality scenes, augmented reality scenes, virtual tourism scenes, virtual meeting scenes, virtual shopping scenes, virtual fitness scenes, etc. that require a voice conversion process, and the embodiment of the present application does not limit this.

[0056] In an optional embodiment, the control method of a virtual character is applied to a terminal game as an example for description.

[0057] Schematically, a game application is installed in the terminal. When the game application is running, a virtual environment in which players play virtual games is displayed. The virtual environment includes at least one of the main virtual character and the NPC, and the NPC in the virtual scene can be commanded / controlled by voice control. The terminal obtains the natural language command issued by the player in the form of voice, and the natural language command includes natural semantics for instructing the NPC. When the natural language command is received, environmental perception information is obtained according to the perception of the virtual environment by the main virtual character and / or the NPC, such as the perception of virtual stables, virtual obstacles, etc. around; then, the natural language command is recognized with the assistance of environmental perception information to obtain the behavioral intention, and the NPC is controlled to perform virtual activities through the behavioral intention. Due to the constraints of the environmental perception information, the behavioral intention is more matched with the virtual environment, which helps to improve the recognition accuracy of the behavioral intention, and then more accurately control the NPC to perform virtual activities, and improve the efficiency and accuracy of controlling the NPC.

[0058] In an optional embodiment, the control method of a virtual character is applied to a virtual reality scene as an example for description.

[0059] Schematically, the virtual environment received from the computer device can be transmitted and presented through the VR head-mounted display. The player observes the virtual environment by wearing the VR head-mounted display. The player can also move to other locations in the virtual environment to explore the virtual scene by moving, adjusting changes in body parts, etc. Taking the player as the main virtual character when wearing the VR head-mounted display as an example, there can also be virtual characters and NPCs controlled and / or directed by other players in the virtual environment presented by the VR head-mounted display. Optionally, the player can control the NPC through natural language commands in the form of voice; the VR head-mounted display can receive natural language commands in the form of voice through the deployed microphone array, analyze the natural language commands autonomously or send the natural language commands to the computer device connected to it for analysis, and the natural language commands include natural semantics for instructing the NPC. In the case of receiving natural language commands, environmental perception information is obtained according to the perception of the virtual environment by the main virtual character and / or NPC, such as the perception of the presence of enemy virtual characters, teammate virtual characters, virtual kitchen knives, etc.; then the natural language command recognition is assisted by environmental perception information to obtain behavioral intentions, and the NPC is controlled to perform virtual activities through behavioral intentions. Due to the constraints of environmental perception information, behavioral intentions are more closely matched with the virtual environment, which helps to improve the accuracy of behavioral intention recognition, and then more accurately control NPCs to perform virtual activities, enriching the fun of virtual reality scenes and enabling players to get a more realistic and in-depth virtual experience.

[0060] It should be noted that before collecting relevant data of users and during the process of collecting relevant data of users, this application can display a prompt interface, pop-up window or output voice prompt information, and the prompt interface, pop-up window or voice prompt information is used to prompt the user that its relevant data is currently being collected, so that this application only starts to execute the relevant steps of obtaining relevant data of users after obtaining the confirmation operation issued by the user to the prompt interface or pop-up window, otherwise (that is, when the confirmation operation issued by the user to the prompt interface or pop-up window is not obtained), the relevant steps of obtaining relevant data of users are terminated, that is, the relevant data of users are not obtained. In other words, all user data collected by this application are collected with the consent and authorization of the user, and the collection, use and processing of relevant user data need to comply with the relevant laws, regulations and standards of the relevant regions.

[0061] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant regions. For example, the natural language commands and other contents involved in this application are all obtained with full authorization.

[0062] The embodiments of the present application provide a solution for users to use natural language voice commands in the form of natural language speech to command and control NPCs (Non Player Characters, NPCs) in a virtual environment. In this solution, in addition to the main virtual character controlled by the user, the user usually has one or more NPCs as teammates. The user can use a relatively casual human conversation method rather than a mechanical command to command the NPC to make the desired feedback to the scene objects in the virtual environment, thereby commanding the NPC to work with the main virtual character to complete the task. Exemplary reference Figure 1 , the user controls the main virtual character 10 to play the game, and the main virtual character 10 has an NPC teammate 20. There is a car 30 in the field of vision of the main virtual character 10. The user speaks a natural voice command in the form of natural language: "No. 2, you go to ambush behind the car 30", and then the NPC teammate 20 will move to the back of the car 30 to ambush. It should be noted that Figure 1 There may be multiple cars in the scene shown, and the NPC teammate 20 will accurately understand that the car the user is talking about is the car 30 located within the user's field of vision. That is, the NPC teammate 20 has a relatively intelligent natural language understanding ability.

[0063] On the one hand, compared with the traditional technology of using mechanical fixed instructions to command NPC, the embodiment of the present application uses relatively random natural language commands to command NPC, which can provide users with more natural, more complex and more flexible language command capabilities.

[0064] On the other hand, the intelligence of the natural language understanding capability provided by the embodiment of the present application is reflected in that when understanding natural language commands, NPC teammates not only consider the literal information of natural language commands. NPC teammates will also combine the main virtual character and / or the NPC's own environmental perception information in the virtual environment to assist in understanding the semantics in natural language commands. That is, NPC not only considers the information of the natural language command mode, but also considers the perception information of other modes such as vision, hearing, radar, etc. to assist in the understanding and execution of natural language commands. Since natural language commands in the form of voice are spoken commands rather than written commands, there may be multiple candidate understanding methods or unclear places when considering the literal information of natural language commands alone. NPC teammates can determine a reasonable understanding method from multiple candidate understanding methods, or eliminate doubts about unclear places, by combining the main virtual character and / or the NPC's own environmental perception information in the virtual environment, thereby achieving a more intelligent natural language understanding capability. The above-mentioned environmental perception information includes, but is not limited to, at least one of the following:

[0065] Visual information perceived within the field of vision of the controlling virtual character;

[0066] Auditory information perceived within the hearing range of the controlling virtual character;

[0067] The information perceived by the master virtual character’s perception skills or perception tools;

[0068] Visual information perceived within the NPC's field of view;

[0069] Auditory information perceived within the NPC's hearing range;

[0070] Information perceived by the NPC's perception skills or perception tools.

[0071] Combined with reference Figure 1 The program includes at least one of the following five stages:

[0072] Phase 1: Preprocessing of spatial data;

[0073] The server will pre-process the scene objects in the virtual scene and construct the spatial data of the virtual scene.

[0074] The spatial data of the virtual scene includes the matching labels of the scene objects, visual labels and the spatial positions of the scene objects. In some examples, the spatial data also includes the appearance images of the scene objects. The scene objects are any objects that appear in the virtual scene, such as Figure 1 Cars, walls, boxes, etc.

[0075] First, the pre-acquisition process of matching label visual labels for scene objects is introduced. First, the attribute information of the scene objects in the virtual scene is obtained, such as at least one of the name, size and other information of the scene objects; and the appearance image of the scene objects is obtained. The appearance image includes an image obtained by observing the scene objects from at least two perspectives. Observing the scene objects from multiple perspectives can ensure that the appearance image carries comprehensive appearance information of the scene objects.

[0076] The server calls the multimodal model to predict the attribute information and appearance image of the scene object. Based on the multimodal model's ability to process text dimension and image dimension information, it extracts hidden features from the attribute information of the text dimension and the appearance image of the image dimension, and decodes the hidden features to predict the visual object label of the scene object. The visual object label is used to describe the scene object in at least one dimension in a textual manner; for example, the scene object is described in dimensions such as material, transparency, color, and shape. The natural language model is called to perform label optimization on the visual object label to obtain the matching label visual label of the scene object; the natural language model has the ability to generate text, and rewrites the visual object label input into the natural language model in a manner that conforms to the spoken expression of natural language, realizes label optimization of the visual object label, and obtains the optimized matching label visual label of the scene object. The optimized visual label conforms to the spoken expression of natural language, and describes the scene object in at least one dimension in a manner close to the spoken expression.

[0077] For example, when describing a virtual bed in a virtual scene in spoken language, people usually pay attention to the color, material, and placement of the virtual bed, but usually neglect to pay attention to the craftsmanship used for the headboard and the side panels of the bed, such as carving, embossing, etc. The purpose of performing pseudo-spoken rewriting on the visual labels of scene objects is to obtain matching labels that are closer to spoken expressions; for example, using multiple words with the same semantics to enrich the labels of dimensions such as color, material, and placement that are paid attention to in spoken expressions, and deleting the labels of the surface feature dimensions of the headboard and the side panels of the bed that are neglected in spoken expressions.

[0078] Next, the spatial position of the scene object is introduced. The spatial position includes at least one of the following of the scene object: coordinate position (Location), such as the coordinate information of the center point or preset point of the scene object in the virtual scene; orientation (Rotation), such as the direction the front of the scene object faces in the virtual scene; bounding box (Bounding Box), used to indicate the size of the scene object in the virtual scene; cover points (Cover points), used to indicate the recommended virtual character position points when the virtual character approaches the scene object, so that the scene object can cover the virtual character.

[0079] Next, the process of pre-acquiring the appearance image of the scene object is introduced. The appearance image is acquired in the process of predicting the matching label visual label of the scene object, and the appearance image includes images obtained by observing the scene object from at least two perspectives.

[0080] Optionally, the spatial data of each scene object in the entire virtual environment may be pre-acquired and organized into a data set for use in a query process in a subsequent stage.

[0081] Phase 2: Speech recognition and behavioral intention recognition;

[0082] When the user controls the main virtual character, the terminal receives the natural language command input by the user in the form of voice; the server performs voice recognition and behavior intention recognition on the natural language command to analyze the user's input instruction;

[0083] Speech recognition is to perform text conversion on the natural language instructions input by the user through voice, obtain the instruction text corresponding to the natural language instructions, and convert the speech information into text information. Call the encoding network to perform feature encoding on the instruction text to obtain the feature representation of the natural language instructions in the latent space; call the text segmentation network to perform segmentation on the feature representation to obtain multiple clauses in the instruction text, and then perform behavioral intention recognition on each of the multiple clauses. In the case that the natural language instructions input by the user through voice are long sentences, the segmentation of the instruction text corresponding to the natural language instructions can be achieved to meet the needs of analyzing the user's input instructions in the long sentence scenario.

[0084] The following introduces the behavioral intention recognition of each clause; the user's input instruction is obtained through behavioral intention recognition, and the user's input instruction includes an entity, a semantic type, a subject type, and an intention type in at least one clause.

[0085] On the one hand, the Conditional Random Field (CRF) method is used to identify entities in clauses, such as identifying entities in clauses as scene objects such as buildings, virtual items, and virtual vegetation in virtual scenes. The entities in the clauses are used to indicate the scene objects targeted by the virtual activities. For example, if the entity in the clause is a virtual building, and the clause carries the semantics of instructing the NPC to launch a virtual attack, then the virtual attack is executed against the virtual building, that is, a virtual attack is launched against the virtual building. On the other hand, the prediction network is called to predict the semantic type, subject type, and intention type in the clause respectively. Among them, the semantic type includes whether to instruct the NPC to launch a virtual attack; the subject type is used to indicate that the subject of the clause is one or more NPCs, or a master virtual character controlled by the user; the intention type is the expected virtual activity indicated by the clause, such as moving, launching a virtual attack, etc.

[0086] Phase 3: Query of spatial data;

[0087] Taking the user's input as input, and the spatial data of the virtual scene and the runtime information corresponding to the master virtual character as reference, the AI ​​model is used to determine the target entity in the virtual space and the control instructions for the NPC;

[0088] For the entity in the input instruction, it is necessary to determine the spatial position of the entity in the virtual environment. For example, the text input instruction is first parsed to parse the entity in the text instruction. In the case where there are multiple candidate entities in the virtual environment that match the entity in the input instruction, the target entity that matches the input instruction can be uniquely determined from the multiple candidate entities in combination with the environmental perception information of the main virtual character and / or NPC.

[0089] For example, the environment perception information is the field of view information of the main virtual character, and then the spatial data of the scene in the virtual environment is used as the query range, and the real-time position and orientation of the main virtual character is used as the query reference condition to query the spatial position of the target entity corresponding to the input instruction in the virtual environment, so as to subsequently control the NPC to move to the vicinity of the target entity position, or to perform the behavior indicated by the input instruction on the target entity position.

[0090] However, since the main virtual character and / or NPC may have multiple environmental perception information, multiple environmental perception information can be integrated to assist the determination process of the above-mentioned target entity, or multiple environmental perception information can be used according to priority to assist the determination process of the above-mentioned target entity, and there is no limitation on this.

[0091] Stage 4: NPC voice feedback;

[0092] NPC will give voice feedback to the user. The types of voice feedback include: at least one of instant feedback, command execution feedback, and dynamic feedback. Among them, instant feedback means that after the user issues a natural language command to the NPC, the NPC immediately gives voice feedback on the natural language command, such as "received", "start execution", "OK", etc.; execution command feedback means that after the NPC executes the control command indicated by the intention of the natural language command, the NPC gives voice feedback on the execution result of the control command, such as "successful execution", "failed to execute successfully", etc.; dynamic feedback means that when the user does not issue a natural language command to the NPC, the NPC gives voice feedback based on real-time environmental perception.

[0093] The above-mentioned voice feedback is obtained by converting text content into speech. The text content is inferred by a large language model combined with the above-mentioned spatial data and dynamic runtime information environment perception information. Using a large language model to dynamically generate the text content of voice feedback can, on the one hand, break through the monotonous, mechanical, and easily repetitive voice feedback of fixed templated voice feedback; on the other hand, it does not need to save too many pre-made template voices in advance, which can avoid the problem of too large a total data volume on the client.

[0094] Stage 5: NPC behavior control;

[0095] Taking the user's input as input, and the spatial data of the virtual scene and the runtime information corresponding to the master virtual character as reference, the AI ​​model is used to determine the target entity in the virtual space and the control instructions for the NPC;

[0096] The above natural language commands usually also include control instructions, which are instructions that can be used for the NPC's behavior tree to execute; the control instruction indicates that the NPC needs to perform a subsequent action, which can be a single action or a sequence of actions. In the case where the control instruction is a multi-action sequence instruction, the multi-action sequence instruction is used to instruct the NPC to perform an action sequence, and the data structure storing the multi-action sequence instruction is in the form of a list, and the instructions to be executed in the sequence instruction are cached. When the NPC's behavior tree executes the current instruction, the cache is called back, and the instructions to be executed in the cache are continued to be sent for execution.

[0097] Hot word update function:

[0098] For phase 2, this solution also introduces a hot word system to help improve the accuracy of sound recognition.

[0099] The hot word system is used to provide scene hot words related to the scene involved in the natural language instruction during the speech recognition process, so that when a natural language instruction input by voice is received, the hot word library can be preferentially searched for words as the speech recognition result instruction analysis result based on the currently running virtual scene, so that the matching degree between the instruction analysis result and the virtual scene is higher.

[0100] The virtual scene includes at least one of multiple scene types, such as at least one of multiple virtual scenes such as virtual battle scene type, virtual transaction scene type, virtual office scene type, virtual kitchen scene type, etc. Frequently used words in multiple virtual scene types, specific virtual elements existing in virtual scene types, etc. (such as virtual buildings, virtual character objects, etc. that only exist in specific scene types) are pre-analyzed as scene hot words, and a scene hot word library composed of multiple scene hot words is obtained, and multiple scene hot words correspond to virtual scene types respectively. In addition, different scene language models are pre-trained for different virtual scene types, such as the scene language model in the battle scene pays more attention to words related to battle, and the scene language model in the transaction scene pays more attention to words related to transaction; the scene language model can perform more targeted analysis on natural language instructions and natural language commands received in virtual scene types.

[0101] For example: in the process of the user controlling the main non-player controlled virtual object character, a natural language instruction is received. Based on the game state, the object position and the current task completion status, it is determined that the first virtual scene currently located by the main control virtual character and / NPC is a virtual battle scene, the target scene type is a battle scene type, and the battle scene language model corresponding to the virtual battle scene type is obtained, and based on the field of view of the main control target virtual object character, the target first scene hot words under the virtual battle scene type are obtained, including: "truck", "virtual grass", "virtual house", "enter", "open truck", "defense", "desert", "attack", "oak tree", "stable", etc., "enemy A", "attack", etc. within the field of view, and the natural language instruction is decoded and processed by a pre-trained natural language analysis model (including an acoustic model network, a target scene language network model, a preset dictionary, etc.) decoding network under the constraint of the first target scene hot words, and the output text content is "give me attack", rather than the result obtained by voice recognition under normal circumstances-"give me supply", which improves the accuracy of the user's input instructions, and outputs the instruction analysis results to accurately control the NPC. The instruction analysis results are illustrated as follows.

[0102] (1) The command analysis result is "No. 1, go to the front and defend." This command analysis result can specifically control No. 1 to move forward and take a defensive posture to prevent the enemy virtual character from attacking the main control virtual character first. If there is no constraint of the hot words in the first scene, the natural language command is easily recognized as "No. 1, go to the front and counterattack and make a sound," thereby affecting the player's virtual combat plan.

[0103] (2) The command analysis result is “No. 2, go explore the desert.” This command analysis result can specifically control No. 2 to move to the desert to explore whether there are virtual elements such as enemy virtual characters or virtual treasure chests. If there is no constraint of the hot words in the first scene, the natural language command is easily recognized as “No. 2, go explore the mountains,” causing No. 2 to move to the wrong location, resulting in low human-computer interaction efficiency.

[0104] (3) The command analysis result is “No. 1, No. 2, attack me”. This command analysis result can be used to specifically control No. 1 and No. 2 to carry out virtual attacks on nearby attackable virtual characters. If there is no constraint of the hot words in the first scene, the natural language command is easily recognized as “No. 1, No. 2, supply me”, which causes No. 1 and No. 2 to make incorrect behaviors that do not meet the player’s expectations, and they can neither protect the main virtual character nor easily allow the enemy virtual character to win the virtual game.

[0105] (4) The command analysis result is “No. 1, go find a nearby oak tree”. This command analysis result can be used to specifically control No. 1 to search for oak trees in the nearby area so that the game task can be completed or an oak tree that can be used to avoid attacks can be found. If there is no constraint of the hot words in the first scene, the natural language command is easily recognized as “No. 1, go find a nearby project book”, which causes No. 1 to search nearby for virtual elements that do not meet the player’s expectations and fail to meet the player’s virtual combat needs.

[0106] (5) The command analysis result is “No. 1, No. 2, go to the stable over there”. This command analysis result can specifically control No. 1 and No. 2 to search for the nearby stable and move to the location of the stable. If there is no constraint of the hot words in the first scene, it is easy to recognize the natural language command as “No. 1, No. 2, go over there, it will be ready soon”, which will cause No. 1 and No. 2 to mistakenly believe that the main virtual character currently hopes to complete the virtual game by themselves, and therefore cannot provide good game assistance to the main virtual character.

[0107] Ambient sound function:

[0108] In addition, the solution also provides a spatial audio enhancement solution for the ambient sound effects of the entire virtual scene.

[0109] The audio played by the terminal includes environmental audio and NPC audio. Environmental audio is audio generated based on the scene characteristics of the virtual scene where the main virtual object is currently located, so that the user can feel immersive; NPC audio is audio generated based on the characteristics of the NPC, so that the user can intuitively feel the character's emotions and physical state through the sound heard.

[0110] Environmental audio: Identify the scene elements in the virtual scene where the master virtual object is located, generate appropriate element sound effects in real time based on the scene elements / select appropriate element sound effects from the sound effect library, and synthesize the element sound effects to obtain environmental audio. For example, if the virtual scene is a forest at night, the scene elements include trees, owls, insects, etc., and the environmental audio includes the rustling of leaves, the calls of owls, the chirping of insects, etc.

[0111] NPC audio: Determine the NPCs included in the virtual scene, and generate corresponding character voices based on the NPC's character type, current emotion, and current behavior. For example, if the NPC is a middle-aged man running, the generated character voice is a deep male voice with a running panting sound effect.

[0112] When the terminal plays the ambient audio and / or NPC audio to the user, the terminal performs audio enhancement processing on the ambient audio and / or NPC audio to improve the realism of the audio.

[0113] In some embodiments, the control system involved in the embodiments of the present application is described. The control method of the virtual character provided in the embodiments of the present application can be implemented by the terminal alone, or by the server, or by the terminal and the server through data interaction, which is not limited in the embodiments of the present application. Optionally, the control method of the virtual character executed interactively by the terminal and the server is described as an example.

[0114] For illustration, please refer to Figure 2 The control system involves a terminal 210 and a server 220 , and the terminal 210 and the server 220 are connected via a communication network 230 .

[0115] In some embodiments, the terminal 210 is installed with a game application. During the operation of the game application, at least one of the main virtual character and the NPC is displayed in the virtual environment. The NPC includes at least one of the multiple virtual characters such as virtual soldiers, virtual heroes, virtual monsters, virtual mounts, etc.; the player can voice control the NPC through voice control.

[0116] Optionally, taking the example of a player directing an NPC by voice, the terminal 210 is deployed with a microphone array, which can obtain natural language commands in the form of voice, and the natural language commands include natural semantics for instructing the NPC.

[0117] In some embodiments, the terminal 210 performs self-analysis based on the natural language command; alternatively, the terminal 210 sends the natural language command to the server 220 via the communication network 230, and the server 220 performs analysis on the natural language command.

[0118] Optionally, the example in which the terminal 210 sends a natural language command to the server 220 and the server 220 analyzes the natural language command is used for description.

[0119] The server 220 determines environmental perception information based on information perceived by at least one of the main virtual character and the NPC about the virtual environment, then identifies behavioral intentions from natural language commands based on the environmental perception information, and sends the behavioral intentions to the terminal 210 via the communication network 230.

[0120] In some embodiments, the terminal 210 controls the NPC to perform a virtual activity in response to the behavioral intention indicated by the natural language command, and the terminal 210 renders and displays the activity animation corresponding to the activity animation data on the screen interface.

[0121] It is worth noting that the above-mentioned terminals include but are not limited to mobile terminals such as mobile phones, tablet computers, portable laptops, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, etc., and can also be implemented as desktop computers, etc.; the above-mentioned servers can be independent physical servers, or they can be server clusters or distributed systems composed of multiple physical servers, or they can also be cloud servers.

[0122] In combination with the above-mentioned noun introduction and application scenarios, the control method of the virtual character provided by the present application is described, and the method is applied to a terminal as an example. Figure 3 As shown, the method includes the following steps 310 to 330.

[0123] Step 310: display at least one of a main virtual character and an NPC in a virtual environment.

[0124] Optionally, the virtual environment is a virtual scene displayed when the application is running on the terminal; or, the virtual environment is a virtual scene displayed by projection. The virtual environment can be a simulation of a real scene, a semi-simulation and semi-fictional scene, or a purely fictional scene. The virtual environment can be any one of a two-dimensional virtual scene, a 2.5-dimensional virtual scene, and a three-dimensional virtual scene.

[0125] Indicatively, the main virtual character is a virtual character manually controlled by the player, such as a virtual hero, a virtual mount, etc. NPC is a virtual character displayed in the game, or, NPC is a virtual character displayed in other virtual environments such as virtual reality scenes, augmented display scenes, etc. NPC has autonomous action capabilities and can move around in the virtual environment on its own.

[0126] In some embodiments, the NPC is a virtual character that can be controlled by the player. When the NPC is not controlled by the player, it moves according to a preset trajectory in the virtual environment, or the NPC attacks the enemy virtual character according to the preset game logic in the virtual environment, assists the main virtual character to perform game tasks, etc.; when the NPC is controlled by the player, the NPC will perform targeted virtual activities in the virtual environment according to the command to more purposefully assist the main virtual character.

[0127] Usually, a virtual game includes a main virtual character and at least one NPC. There are certain differences between the main virtual character and the NPC.

[0128] Optionally, the main virtual character is the virtual character mainly controlled by the player, and the NPC is the virtual character chosen by the player to command; or, the main virtual character is the virtual character selected by the player, and the NPC is the virtual character configured by the system by default; or, the main virtual character and the NPC are both virtual characters selected by the player, among which the main virtual character is the virtual character with the highest priority, and the NPC is the virtual character with a lower priority, etc.

[0129] For example, when the NPC is a virtual character selected by the player, multiple candidate control characters with different personalities and / or autonomous behavior capabilities can be provided. Personality can show the attack strength of the candidate control character in the virtual game (e.g., the attack strength of the candidate control character with a violent personality is greater, and the attack strength of the candidate control character with a gentle personality is smaller, etc.); autonomous behavior capabilities represent the skills of the candidate control character in the virtual game, such as candidate control character 1 can use a large move A to attack the enemy virtual character in the circular area, and candidate control character 2 can use a large move B to defend against the attack of the nearby enemy virtual character, etc. The player can select a character from multiple candidate control characters and use at least one candidate control character as an NPC that can be commanded in the game.

[0130] Step 320, obtaining a natural language command in the form of speech.

[0131] Speech is a form of expression in the form of speech or sound, that is, information or intention is conveyed through speech or sound. In the field of computer science, especially in human-computer interaction and natural language processing, speech usually refers to the way in which users use spoken language to input information.

[0132] Illustratively, natural language commands are usually implemented as voice commands, such as audio data. For example, when a player is operating a terminal, the terminal automatically collects the voice based on a microphone array and obtains audio data as a natural language command in the form of voice; or, when a player is operating a terminal, the terminal triggers an audio collection control and then issues a voice, and the terminal receives the voice and obtains audio data as a natural language command in the form of voice.

[0133] Among them, the natural language command includes natural semantics for instructing the NPC.

[0134] Illustratively, natural semantics is the semantic information expressed by the natural language commands used to control NPCs. Based on natural semantics, NPCs can be controlled to perform virtual activities in a targeted manner.

[0135] Optionally, the natural semantics include identification semantics for indicating the NPC and intention semantics for indicating the NPC activity. Illustratively, the identification semantics include at least one of the NPC's name (such as "Hero A"), the NPC's number (such as "No. 1"), and the NPC's location information (such as "next to the house").

[0136] Optionally, the NPC is a virtual character commanded by the player through voice, including at least one of multiple types of virtual characters such as virtual soldiers, virtual heroes, virtual monsters, virtual mounts, etc. Based on the natural language commands issued by the player in the form of voice during the game, the NPC in the virtual environment can be commanded to act according to the natural language commands.

[0137] In some embodiments, the process of commanding an NPC is a process of unlocking a game account after reaching a preset level, and the game account is the account logged in by the player.

[0138] For example: the preset level is 10. When the game account has not reached the preset level, the NPC is an uncontrollable virtual character, that is, the player cannot command the NPC by voice; when the game account reaches the preset level, the NPC is converted into a player-controllable virtual character, that is, the player can currently command the NPC by voice, etc.

[0139] Step 330 , in response to the behavioral intention indicated by the natural language command, control the NPC to perform a virtual activity.

[0140] In schematic form, behavioral intent is the operational intention or purpose expressed by natural language commands; accurate understanding of behavioral intent will help control NPCs to perform virtual activities in a targeted manner.

[0141] The behavioral intention is identified from the natural language command based on the environmental perception information, and the environmental perception information includes the information perceived by at least one of the main virtual character and the NPC about the virtual environment.

[0142] Illustratively, since the natural language command is voice information received during the process of displaying the virtual environment, when analyzing the natural language command, the virtual environment may be comprehensively considered to perform correlation analysis on the natural language command.

[0143] In some embodiments, environmental perception information is obtained based on information perceived by at least one of the main virtual character and the NPC about the virtual environment.

[0144] In an illustrative manner, environmental perception information is obtained based on information perceived by the controlling virtual character about the virtual environment; or, environmental perception information is obtained based on information perceived by the NPC about the virtual environment; or, environmental perception information is obtained based on information perceived by the controlling virtual character and the NPC together about the virtual environment.

[0145] Optionally, the environmental perception information is used to represent the environmental information perceived and acquired by the controlling virtual character and / or NPC within the object perception range, and the object perception range is the three-dimensional space range in which the controlling virtual character and / or NPC perceives the existence of other scene elements.

[0146] Optionally, the environmental perception information includes virtual elements determined by emitting rays, and may also include regional structure information determined by analyzing the geometric structure and layout information of the virtual environment.

[0147] Illustratively, the virtual environment also includes multiple virtual elements. When determining environmental perception information, the virtual elements perceived by the controlling virtual character and / or NPC in the virtual environment are determined as environmental perception information; or, when determining environmental perception information, the object perception range of the controlling virtual character and / or NPC in the virtual environment is determined, and the virtual elements within the object perception range are used as environmental perception information, etc.

[0148] In some embodiments, intention recognition is performed on natural language commands under the constraints of environmental perception information to obtain behavioral intentions, and the NPC is controlled to perform virtual activities through the behavioral intentions.

[0149] In schematic form, since environmental perception information is information obtained based on the controlling virtual character and / or NPC, this information is not only closely related to the virtual environment, but also closely related to the controlling virtual character and / or NPC in the virtual environment. Constraining the analysis process of natural language commands through environmental perception information helps to refine the acquisition of intention information expressed in natural language commands.

[0150] Optionally, the behavior intention recognition process is performed through a pre-trained machine learning model, the environmental perception information and natural language commands are used as inputs of the machine learning model, and the encoding process and the decoding process are performed through the machine learning model to output the behavior intention.

[0151] Illustratively, the machine learning model includes at least one of a variety of networks such as Recurrent Neural Networks (RNNs), Long Short-Term Memory Networks (LSTM), Gated Recurrent Unit (GRU), Convolutional Neural Networks (CNNs), etc., which are not limited here.

[0152] In summary, after obtaining the natural language command in the form of voice for commanding the NPC, the environmental perception information obtained by the master virtual character and at least one of the NPCs from the virtual environment is used to identify the behavioral intention from the natural language command, so as to control the NPC to perform virtual activities through the behavioral intention. With the help of the environmental perception information of the master virtual character and / or the NPC, it is helpful to better understand the natural language command, improve the accuracy of the analysis of the natural language command, and then improve the accuracy of the recognized behavioral intention, so that the behavioral intention is more matched with the virtual environment, which greatly avoids invalid or wrong commands to the NPC, improves the accuracy of the control of the NPC, and can also provide players with a more efficient and comfortable operating environment, improving the efficiency of human-computer interaction.

[0153] In an optional embodiment, a method for controlling a virtual character executed by a terminal is used as an example for explanation. The terminal performs speech-to-text processing on a natural language command to obtain a command analysis result in text form, and the command analysis result can be displayed on a terminal interface; the behavior intention is recognized through the command analysis result in text form to obtain the behavior intention and control the NPC to perform a virtual activity. Schematically, Figure 4 As shown above Figure 3 The illustrated step 330 may also be implemented as the following steps 410 to 420 .

[0154] Step 410: convert the natural language command into a command analysis result in text form.

[0155] Illustratively, the terminal receives a natural language command in voice form, and first converts the natural language command in voice form into a command analysis result in text form through voice-to-text technology.

[0156] Among them, the command analysis result is the result of converting natural language command speech into text based on environmental perception information.

[0157] Illustratively, in the process of voice-to-text conversion of natural language commands, the environmental perception information of the main control virtual character and / or NPC is used to constrain the natural language commands, thereby converting the natural language commands into text form, that is, obtaining the command analysis results.

[0158] Among them, the command analysis results are used to obtain behavioral intentions through behavioral intention recognition.

[0159] Schematically, the terminal performs a behavioral intention recognition process based on command analysis results in text form; the text form is more intuitive and clear than the voice form, which facilitates the accuracy of behavioral intention recognition.

[0160] Among them, behavioral intention is the intention information expressed in the command analysis result. For example, if the command analysis result is "No. 1, move to A", the behavioral intention is determined to be "No. 1" and "move to A" based on behavioral intention recognition; that is, through behavioral intention recognition, the intention information for controlling the NPC to perform virtual activities is obtained from the command analysis result.

[0161] In an optional embodiment, if Figure 5 As shown, the above step 410 can be implemented as the following steps 411 to 412.

[0162] Step 411, based on the first virtual scene where the main control virtual character and / or the NPC is located, obtain multiple first scene hot words.

[0163] Among them, the plurality of first scene hot words are scene-related words of the first virtual scene, that is, there is an association relationship between the first scene hot words and the first virtual scene, and the first scene hot words are words that describe the first virtual scene.

[0164] Optionally, the first scene hot words include the scene status of the first virtual scene, such as: the first virtual scene is a virtual restaurant, and the first scene hot words include business status, rest status, closed status, etc.

[0165] Optionally, the first scene hot words include element names of virtual elements in the first virtual scene, such as: the first virtual scene is a virtual restaurant, and the first scene hot words include virtual tables, virtual chairs, virtual kitchen utensils, virtual plates, etc.

[0166] Optionally, the first scene hot words include interactive vocabulary for interacting with virtual elements in the first virtual scene, such as: the first virtual scene is a virtual battlefield, and the first scene hot words include attack, attack virtual character, defense, building barriers, etc.

[0167] In some embodiments, after receiving a natural language command, multiple first scene hot words are collected based on the first virtual scene; or, after receiving a natural language command, multiple first scene hot words corresponding to the first virtual scene are filtered from multiple scene hot words pre-acquired based on the first virtual scene, etc.

[0168] In an optional embodiment, a first scene type corresponding to a first virtual scene where the main control virtual character and / or the NPC is located is obtained.

[0169] Illustratively, the first scene type is the scene type corresponding to the first virtual scene, and the scene type is used to characterize the scene state of the virtual scene. For example, the scene types include office type, battle type, kitchen type, bedroom type, classroom type, store type, factory type and other types that describe the scene state.

[0170] Optionally, the first virtual scene corresponds to a first scene type, and the first scene type is used to describe the scene state represented by the first virtual scene. For example, if the first virtual scene is a virtual office, the first scene type is an office type; or, if the first virtual scene is a virtual battle area, the first scene type is a battle type; or, if the first virtual scene is a virtual store, the first scene type is a store type, etc.

[0171] In an optional embodiment, a hot word set corresponding to the first scene type is obtained, and the words in the hot word set are used as multiple first scene hot words. The hot word set is a set of scene-related words collected based on the first scene type.

[0172] Schematically, multiple scene types correspond to a hot word set respectively, and the hot word set represents a set of scene-related vocabulary collected based on the scene type, and also represents a set of scene-related vocabulary collected based on at least one virtual scene under the scene type.

[0173] For example: the office type corresponds to hot word set 1, and the hot word set 1 includes multiple scene-related words collected based on at least one virtual office scene under the office type. For example, the hot word set 1 includes scene-related words collected based on virtual office 1, and includes scene-related words collected based on virtual office 2. Virtual office 1 and virtual office 2 are two virtual offices under the office type.

[0174] For example: the battle type corresponds to hot word set 2, and hot word set 2 includes multiple scene-related words collected based on at least one virtual battle scene under the battle type. For example, hot word set 2 includes scene-related words collected based on virtual battle scene 1, and includes scene-related words collected based on virtual battle scene 2. Virtual battle scene 1 and virtual battle scene 2 are two virtual battle scenes under the battle type.

[0175] Optionally, after determining the first scene type corresponding to the first virtual scene, obtain a hot word set corresponding to the first scene type. The hot word set is a set of scene-related words collected based on the first scene type, that is, the hot word set corresponding to the first scene type is a set of scene-related words collected based on the first scene type. For example, the first scene type of the first virtual scene is a battle type, and the hot word set 2 corresponding to the battle type is used as the hot word set corresponding to the first scene type.

[0176] In some embodiments, the words in the hot word set are called scene-related words, or hot word words, or words. After obtaining the hot word set corresponding to the first scene type, the words in the hot word set are used as multiple first scene hot words, that is, when multiple first scene hot words are obtained based on the first virtual scene, the multiple words in the hot word set corresponding to the first scene type to which the first virtual scene belongs are used as multiple first scene hot words corresponding to the first virtual scene.

[0177] Schematically, the first virtual scene is virtual kitchen 1, and the scene type corresponding to virtual kitchen 1 is determined to be kitchen type. A hot word set corresponding to the kitchen type is obtained and the words therein are used as multiple first scene hot words; the hot word set corresponding to the kitchen type includes both words obtained based on virtual kitchen 1 and words obtained based on virtual kitchen 2, so the multiple first scene hot words obtained in this way have a wider range of characteristics.

[0178] In an optional embodiment, the object perception range of the main virtual character and / or the NPC in the first virtual scene is obtained.

[0179] Optionally, the object perception range of the main control virtual character in the first virtual scene is obtained; or, the object perception range of the NPC in the first virtual scene is obtained; or, the object perception range of the main control virtual character in the first virtual scene is obtained and the object perception range of the NPC in the first virtual scene is obtained.

[0180] The object perception range is the three-dimensional space range in which the main virtual character and / or NPC perceives the existence of other scene elements.

[0181] Illustratively, the main virtual character and the NPC are virtual characters in the first virtual scene, and can be regarded as a virtual element in the first virtual scene. The first virtual scene may also include other scene elements besides the main virtual character and the NPC; the object perception of the main virtual character and / or the NPC represents the perception of the main virtual character and / or the NPC of other scene elements, and the object perception range is the range of object perception, which is implemented as a three-dimensional space range in the first virtual scene.

[0182] In some embodiments, the object perception range includes at least one of the following.

[0183] (1) The visual perception range of the main virtual character and / or NPC.

[0184] It is usually used to describe the size of the area that an organism or technical device can visually cover or perceive. The visual perception range of the controlling virtual character and / or NPC is the three-dimensional space range perceived by the controlling virtual character and / or NPC as the observer.

[0185] Optionally, the visual perception range is acquired based on the position and visual observation ability of the main virtual character and / or NPC in the first virtual scene.

[0186] Schematically, the position is used to indicate the position information of the controlling virtual character and / or NPC relative to the first virtual scene, such as expressed by position coordinates; the visual observation ability is used to indicate the ability of the controlling virtual character and / or NPC to observe the first virtual scene, such as visual observation ability including visual distance (i.e., the farthest distance that can be observed and perceived) and visual angle (i.e., the angle range that can be observed and perceived), etc. Combining the position and visual observation ability, the visual perception range of the controlling virtual character and / or NPC can be roughly determined from the first virtual scene.

[0187] In some embodiments, when the object perception range includes a visual perception range, a preset visual cone is generated as the visual perception range with the position of the controlling virtual character and / or NPC as the starting point and the direction of the controlling virtual character and / or NPC as the center line of the cone.

[0188] Among them, the preset visual cone is a three-dimensional area that measures the range of visual perception.

[0189] Illustratively, when the object perception range includes the visual perception range, the position of the controlling virtual character and / or NPC and the orientation of the controlling virtual character and / or NPC are determined, with the position as the starting point and the orientation as the cone midline, and the preset visual cone as the visual perception range determined based on the visual perception ability.

[0190] Optionally, the preset viewing frustum is a tetrahedron; or, the preset viewing frustum is a three-dimensional prism (such as the preset viewing frustum is obtained by cutting a small tetrahedron from a large tetrahedron); or, the preset viewing frustum is a cone shape; or, the preset viewing frustum is a polygonal prism shape, etc.

[0191] (2) The auditory perception range of the controlling virtual character and / or NPC.

[0192] In schematic form, the auditory perception range refers to the spatial range that a listener can perceive or understand, and is usually used to describe the range of sounds that an organism or technical device can cover or perceive. The auditory perception range of the controlling virtual character and / or NPC is the three-dimensional spatial range that the controlling virtual character and / or NPC can perceive through the auditory perception ability of the controlling virtual character and / or NPC, with the controlling virtual character and / or NPC as the listener.

[0193] Optionally, the auditory perception range is acquired based on the position and auditory perception ability of the main virtual character and / or NPC in the first virtual scene.

[0194] In schematic form, the position is used to indicate the position information of the controlling virtual character and / or NPC relative to the first virtual scene, such as by position coordinates; the auditory perception ability is used to indicate the ability of the controlling virtual character and / or NPC to perceive the sound changes in the first virtual scene, such as auditory perception ability including auditory distance (i.e., the farthest distance of the sound that can be heard), auditory angle (i.e., the direction of the sound source can be determined or the coverage angle range of the sound heard), frequency range (i.e., the frequency range of the sound that can be perceived), etc. Combining the position and auditory perception ability, the auditory perception range of the controlling virtual character and / or NPC can be roughly determined from the first virtual scene.

[0195] (3) The olfactory perception range of the main virtual character and / or NPC.

[0196] In schematic form, the olfactory perception range refers to the spatial range of the odor perceived or recognized by the sense of smell. The olfactory perception range of the controlling virtual character and / or NPC is the three-dimensional spatial range that the controlling virtual character and / or NPC can perceive by the sense of smell.

[0197] Optionally, the olfactory perception range is acquired based on the position and olfactory perception ability of the main control virtual character and / or NPC in the first virtual scene.

[0198] Schematically, the position is used to indicate the position information of the controlling virtual character and / or NPC relative to the first virtual scene, such as by position coordinates; the olfactory perception range is used to indicate the ability of the controlling virtual character and / or NPC to perceive the change of odor in the first virtual scene, such as the olfactory perception range includes olfactory sensitivity (i.e., the ability to react to odor molecules to different degrees), olfactory perception distance (indirectly characterizing the source of the odor, for example, the controlling virtual character and / or NPC), etc. Combining the position and olfactory perception ability, the olfactory perception range of the controlling virtual character and / or NPC can be roughly determined from the first virtual scene.

[0199] (4) The range of perception of the control virtual character and / or NPC’s perception skills or perception tools.

[0200] Illustratively, the perception skill is the ability of the main virtual character and / or NPC to perceive the first virtual scene, and the perception skill includes at least one of a perception acquisition form and a perception enhancement form.

[0201] Optionally, the perception acquisition form is used to indicate the process of acquiring a perception skill that is not possessed. After possessing the perception skill through the perception acquisition form, the main virtual character and / or NPC can perform targeted scene perception of the first virtual scene through the perception skill. The perception enhancement form is used to indicate the process of enhancing a certain perception ability by possessing the perception skill when possessing the perception ability.

[0202] Illustratively, the sensing prop is a virtual prop used to perceive the first virtual scene, and the sensing prop includes at least one of a sensing acquisition form and a sensing enhancement form; for example, the sensing prop in the sensing acquisition form is a first sensing prop, and the sensing prop in the sensing enhancement form is a second sensing prop.

[0203] Optionally, the main virtual character and / or NPC can obtain a sensing tool based on the process of completing a specific game task or reaching a certain game level. Optionally, different sensing skills or sensing tools each correspond to a preset effective range, representing the range that can be sensed when the sensing skill or sensing tool is applied, and the preset effective range represents the object's sensing range.

[0204] In some embodiments, the object perception range is determined based on a first position of the master virtual character in the first virtual scene.

[0205] In some embodiments, the object perception range is determined based on a second position of the NPC in the first virtual scene.

[0206] In some embodiments, when the main virtual character and the NPC are in the same first virtual scene, the first object perception range of the main virtual character in the first virtual scene is obtained, and the second object perception range of the NPC in the first virtual scene is obtained, and the first object perception range and the second object perception range are used together as the object perception range.

[0207] It is worth noting that the above object perception range is only an illustrative example and is not limited to this in the embodiments of the present application.

[0208] In an optional embodiment, a plurality of first scene hot words are acquired based on environmental perception information within a perception range of the object.

[0209] Schematically, the environmental perception information is used to represent the environmental information perceived and acquired by the main virtual character and / or NPC within the object perception range. Optionally, the environmental perception information includes virtual elements determined by emitting rays, and may also include regional structure information determined by analyzing the geometric structure and layout information of the virtual environment.

[0210] Illustratively, after determining the object perception range, based on the range of the object perception range in the first virtual scene, and the environmental perception information determined based on perception of the main virtual character and / or NPC within the object perception range, multiple first scene hot words representing the state of the main virtual character and / or NPC are obtained.

[0211] In an optional embodiment, the environmental perception information includes virtual elements; and the virtual elements in the first virtual scene that are within the perception range of the object are determined.

[0212] Schematically, the object perception range is a part of the three-dimensional space range in the first virtual scene; the first virtual scene includes a large number of virtual elements, and the virtual elements are elements that constitute the virtual scene, such as virtual elements including virtual ground, virtual buildings, virtual trees, virtual characters, virtual stones, etc.

[0213] Optionally, when determining the object perception range, the virtual elements within the object perception range are determined from a plurality of virtual elements corresponding to the first virtual scene.

[0214] Indicative, such as Figure 6 As shown, taking the visual perception range as the object perception range as an example, if the first virtual scene in which the master virtual character is located is displayed from the first-person perspective, then based on the position and orientation of the master virtual object, it can be determined as follows: Figure 6 The frustum shown is the range of visual perception.

[0215] Optionally, the position and orientation of the master virtual object are represented based on the position and orientation of the virtual camera 610; taking the position of the virtual camera 610 as the starting point and the orientation of the virtual camera as the cone midline, the following is obtained: Figure 6 The tetrahedron frustum shown serves as the object perception range.

[0216] Alternatively, the two limiting planes in the preset camera rendering are Figure 6 The near clipping plane 621 and the far clipping plane 622 are parallel to the XY plane of the virtual camera, and are separated by a certain distance along the center line. Any virtual element closer to the virtual camera than the near clipping plane 621 and any virtual element farther from the virtual camera than the far clipping plane 622 will not be rendered. Figure 6The small tetrahedron 632 is truncated from the large tetrahedron 631, and the obtained three-dimensional prism is used as a viewing cone.

[0217] Schematically, the distance between the far clipping plane 622 and the virtual camera 610 is the farthest distance observed by the main virtual object, and the viewing cone is used as the object perception range. The object perception range includes virtual grass, virtual buildings, virtual cars and other virtual elements in the three-dimensional scene, that is, virtual cars, virtual grass, virtual buildings and other virtual elements are virtual elements within the object perception range. Figure 6 As shown, multiple virtual elements are presented by projecting them onto the far clipping plane 622 as an example. In fact, the multiple virtual elements may be displayed dispersedly at multiple positions within the visual perception range (such as within the three-dimensional prism surrounded by the near clipping plane 621 and the far clipping plane 622).

[0218] Optionally, if the visual perception range as the object perception range is determined from a third-person perspective, the virtual camera is located in an area outside the main virtual character, and can not only observe multiple virtual elements, but also observe the main virtual character active in the virtual environment; if the visual perception range is determined at this time, the range captured by the virtual camera can be used as the visual perception range (wherein the virtual camera moves with the main virtual character, and the visual perception range is still related to the main virtual character); or, the visual cone determined by the position and orientation of the main virtual character is used as the visual perception range (refer to Figure 6 As shown, taking the case where the viewing angle of the main virtual character is consistent with the viewing angle of the virtual camera 610 as an example), the process of determining the visual perception range is not limited here.

[0219] In some embodiments, virtual elements in the first virtual scene that are within the perception range of the object are highlighted.

[0220] Optionally, after the virtual elements within the object perception range are removed from the first virtual scene, the virtual elements within the object perception range are highlighted, so that different virtual elements inside and outside the object perception range are presented distinctively, which also helps to prompt the player with the situation of virtual elements related to the analysis of natural language commands.

[0221] In an optional embodiment, the element name of the virtual element is used as the first scene hot word; or, the action name of the interactive action corresponding to the virtual element is used as the first scene hot word.

[0222] Schematically, a virtual element corresponds to an element name, and the element name is used to represent the virtual element itself. The element name can be one or more, that is, multiple element names can represent and indicate virtual elements, etc. For example, the element name of the virtual element 621 is truck or car; the element name of the virtual element 622 is grass or bush, etc.

[0223] Optionally, after determining the virtual elements within the object perception range, the element names of the virtual elements are summarized and the multiple element names are used as first scene hot words corresponding to the first virtual scene to obtain multiple first scene hot words.

[0224] Indicatively, during the game, the player can command the main virtual character or NPC to interact with the virtual elements, that is, there are interactive actions to achieve the interactive process, and the action name of the interactive action is used as the first scene hot word.

[0225] For example: the element name of the virtual element 621 is truck or car, and the player can control the main virtual character or command the NPC to interact with the virtual element 621, such as riding, getting off, hitting the truck, climbing onto the truck, etc., which are the action names of the interactive actions; or, the element name of the virtual element 622 is grass or bushes, and the player can control the main virtual character or command the NPC to interact with the virtual element 622, such as hiding in the bushes, trimming the bushes, stepping on the bushes, etc., which are the action names of the interactive actions.

[0226] Optionally, action names that may generate interactive actions with virtual elements are summarized, and at least one action name corresponding to multiple virtual elements is used as a first scene hot word corresponding to the first virtual scene to obtain multiple first scene hot words.

[0227] In some embodiments, the first scene hot word corresponding to the virtual element is highlighted next to the virtual element.

[0228] In an illustrative manner, after determining the first scene hot words corresponding to the multiple virtual elements, the first scene hot words associated with the virtual elements may be displayed around the multiple virtual elements in the interface. For example, the text word "stable" is displayed around the virtual element "stable", and the text word "attack him" is displayed around the virtual element "enemy virtual character".

[0229] In an optional embodiment, a first scene type corresponding to the first virtual scene is obtained; and a hot word set corresponding to the first scene type is obtained.

[0230] Illustratively, the first scene type is the scene type corresponding to the first virtual scene, and the scene type is used to characterize the scene state of the virtual scene. For example, the scene types include office type, battle type, kitchen type, bedroom type, classroom type, store type, factory type and other types that describe the scene state.

[0231] Schematically, multiple scene types correspond to a hot word set respectively, and the hot word set represents a set of scene-related vocabulary collected based on the scene type, and also represents a set of scene-related vocabulary collected based on at least one virtual scene under the scene type.

[0232] Optionally, after determining the first scene type corresponding to the first virtual scene, obtain a hot word set corresponding to the first scene type. The hot word set is a set of scene-related vocabulary collected based on the first scene type. For example, if the first scene type of the first virtual scene is a battle type, the hot word set 2 corresponding to the battle type is used as the hot word set corresponding to the first scene type.

[0233] In an optional embodiment, the object perception range of the main virtual character and / or the NPC in the first virtual scene is obtained.

[0234] In an exemplary embodiment, the controlling virtual character and / or NPC includes at least one of the controlling virtual character and the NPC. When obtaining the object perception range based on the controlling virtual character and / or the NPC, when the controlling virtual character and / or the NPC is the controlling virtual character, the object perception range of the controlling virtual character in the first virtual scene is obtained; when the controlling virtual character and / or the NPC is the NPC, the object perception range of the NPC in the first virtual scene is obtained; or, when the controlling virtual character and / or the NPC is the controlling virtual character and the NPC, the object perception range of the controlling virtual character in the first virtual scene is obtained and the object perception range of the NPC in the first virtual scene is obtained.

[0235] Optionally, the object perception range includes at least one of the visual perception range, auditory perception range, olfactory perception range of the controlling virtual character and / or NPC, and the range perceived by the perception skills or perception tools possessed by the controlling virtual character and / or NPC.

[0236] In an optional embodiment, at least two words are obtained from the hot word set as a plurality of first scene hot words based on the environmental perception information within the object perception range.

[0237] Optionally, after determining the object perception range, a virtual element within the object perception range is determined, and words that have an element association relationship with the virtual element are selected from the hot word set as a plurality of first scene hot words.

[0238] Schematically, the element association relationship is used to characterize the relationship that reflects the state of the virtual element. For example, the vocabulary including the element name of the virtual element is obtained from the hot word set as the first scene hot word; or, the vocabulary used to trigger the virtual element is obtained from the hot word set as the first scene hot word; or, the interactive vocabulary used to interact with the virtual element is obtained from the hot word set as the first scene hot word, etc.

[0239] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0240] In an optional embodiment, based on a first virtual game in which a main control virtual character and / or an NPC participates, a first virtual scene corresponding to the first virtual game is determined.

[0241] Among them, the multiple virtual games correspond to a virtual scene respectively, and the multiple virtual games are simulated battle environments provided for the main control virtual character and / or NPC.

[0242] Indicatively, in addition to determining the first virtual scene based on the position of the main virtual character and / or NPC in the virtual environment, the virtual scene corresponding to the first virtual game can also be used as the first virtual scene according to the first virtual game in which the main virtual character and / or NPC participates. For example: the virtual environment includes multiple virtual games, each virtual game corresponds to a virtual scene (such as virtual game 1 is a virtual snow scene, virtual game 2 is a virtual forest scene, etc.), if the virtual game in which the main virtual character and / or NPC participates is the first virtual game, the virtual scene corresponding to the first virtual game is used as the first virtual scene (if the main virtual character participates in virtual game 1 with the enemy virtual character in the virtual environment, the virtual scene is used as the first virtual scene, etc.); or, before the game starts, multiple virtual games are provided, each virtual game corresponds to a virtual scene, if the player chooses the main virtual character to participate in the first virtual game, the NPC accompanies the main virtual character to participate in the first virtual game, and the virtual scene corresponding to the first virtual game is used as the first virtual scene, etc.

[0243] Optionally, the virtual game includes game tasks (such as task 1 corresponds to virtual scene 1, task 2 corresponds to virtual scene 2, etc.), game levels (such as game level 1 corresponds to virtual scene 1, game level 2 corresponds to virtual scene 2, etc.), etc., which are not limited here.

[0244] In an optional embodiment, multiple scene hot words are obtained, and the multiple scene hot words correspond to at least two virtual scenes.

[0245] Illustratively, scene hot words corresponding to multiple virtual scenes in the virtual environment are obtained in advance. Optionally, based on the virtual elements in each virtual scene, at least one scene hot word corresponding to the multiple virtual scenes is obtained, and multiple scene hot words are stored.

[0246] Optionally, the element name of the virtual element in the virtual scene is obtained as the scene hot word; or, the action name of the interactive action interacting with the virtual element in the virtual scene is obtained as the scene hot word; or, the state name used to describe the element state of the virtual element in the virtual scene is obtained as the scene hot word, etc.

[0247] In an optional embodiment, based on the first virtual scene where the main virtual character and / or the NPC is located, a plurality of first scene hot words corresponding to the first virtual scene are obtained from a plurality of scene hot words.

[0248] Optionally, multiple scene hot words correspond to scene identifiers, and the scene identifier is used to indicate the virtual scene when the scene hot words are collected. For example: if scene hot words 1 and 2 are obtained based on virtual scene A, scene identifier a corresponding to virtual scene A is marked for scene hot word 1, and scene identifier a is also marked for scene hot word 2; if scene hot word 3 is obtained based on virtual scene B, scene identifier b corresponding to virtual scene B is marked for scene hot word 3, and so on.

[0249] Optionally, based on the first virtual scene where the main control virtual character and / or the NPC is located, a plurality of candidate scene hot words with a first scene identifier are obtained from a plurality of scene hot words, where the first scene identifier is a scene identifier corresponding to the first virtual scene;

[0250] Illustratively, after determining the first virtual scene in which the main virtual character and / or NPC is located, based on the fact that multiple scene hot words are each labeled with a scene identifier, scene hot words labeled with the first scene identifier can be screened from the multiple scene hot words as candidate scene hot words.

[0251] In some embodiments, based on the object perception range of the main virtual character and / or the NPC in the first virtual scene, at least two candidate scene hot words are selected from a plurality of candidate scene hot words as the first scene hot words.

[0252] Illustratively, after obtaining multiple candidate scene hot words based on the first virtual scene, the object perception range is a partial range in the first virtual scene, so the multiple candidate scene hot words can be further screened based on the object perception range, and at least two candidate scene hot words related to the object perception range are selected as the first scene hot words.

[0253] Optionally, a virtual element within the object's perception range is determined as a perceived virtual element, that is, the perceived virtual element is a virtual element that can be perceived by the main virtual character and / or the NPC in the first virtual scene; the element name of the perceived virtual element is filtered from multiple candidate scene hot words as the first scene hot word; or, the action name of the interactive action corresponding to the perceived virtual element is filtered from multiple candidate scene hot words as the first scene hot word, etc.

[0254] Step 412: converting the natural language command into a command analysis result in text form based on the multiple first scenario hot words.

[0255] In an optional embodiment, the first virtual scene is one of multiple virtual scenes in the virtual environment; the virtual environment is a three-dimensional simulation environment of a virtual game in which the main virtual character and the NPC jointly participate.

[0256] Illustratively, multiple virtual characters participating in a virtual game are in a virtual environment, which includes multiple virtual scenes. The multiple virtual scenes together constitute the virtual environment, for example, the virtual environment includes a virtual kitchen 1, a virtual kitchen 2, a virtual office, and a virtual battle scene.

[0257] Optionally, based on the position of the controlling virtual character and / or NPC in the virtual environment, a first virtual scene where the controlling virtual character and / or NPC is located is determined from a plurality of virtual scenes.

[0258] Optionally, the virtual scene where the controlling virtual character is located is determined from multiple virtual scenes as the first virtual scene; or, the virtual scene where the NPC is located is determined from multiple virtual scenes as the first virtual scene; or, the virtual scene where the controlling virtual character and the NPC are located together is determined from multiple virtual scenes as the first virtual scene.

[0259] In some embodiments, when no command analysis result is obtained, in response to the main virtual character and / or NPC moving from the first virtual scene to the second virtual scene, a plurality of second scene hot words corresponding to the second virtual scene are obtained.

[0260] Schematically, the virtual environment includes multiple virtual scenes, and the second virtual scene is another virtual scene different from the first virtual scene. As virtual characters participating in the virtual game, the position of the main virtual character and / or NPC may change in the virtual environment, such as moving from A to B. If the main virtual character and / or NPC moves from the first virtual scene to the second virtual scene during the process of receiving a natural language command but before obtaining the command analysis result, it is necessary to determine the scene hot words used to constrain the generation of the command analysis result based on the second virtual scene that has been moved to most recently.

[0261] For example: in the process of receiving a natural language command but not yet generating a command analysis result, if the main virtual character and / or NPC moves from the current first virtual scene to the second virtual scene, that is, the position of the main virtual character and / or NPC moves from the first virtual scene to the second virtual scene, then in order to be able to more flexibly analyze the status of the virtual scene after the move, multiple second scene hot words corresponding to the second virtual scene are obtained.

[0262] In some embodiments, a command analysis result is obtained by converting a natural language command into text form based on a plurality of second scenario hot words.

[0263] Illustratively, after acquiring multiple second-scenario hot words, the natural language commands are converted into command analysis results in text form through a pre-trained natural language analysis model with the multiple second-scenario hot words as constraints.

[0264] Optionally, a natural language command and multiple second-scenario hot words are used as inputs of the model, and the instruction feature representation of the natural language command and the hot word feature representation of the multiple second-scenario hot words are extracted through the natural language analysis model. The instruction feature representation and the hot word feature representation are fused to obtain a fused feature representation, and the command analysis result in text form is output based on the fused feature representation.

[0265] In an optional embodiment, a pre-trained natural language analysis model is obtained.

[0266] Among them, the pre-trained natural language analysis model includes an acoustic network, a language network and a preset dictionary, and the preset dictionary includes multiple first-scene hot words, other-scene hot words and common vocabulary.

[0267] Optionally, the natural language analysis model can also be called a natural language analysis system, and the acoustic network, language network and preset dictionary in the natural language analysis model can also be called the acoustic model, language model and preset dictionary in the natural language analysis system. The names of the natural language analysis model and its related network are not limited here.

[0268] Schematically, the acoustic network can also be called an acoustic model, which is used to process input in the form of speech to convert sound signals into potential speech units, such as phonemes or segments. Phonemes are the smallest speech units in speech and are phonemes in language. Phonemes are related to the acoustic features of pronunciation; the cloud is a specific sound unit in the actual speech signal, which describes the pronunciation of sound in actual speech and the acoustic details of pronunciation (such as duration, frequency, sound intensity, etc.). By analyzing natural language commands in the form of speech through the acoustic network, possible sequences of speech units can be identified based on the acoustic features of natural language commands (such as spectrum, tone, speech speed, etc.); the acoustic network can learn the mapping from acoustic features to speech units through machine learning techniques such as deep neural networks (DNN) or recurrent neural networks (RNN).

[0269] Schematically, the language network can also be called a language model, which is used to improve the probability score of the language unit sequence with high probability when understanding text input. The language network predicts the rationality and fluency of a given string of words or speech unit sequences based on the grammatical rules and statistical information of the language; the language network includes n-gram model (n-gram), recurrent neural network language model (RNNLM) or transformer model (Transformer), etc., which are used to model and generate text sequences.

[0270] Schematically, the preset dictionary contains words and their corresponding pronunciation or language feature information that may be encountered in speech recognition or text understanding. It provides a way to map text words to their pronunciation forms or language features; in speech recognition, the preset dictionary helps reduce possible pronunciation ambiguity and improve the recognition accuracy of the system.

[0271] Among them, the preset dictionary includes multiple first-scene hot words, other-scene hot words and common words.

[0272] Illustratively, the first scene hot words are scene hot words corresponding to the first virtual scene, other scene hot words are scene hot words corresponding to other virtual scenes except the first virtual scene, and common words are words frequently used in daily life, such as "we", "I", "here", "there", "big", "small", etc. The preset dictionary is a collection of multiple words collected in advance, including the first scene hot words, other scene hot words, and commonly used common words.

[0273] In an optional embodiment, multiple speech units corresponding to the natural language command are acquired through an acoustic network.

[0274] Among them, phonetic units are the basic building blocks of vocabulary pronunciation.

[0275] In some embodiments, instruction feature representations corresponding to natural language commands are extracted.

[0276] Schematically, the feature extraction process is used to indicate that a continuous sound signal - a natural language command is divided into multiple short time periods, and acoustic features such as spectrum, pitch, and speaking rate are calculated within multiple time periods. Each time period corresponds to an acoustic feature. The command feature representation can represent an acoustic feature sequence obtained by arranging multiple acoustic features according to time periods, and can also represent a feature representation after the fusion of multiple acoustic features, which is not limited here.

[0277] In some embodiments, the instruction signature is analyzed via an acoustic network.

[0278] Schematically, taking the instruction feature representation as an example, the goal of the acoustic network is to map a continuous sequence of acoustic features to a sequence of speech units, such as phonemes or segments. It learns to predict possible sequences of speech units (phonemes or segments) given acoustic features.

[0279] Optionally, the acoustic network outputs one or more possible speech unit sequences, wherein the speech unit sequence includes multiple speech units. Different speech unit sequences may contain different speech units, or may just have different orders of speech units. The multiple speech unit sequences represent the arrangement of speech units that the language network considers to be most likely when understanding the input natural language command.

[0280] In some embodiments, in the process of analyzing instruction feature representations through an acoustic network, a context analysis network is additionally introduced. The context analysis network is used to analyze the context and helps to obtain a more reliable speech unit sequence by considering the temporal dependencies between multiple acoustic features.

[0281] In an optional embodiment, the sequence matching relationship between the plurality of speech units and the selected words in the preset dictionary is analyzed through a language network to obtain the command analysis result in text form.

[0282] Illustratively, the preset dictionary includes multiple words, each word corresponds to at least one speech unit. After obtaining at least one speech unit sequence, the at least one speech unit sequence can be matched with the words in the preset dictionary to analyze at least one word represented by the speech unit sequence.

[0283] Optionally, considering the process of obtaining multiple first scene hot words based on the first virtual scene, the first scene hot words are used to constrain the command analysis results. Therefore, when matching words through the speech unit sequence, at least multiple first scene hot words can be used as selected words that need to be applied in the preset dictionary matching process; in addition, considering that some commonly used general words are usually involved in the speech recognition process, general words can also be used as selected words during vocabulary matching, that is, at least one speech unit sequence is matched with the first scene hot words and general words in the selected words to analyze at least one word represented by the speech unit sequence. At least one word is more likely to include the first scene hot words to avoid interference from other scene hot words. That is, the selected words include at least multiple first scene hot words and general words.

[0284] In an optional embodiment, the selected vocabulary also includes other scene hot words.

[0285] Illustratively, if the selected vocabulary is a first scene hot word and a common vocabulary, at least one speech unit sequence can be matched with the first scene hot word and the common vocabulary; if the selected vocabulary is a first scene hot word, other scene hot words and a common vocabulary, at least one speech unit sequence can be matched with the first scene hot word, other scene hot words and a common vocabulary.

[0286] Among them, the vocabulary in the preset dictionary includes analysis weights, and the first analysis weights of multiple first-scene hot words are higher than the second analysis weights of other scene hot words. The analysis weight refers to the degree of attention paid to the vocabulary when it participates in sequence matching.

[0287] Indicatively, each word in the preset dictionary corresponds to an analysis weight. The larger the analysis weight, the more attention the word will receive when it participates in sequence matching, that is, the easier it will be to be paid attention to. The smaller the analysis weight, the more attention the word will receive when it participates in sequence matching, that is, the harder it will be to be paid attention to. The analysis weight can be used to adjust the degree of attention paid to different words in the sequence matching process, so that words with larger analysis weights are matched with the word sequence first, and at the same time, words with larger analysis weights are more likely to be selected as words that constitute the command analysis results.

[0288] Optionally, multiple first-scene hot words are obtained based on the first virtual scene in which the main virtual character and / or NPC is located. The first-scene hot words can better reflect the situation of the first virtual scene and have a greater probability of being associated with natural language commands. Therefore, the first analysis weight of the first scene hot words is set to be higher than the second analysis weights of other scene hot words, so that when the words participate in sequence matching, they are more likely to pay attention to the first scene hot words, and are more likely to be selected as the words that constitute the command analysis results.

[0289] Optionally, the common vocabulary corresponds to a third analysis weight, the third analysis weight is less than the first analysis weight and the third analysis weight is greater than the second analysis weight; or, the third analysis weight is equal to the first analysis weight and the third analysis weight is greater than the second analysis weight; or, the third analysis weight is less than the first analysis weight and the third analysis weight is equal to the second analysis weight; or, the third analysis weight is less than the first analysis weight and the third analysis weight is less than the second analysis weight, etc. The numerical value of the third analysis weight is not limited here.

[0290] Among them, the first analysis weight, the second analysis weight and the third analysis weight are usually values ​​between 0 and 1. Under the condition that the first analysis weight is greater than the second analysis weight, the specific values ​​of the analysis weights are not limited here.

[0291] In an optional embodiment, the matching relationship between the plurality of speech units and the selected words in the preset dictionary is analyzed to obtain a plurality of candidate word sequences.

[0292] Schematically, the process of analyzing the matching relationship represents the process of analyzing the matching between the selected vocabulary and the speech unit. Schematically, each selected vocabulary corresponds to at least one speech unit, and the conversion probability of the speech unit to the selected vocabulary is determined by analyzing the multiple speech units and the selected vocabulary. The conversion probability represents the probability of matching between the selected vocabulary and the speech unit. The larger the conversion probability, the closer the speech unit is to the selected vocabulary, that is, the greater the possibility that the speech unit may express the selected vocabulary, and the stronger the matching relationship between the selected vocabulary and the speech unit.

[0293] Optionally, a plurality of speech units form at least one speech unit sequence, and based on a matching analysis process between the speech units and the selected vocabulary, at least one candidate vocabulary sequence corresponding to the at least one speech unit sequence is determined, and a plurality of candidate vocabulary sequences are obtained.

[0294] Schematically, after analyzing the natural language command based on the sound network, a speech unit sequence 1 and a speech unit sequence 2 are obtained, wherein the speech unit sequence 1 is composed of speech unit a—speech unit b—speech unit c, and the speech unit sequence 2 is composed of speech unit a—speech unit d—speech unit c, and it can be regarded that the multiple speech units include speech unit a, speech unit b, speech unit c, and speech unit d. After matching and analyzing the multiple speech units and the selected words in the preset dictionary, the candidate vocabulary sequence obtained based on the speech unit sequence 1 includes a candidate vocabulary sequence 11 and a candidate vocabulary sequence 12, wherein the candidate vocabulary sequence 11 is "help me attack this place", and the candidate vocabulary sequence 12 is "help me attack the enemy"; the candidate vocabulary sequence obtained based on the speech unit sequence 2 includes a candidate vocabulary sequence 21 and a candidate vocabulary sequence 22, wherein the candidate vocabulary sequence 21 is "help me supply this place", and the candidate vocabulary sequence 22 is "help me attack the enemy", that is, multiple candidate vocabulary sequences are obtained, including the candidate vocabulary sequence 11, the candidate vocabulary sequence 12, the candidate vocabulary sequence 21, and the candidate vocabulary sequence 22.

[0295] The plurality of candidate vocabulary sequences include at least one of the plurality of selected vocabulary, and the candidate vocabulary sequence is a vocabulary sequence obtained based on a change relationship between a plurality of speech units.

[0296] Illustratively, since the candidate vocabulary sequence is a vocabulary sequence obtained based on selected vocabulary in a preset dictionary, the vocabulary in the candidate vocabulary sequence is the vocabulary in the selected vocabulary; the candidate vocabulary sequence can be a sequence obtained from a single vocabulary, or a sequence obtained from a combination of multiple vocabulary, which is not limited here.

[0297] Since the candidate vocabulary sequence is a sequence obtained after matching analysis between multiple phonetic units and selected vocabulary, the phonetic unit transformation of at least one vocabulary in the candidate vocabulary sequence matches the transformation between multiple phonetic units, that is, the candidate vocabulary sequence is a vocabulary combination sequence predicted based on the changes between multiple phonetic units.

[0298] In an optional embodiment, the sequence semantics of multiple candidate vocabulary sequences are analyzed through a language network, and at least one candidate vocabulary sequence is obtained from the multiple candidate vocabulary sequences as a command analysis result in text form.

[0299] Schematically, sequence semantics is used to represent the semantic information expressed by the candidate vocabulary sequence, which can not only reflect the semantic changes of at least one word in the candidate vocabulary sequence, but also reflect the sentence semantics of the entire candidate vocabulary sequence.

[0300] Schematically, the language network evaluates multiple candidate vocabulary sequences respectively, so as to take at least one candidate vocabulary sequence that is more consistent with the current semantic situation as the command analysis result.

[0301] Optionally, probability evaluation is performed on multiple candidate vocabulary sequences through a language network to obtain prediction probabilities corresponding to the multiple candidate vocabulary sequences respectively; and at least one candidate vocabulary sequence with the largest prediction probability is used as the command analysis result.

[0302] Indicatively, the prediction probability is used to characterize the probability estimation result of the candidate vocabulary sequence being correct under the constraints of multiple speech units. For example, the prediction probability of candidate vocabulary sequence 11 is 0.32, the prediction probability of candidate vocabulary sequence 12 is 0.8, the prediction probability of candidate vocabulary sequence 22 is 0.45; if the candidate vocabulary sequence with the largest prediction probability is used as the command analysis result, the candidate vocabulary sequence 12 - help me attack the enemy is output as the command analysis result.

[0303] In some embodiments, after analyzing the sequence semantics of multiple candidate vocabulary sequences through a language network, probability distributions corresponding to the multiple candidate vocabulary sequences are generated; the multiple probability distributions and multiple speech units are input into a decoding network, and based on the constraints of the multiple speech units, at least one candidate vocabulary sequence is output as a command analysis result.

[0304] Schematically, the probability distribution is used to describe the naturalness between at least one word in a candidate vocabulary sequence; the decoding network (or decoder) is used to combine multiple candidate vocabulary sequences, corresponding probability distributions and multiple language units to select the best candidate vocabulary sequence as the command analysis result.

[0305] Optionally, the decoding network may use beam search or other optimization algorithms to search among multiple candidate vocabulary sequences. Considering the probability distribution of the language network output and the constraints of multiple speech units (phoneme or segment sequences) output by the acoustic network, the beam search can find the most likely candidate vocabulary sequence in the potential sequence space as the command analysis result, thereby improving the accuracy of speech recognition.

[0306] It is worth noting that the decoding network can be configured outside the sound network and the language network, or within the language network; the preset dictionary can be configured within the language network, or outside the language network as an independent neural network layer; and the network structure of the natural language analysis model is not limited here.

[0307] In some embodiments, the language network includes language sub-networks corresponding to at least two virtual scenes respectively.

[0308] Schematically, multiple virtual scenes correspond to a language subnetwork respectively, and different language subnetworks are used to analyze the corresponding virtual scenes in a targeted manner. For example, virtual scene A corresponds to language subnetwork 1, and virtual scene B corresponds to language subnetwork 2. When analysis is required based on virtual scene A, language subnetwork 1 is used for language analysis, and when analysis is required based on virtual scene B, language subnetwork 2 is used for language analysis, and so on.

[0309] Optionally, if the natural language analysis model is called a natural language analysis system, language models corresponding to at least two virtual scenes can be configured, and multiple language models correspond one-to-one to multiple virtual scenes. There is no limitation on the model structure or system architecture here, and the above content is merely a schematic illustration of the analysis process.

[0310] In some embodiments, based on a first virtual scene where the main virtual character and / or the NPC is located, a first language sub-network corresponding to the first virtual scene is determined from at least two language sub-networks.

[0311] Illustratively, the first virtual scene is a virtual scene among at least two virtual scenes. In addition to determining the first scene hot words based on the first virtual scene in which the main virtual character and / or NPC is located, a first language subnetwork corresponding to the first virtual scene is determined from at least two language subnetworks based on the first virtual scene. The first language subnetwork is a subnetwork layer that analyzes a candidate vocabulary sequence based on the first virtual scene.

[0312] In some embodiments, the sequence semantics of multiple candidate vocabulary sequences are analyzed by the first language sub-network, and at least one candidate vocabulary sequence is obtained from the multiple candidate vocabulary sequences as a command analysis result in text form.

[0313] Optionally, different language sub-networks are language sub-networks pre-trained based on corresponding virtual scenes, wherein the first language sub-network is a language sub-network trained based on the first virtual scene.

[0314] In an illustrative example, taking the first language sub-network obtained by training the first virtual scene as an example, the first scene hot words corresponding to the first virtual scene are obtained, and the first scene hot words and common words are used as sample data to train the first language sub-network; or, a plurality of scene description words used to describe the first virtual scene are obtained, and the scene description words are used as sample data to train the first language sub-network, and the scene description words include a plurality of words describing the virtual scene state and describing the virtual elements therein, etc. Other language sub-networks are also trained by corresponding virtual scene-related words (scene hot words, common words, scene description words, etc.), which will not be elaborated here.

[0315] Optionally, after obtaining multiple candidate vocabulary sequences under the constraint of the hot words of the first scene, semantic analysis is performed on the multiple candidate vocabulary sequences respectively through the first language sub-network corresponding to the first virtual scene to obtain prediction probabilities corresponding to the multiple candidate vocabulary sequences, and at least one candidate vocabulary sequence with the largest prediction probability is used as the command analysis result in text form.

[0316] It is worth noting that the above process of determining the corresponding first language sub-network based on the first virtual scene in which the main virtual character and / or NPC is located is only an illustrative example. Multiple virtual scenes can also be configured to correspond to a sound network respectively (such as a sound network is a network trained based on the corresponding virtual scene), and the corresponding first sound sub-network is selected for analysis based on the first virtual scene, or the corresponding first sound sub-network and first language sub-network are selected for analysis based on the first virtual scene. This is not limited here.

[0317] In an optional embodiment, the command analysis results are displayed in text form.

[0318] In an illustrative manner, after performing a speech-to-text process on a natural language command in speech form, a command analysis result in text form is obtained, and the terminal may display the command analysis result on a terminal interface to prompt the player that the terminal has successfully received and analyzed the natural language command.

[0319] Optionally, the command analysis results are displayed in a preset area on the terminal interface, such as in a preset dialog area, which is used to show the simulated dialogue process between the main virtual character and the NPC; or, the command analysis results are displayed in the central area of ​​the terminal interface, such as in the center of the terminal interface with a preset font size and preset color.

[0320] In an optional embodiment, the NPC's reply statement in response to the command analysis result is displayed.

[0321] Among them, the reply statement is a text sentence generated after performing semantic analysis based on the command analysis result. Schematically, after analyzing the command analysis result, the terminal calls the pre-trained semantic analysis model to analyze the command analysis result, and generates a reply statement for replying to the command analysis result. For example: the command analysis result is "No. 1, go to A to have a look", and the reply statement generated for the command analysis result based on the semantic analysis model is "No. 1 received" or "OK", etc. The form of the reply statement is not limited here.

[0322] Optionally, the reply statement is displayed below the command analysis result; or, the reply statement is displayed after the command analysis result is canceled.

[0323] Step 420 , controlling the NPC to perform virtual activities based on the behavioral intention.

[0324] In an illustrative manner, after converting the command analysis results into text form, the command analysis results are subjected to behavioral intention recognition to obtain the behavioral intention, and then the NPC is controlled to execute the virtual activity indicated by the behavioral intention based on the behavioral intention.

[0325] Schematically, the command analysis result is text content in text form. Compared with natural language commands in voice form, the command analysis result is more intuitive and can provide more targeted control over the NPC.

[0326] Optionally, the form of directing NPC activities based on behavioral intentions includes at least one of the following: (1) directing NPCs to move in a virtual environment; (2) directing NPCs to search for designated items; (3) directing NPCs to cooperate with the main virtual character to defend against attacks by enemy virtual characters; (4) directing NPCs to attack enemy virtual characters; (5) directing NPCs to change props, accessories, clothing accessories, etc. The form of activities is not limited here.

[0327] For example, after analyzing the natural language command, the command analysis result is "attack me". Based on behavioral intention recognition, the behavioral intention of the natural language command is determined to be: commanding the NPC to make a virtual attack on a virtual character that can be attacked in a virtual environment. Therefore, based on the behavioral intention, the NPC can be controlled to make a virtual attack on other virtual characters.

[0328] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0329] In summary, after obtaining the natural language command in the form of voice for commanding the NPC, the environmental perception information obtained by the master virtual character and at least one of the NPCs from the virtual environment is used to identify the behavioral intention from the natural language command, so as to control the NPC to perform virtual activities through the behavioral intention. With the help of the environmental perception information of the master virtual character and / or the NPC, it is helpful to better understand the natural language command, improve the accuracy of the analysis of the natural language command, and then improve the accuracy of the recognized behavioral intention, so that the behavioral intention is more matched with the virtual environment, which greatly avoids invalid or wrong commands to the NPC, improves the accuracy of the control of the NPC, and can also provide players with a more efficient and comfortable operating environment, improving the efficiency of human-computer interaction.

[0330] In an embodiment of the present application, it is introduced that the first virtual scene is determined by the position of the main virtual character and / or NPC, and then the object perception range is determined from the first virtual scene based on the position and orientation and the content of the first scene hot words is obtained. If multiple virtual characters are active in a virtual environment including multiple virtual scenes, the first virtual scene can be specifically determined from the virtual environment according to the position of the main virtual character and / or NPC, so that the object perception range is determined in the first virtual scene in combination with the orientation of the main virtual character and / or NPC, which overcomes the problem in the related art that only broad command analysis results can be obtained through the voice recognition system, and the environmental perception information represented by the object perception range constrains the text vocabulary used in the speech-to-text process (such as giving priority to the use of scene hot words), thereby helping to improve the text accuracy of the command analysis results.

[0331] In addition, even if the command analysis results are not generated, the second scene hot words can be obtained based on the process of the main virtual character and / or NPC moving to the second virtual scene to constrain the analysis process of the command analysis results. This not only improves the flexibility of obtaining scene hot words used to constrain the analysis process, but also helps to further improve the flexibility of obtaining command analysis results, making it easier to command the NPC more accurately and flexibly through the command analysis results in text form.

[0332] In the embodiment of the present application, in addition to determining the first scene hot words based on the first virtual scene where the main virtual character and / or NPC is located, a process for determining the corresponding first language sub-network for targeted analysis based on the first virtual scene is also introduced. The corresponding language sub-networks are trained for multiple virtual scenes respectively, so that the language sub-network can perform a more purposeful analysis of the candidate vocabulary sequence under the constraints of the virtual scene, avoiding the blindness problem when obtaining the command analysis results; the process of obtaining the candidate vocabulary sequence through the first scene hot word constraints improves the accuracy of the candidate vocabulary sequence, and then further constrains the analysis process of the candidate vocabulary sequence in combination with the first language sub-network, which helps to further improve the correlation between the command analysis results and the first virtual scene, improve the accuracy of the analysis of natural language commands, and facilitate the improvement of the command accuracy of NPCs through more accurate command analysis results.

[0333] In an optional embodiment, the control method of a virtual character executed by a terminal is used as an example for explanation. The NPC has multiple behavioral capabilities in a virtual scene, and the multiple behavioral capabilities correspond to multiple classification labels. After the terminal obtains the command analysis result in text form based on speech-to-text conversion, it performs behavioral intention recognition on the command analysis result through a hierarchical prediction network to obtain the behavioral intention and control the NPC to perform virtual activities. Schematically, Figure 7 As shown above Figure 4 The illustrated step 420 may also be implemented as the following steps 710 to 720 .

[0334] Step 710, calling at least one hierarchical prediction network to perform behavioral intention recognition on the command analysis result, and obtaining a first classification label of the command analysis result intended to command the NPC as the behavioral intention.

[0335] In a schematic manner, behavioral intention recognition is a recognition process for identifying behavioral intention from command analysis results. Behavioral intention is the intention information expressed in the command analysis results, and behavioral intention includes at least one of subject information, semantic information, and behavioral information. Subject information is used to indicate the identity information of the NPC pointed to by the command analysis results (such as the name and nickname of the NPC included); semantic information is used to indicate the meaning and ideas conveyed by the command analysis results (such as analyzing whether the command analysis results command the NPC to launch a virtual attack, etc.); behavioral information is used to indicate the behavior of the virtual activities that the NPC pointed to by the command analysis results needs to perform (such as analyzing whether the command analysis results command the NPC to move and jump in the virtual environment, etc.).

[0336] The hierarchical prediction network includes at least two layers of sub-networks, and at least two layers of sub-networks are constructed based on a tree structure of multiple classification labels.

[0337] Exemplarily, the hierarchical prediction network has the ability to predict the classification label corresponding to the command analysis result. Exemplarily, the hierarchical prediction network includes at least two layers of sub-networks, the upper network and the lower network are cascaded, and the lower network further performs the prediction of the first classification label according to the prediction result output by the upper network.

[0338] In this embodiment, at least one hierarchical prediction network is called to perform behavioral intention recognition on the fusion features of the command analysis results to obtain a first classification label of the command analysis result intended to command the NPC.

[0339] Figure 8 A schematic diagram of performing behavioral intention recognition on a command analysis result provided by an exemplary embodiment of the present application is shown. An encoding network 460 based on an attention mechanism is called to perform feature encoding on the command analysis result 450 to obtain a first feature vector 451 of the command analysis result in the latent space; the first feature vector 451 can indicate the natural semantics of the characters in the command analysis result 450.

[0340] In this embodiment, the information structure prediction network includes a first fully connected layer 462 and a first activation function 464; wherein the first fully connected layer 462 is used to perform dimension encoding on the first feature vector 451 to obtain a second feature vector 452 in the control dimension; the second feature vector 452 can describe the natural semantics of the command analysis result 450 in the command NPC dimension. Based on the first activation function 464, the second feature vector 452 is activated to obtain a first output 465, which is used to indicate whether the command analysis result 450 carries the natural semantics of the command NPC.

[0341] When the first output 465 is used to indicate that the command analysis result 450 carries the natural semantics of the command NPC, the first bilinear transformation layer 466 is called to perform a bilinear transformation on the first feature vector 451 and the second feature vector 452, and the natural language semantics of the command analysis result 450 in two dimensions are integrated to obtain the third feature vector 453.

[0342] Next, the behavior intention recognition is performed on the command analysis result by calling the hierarchical prediction network, which includes two layers of sub-networks. First, for the high-level first-layer sub-network, the first-layer sub-network includes a second fully connected layer 472 and a second activation function 474; the second fully connected layer 472 encodes the third feature vector 453 from the dimension of the intent feature to obtain a fourth feature vector 454; the intent dimension corresponding to the fourth feature vector 454 includes any one of the subject dimension, the semantic dimension, and the behavior dimension. Based on the second activation function 474, the fourth feature vector 454 is activated to obtain a second output 475, which indicates the first-level behavior label of the command analysis result 450 on an intent dimension.

[0343] The second bilinear transformation layer 476 is called to perform bilinear transformation on the third feature vector 453 and the fourth feature vector 454 , and the natural language semantics of the command analysis result 450 in two dimensions are integrated to obtain the fifth feature vector 455 .

[0344] Next, the second-layer network of the lower layer of the hierarchical prediction network is used to predict the secondary behavior label corresponding to the command analysis result 450 in the primary behavior label; wherein, for the lower-layer second-layer network: the second-layer network includes a third fully connected layer 482 and a third activation function 484; the third fully connected layer 482 encodes the fifth feature vector 455 from the dimension of the intention feature, and performs activation processing on the encoded result based on the second activation function 474 to obtain a third output 485, and the third output 485 indicates the secondary behavior label corresponding to the command analysis result 450 in the primary behavior label.

[0345] In an optional embodiment, the behavioral intention includes at least one intention dimension; each hierarchical prediction network in at least one hierarchical prediction network is used to predict the command intention of the command analysis result for the NPC on an intention dimension; the intention dimension includes: at least one of a subject dimension, a semantic dimension, and a behavioral dimension.

[0346] In one example, at least one hierarchical prediction network includes a subject prediction network, a behavior prediction network, and a semantic prediction network to realize the tactical command understanding of the command analysis results in parallel; the command intention against the NPC is described from three dimensions: subject dimension, semantic dimension, and behavior dimension attack. It can be understood that performing behavioral intention recognition on the command analysis results is also called performing tactical command understanding on the command analysis results; by calling at least one hierarchical prediction network to perform tactical command understanding on the command analysis results, the first classification label of the command analysis result intention to command the NPC is obtained. Exemplarily, natural language has rich semantics, and there are rich semantic elements in the command analysis results in natural language form. Each hierarchical prediction network predicts the command intention in the command analysis results from different dimensions. On the one hand, it reduces the number of label types that need to be classified in the hierarchical prediction network prediction task, and reduces the complexity of the classification prediction task; on the other hand, the hierarchical prediction network only needs to understand the natural language semantics in the command analysis results in one dimension, which reduces the difficulty of extracting the natural language semantics in the command analysis results.

[0347] Exemplarily, in at least one hierarchical prediction network, the i-th layer subnetwork in each hierarchical prediction network is used to predict the first-level behavior label of the command analysis result on an intent dimension, and the i+1-th layer subnetwork in the hierarchical prediction network is used to predict the second-level behavior label corresponding to the command analysis result in the first-level behavior label, where i is a positive integer; the high-level behavior label (such as the first-level behavior label) predicted by the high-level subnetwork (such as the i-th layer subnetwork) in the hierarchical prediction network includes multiple sublabels, or subordinate lower-level labels; the corresponding low-level subnetwork (such as the i+1-th layer subnetwork) needs to be called to further perform classification prediction (such as predicting the second-level behavior label). The prediction task of a wide variety of classification labels is split into prediction subtasks performed by at least two layers of subnetworks to reduce the complexity of classification prediction in each layer of subnetwork.

[0348] In an optional embodiment, behavioral intention includes at least one intention dimension among subject dimension, semantic dimension and behavioral dimension; the subject dimension analysis process is called subject prediction process; the semantic dimension analysis process is called semantic prediction process; the behavioral dimension analysis process is called behavioral prediction process; the prediction process of each dimension is described as follows.

[0349] (1) Subject prediction process

[0350] In some embodiments, the i-th layer sub-network in the subject prediction network is called to perform subject prediction on the command analysis result, and the command analysis result is predicted to correspond to a first-level behavior label among multiple first-level candidate labels.

[0351] In this embodiment, the intention dimension includes a subject dimension, and the hierarchical prediction network includes a hierarchical subject prediction network, which has the ability to predict the subject type in the command analysis result; the subject type is used to indicate the identity of the NPC commanded by the command analysis result, that is, to characterize the subject information in the command analysis result. Exemplarily, the subject prediction network is used to predict which role the NPC commanded by the command analysis result is among a plurality of NPCs, and the number of NPCs commanded by the command analysis result can be one or more.

[0352] Exemplarily, the i-th layer subnetwork is used to predict the first-level behavior label corresponding to the command analysis result from multiple first-level candidate labels. Multiple first-level candidate labels are used to indicate whether there is a subject in the command analysis result; the i-th layer subnetwork is used to perform a binary classification task of whether there is a subject. In an optional example, the i-th layer subnetwork includes an encoding layer for extracting the feature representation of the command analysis result in the subject dimension, and then performing classification prediction on the feature representation of the subject dimension to obtain the first-level behavior label corresponding to the command analysis result.

[0353] In some embodiments, the i+1th layer subnetwork corresponding to the primary behavior label in the subject prediction network is called to perform subject prediction on the command analysis result to obtain a secondary behavior label corresponding to the command analysis result among multiple secondary candidate labels.

[0354] Exemplarily, when the primary behavior label is used to indicate that there is a subject in the command analysis result, multiple secondary candidate labels are used to indicate that the subject of the command analysis result is the main virtual character controlled by the sender of the command analysis result, and / or one or more NPCs in the virtual camp where the main virtual character is located. Multiple secondary candidate labels are used to indicate the identity of the virtual character commanded by the command analysis result. In one example, when the secondary behavior label indicates that the subject of the command analysis result is the main virtual character, although there are natural semantics for controlling the virtual character in the command analysis result, it is not a control command for the NPC, but only the user's soliloquy, or a communication text between multiple users to inform each other of the activities of the main virtual character. When the secondary behavior label indicates that the subject of the command analysis result is one or more NPCs, the secondary behavior label can determine the identity of the NPC that needs to perform virtual activities according to the command analysis result.

[0355] In one example, Fig. 9 A schematic diagram of a first-level candidate label provided by an exemplary embodiment of the present application is shown. The i-th layer subnetwork in the subject prediction network is called to perform subject prediction on the command analysis result, and the command analysis result is predicted to correspond to a first-level behavior label in multiple first-level candidate labels; the multiple first-level candidate labels include a positive label 902 and a negative label 904; the positive label 902 is used to indicate that there is a subject in the command analysis result, and the negative label 904 is used to indicate that there is no subject in the command analysis result. The positive label 902 includes multiple second-level candidate labels: an all-role label 905, a master virtual character label 906, and an NPC label 907; the all-role label 905 is used to indicate that the subject in the command analysis result is all virtual characters in the virtual camp; the master virtual character label 906 is used to indicate that the subject in the command analysis result is a master virtual character controlled by the user; the NPC label 907 is used to indicate that the subject in the command analysis result is an NPC.

[0356] In some embodiments, when the primary behavior label is used to indicate that there is no subject in the command analysis result, the first classification label of the command analysis result is determined to be the first label.

[0357] Exemplarily, when the primary behavior label is used to indicate that there is no subject in the command analysis result, the command analysis result does not clearly indicate the identity of the NPC that needs to perform the virtual activity. The first classification label of the command analysis result is the first label, and the first label is used to indicate that the subject of the command analysis result is a historical subject in a historical command, and the historical command is an adjacent command input before the command analysis result; the historical subject in the previously input historical command is continued, so that continuous control of one or more NPCs can be achieved with reference to the historical subject.

[0358] Optionally, when the first-level behavior label is used to indicate that there is no subject in the command analysis result, and the input time interval between the command analysis result and the historical command does not exceed the duration threshold, the first classification label of the command analysis result is determined to be the first label; wherein, when the input time interval between the command analysis result and the historical command does not exceed the duration threshold, there is sentence continuity between the command analysis result and the historical command, which is a continuous control command for one or more NPCs.

[0359] In some embodiments, when the primary behavior label is used to indicate that there is no subject in the command analysis result, the first classification label of the command analysis result is determined to be the second label.

[0360] Similarly, when the first-level behavior label is used to indicate that there is no subject in the command analysis result, the command analysis result does not clearly indicate the identity of the NPC that needs to perform the virtual activity. The first classification label of the command analysis result is the second label, and the second label is used to indicate that the subject of the command analysis result is all the NPCs in the virtual camp where the controlling virtual character is located. In the oral expression of natural language, when speaking to a group of objects, the subject is usually omitted, and the subject of the command analysis result is determined to be all the NPCs in the virtual camp where the controlling virtual character is located, which is consistent with the oral expression of natural language.

[0361] Optionally, when the first-level behavior label is used to indicate that there is no subject in the command analysis result, and there are no historical commands in the time threshold before the input time of the command analysis result, the first classification label of the command analysis result is determined to be the second label; wherein, since there are no historical commands in the time threshold before the input time of the command analysis result, the command analysis result is isolated command information and not a continuous expression of the previous information, and according to the oral expression habits, the subject of the command analysis result is determined to be all NPCs.

[0362] (2) Semantic prediction process

[0363] In an optional embodiment, the i-th layer sub-network in the semantic prediction network is called to perform semantic prediction on the command analysis result, and the command analysis result is predicted to correspond to a first-level behavior label among multiple first-level candidate labels.

[0364] In this embodiment, the intention dimension includes a semantic dimension, and the hierarchical prediction network includes a semantic prediction network of a hierarchical structure. The semantic prediction network has the ability to predict the semantic type in the command analysis result; the semantic type is used to indicate the control method of the NPC for launching a virtual attack, that is, to characterize the semantic information in the command analysis result. Exemplarily, the subject prediction network is used to predict whether the virtual activity performed by the NPC commanded by the command analysis result is related to launching a virtual attack. It should be noted that the virtual scene provides a space for virtual competition between virtual characters; in the process of virtual competition, it is necessary to maintain the survival of the virtual character. Initiating a virtual attack is highly related to maintaining the survival of the virtual character. For example, when actively launching a virtual attack, you can obtain (or pick up) additional virtual supplies by defeating the enemy virtual character, but you will be counterattacked by the enemy virtual character, and there is a risk of virtual life value reduction; so whether to launch a virtual attack is highly related to maintaining the survival of the virtual character. In this embodiment, for the scene of virtual competition in the virtual scene, the semantic dimension of whether to launch a virtual attack is predicted as a separate dimension.

[0365] Exemplarily, the i-th layer subnetwork is used to predict the first-level behavior label corresponding to the command analysis result from multiple first-level candidate labels. Multiple first-level candidate labels are used to indicate whether there is semantics of launching a virtual attack in the command analysis result; the i-th layer subnetwork is used to perform a binary classification task of whether there is semantics of launching a virtual attack. In an optional example, the i-th layer subnetwork includes an encoding layer for extracting feature representations of the command analysis results in the semantic dimension, and then performing classification prediction on the feature representations of the semantic dimension to obtain the first-level behavior label corresponding to the command analysis result.

[0366] In an optional embodiment, the i+1th layer subnetwork corresponding to the primary behavior label in the semantic prediction network is called to perform semantic prediction on the command analysis result to obtain a secondary behavior label corresponding to the command analysis result among multiple secondary candidate labels.

[0367] Exemplarily, when the first-level behavior label is used to indicate that there is no natural semantics of launching a virtual attack in the command analysis result, multiple second-level candidate labels are used to indicate that the natural semantics in the command analysis result is to avoid launching a virtual attack, or other semantics unrelated to a virtual attack.

[0368] For example, the first-level behavior label is used to indicate that the command analysis result does not indicate the active initiation of a virtual attack, but the way to deal with the virtual attack is predicted by the i+1th layer sub-network. For example, avoiding launching a virtual attack is used to indicate the avoidance and evasion of virtual attacks, intentionally reducing the number of virtual attacks. Other semantics unrelated to virtual attacks are used to indicate a neutral approach to virtual attacks, and do not indicate the active initiation or deliberate avoidance of virtual attacks.

[0369] In one example, Fig. 9 A schematic diagram of a first-level candidate label provided by an exemplary embodiment of the present application is shown. The i-th layer subnetwork in the semantic prediction network is called to perform subject prediction on the command analysis result, and the command analysis result is predicted to correspond to a first-level behavior label in multiple first-level candidate labels; the multiple first-level candidate labels include a positive semantic label 912 and a negative semantic label 914; the positive semantic label 912 is used to indicate that a virtual attack is actively initiated, and the negative semantic label 914 is used to indicate that there is no active virtual attack. Negative semantic label 914 includes multiple second-level candidate labels: stop label 915 and other labels 916; stop label 915 is used to indicate that the virtual attack is avoided and evaded, and the number of virtual attacks is intentionally reduced; other labels 916 are used to indicate that the virtual attack is neutral, and does not indicate that the virtual attack is actively initiated or deliberately avoided.

[0370] (3) Behavior prediction process

[0371] In an optional embodiment, the i-th layer sub-network in the behavior prediction network is called to perform behavior prediction on the command analysis result, and the command analysis result is predicted to correspond to a first-level behavior label among multiple first-level candidate labels.

[0372] In this embodiment, the intention dimension includes a behavior dimension, and the hierarchical prediction network includes a hierarchical behavior prediction network, which has the ability to predict the behavior type in the command analysis result; the behavior type is used to indicate the behavior mode of the NPC for performing virtual activities, that is, to characterize the behavior information in the command analysis result. Exemplarily, the behavior prediction network is used to predict what virtual activity the NPC performs.

[0373] Exemplarily, the i-th layer sub-network is used to predict the first-level behavior label corresponding to the command analysis result from multiple first-level candidate labels. Multiple first-level candidate labels are used to indicate the virtual activity type; the virtual activity type predicted by the i-th layer sub-network includes multiple sub-labels, or subordinate lower-layer labels; it is necessary to call the i+1-th layer sub-network corresponding to the first-level behavior label for further prediction.

[0374] In an optional embodiment, the i+1th layer subnetwork corresponding to the primary behavior label in the behavior prediction network is called to perform behavior prediction on the command analysis result to obtain a secondary behavior label corresponding to the command analysis result among multiple secondary candidate labels.

[0375] Exemplarily, the multiple secondary candidate tags are multiple sub-tags of the primary behavior tag predicted by the i-th layer sub-network, or subordinate lower-layer tags. The multiple secondary candidate tags are used to indicate multiple behavior entries in the virtual activity type corresponding to the primary behavior tag, and the behavior entries are used to indicate the way in which the NPC performs the virtual activity.

[0376] In an optional implementation, the secondary candidate tags are introduced as follows.

[0377] Fig.10 A schematic diagram of first-level candidate tags provided by an exemplary embodiment of the present application is shown. In one example, the i-th layer subnetwork in the behavior prediction network is called to perform subject prediction on the command analysis result, and the command analysis result is predicted to correspond to a first-level behavior tag in multiple first-level candidate tags; the multiple first-level candidate tags include throwing virtual props 921, triggering virtual skills 922, adjusting movement characteristics 923, interacting with virtual items in a virtual scene 924, moving in a virtual scene 925, and answering inquiries 926.

[0378] In the case where the primary behavior label is used to indicate that the type of virtual activity is throwing virtual props, multiple secondary candidate labels are used to describe the types of virtual props that need to be thrown. Exemplarily, taking a shooting game as an example, multiple secondary candidate labels include but are not limited to at least one of virtual explosives label 931, virtual smoke bomb label 932, and virtual flash bomb label 933. The present application does not limit the type of virtual props thrown. Virtual props can be virtual props equipped before entering the virtual scene, or virtual props picked up in the virtual scene. It should be noted that the process of taking out virtual props from the storage space of the NPC virtual character (such as a virtual backpack) and triggering the use of virtual props by giving up holding the virtual props belongs to throwing virtual props, such as taking out the virtual turret in the virtual backpack and arranging the virtual turret in the virtual scene in a way that the virtual turret is no longer held; the corresponding throwing virtual props is also called deploying virtual props.

[0379] And / or, in the case where the primary behavior tag is used to indicate that the type of virtual activity is to trigger a virtual skill, multiple secondary candidate tags are used to describe the type of virtual skill that needs to be triggered; illustratively, taking a MOBA game as an example, multiple secondary candidate tags include at least one of an attack virtual skill tag 934, a transfer virtual skill tag 935, a shield virtual skill tag 936, etc. A virtual skill can be triggered for one or more virtual objects, or it can be triggered for an effective range / effective direction. Exemplarily, an NPC has three virtual skills, such as: skill one, skill two, and skill three (or ultimate skill). The type of virtual skill indicated by the secondary candidate tag can be the name of the virtual skill, or it can be the sequence number of the virtual skill among the three virtual skills possessed by the NPC.

[0380] And / or, in the case where the primary behavior tag is used to indicate that the type of virtual activity is to adjust the motion characteristics, the multiple secondary candidate tags include adjusting the NPC's body posture tag 937 or the moving speed tag 938; illustratively, the body posture includes but is not limited to at least one of jumping, standing, squatting, lying prone, aiming, etc. The moving speed includes but is not limited to at least one of running speed, walking speed, slow walking speed, crawling forward speed, and staggering speed. Adjusting the motion characteristics is used to indicate adjusting the NPC's appearance style (for example, different body postures have different projected observation areas) and / or moving characteristics.

[0381] And / or, in the case where the primary behavior tag is used to indicate that the virtual activity type is to interact with a virtual item in a virtual scene, the plurality of secondary candidate tags include an open tag 939 or a close tag 940; the virtual item in this embodiment is an item with different switch states, such as at least one of a virtual door, a virtual window, a virtual safe, a virtual box, a virtual computer, and a virtual door. For example, if the virtual item is a virtual door in a virtual scene, the virtual door has a switch state, and has different appearance styles (the door panel angles in the open state and the closed state are different) / different human-computer interaction modes (the door panel angles in the open state and the closed state are different) in different switch states.

[0382] And / or, in the case where the primary behavior tag is used to indicate that the virtual activity type is moving in a virtual scene, the multiple secondary candidate tags include a moving path tag 941 or a tag 942 for accompanying activities during movement; illustratively, the secondary candidate tags describing the moving path include but are not limited to at least one of the following: moving between two locations based on the line between the two locations, moving in a way that bypasses a virtual object in the virtual environment, and multiple NPCs moving with the purpose of surrounding a virtual object (surrounding a virtual object). The secondary candidate tags describing accompanying activities include but are not limited to at least one of the following: accompanying the initiation of a virtual attack, accompanying the avoidance of a virtual attack, and accompanying the search for a cover object in the virtual environment.

[0383] And / or, in the case where the primary behavior tag is used to indicate that the virtual activity type is answering a query, the multiple secondary candidate tags include a location query tag 943 including location information in a virtual scene in the answer sentence or an information query tag 944 for description information of the virtual scene. In the case where the secondary candidate tag is used to describe that the answer sentence includes location information in a virtual scene, the purpose of the query sentence is to inquire about the location of the NPC in the virtual environment; in the case where the secondary candidate tag is used to describe that the answer sentence includes description information in a virtual scene, the purpose of the query sentence is to inquire about the observation results of the NPC in the virtual environment.

[0384] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0385] In an optional embodiment, the natural language command is converted into a command analysis statement group including a plurality of command analysis statements based on speech-to-text technology, wherein each command analysis statement can be used as a command analysis result for behavioral intention recognition.

[0386] In some embodiments, the process of performing intent analysis based on a command analysis statement group is described. Fig.11 , which is a schematic diagram of a method for processing command analysis statements provided by an exemplary embodiment of the present application.

[0387] Obtain command analysis statement group 1102, the text content of command analysis statement group 1102 includes: "You set up the gun on the first floor and I will go to the front to get the treasure chest"; call text segmentation network 1110 to perform semantic segmentation on command analysis statement group 1102, and obtain command analysis statement group 1102 including two command analysis statements: a first command analysis statement 1104 and a second command analysis statement 1106; wherein, the text content of the first command analysis statement 1104 includes: "You set up the gun on the first floor"; the text content of the second command analysis statement 1106 includes: "I will go to the front to get the treasure chest".

[0388] The following is an introduction to the behavioral intention recognition and entity recognition process of the first command analysis statement 1104. First, the information structure prediction network 1120 is called to perform classification prediction on the first command analysis statement 1104 to obtain a control label 1125. The control label 1125 is used to indicate whether the first command analysis statement 1104 carries the natural semantics of commanding the NPC. When the control label 1125 indicates that the first command analysis statement 1104 carries the natural semantics of commanding the NPC, the hierarchical prediction network 1100 and the entity prediction network 1160 are called respectively to perform behavioral intention recognition and entity recognition on the first command analysis statement 1104. The NPC is a server or client or is controlled by artificial intelligence (AI), and the NPC has autonomous behavioral capabilities. The command is given by the player, and the NPC understands and executes the command according to its own autonomous behavioral capabilities.

[0389] For the behavioral intention recognition of the first command analysis statement 1104, the hierarchical prediction network 1100 includes three parts, which predict the command intention for the NPC from three intention dimensions. Among them, the subject prediction network 1130 is used to predict the subject label 1135 of the first command analysis statement 1104, and the subject label 1135 is used to indicate the subject type, and the subject type is used to indicate the identity of the NPC commanded by the command analysis statement. In the first command analysis statement 1104, the subject label 1135 indicates that the command analysis statement commands the teammate role. The semantic prediction network 1140 is used to predict the semantic label 1145 of the first command analysis statement 1104, and the semantic label 1145 is used to indicate the semantic type, and the semantic type is used to indicate the control method of the NPC for launching a virtual attack. In the first command analysis statement 1104, the semantic label 1145 indicates that there is a semantics of launching a virtual attack. The behavior prediction network 1150 is used to predict the behavior label 1155 of the first command analysis statement 1104, and the behavior label 1155 is used to indicate the intention type, and the intention type is the behavior method of the NPC for performing virtual activities. In the first command analysis statement 1104 , the behavior tag 1155 indicates moving in the virtual environment to go to a designated location.

[0390] Exemplarily, each of the subject prediction network 1130, the semantic prediction network 1140, and the behavior prediction network 1150 includes at least two layers of subnetworks, and at least two layers of subnetworks are constructed based on a tree structure of multiple classification labels. Taking the subject prediction network 1130 as an example, the high-level subnetwork of the subject prediction network 1130 is used to predict whether there is a subject in the first command analysis statement 1104. When there is a subject in the first command analysis statement 1104, the low-level subnetwork is used to predict that the subject of the first command analysis statement 1104 is the main control virtual character (also called the player character), and / or one or more NPCs (also called teammate characters) in the virtual camp where the main control virtual character is located.

[0391] For entity recognition of the first command analysis statement 1104, the entity prediction network 1160 can be implemented as a conditional random field to predict the entity tag 1165 of the first command analysis statement 1104. In the first command analysis statement 1104, the word "you" corresponds to the person entity tag, the word "first floor" corresponds to the building entity tag, and the non-existent word corresponds to the modifier tag.

[0392] Similarly, the behavior intention recognition and entity recognition process for the second sentence 1106 are similar to the first sentence. In the second sentence 1106, the subject tag 1135 indicates that the command analysis sentence is commanding the player character, the semantic tag 1145 indicates that there is a semantic meaning of launching a virtual attack, and the behavior tag 1155 indicates moving in the virtual environment to go to a designated location. In the second sentence 1106, the word "I" corresponds to the character entity tag, the non-existent word corresponds to the building entity tag, and the word "front" corresponds to the orientation tag in the modifier.

[0393] In the process of behavioral intention recognition, by calling the two-layer sub-networks of each part of the hierarchical prediction network 1100 in sequence, the classification labels of the command analysis sentence in the subject dimension, semantic dimension, and behavior dimension are predicted, and the type of the control object carried in the command analysis sentence is determined; at least two layers of sub-networks in the hierarchical prediction network are constructed based on a tree structure of multiple classification labels, which reduces the complexity of classification prediction of each layer of sub-network; in the process of entity recognition prediction, words corresponding to person entity labels, building entity labels, and modifier labels are realized based on conditional random fields. The accuracy of classification prediction for natural semantics in command analysis sentences is improved.

[0394] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0395] In an optional embodiment, the content of the command analysis statement obtained from the command analysis statement group is described.

[0396] In some embodiments, the encoder is called to perform feature encoding on the command analysis statement group to obtain a first feature representation of the command analysis result in the latent space.

[0397] Exemplarily, the encoder has the ability to extract feature representations of command analysis results in the latent space, and encode the command analysis results in natural language form into at least one of eigenvalues, eigenvectors, and feature matrices. Exemplarily, the encoder can be implemented as a natural language model, such as an encoder based on bidirectional transformation (Bidirectional Encoder Representations from Transformers, BERT).

[0398] In some embodiments, a segmenter is called to perform semantic segmentation on the first feature representation to obtain at least two sub-feature representations.

[0399] Illustratively, the segmenter determines whether a character in the command analysis statement group is the starting position of a statement based on the first feature representation obtained by encoding, and then splits the first feature representation into at least two sub-feature representations based on the character that is the starting position of the statement.

[0400] In some embodiments, the decoder is called to perform feature decoding on the at least two sub-feature representations respectively to obtain at least two command analysis results, and the at least two command analysis results correspond one-to-one to the at least two sub-feature representations.

[0401] Exemplarily, the decoder has the ability to decode latent features of natural language into natural language, and performs feature decoding on at least two sub-feature representations respectively to obtain at least two command analysis results; wherein, the at least two command analysis results correspond one-to-one to the at least two sub-feature representations to achieve the splitting of the command analysis result group.

[0402] Fig.12 A schematic diagram of the segmented command analysis results provided by an exemplary embodiment of the present application is shown. The command analysis result group 1202 is used to continuously command the NPC to perform virtual activities; the command analysis result group 1202 includes a silent start symbol [CLS] and N words (Token, Tok). BERT 1210 is called to perform feature encoding on the command analysis result group 1202, wherein BERT 1210 performs a linear transformation on the start symbol [CLS] and the N words to obtain a linear encoding: E CLS and E1 to E N BERT1210 performs feature encoding on the linear encoding to obtain the feature representation of the command analysis result in the hidden space: T CLS and T1 to T N The text segmenter 1220 is called to perform semantic segmentation on the first feature representation to obtain two sub-feature representations; the two sub-feature representations include E1 to E3 and E'1 to E'3; the above two sub-feature representations correspond to the two command analysis sentences. The text segmenter 1220 has the function of determining whether the N words in the command analysis result group 1202 are related to the starting position of the sentence. For example, the text segmenter 1220 determines that the segmentation label of the word in the command analysis result group 1202 is S B In the case of , the word is related to the starting position of the sentence; and in the case of the word segmentation label S O In the case of , the word is not related to the starting position of the sentence. The decoder is called to perform feature decoding on the at least two sub-feature representations respectively to obtain at least two command analysis results 1242.

[0403] In some embodiments, a first command analysis result of the at least two command analysis results is determined as the command analysis result.

[0404] In this embodiment, the various steps below are introduced by taking the prediction of the first classification label of a command analysis result as an example, and the first classification label of each command analysis statement intended to command the NPC is obtained by repeatedly performing the behavioral intention recognition of each command analysis statement in at least two command analysis results.

[0405] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0406] In some embodiments, calling an encoding network based on an attention mechanism to perform feature encoding on the command analysis result to obtain a hidden feature representation of the command analysis result in a hidden space;

[0407] Exemplarily, the encoding network in this step and the network structure of the encoder described above can be the same or different. In the above content, the hidden features extracted by the encoder can indicate whether the characters in the command analysis result group are the starting position of the sentence; in this step, the encoding network based on the attention mechanism can indicate the natural semantics of the characters in the command analysis result; it can be seen that the encoding purposes of the two are different, and the corresponding network parameters of the two are usually different. The difference in the above network parameters is caused by the use of different loss functions during the network training process. Exemplarily, the hidden feature representation includes at least one of eigenvalues, eigenvectors, and feature matrices.

[0408] It should be noted that, in the above new embodiment, the classification prediction performed by calling the hierarchical prediction network is performed on the hidden layer feature representation of the command analysis result.

[0409] In an optional embodiment, the call information structure prediction network performs classification prediction on the command analysis result, and predicts that the command analysis result corresponds to the first control label.

[0410] The information structure prediction network has the ability to predict whether the command analysis result carries the natural semantics of commanding the NPC, and the first control tag is used to indicate that the command analysis result carries the natural semantics of commanding the NPC. When the command analysis result is predicted to correspond to the first control tag, it indicates that the command analysis result is used to control the NPC, rather than the chat content, and then it is necessary to perform behavioral intention recognition to obtain the first classification label of the intention to command the NPC. Furthermore, the structure prediction network is used to perform a binary classification task of whether it belongs to the chat content.

[0411] In an optional embodiment, a feature encoding network is called to perform dimensional encoding on the command analysis result to obtain a dimensional feature representation of the command analysis result.

[0412] Exemplarily, the dimensional feature representation is used to indicate the feature representation of the command analysis result in at least one of the subject dimension, semantic dimension, behavior dimension, and control dimension; wherein the control dimension is used to indicate whether the command analysis result carries the natural semantics of commanding the NPC. In some examples, the dimension corresponding to the dimensional encoding performed by the feature encoding network is different from the intention dimension for performing behavioral intention recognition. By performing encoding on different dimensions, feature information with more information load can be obtained, which increases the information source basis for performing behavioral intention recognition.

[0413] In an optional embodiment, a linear transformation is performed on the latent feature representation and the dimensional feature representation of the command analysis result to obtain a fusion feature of the command analysis result.

[0414] Exemplarily, by performing a linear transformation on the latent feature representation and the dimensional feature representation, the natural semantic dimension and the dimensional feature (at least one of the subject dimension, semantic dimension, behavior dimension, and control dimension introduced above) are integrated, providing more information sources for the identification of behavioral intentions. Accordingly, in this embodiment, the classification prediction performed by calling the hierarchical prediction network is performed on the fused features of the command analysis results.

[0415] It should be noted that, in the above new embodiment, the classification prediction performed by calling the hierarchical prediction network is performed on the fusion features of the command analysis results.

[0416] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0417] In an optional embodiment, the training process of the hierarchical prediction network is described.

[0418] In some embodiments, sample commands and sample labels of the sample commands are obtained, where the sample labels are type labels of the sample commands; at least one hierarchical prediction network is called to perform behavioral intention recognition on the sample commands to obtain a prediction label of the sample command's intention to command the NPC.

[0419] Exemplarily, the sample command is a command analysis result that has been annotated with a sample label, and the sample label is a type label of the sample command; the sample command and the sample label can be obtained by manual labeling and used to train a hierarchical prediction network.

[0420] Exemplarily, the hierarchical prediction network has the ability to predict the classification label corresponding to the command analysis result. Exemplarily, the hierarchical prediction network includes at least two layers of sub-networks, and at least two layers of sub-networks are constructed based on the tree structure of multiple classification labels. The hierarchical prediction network constructs at least two layers of sub-networks corresponding to the hierarchical structure for the tree structure of the classification label, and splits the prediction task of a wide variety of classification labels into prediction sub-tasks performed by at least two layers of sub-networks, so as to reduce the complexity of classification prediction of each layer of sub-network.

[0421] In some embodiments, based on the difference between the predicted label and the sample label, backward error propagation training is performed on the hierarchical prediction network to obtain a trained hierarchical prediction network.

[0422] Exemplarily, the difference between the predicted label and the sample label is used to indicate the accuracy of the hierarchical prediction network in performing behavioral intention recognition. According to the difference between the predicted label and the sample label, the hierarchical prediction network is trained by backward error propagation, and the network parameters of the hierarchical prediction network are adjusted with minimizing the difference between the predicted label and the sample label as the training goal, thereby obtaining a trained hierarchical prediction network.

[0423] It should be noted that the behavioral intention includes at least one intention dimension; each hierarchical prediction network in the at least one hierarchical prediction network in this embodiment is used to predict the command intention of the command analysis result for the NPC on an intention dimension; the intention dimension includes: at least one of the subject dimension, the semantic dimension, and the behavioral dimension. For the introduction of the subject dimension, the semantic dimension, and the behavioral dimension, please refer to the various embodiments above, which will not be repeated here one by one.

[0424] Exemplarily, the backward error propagation training performed on the hierarchical prediction network in this step may be performed only on the hierarchical prediction network of one intent dimension, or may be performed on multiple hierarchical prediction networks together to obtain multiple trained hierarchical prediction networks at the same time.

[0425] Optionally, obtain the distinction loss between the prediction label and the sample label; obtain a first quantity, and determine a weighting parameter based on the first quantity, the first quantity and the weighting parameter are negatively correlated, the first quantity is the number of training commands, the training commands and the sample commands correspond to the same sample labels, and the training commands are used to train the hierarchical prediction network; according to the product of the distinction loss and the weighting parameter, and based on the product, perform backward error propagation training on the hierarchical prediction network to obtain the trained hierarchical prediction network.

[0426] In some embodiments, when there is no corresponding i+1th layer sub-network for the first prediction label, backward error propagation training is performed on the hierarchical prediction network according to the difference between the first prediction label and the sample label to obtain a trained hierarchical prediction network.

[0427] Exemplarily, when there is no corresponding i+1th layer sub-network for the first prediction label, only one layer of sub-network in the hierarchical prediction network is needed to realize the prediction of the prediction label, and the loss function used to train the hierarchical prediction network is determined based on the difference between the first prediction label and the sample label.

[0428] like Fig.13 As shown, a schematic diagram of a training method for a prediction network provided by an exemplary embodiment of the present application. The long sentence set 1302 includes long sentences that continuously command NPCs in a virtual scene, also known as a sample command group; the long sentence set 1302 corresponds to a clause set 1304 and a label set 1306; taking a long sentence in the long sentence set 1302 as an example, the clause set 1304 is used to indicate each clause after the long sentence is segmented. The label set 1306 is used to indicate the classification label of each word in the long sentence, and the classification label includes labels in multiple dimensions, which will be explained in detail below.

[0429] The long sentences in the long sentence set 1302 are input into the text segmentation network 1310 to obtain the segmentation results 1315 of the long sentences. For example, please refer to the encoder, segmenter, and decoder above for the introduction of the text segmentation network 1310; they will not be repeated here. The segmentation result 1315 can be a predicted segmentation label for each word in the long sentence, indicating whether each word in the long sentence is the starting position of the sentence; the first loss function 1316 is obtained by comparing the segmentation result 1315 with the actual segmentation labels marked for each word in the long sentence in the label set 1306; for example, the first loss function 1316 is a cross entropy loss function.

[0430] Next, we will introduce the entity prediction and intent prediction process for a clause. To achieve entity prediction and intent prediction for each clause in a long sentence, you only need to repeat the process and perform entity prediction and intent prediction for each clause in the long sentence.

[0431] First, the clauses in the clause set 1304 are input into the information structure prediction network 1320 to obtain the control label 1325; the control label 1325 is used to indicate whether the clause carries the natural semantics of commanding the NPC. When the control label 1325 is used to indicate that the clause carries the natural semantics of commanding the NPC, the subject prediction network 1330, the semantic prediction network 1340, the behavior prediction network 1350, and the entity prediction network 1360 are called to perform predictions on the clauses respectively. Based on the prediction of the subject prediction network 1330, the subject label 1335 is obtained, based on the prediction of the semantic prediction network 1340, the semantic label 1345 is obtained, and based on the prediction of the behavior prediction network 1350, the behavior label 1355 is obtained; the above-mentioned subject prediction network 1330, the semantic prediction network 1340, and the behavior prediction network 1350, a total of three networks, are behavior intention recognition on different intent dimensions. It should be noted that each of the above three networks includes at least two layers of sub-networks; for the introduction of the above three networks, please refer to the various embodiments of the method for processing command analysis results in the above text, which will not be repeated one by one. Based on the prediction of the entity prediction network 1360, the entity label 1365 is obtained, and the entity label 1365 corresponds to the environmental element in the virtual scene. The entity label 1365 is used to describe the clause indication to instruct the NPC to perform virtual activities on which environmental element in the virtual environment. Exemplarily, the entity prediction network 1360 is implemented as a conditional random field; for the incomplete introduction of the entity prediction network 1360, please refer to step 530 in the above text, which will not be repeated one by one.

[0432] The label set 1306 includes the real labels of each word in the clause in the subject dimension, semantic dimension, behavior dimension, control dimension, and entity dimension. The difference between the real label and the predicted label is compared to obtain the second loss function 1326. According to the sum of the first loss function 1316 and the second loss function 1326, the text segmentation network 1310, the information structure prediction network 1320, the subject prediction network 1330, the semantic prediction network 1340, the behavior prediction network 1350, and the entity prediction network 1360 are overall performed with backward error propagation training to achieve model training.

[0433] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0434] Step 720: Control the NPC to perform a virtual activity according to the first intention corresponding to the first classification tag.

[0435] Illustratively, based on the above process, the first classification label is determined, and based on the correspondence between the classification label and the intention, the NPC can be controlled to perform a virtual activity through the corresponding first intention when the first classification label is determined.

[0436] Optionally, the first classification label is a classification label determined based on at least one of the subject dimension, the semantic dimension, and the behavior dimension, and thus can more comprehensively display the intention information, making the control of the virtual activities performed by the NPC more accurate.

[0437] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0438] In summary, after obtaining the natural language command in the form of voice for commanding the NPC, the environmental perception information obtained by the master virtual character and at least one of the NPCs from the virtual environment is used to identify the behavioral intention from the natural language command, so as to control the NPC to perform virtual activities through the behavioral intention. With the help of the environmental perception information of the master virtual character and / or the NPC, it is helpful to better understand the natural language command, improve the accuracy of the analysis of the natural language command, and then improve the accuracy of the recognized behavioral intention, so that the behavioral intention is more matched with the virtual environment, which greatly avoids invalid or wrong commands to the NPC, improves the accuracy of the control of the NPC, and can also provide players with a more efficient and comfortable operating environment, improving the efficiency of human-computer interaction.

[0439] In an embodiment of the present application, if the subject label corresponding to the command analysis result is predicted through two layers of sub-networks in the subject prediction network, the identity of the NPC commanded by the command analysis result can be determined; at least two layers of sub-networks in the subject prediction network are constructed based on a tree structure of multiple subject labels, reducing the complexity of classification prediction of each layer of sub-network; in the i-th layer sub-network, it is predicted whether there is a subject in the command analysis result, and in the i+1-th layer sub-network, it is predicted that the identity of the specific character commanded by the command analysis result; in the scenario of controlling the NPC, according to the classification labels of the tree structure of the subject labels, a subject prediction network is constructed in a targeted manner, thereby improving the accuracy of classification prediction of natural semantics in the command analysis results.

[0440] In an embodiment of the present application, if the semantic labels corresponding to the command analysis results are predicted through the two-layer sub-network in the semantic prediction network, the control method for initiating a virtual attack can be determined; at least two layers of sub-networks in the semantic prediction network are constructed based on a tree structure of multiple semantic labels, reducing the complexity of classification prediction of each layer of sub-network; the i-layer sub-network predicts whether there is semantics of initiating a virtual attack in the command analysis results, and the i+1-layer sub-network predicts that the command analysis results specifically indicate to avoid initiating a virtual attack or are unrelated to the virtual attack; in the scenario of controlling NPCs, a semantic prediction network is constructed in a targeted manner according to the classification labels of the tree structure of semantic labels, thereby improving the accuracy of classification prediction of natural semantics in the command analysis results.

[0441] In an embodiment of the present application, if the behavior label corresponding to the command analysis result is predicted through two layers of sub-networks in the behavior prediction network, the type of virtual activity can be determined; at least two layers of sub-networks in the behavior prediction network are constructed based on a tree structure of multiple behavior labels, reducing the complexity of classification prediction of each layer of sub-network; the virtual activity type of the command analysis result is predicted in the i-layer sub-network, and the virtual activity entry of the command analysis result is predicted in the i+1-layer sub-network; in the scenario of controlling NPCs, the behavior prediction network is constructed in a targeted manner according to the classification labels of the tree structure of the behavior labels, thereby improving the accuracy of classification prediction of natural behaviors in the command analysis results.

[0442] In addition, at least two layers of sub-networks in the hierarchical prediction network are constructed based on a tree structure of multiple classification labels, which reduces the complexity of classification prediction of each layer of sub-networks; among them, the classification label is used as label information to characterize the NPC's behavioral ability. The tree structure constructed by multiple classification labels performs behavioral intention recognition on the command analysis results, which can better utilize the hierarchical relationship to achieve efficient overall recognition process, and is helpful to extract behavioral intentions that are more matched with the NPC's behavioral ability from the command analysis results, and enhance the influence of environmental perception information in the virtual environment on behavioral intention recognition, overcome the problem that the related technology cannot extract behavioral intentions from text content in a targeted manner, and help to fully improve the recognition accuracy of behavioral intentions; in the scenario of controlling the NPC, according to the classification labels of the tree structure of the NPC control, a hierarchical prediction network is constructed in a targeted manner, which improves the accuracy of classification prediction of the natural semantics in the command analysis results, and obtains more accurate behavioral intentions, which in turn helps to better control the NPC to perform virtual activities through more accurate behavioral intentions; in addition, the encoder, segmenter and decoder can also be used to segment long sentences to adapt to the method of continuously commanding the NPC.

[0443] In an optional embodiment, the control method of a virtual character executed by a server is used as an example for explanation. After receiving a natural language command in the form of voice, the terminal sends the natural language command to the server, and the server performs voice-to-text processing on the natural language command to obtain a command analysis result in the form of text. The command analysis result can be sent to the terminal and displayed on the terminal interface; in addition, the server can also perform behavioral intention recognition through the command analysis result in the form of text to obtain the behavioral intention and control the NPC to perform virtual activities. Schematically, as shown in FIG. Fig.14 As shown above Figure 3 The illustrated step 330 may also be implemented as the following steps 1410 to 1430 .

[0444] Step 1410, obtaining a natural language command in speech form.

[0445] Illustratively, natural language commands are usually implemented as voice commands, such as audio data. For example, when a player is operating a terminal, the terminal automatically collects the voice based on a microphone array and obtains audio data as a natural language command in the form of voice; or, when a player is operating a terminal, the terminal triggers an audio collection control and then issues a voice, and the terminal receives the voice and obtains audio data as a natural language command in the form of voice.

[0446] Among them, the natural language command includes natural semantics for instructing the NPC.

[0447] Illustratively, natural semantics is the semantic information expressed by the natural language commands used to control NPCs. Based on natural semantics, NPCs can be controlled to perform virtual activities in a targeted manner.

[0448] Optionally, the natural semantics include identification semantics for indicating the NPC and intention semantics for indicating the NPC activity. Illustratively, the identification semantics include at least one of the NPC's name (such as "Hero A"), the NPC's number (such as "No. 1"), and the NPC's location information (such as "next to the house").

[0449] Step 1420, converting the natural language command into a command analysis result in text form.

[0450] Among them, the command analysis result is the result of converting the natural language command speech into text based on the environmental perception information, and the environmental perception information includes the information perceived by at least one of the main virtual character and the NPC about the virtual environment.

[0451] Optionally, the environmental perception information is used to represent the environmental information perceived and acquired by the controlling virtual character and / or NPC within the object perception range, and the object perception range is the three-dimensional space range in which the controlling virtual character and / or NPC perceives the existence of other scene elements.

[0452] In some embodiments, behavioral intention recognition is performed on natural language commands under the constraints of environmental perception information to obtain behavioral intentions, and the NPC is controlled to perform virtual activities through the behavioral intentions.

[0453] Optionally, the behavior intention recognition process is performed through a pre-trained machine learning model, the environmental perception information and natural language commands are used as inputs of the machine learning model, and the encoding process and the decoding process are performed through the machine learning model to output the behavior intention.

[0454] In an optional embodiment, a plurality of first scene hot words are obtained based on the first virtual scene where the main virtual character and / or the NPC is located.

[0455] Among them, the multiple first scene hot words are scene-related words of the first virtual scene.

[0456] Optionally, the first virtual scene is one of multiple virtual scenes in the virtual environment; based on the position of the main virtual character and / or NPC in the virtual environment, the first virtual scene where the main virtual character and / or NPC is located is determined from the multiple virtual scenes.

[0457] In some embodiments, the position coordinates of the master virtual character and / or NPC in the virtual environment are obtained; the scene coordinate intervals corresponding to multiple virtual scenes are obtained; based on the relative position relationship between the position coordinates and the multiple scene coordinate intervals, the first virtual scene where the master virtual character and / or NPC is located is determined from the multiple virtual scenes.

[0458] The scene coordinate interval is used to represent the position range of the virtual scene in the virtual environment.

[0459] Illustratively, the position coordinates of the main virtual character and / or NPC are the content of the position information represented by the coordinate data. Optionally, the position coordinates corresponding to the main virtual character and / or NPC are determined based on the world coordinate system corresponding to the virtual environment.

[0460] Schematically, multiple virtual scenes in the virtual environment correspond to a scene coordinate interval respectively, and the scene coordinate interval is used to limit the three-dimensional space range corresponding to the virtual scene from three dimensions, and the three dimensions represent the horizontal direction, vertical direction and depth direction in the three-dimensional space. The scene coordinate intervals corresponding to different virtual scenes are independent and do not overlap with each other. Optionally, the scene coordinate intervals corresponding to the multiple virtual scenes are determined based on the world coordinate system corresponding to the virtual environment.

[0461] In an illustrative manner, after obtaining the position coordinates, the position coordinates are matched with a plurality of scene coordinate intervals, and the virtual scene corresponding to the scene coordinate interval including the position coordinates is used as the first virtual scene in which the master virtual character and / or the NPC is located. For example, the scene coordinate interval including the position coordinates is referred to as the first scene coordinate interval, the first virtual scene corresponds to the first scene coordinate interval, and the object coordinates are located within the first scene coordinate interval.

[0462] Optionally, if the master virtual character is the master virtual character and / or the NPC, the first position coordinates of the master virtual character in the virtual environment are obtained; based on the relative position relationship between the first position coordinates and multiple scene coordinate intervals, the first virtual scene where the master virtual character and / or the NPC is located is determined from multiple virtual scenes.

[0463] Optionally, if the NPC is a controlling virtual character and / or an NPC, the first position coordinates of the NPC in the virtual environment are obtained; based on the relative position relationship between the first position coordinates and multiple scene coordinate intervals, the first virtual scene where the controlling virtual character and / or the NPC is located is determined from multiple virtual scenes.

[0464] Optionally, when no command analysis result is obtained, in response to the main virtual character and / or NPC moving from the first virtual scene to the second virtual scene, multiple second scene hot words corresponding to the second virtual scene are obtained; and based on the multiple second scene hot words, the natural language command is converted into a command analysis result in text form.

[0465] In an optional embodiment, based on a first virtual game in which a main control virtual character and / or an NPC participates, a first virtual scene corresponding to the first virtual game is determined.

[0466] Among them, the multiple virtual games correspond to a virtual scene respectively, and the multiple virtual games are simulated battle environments provided for the main control virtual character and / or NPC.

[0467] In some embodiments, a first scene type corresponding to a first virtual scene in which a master virtual character and / or an NPC is located is obtained.

[0468] Illustratively, the first scene type is the scene type corresponding to the first virtual scene, and the scene type is used to characterize the scene state of the virtual scene. For example, the scene types include office type, battle type, kitchen type, bedroom type, classroom type, store type, factory type and other types that describe the scene state.

[0469] Optionally, the first virtual scene corresponds to a first scene type, and the first scene type is used to describe the scene state represented by the first virtual scene. For example, if the first virtual scene is a virtual office, the first scene type is an office type; or, if the first virtual scene is a virtual battle area, the first scene type is a battle type; or, if the first virtual scene is a virtual store, the first scene type is a store type, etc.

[0470] In some embodiments, a hot word set corresponding to the first scene type is obtained, and the words in the hot word set are used as multiple first scene hot words. The hot word set is a set of scene-related words collected based on the first scene type.

[0471] Schematically, multiple scene types correspond to a hot word set respectively, and the hot word set represents a set of scene-related vocabulary collected based on the scene type, and also represents a set of scene-related vocabulary collected based on at least one virtual scene under the scene type.

[0472] Optionally, multiple scene hot words are obtained, and the multiple scene hot words correspond to at least two virtual scenes.

[0473] In an illustrative manner, scene hot words corresponding to multiple virtual scenes in a virtual environment are obtained in advance. Optionally, based on the virtual elements in each virtual scene, at least one scene hot word corresponding to the multiple virtual scenes is obtained, and multiple scene hot words are stored. Optionally, the element name of the virtual element in the virtual scene is obtained as the scene hot word; or, the action name of the interactive action interacting with the virtual element in the virtual scene is obtained as the scene hot word; or, the state name used to describe the element state of the virtual element in the virtual scene is obtained as the scene hot word, etc.

[0474] Optionally, based on a first virtual scene in which the main control virtual character and / or the NPC is located, a plurality of first scene hot words corresponding to the first virtual scene are obtained from a plurality of scene hot words.

[0475] Optionally, based on the first virtual scene in which the main virtual character and / or NPC is located, multiple candidate scene hot words with a first scene identifier are obtained from multiple scene hot words, wherein the first scene identifier is the scene identifier corresponding to the first virtual scene; based on the object perception range of the main virtual character and / or NPC in the first virtual scene, at least two candidate scene hot words are selected from the multiple candidate scene hot words as the first scene hot words.

[0476] For example: if scene hot words 1 and scene hot words 2 are obtained based on virtual scene A, scene identifier a corresponding to virtual scene A is marked for scene hot word 1, and scene identifier a is also marked for scene hot word 2; if scene hot word 3 is obtained based on virtual scene B, scene identifier b corresponding to virtual scene B is marked for scene hot word 3, and so on.

[0477] In some embodiments, the object perception range of the master virtual character and / or NPC in the first virtual scene is obtained, and the object perception range is the three-dimensional spatial range in which the master virtual character and / or NPC perceives the existence of other scene elements; and multiple first scene hot words are obtained based on the environmental perception information within the object perception range.

[0478] Illustratively, after determining the object perception range, based on the range of the object perception range in the first virtual scene, and the environmental perception information determined based on perception of the main virtual character and / or NPC within the object perception range, multiple first scene hot words representing the state of the main virtual character and / or NPC are obtained.

[0479] Optionally, the object perception range includes at least one of the following: the visual perception range of the controlling virtual character and / or NPC; the auditory perception range of the controlling virtual character and / or NPC; the olfactory perception range of the controlling virtual character and / or NPC; the range perceived by the perception skills or perception tools possessed by the controlling virtual character and / or NPC.

[0480] Optionally, when the object perception range includes the visual perception range, a preset visual cone is generated as the visual perception range with the position of the main virtual character and / or NPC as the starting point and the direction of the main virtual character and / or NPC as the center line of the cone. The preset visual cone is a three-dimensional area that measures the visual perception range.

[0481] Illustratively, when the object perception range includes the visual perception range, the position of the controlling virtual character and / or NPC and the orientation of the controlling virtual character and / or NPC are determined, with the position as the starting point and the orientation as the cone midline, and the preset visual cone as the visual perception range determined based on the visual perception ability.

[0482] In some embodiments, a first scene type corresponding to a first virtual scene is obtained; a hot word set corresponding to the first scene type is obtained; an object perception range of a master virtual character and / or NPC in the first virtual scene is obtained; and at least two words are obtained from the hot word set as multiple first scene hot words based on environmental perception information within the object perception range.

[0483] Optionally, the environmental perception information includes virtual elements; determining the virtual elements within the object perception range in the first virtual scene; using the element name of the virtual element as the first scene hot word; or using the action name of the interactive action corresponding to the virtual element as the first scene hot word.

[0484] Schematically, the element association relationship is used to characterize the relationship that reflects the state of the virtual element. For example, the vocabulary including the element name of the virtual element is obtained from the hot word set as the first scene hot word; or, the vocabulary used to trigger the virtual element is obtained from the hot word set as the first scene hot word; or, the interactive vocabulary used to interact with the virtual element is obtained from the hot word set as the first scene hot word, etc.

[0485] In an optional embodiment, a command analysis result is obtained by converting a natural language command into text form based on a plurality of first scenario hot words.

[0486] In some embodiments, a pre-trained natural language analysis model is obtained, the pre-trained natural language analysis model includes an acoustic network, a language network and a preset dictionary, the preset dictionary includes a plurality of first scene hot words, other scene hot words and common vocabulary.

[0487] In some embodiments, multiple speech units corresponding to natural language commands are obtained through an acoustic network, where a speech unit is the basic building block of vocabulary pronunciation; the sequence matching relationship between the multiple speech units and selected vocabulary in a preset dictionary is analyzed through a language network to obtain a command analysis result in text form.

[0488] The selected words include at least a plurality of first-scenario hot words and common words.

[0489] Optionally, the selected vocabulary also includes other scene hot words.

[0490] Illustratively, if the selected vocabulary is a first scene hot word and a common vocabulary, at least one speech unit sequence can be matched with the first scene hot word and the common vocabulary; if the selected vocabulary is a first scene hot word, other scene hot words and a common vocabulary, at least one speech unit sequence can be matched with the first scene hot word, other scene hot words and a common vocabulary.

[0491] Among them, the vocabulary in the preset dictionary includes analysis weights, and the first analysis weights of multiple first-scene hot words are higher than the second analysis weights of other scene hot words. The analysis weight refers to the degree of attention paid to the vocabulary when it participates in sequence matching.

[0492] In schematic form, the larger the analysis weight, the more attention the word will receive when it participates in sequence matching, that is, the easier it will be to be noticed; the smaller the analysis weight, the less attention the word will receive when it participates in sequence matching, that is, the harder it will be to be noticed; the analysis weight can be used to adjust the degree of attention paid to different words in the sequence matching process, so that words with larger analysis weights are matched with the word sequence first, and at the same time, words with larger analysis weights are more likely to be selected as words that constitute the command analysis results.

[0493] Optionally, the matching relationship between the plurality of speech units and the selected words in the preset dictionary is analyzed to obtain a plurality of candidate word sequences, wherein the plurality of candidate word sequences include at least one word from the plurality of selected words, and the candidate word sequence is a word sequence obtained based on the changing relationship between the plurality of speech units.

[0494] Schematically, the process of analyzing the matching relationship represents the process of analyzing the matching between the selected vocabulary and the speech unit.

[0495] Optionally, the sequence semantics of multiple candidate vocabulary sequences are analyzed through a language network, and at least one candidate vocabulary sequence is obtained from the multiple candidate vocabulary sequences as a command analysis result in text form.

[0496] Schematically, sequence semantics is used to represent the semantic information expressed by the candidate vocabulary sequence, which can not only reflect the semantic changes of at least one word in the candidate vocabulary sequence, but also reflect the sentence semantics of the entire candidate vocabulary sequence. Optionally, a probability evaluation is performed on multiple candidate vocabulary sequences through a language network to obtain the prediction probabilities corresponding to the multiple candidate vocabulary sequences respectively; at least one candidate vocabulary sequence with the largest prediction probability is used as the command analysis result.

[0497] Optionally, the language network includes language sub-networks corresponding to at least two virtual scenes respectively.

[0498] Schematically, multiple virtual scenes correspond to a language subnetwork respectively, and different language subnetworks are used to analyze the corresponding virtual scenes in a targeted manner. For example, virtual scene A corresponds to language subnetwork 1, and virtual scene B corresponds to language subnetwork 2. When analysis is required based on virtual scene A, language subnetwork 1 is used for language analysis, and when analysis is required based on virtual scene B, language subnetwork 2 is used for language analysis, and so on.

[0499] Illustratively, based on a first virtual scene in which a main control virtual character and / or an NPC is located, a first language subnetwork corresponding to the first virtual scene is determined from at least two language subnetworks; the sequence semantics of multiple candidate vocabulary sequences are analyzed through the first language subnetwork, and at least one candidate vocabulary sequence is obtained from the multiple candidate vocabulary sequences as a command analysis result in text form.

[0500] Optionally, different language sub-networks are language sub-networks pre-trained based on corresponding virtual scenes, wherein the first language sub-network is a language sub-network trained based on the first virtual scene.

[0501] Optionally, after obtaining multiple candidate vocabulary sequences under the constraint of the hot words of the first scene, semantic analysis is performed on the multiple candidate vocabulary sequences respectively through the first language sub-network corresponding to the first virtual scene to obtain prediction probabilities corresponding to the multiple candidate vocabulary sequences, and at least one candidate vocabulary sequence with the largest prediction probability is used as the command analysis result in text form.

[0502] Step 1430, perform behavior intention recognition on the command analysis result to obtain the behavior intention.

[0503] In some embodiments, at least one hierarchical prediction network is called to perform behavioral intention recognition on the command analysis result, and a first classification label of the command analysis result that intends to command the NPC is obtained as the behavioral intention.

[0504] The hierarchical prediction network includes at least two layers of sub-networks, and at least two layers of sub-networks are constructed based on a tree structure of multiple classification labels.

[0505] In some embodiments, behavioral intention includes at least one intention dimension; each hierarchical prediction network in at least one hierarchical prediction network is used to predict the command intention of the command analysis result for the NPC on an intention dimension; the i-th layer subnetwork in each hierarchical prediction network is used to predict the first-level behavior label of the command analysis result on an intention dimension, and the i+1-th layer subnetwork in the hierarchical prediction network is used to predict the second-level behavior label corresponding to the command analysis result in the first-level behavior label, where i is a positive integer; wherein the intention dimension includes: at least one of: subject dimension, semantic dimension, and behavior dimension.

[0506] In some embodiments, the intent dimension includes a subject dimension, the hierarchical prediction network includes a hierarchical subject prediction network, and the subject prediction network has the ability to predict the subject type in the command analysis result; the subject type is used to indicate the identity of the NPC commanded by the command analysis result; and / or, the intent dimension includes a semantic dimension, the hierarchical prediction network includes a hierarchical semantic prediction network, and the semantic prediction network has the ability to predict the semantic type in the command analysis result; the semantic type is used to indicate the NPC's control method for launching a virtual attack; and / or, the intent dimension includes a behavior dimension, the hierarchical prediction network includes a hierarchical behavior prediction network, and the behavior prediction network has the ability to predict the behavior type in the command analysis result; the behavior type is used to indicate the NPC's behavior method for performing virtual activities.

[0507] It is worth noting that the content of performing behavioral intention recognition in step 1430 can refer to that shown in the above step 710 and will not be repeated here.

[0508] Among them, the behavior intention is used to control the NPC to perform virtual activities. Schematically, the behavior intention instructs the NPC to perform virtual activities based on the behavior type. For example, instructing the NPC to perform a throwing action; or instructing the NPC to adjust the movement characteristics; or instructing the NPC to interact with virtual objects, etc.

[0509] In some embodiments, after the server controls the NPC to perform virtual activities based on the behavioral intention, the activity animation data is sent to the terminal so that the activity animation of the NPC performing the virtual activity is rendered and displayed on the terminal; or, the server sends the behavioral intention to the terminal, and the terminal controls the NPC to perform the virtual activity based on the behavioral intention, and renders and displays the corresponding activity animation, etc.

[0510] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0511] In summary, after obtaining the natural language command in the form of voice for commanding the NPC, the environmental perception information obtained by the master virtual character and at least one of the NPCs from the virtual environment is used to identify the behavioral intention from the natural language command, so as to control the NPC to perform virtual activities through the behavioral intention. With the help of the environmental perception information of the master virtual character and / or the NPC, it is helpful to better understand the natural language command, improve the accuracy of the analysis of the natural language command, and then improve the accuracy of the recognized behavioral intention, so that the behavioral intention is more matched with the virtual environment, which greatly avoids invalid or wrong commands to the NPC, improves the accuracy of the control of the NPC, and can also provide players with a more efficient and comfortable operating environment, improving the efficiency of human-computer interaction.

[0512] In the embodiment of the present application, the process of voice-to-text processing of natural language commands by the server is introduced, as well as the process of behavioral intention recognition by the server is introduced. The natural language commands are analyzed by the server to obtain the behavioral intention for controlling the NPC to perform virtual activities, which utilizes the powerful computing power of the server, not only reduces the pressure of data analysis on the terminal, but also helps to continuously improve the analysis process and analysis efficiency with the help of the algorithm integration capability of the server, and improves the privacy of data analysis.

[0513] In an optional embodiment, the above-mentioned virtual character control method can be applied to a game scene, where the player issues a natural language command in the form of voice during the game, so that the terminal can achieve the purpose of voice commanding the NPC based on the behavioral intention in the natural language command. Fig.15 As shown, it is a technical flow chart of the voice conversion method based on virtual scene, which includes the following steps 1510 to 1590.

[0514] Step 1510, the player enters the game level.

[0515] Schematically, a game level is a specific section or stage in a video game where the player needs to complete specific goals or tasks in order to move on to the next level. Levels are usually designed to gradually increase in difficulty to challenge the player's skills and strategies. Each level may contain different environments, enemies, puzzles, and rewards, designed to provide a diverse gaming experience.

[0516] Optionally, the player can control the main virtual character to participate in the virtual game during the game, and the player can also command other NPCs to cooperate with the main virtual character to participate in the virtual game. NPC is a virtual character other than the main virtual character, such as virtual soldiers and virtual heroes deployed by the system by default in the virtual game. For example: the player can manually control the main virtual character to fight in the virtual game during the game, and can also command the NPC to fight in the virtual game through voice.

[0517] Step 1520 , prepare a hot word list based on virtual elements in the virtual scene of the game level.

[0518] Schematically, a game usually provides multiple different game levels. A game level corresponds to a virtual environment. A virtual environment includes multiple virtual scenes. The game scene refers to the three-dimensional space in which the main virtual character and / or NPC is located in a specific game level. The virtual scene is not only a visual presentation, but also includes virtual elements, game sound effects and other details, which together constitute the player's gaming experience. A well-designed virtual scene can greatly enhance the immersion and fun of the game. Multiple virtual scenes correspond to a scene type respectively. The scene types of multiple virtual scenes may be the same or different, which is not limited here.

[0519] Optionally, each scene type has its own unique virtual elements and atmosphere, through which the story and interactivity of the game can be enhanced. For example, in a forest scene, the player may need to avoid wild animals or find hidden paths; while in a future city scene, the player may need to use high-tech equipment to complete the task, etc.

[0520] Optionally, each game level may have some specific virtual elements, such as keys for unlocking specific doors or treasure chests, that is, doors leading to new areas or game levels that require specific keys or completion of specific tasks to open. Virtual props obtained in specific game levels may have unique attack effects or capabilities, such as: providing additional defense or special resistance to help players deal with specific hostile virtual characters, etc. There are unique virtual props and consumables, collectibles, gems / gold coins in the game levels; in addition, the game levels also contain puzzle props, specific means of transportation, specific abilities (magic, skills, etc.); and items that interact with the environment, such as virtual explosives, virtual obstacles, virtual buildings, etc. Among them, the virtual elements unique to each specific level in the game level usually have unique functions or effects that can affect the player's gaming experience.

[0521] In some embodiments, based on the virtual elements respectively contained in the multiple virtual scenes, hot word lists corresponding to the multiple virtual scenes are prepared; or, a hot word list corresponding to the multiple virtual scenes is prepared, and the hot word list includes scene hot words corresponding to the multiple virtual scenes respectively.

[0522] Optionally, in order to obtain more abundant scene hot words, this goal can be achieved by designing a hot word management system, which has pre-processed all game assets and can calculate the position and direction of the main virtual character and / or NPC when the player interacts in real time. A hot word library (or hot word list) is generated based on the asset pre-processing, which contains scene hot words of various virtual scenes.

[0523] In some embodiments, virtual elements existing in the object perception range are determined from the first virtual scene where the master virtual character and / or the NPC is located based on the object perception range, that is, virtual elements within the object perception range are determined.

[0524] In an illustrative manner, when the virtual scene changes, the hotword management system can automatically adjust the required scene hotwords, and according to the current scene, first select candidate scene hotwords from a predefined hotword library, and then perform a secondary selection according to the player's field of view (object perception range).

[0525] Optionally, taking the field of view as an example of the object perception range, when determining the field of view of the main virtual character and / or NPC, the field of view is determined by the camera's frustum. The frustum is a spatial area defined by the camera position and field of view (FOV). To improve efficiency, spatial segmentation techniques such as quadtree, octree or grid can be used to organize virtual elements in the virtual scene. These data structures can help quickly determine which virtual elements are within the field of view. After the spatial range is determined, frustum culling will be performed to determine which virtual elements need to be rendered by calculating which virtual elements are within the frustum.

[0526] Indicatively, six planes (upper, lower, left, right, front, and back) are extracted from the camera's view cone to extract the view cone plane. For each virtual element, its bounding box is used to test the view cone plane. If the bounding box is completely outside the view cone, the element name, action name, etc. of the virtual element do not need to be loaded into the first scene hotword.

[0527] Indicatively, based on the result of frustum culling, it is determined which virtual elements' corresponding element names, action names, etc. need to be loaded into the first scene hot words to be used, and which virtual elements' corresponding hot words can be unloaded from the memory. Optionally, thresholds for hot word loading and unloading can also be set to avoid frequent hot word loading and unloading operations.

[0528] Hierarchical Frustum Culling: Combined with the space segmentation technology, the element names and action names of the virtual elements in the large area are first unloaded, and then the element names and action names of the virtual elements in the small area are unloaded.

[0529] Among them, scenario hot words are words frequently used in specific application scenarios. By setting scenario hot words, the system can recognize these words more accurately, thereby improving the overall recognition accuracy. These hot words either only appear in a specific game level of a certain game, or are words frequently used by players in a specific game level. These words are difficult to translate into text based on pronunciation alone, or these words are used very frequently, which affects the user experience.

[0530] Optionally, an initial hot word list is collected, which includes the following contents.

[0531] (1) Scene-specific object terms: such as “Area A”, “virtual stable”, “virtual crash site”, virtual “oil tank truck”, etc.

[0532] (2) Scene-specific character words: such as “virtual hero 1”, “virtual soldier 2”, “virtual monster”, etc.;

[0533] (3) Common commands: such as "attack", "defense", "jump", "run", etc.;

[0534] (4) Specific task words: such as "open the door", "unlock the treasure chest", "dialogue", etc.;

[0535] (5) Environmental interaction words: such as "view map", "use props", "save game", etc.

[0536] Step 1530, matching hot word list.

[0537] Illustratively, after determining the virtual element within the object perception range, the hot word list is matched to determine the scene hot word corresponding to the virtual element as the first scene hot word.

[0538] Optionally, taking the example that multiple virtual scenes correspond to one hot word list respectively, a first hot word list corresponding to the first virtual scene is determined based on the first virtual scene, and multiple first scene hot words are determined from the first hot word list based on the virtual elements within the object perception range. For example, the element name describing the virtual element is used as the first scene hot word, or the action name of the interactive action interacting with the virtual element is used as the first scene hot word, etc.

[0539] Optionally, taking multiple virtual scenes corresponding to one hot word list as an example, multiple first scene hot words are determined from the hot word list based on virtual elements within the object perception range. For example, the element name describing the virtual element is used as the first scene hot word, or the action name of the interactive action interacting with the virtual element is used as the first scene hot word, etc.

[0540] Step 1540, ASR system hot word weight.

[0541] Schematically, the ASR system, or Automatic Speech Recognition system, is a commonly used system in speech recognition technology, which aims to convert instructions in the form of human voice into instructions in the form of text. It involves the process of processing, analyzing and understanding sound signals so that computers can "understand" and perform corresponding operations.

[0542] The ASR system is the pre-trained natural language analysis model mentioned above. The ASR system includes the following key components.

[0543] (1) Speech recognition acoustic model (the acoustic network in the above natural language analysis model);

[0544] The acoustic model is used to map the feature vector of the audio signal to the basic speech unit (such as phoneme). It is usually trained through machine learning techniques (such as deep neural network), using a large amount of labeled speech data to learn the relationship between speech signals and phonemes.

[0545] (2) Speech recognition language model (language network in the above natural language analysis model);

[0546] The language model is used to predict the probability of a word sequence, helping the system select the most likely one among multiple possible word sequences. Usually, certain language model information is already included in the end-to-end network, and decoding constraints based on statistical methods (n-gram model) are used to improve accuracy.

[0547] (3) Speech recognition hot words (the first scenario hot words in the preset dictionary in the above natural language analysis model, and can also be selected words including common words);

[0548] Hot words refer to words or phrases that require special attention and priority recognition in a specific virtual scene. For example, in a game scene, "virtual stable" or "virtual crash site" are hot words. Hot words usually require higher recognition accuracy, so there may be special models or algorithms to handle them. These hot words often have scene associations with virtual scenes in the game, and their corresponding texts cannot be written based solely on pronunciation.

[0549] (4) Speech recognition decoder (the decoding network mentioned above);

[0550] The decoder is one of the core components of the speech recognition system. It combines the output of the acoustic model and the language model and uses a search algorithm (such as the Viterbi algorithm) to find the most likely sequence of words. The decoder's task is to convert the feature vector sequence into the final text output.

[0551] In an optional embodiment, a weighted finite-state transducer (WFST) is used to process the relationship between states and transitions. In the speech recognition process, states can represent different meanings based on different inputs, and transitions represent the transformation between states. For example, when processing multiple speech units, each speech unit can be regarded as a state, and the transformation between speech units (such as the transition from speech unit 1 to speech unit 2) is the transformation between states; in the analysis of candidate vocabulary sequences, each word in the candidate vocabulary sequence can be regarded as a state, and the transformation between words (such as the transition from word 1 to word 2) is the transformation between states, etc.

[0552] Schematically, WFST can assign a weight to each transition. The weight usually represents the cost or probability of the transition, such as the weight of the transition from speech unit 1 to speech unit 2; or the weight of the transition from vocabulary 1 to vocabulary 2, etc.

[0553] In some embodiments, WFST is used to represent various models and transformation relationships, such as acoustic models, language models, preset dictionaries, etc. In speech recognition systems, WFST is used in speech recognition to combine different models (such as acoustic models, language models, and preset dictionaries) into a unified decoding network that can efficiently search for the most likely word sequence.

[0554] Schematically, the acoustic model represents the probability of phonemes or other basic speech units as WFST; the language model represents the probability of candidate vocabulary sequences as WFST; the preset dictionary represents the mapping between vocabulary and its corresponding phoneme sequence (speech unit sequence) as WFST. The relationship between the acoustic model, language model and WFST is: In the speech recognition system, the acoustic model, language model and dictionary are usually represented as independent WFSTs, and then they are merged into a comprehensive WFST through specific combination and optimization algorithms. This comprehensive WFST is used for decoding and recognition.

[0555] Optionally, the acoustic model (H) maps multiple acoustic features corresponding to the natural language command in the form of speech to phonemes or other basic speech units to obtain multiple speech unit sequences. This mapping can be represented as a WFST, called H. The context dependency model (C) is usually used in the acoustic model to obtain a speech unit sequence that is more closely related to the context.

[0556] The preset dictionary (L) maps vocabulary to a sequence of speech units to obtain multiple candidate vocabulary sequences. This mapping can also be expressed as a WFST, called L.

[0557] The language model (G) represents the probability of candidate word sequences as a WFST, called G.

[0558] By combining H, C, L and G, the HCLG corresponding to the speech recognition system is obtained, that is, a comprehensive WFST is generated, called HCLG. This comprehensive WFST represents the complete mapping from feature vectors to word sequences. During the decoding process, the ASR system uses the HCLG, a comprehensive WFST, to search for the most likely candidate word sequence as the command analysis result.

[0559] Schematically, a natural language command in speech form is converted into multiple acoustic features, and the acoustic model (H) and context dependency model (C) in HCLG are used to score the multiple acoustic features to generate phoneme probabilities; then the preset dictionary (L) in HCLG is used to map the phoneme probabilities to word sequences; finally, the command analysis result is obtained through analysis by the language model (G), such as finding the most likely word sequence as the command analysis result through a decoder through a search algorithm (such as the Viterbi algorithm).

[0560] In an optional embodiment, it is necessary to prepare voice data and text data related to the game level. These data will be used to train the acoustic model and language model. Voice data includes recorded and collected audio data in the form of voice that players may use in the game; text data includes text previously entered by players and text related to the game level entered by the large language model, such as level names, action instructions, etc.

[0561] Optionally, the Mel Frequency Cepstral Coefficients (MFCC) and fbank filter bank extract features from natural language commands in speech form to train the acoustic model. After using the extracted features to train the acoustic model in acoustic model training, the model probability at the level of Connectionist Temporal Classification (CTC) will be obtained. The language model (LM) is trained using the data described above. Commonly used tools include SRI Language Modeling Toolkit, Ken Language Model Toolkit (KenLM), etc. Build a pronunciation dictionary containing all possible words and their corresponding phoneme sequences. The pronunciation dictionary format contains one word and its phoneme representation per line. Then use the composition tools arpa2fst, fstcompose, fstreduced, etc. to construct the speech recognition decoding word graph and obtain them respectively.

[0562] Step 1550: perform input correction based on the large language model.

[0563] In an optional embodiment, a large language model is used to generate sample data for training the ASR system, so that the ASR system is trained with the sample data before the ASR system is applied; or, the ASR system is periodically trained with the sample data during the application of the ASR system; or, the ASR system is trained with the sample data in real time during the application of the ASR system, etc.

[0564] Schematically, a large language model (LLM) refers to a natural language processing (NLP) model that has been trained with large-scale text data and has high complexity and high performance. Large language models use deep learning technology, especially the Transformer architecture, to understand and generate natural language commands. Large language models perform well in various NLP tasks, such as text generation, translation, question answering, summarization, etc. Large-scale data training: Large language models are usually trained using a large amount of text data, which may include billions or even trillions of words. Large-scale data enables the model to learn richer language features and patterns. Complex model architecture: These models usually have complex architectures, including multi-layer transformers, attention mechanisms, etc. to capture the nuances in the language. High performance: Due to the use of large amounts of data and complex model architectures, large language models usually have high understanding and generation capabilities and can perform well in various language tasks.

[0565] The large language model has prompts, which refer to the initial input text provided to the model when using the large language model for natural language processing tasks. The design and selection of prompts have an important impact on the output quality and accuracy of the model. By carefully designing prompts, the model can be guided to generate text that is more in line with expectations. In other words, prompts provide contextual information for the model, helping the model understand the user's intentions and generate relevant text. For example, given a prompt "write an article about climate change", the model will generate an article related to climate change. The output style and format can also be controlled by prompts. For example, prompts can contain specific instructions or format requirements to help control the output style and format of the model. For example, a prompt of "explain quantum mechanics in concise language" will guide the model to generate concise and easy-to-understand explanations.

[0566] In addition, the prompt can also include the content of previous conversations to help the model understand the context of the current conversation and generate a more coherent response. The design of prompts Effective prompts need to consider the following aspects: Clear instructions; prompts should contain clear instructions to tell the model what task needs to be completed. For example: The prompt "Please summarize the main points of the following article" is clearer than the prompt "summarize". Prompts can also provide enough contextual information to help the model understand the task. For example: In a dialogue system, prompts can include the content of previous conversations. In addition, if there are specific format requirements for the output, they can be clearly stated in the prompt. For example: The prompt is "Please list the answers to the following questions in list form."

[0567] Based on the player's in-game instructions, dialogues, descriptions, etc. Provide context: Give the model enough contextual information so that it can generate reasonable speech input. Use examples: Guide the model to generate similar outputs by providing some examples. Provide clear formatting: Make sure it is clearly stated in the prompt words.

[0568] In some embodiments, the process of generating sample data based on a large language model is described as follows.

[0569] For example, suppose there is an adventure game where players can command NPCs through natural language commands in the form of voice. Therefore, it is hoped that the large language model can generate some possible sample data to conduct more targeted training of the natural language analysis model.

[0570] Optionally, preset instruction templates can be spliced ​​to generate text corpus. For example: suppose the template is a string containing placeholders for inserting specific player instruction parameters. Placeholders can be represented by {}. For example: "{player_name} goes to {direction}{location}{action}", which means that the player name (such as the name of the NPC) goes to a certain direction, a certain location and takes a certain action; suppose there are the following player instruction parameters: player_name = "Teammate No. 1 goes"; direction = "front"; location = "in the forest"; action = "throw a grenade"; splicing instructions to obtain template = "Teammate No. 1 {player_name} is at {location}{action}", that is: Teammate No. 1 goes to the forest in front and throws a virtual grenade.

[0571] Alternatively, if the player is a player in an adventure game, he can use natural language commands to command a virtual character. For example, the command is: 1. Go north; 2. Attack the enemy in front; 3. Use healing potion; 4. Open the backpack; 5. Check the map. This command can command the virtual character.

[0572] Based on this example instruction, sample data of some possible speech forms can be generated based on the large language model as data for training the natural language analysis model.

[0573] Indicatively, if you want the generated voice input to be more specific, you can provide more contextual information. For example, the prompt words provided to the large language model are "You are a player in an adventure game, your character is a warrior, and you are currently in the forest. You can command your character by voice input. Here are some example commands: 1. Go north; 2. Attack the enemy in front; 3. Use healing potion; 4. Open the backpack; 5. Check the map; Current game status: Role: Warrior; Location: Forest; Mission: Find hidden treasure; Now, please generate some possible player voice inputs."

[0574] Indicatively, if you want to generate more diverse voice input, you can add more examples and different scenarios to the prompt words. For example: the prompt words provided to the large language model are "You are a player in an adventure game, your character is a warrior, and you are currently in the forest. You can command your character through voice input commands; here are some example commands: 1. Go north; 2. Attack the enemy in front; 3. Use healing potions; 4. Open the backpack; 5. Check the map; 6. "Pick up the gems on the ground", 7. "Talk to the villagers", 8. "Enter the cave"; Current game state: Warrior; Location: Forest; Mission: Find hidden treasures; Now, please generate some possible player voice inputs" and so on.

[0575] Optionally, based on the above prompt words, the large language model may generate the following text: 1. "Go east; 2. Use fireball to attack the enemy; 3. Check the task log; 4. Drink a bottle of magic potion; 5. Search the nearby area; 6. "Trade with the merchant", 7. "Enter the castle", 8. "Equip a new sword", etc.

[0576] Through the above process, a large amount of text simulating the player's speech in the scene can be generated through the human speaking habits built into the large language model, and the text can be used as sample data; and / or, the text can be converted into speech to obtain speech input as sample data, so as to use richer sample data to train the natural language analysis model.

[0577] In an optional embodiment, the text generated by the large language model is used as sample data, and the language model therein is updated by the sample data before being applied in the ASR system.

[0578] Step 1560, activating the language model of the current game level.

[0579] In an optional embodiment, the language model (G) in the HCLG includes multiple language models, and the multiple language models correspond to multiple virtual scenes respectively. Based on the first virtual scene in which the main virtual character and / or NPC is located, the first language model corresponding to the first virtual scene is determined from the multiple language models (that is, the first language sub-network corresponding to the first virtual scene is determined from the language network as mentioned above).

[0580] Optionally, multiple language models are models obtained after training with corresponding virtual scenes. In the process of training the language model using the text generated by the large language model as sample data, the texts corresponding to the multiple virtual scenes are obtained, and the language models corresponding to the virtual scenes are trained with the texts.

[0581] Step 1570, the ASR system selects a language model when decoding.

[0582] Illustratively, when the ASR system performs decoding processing, a first language model corresponding to the first virtual scene is selected so that the content input by the sound model is analyzed through the first language model.

[0583] In some embodiments, multiple speech units corresponding to natural language commands are obtained through the acoustic model in the ASR system, and the speech unit is the basic building block of vocabulary pronunciation; the sequence matching relationship between the multiple speech units and the selected vocabulary in the preset dictionary is analyzed through the language model in the ASR system, and the command analysis result in text form is obtained, which is the decoding result, and the selected vocabulary includes at least multiple first-scene hot words and common vocabulary.

[0584] In an optional embodiment, when the main virtual character and / or NPC controlled by the player moves between different buildings, the spatial query logic will provide complete location information to help the player quickly locate the position of the enemy virtual character. For example: the main virtual character and / or NPC is outside the motel and hears virtual gunshots coming from the second floor of the motel. The system will prompt "the virtual gunshot is on the second floor of the motel". Players move in the same building: When the player moves in the same building, the spatial query logic will simplify the feedback information and only provide information about the sub-level area. For example, the main virtual character and / or NPC is on the first floor of the motel and the enemy virtual character is on the second floor. The system will prompt "the enemy is on the second floor". Players are in the same room: When the main virtual character and / or NPC is in the same room with the enemy virtual character, the spatial query logic will provide more accurate orientation information to help players react quickly. For example, the player and the enemy are both on the second floor of the motel, and the system will prompt "the enemy is in the right front position". Through hierarchical feedback information, it helps players to understand the location of the enemy virtual character more intuitively, and provide the most relevant location information according to the different locations of the players.

[0585] In an optional embodiment, in game speech recognition, loading different language models and scene hot words according to different virtual scenes can significantly improve the accuracy and response speed of speech recognition. The following are steps and methods to achieve this goal.

[0586] First, it is necessary to be able to identify the virtual scene in the current game. The loading of the language model decoding word graph will be achieved through the following methods: (1) Game state detection: by detecting the state variables of the game (such as player position, task progress, current activity, etc.), the first virtual scene where the current main virtual character and / or NPC is located is determined; (2) Event triggering: switching scenes according to specific events (such as entering a certain area, starting a certain task), thereby determining the first virtual scene where the main virtual character and / or NPC is located; (3) Player input: the player can actively inform the current first virtual scene through voice or other input methods.

[0587] Different language models are trained or selected for different virtual scenarios. For example, the language model in the combat scenario can focus more on words and phrases related to the virtual combat, while the language model in the exploration scenario can focus more on words related to navigation and interaction.

[0588] Among them, a set of scene hot words is generated according to the virtual elements (such as virtual items, virtual characters, etc.) contained in the line of sight. These scene hot words are the most frequently triggered or important words in the current player scene. For example: in a battle scene, hot words may include "enemy", "attack", "retreat", etc., while in a transaction scene, hot words may include "buy", "sell", "item", etc.

[0589] In an illustrative manner, the corresponding language model and scene hot words are dynamically loaded according to the recognized virtual scene.

[0590] Among them, when a scene change is detected, the language model of the corresponding virtual scene is switched, which can be achieved by calling the application programming interface (API) of the speech recognition engine; in addition, when a scene change is detected, the currently used hot word list is updated so that the speech recognition engine can recognize these words more accurately.

[0591] In an optional embodiment, in the game, according to the virtual scene where the main virtual character and / or NPC is located, the decoding strategy of the speech recognition system can also be dynamically adjusted to improve the accuracy of speech recognition and user experience. The following are some methods and steps.

[0592] Indicatively, the player's current scene is first identified and classified through information such as game status, player location, and current task.

[0593] Optionally, different decoding strategies are defined according to different virtual scenarios. These strategies include: (1) vocabulary adjustment: adjusting the vocabulary of speech recognition according to the virtual scenario, that is, the selected vocabulary in the above-mentioned preset dictionary; (2) language model adjustment: using different language models to process different virtual scenarios; (3) confidence threshold adjustment: adjusting the confidence threshold of speech recognition according to the virtual scenario, such as manually raising the confidence threshold. Only when the predicted probability is higher than the confidence threshold, the candidate vocabulary sequence corresponding to the maximum preset probability is used as the instruction output result.

[0594] Among them, by dynamically adjusting the decoder during the game running process, the decoding strategy of the speech recognizer can be dynamically adjusted according to the virtual scene where the main virtual character and / or NPC is located, thereby improving the flexibility of decoding.

[0595] In an optional embodiment, after the ASR system is launched, the language model (or the sound model, etc.) can be continuously optimized and adjusted according to the actual usage of the players, so that it can be more adapted to the needs of different virtual scenes. Optimizing the language model includes the following operations: (1) Hot word adjustment: Based on player feedback and usage data, the text recognized by the ASR system is collected and automatically rewritten using the correction engine. The hot word list is dynamically adjusted through the rewritten player text to ensure that it covers the most commonly used and important vocabulary, so that the hot words of the first scene can be obtained more accurately in the future; (2) Model update: Retrain the language model and / or sound model and adjust the parameters of the model; (3) Testing and deployment: The updated model undergoes sufficient control tests to ensure its performance and stability. Once the test passes, the trained model can be deployed in the game; (4) Continuous iteration: Speech recognition is a continuous improvement process. Feedback is regularly collected and analyzed, and then the model is updated to adapt to player behavior and changes in the game. Test in the actual game environment, collect data and feedback, and continuously iterate and improve the speech recognition system to improve the accuracy of command analysis results.

[0596] In an optional embodiment, by adjusting the language model of speech recognition according to the virtual scene, it is helpful to better understand and parse the natural language commands in the form of speech, select an appropriate language model from the predefined language model library, and when the scene changes, the language model for analysis can be automatically adjusted. This process can be achieved through a preset language model management system. In addition, the system can also establish an effective feedback mechanism, players can provide feedback and correct recognition errors or add new scene hot words through the cloud-based large language model. These feedbacks will update and improve the natural language analysis model online, and adjust and update the hot words and language models in time. In addition, in order to ensure the reliability and stability of the system, the existing large language model can also be used for offline testing and evaluation. Combined with the system automatic testing and player feedback data in different scenarios, the hot words and language models are further optimized according to the test feedback data to meet the needs of different scenarios. For example: when a player enters the interior of a city room, "turn on the lights", "turn off the lights", "turn up the temperature", etc. may be the scene hot words of the virtual indoor scene.

[0597] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0598] Step 1580: Perform behavioral intention recognition based on the command analysis results output by the language model, and control the NPC to perform virtual activities through the behavioral intention.

[0599] Schematically, the behavioral intention of the command analysis results in text form is recognized through a hierarchical prediction network, so as to determine the behavioral intention corresponding to the command analysis results, and control the NPC to perform virtual activities that meet the behavioral intention through the behavioral intention.

[0600] In an optional embodiment, taking a virtual soldier including a main virtual character and a non-player virtual character in a virtual environment as an example, the process of obtaining a command analysis result in text form based on a natural language command in voice form is described as follows.

[0601] Indicative, such as Fig.16 As shown, the virtual environment includes a main virtual character 1610 manually controlled by a player, a virtual soldier 1611 (referred to as No. 1) and a virtual soldier 1612 (referred to as No. 2) controlled by voice.

[0602] Optionally, after receiving a natural language command, the first virtual scene in which the main virtual character 1610 is located is determined to be a "virtual outdoor scene", and multiple first scene hot words corresponding to the first virtual scene are obtained, including "truck", "virtual grass", "virtual house", "enter", "open the truck", "defense", "desert", "attack", "oak tree", "stable", etc.

[0603] Without the constraints of the hot words in the first scene, it is easy to recognize the natural language command as "No. 1, go to the front Voice "A moment"; however, based on the command analysis results of converting the natural language command into text form based on multiple hot words in the first scene, the command analysis result is "No. 1, you go to the front Defense "A moment"; if the behavior intention is identified as "front defense" through behavior intention recognition of the command analysis result, the virtual soldier 1611 can be specifically commanded to move forward and take a defensive posture to prevent the enemy virtual character from attacking the main control virtual character first.

[0604] Alternatively, without the constraints of the hot words in the first scenario, it is easy to recognize the natural language command as "No. 2, you go Mountains "Explore it"; however, based on multiple hot words in the first scenario, the natural language command is converted into a text form command analysis result, and the command analysis result is "No. 2, you go desert "Explore it"; if the behavior intention is identified through behavior intention recognition on the command analysis result and the behavior intention is "desert exploration", the virtual soldier 1612 can be specifically commanded to move to the desert location to explore whether there are virtual elements such as enemy virtual characters or virtual treasure chests.

[0605] Alternatively, if there is no constraint on the hot words in the first scenario, it is easy to recognize the natural language command as "No. 1, No. 2, give me for Give "; however, based on multiple hot words in the first scenario, the natural language command is converted into a text form command analysis result, and the command analysis result is "No. 1, No. 2, give me attack "; if the behavior intention is identified as "attack" through the behavior intention recognition of the command analysis result, the virtual soldier 1611 and the virtual soldier 1612 can be specifically directed to carry out virtual attacks on nearby attackable virtual characters.

[0606] Alternatively, if there is no constraint on the hot words in the first scene, it is easy to recognize the natural language command as "No. 1, find nearby Project Book "; however, based on multiple first-scene hot words, the natural language command is converted into a command analysis result in text form, and the command analysis result is obtained as "No. 1, go find a nearby oak tree"; the command analysis result is subjected to behavioral intention recognition to obtain the behavioral intention as "find an oak tree", and the virtual soldier 1611 can be specifically directed to find an oak tree in the nearby area so as to complete the game task or find an oak tree that can avoid attacks.

[0607] Alternatively, without the constraints of the hot words in the first scene, it is easy to recognize the natural language command as "No. 1, No. 2, go over there" ， Right awayOK”; however, based on multiple hot words in the first scenario, the natural language command is converted into a text-based command analysis result, and the command analysis result is “No. 1, No. 2, go over there stable OK”; the behavior intention is identified by performing behavior intention recognition on the command analysis result to obtain the behavior intention of “searching for stables”, then the virtual soldiers 1611 and 1612 can be specifically directed to search for nearby stables and move to the location of the stables.

[0608] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0609] In summary, after obtaining the natural language command in the form of voice for commanding the NPC, the environmental perception information obtained by the master virtual character and at least one of the NPCs from the virtual environment is used to identify the behavioral intention from the natural language command, so as to control the NPC to perform virtual activities through the behavioral intention. With the help of the environmental perception information of the master virtual character and / or the NPC, it is helpful to better understand the natural language command, improve the accuracy of the analysis of the natural language command, and then improve the accuracy of the recognized behavioral intention, so that the behavioral intention is more matched with the virtual environment, which greatly avoids invalid or wrong commands to the NPC, improves the accuracy of the control of the NPC, and can also provide players with a more efficient and comfortable operating environment, improving the efficiency of human-computer interaction.

[0610] An exemplary embodiment of the present application provides a method for controlling a virtual character. The method comprises:

[0611] Displaying at least one of a main control virtual character and an NPC in a virtual environment;

[0612] Exemplarily, the main virtual character is a virtual character that the user directly controls in the virtual environment. There are also one or more non-player characters (NPC) in the virtual environment; the NPC and the main virtual character belong to the same virtual camp and are teammates of the main virtual character. The NPC follows the main virtual character in the virtual environment. In addition, the NPC can also be a follower character, a pet character, etc. controlled by the main virtual character. In addition, the NPC can also be a neutral character, and only cooperates with the main virtual character when certain conditions are met.

[0613] Obtain natural language commands;

[0614] Exemplarily, the natural language command includes the natural semantics of commanding the NPC; the NPC is commanded in the virtual scene through the natural language command, and the natural language command can be directly input by the user or extracted from the voice information input by the user. The present application does not limit the method of obtaining the natural language command.

[0615] The natural language command includes at least one of a behavior intention and a scene entity; the behavior intention is used to indicate the type of virtual activity performed by the NPC, such as indicating what type of virtual activity is performed in the virtual environment; the scene entity includes a first scene object, such as indicating what scene object in the virtual environment the virtual activity is performed on. The behavior intention and scene entity included in the natural language command are partial words in the natural language command.

[0616] In one example, the natural language command controls the NPC in terms of what type of virtual activity to perform and for what scene object in the virtual environment to perform the virtual activity; it is a complex instruction to control the NPC.

[0617] Controlling NPCs to perform virtual activities in response to natural language commands;

[0618] Exemplarily, controlling the NPC to perform virtual activities is determined based on the environmental perception information of the main virtual character and / or NPC. In some examples, the main virtual character and / or NPC obtains environmental perception information within the perception range of vision, hearing, movement trajectory and other perception methods. Exemplarily, the virtual activity is close to the current perception of the virtual environment by the main virtual character and / or NPC, and the NPC is controlled with reference to the current perception of the virtual character and / or NPC of the virtual environment. Exemplarily, the NPC has autonomous behavior capabilities, and the user only directs the NPC. The user gives commands, and the NPC understands and executes the commands based on its own autonomous behavior capabilities.

[0619] To sum up, the method provided in this embodiment determines the virtual activities that need to be performed by facing complex instructions for controlling the NPC in terms of what type of virtual activities to perform and for what scene objects in the virtual environment to perform the virtual activities, with reference to the environmental perception information of the master virtual character and / or the NPC; and ensures that the NPC and the master virtual character perform virtual activities in collaboration with each other by using the environmental perception information of the master virtual character and / or the NPC as a reference.

[0620] Fig.17 A flowchart of a method for controlling a virtual character provided by an exemplary embodiment of the present application is shown. The method comprises:

[0621] Step 1701: Obtain a spatial data set;

[0622] The spatial dataset of the virtual environment includes visual labels of scene objects, and the visual labels are used to describe the visual characteristics of the scene objects in at least one dimension, such as describing the scene objects in dimensions such as material, transparency, color, and shape.

[0623] Step 1702: Display at least one of a main control virtual character and an NPC in a virtual environment;

[0624] Exemplarily, the main control virtual character is a virtual character that the user directly controls in the virtual environment. There are also one or more NPCs in the virtual environment; the NPCs and the main control virtual character belong to the same virtual camp and are teammates of the main control virtual character.

[0625] Step 1703: Acquire a natural language command in voice form;

[0626] The natural language command includes natural semantics for commanding the NPC, and the command information includes behavioral intention and scene entity; the behavioral intention is used to indicate the type of virtual activity performed by the NPC, and the scene entity is used to indicate the target entity of the virtual activity;

[0627] Step 1704: converting the natural language command in voice form into the natural language command in text form;

[0628] Exemplarily, speech-to-text (Automatic Speech Recognition, ASR) processing is performed on the natural language command in the form of speech to determine the natural language command in the form of text. Speech-to-text processing generally includes calling components such as acoustic models and language models to recognize the pronunciation, vocabulary, grammatical structure and other information of the natural language command in the form of speech, and convert it into a natural language command in the form of text.

[0629] Step 1705: performing intent recognition on the natural language command to obtain a first classification label;

[0630] Exemplarily, the first intention indication corresponding to the first classification label includes a command intention for a non-player character on an intention dimension; for example, at least one of the following: indicating which of a plurality of non-player characters the non-player character commanded by the command text is, indicating whether the virtual activity performed by the non-player character is related to initiating a virtual attack, and indicating what kind of virtual activity the non-player character performs.

[0631] Step 1706: Perform entity recognition on the natural language command to obtain a target entity;

[0632] The target entity is determined from the virtual environment in combination with the environmental perception information, which includes information perceived by at least one of the main virtual character and the non-player character from the virtual environment. The target entity may be the entity currently seen by the player, or may be an entity that the player has seen in the past.

[0633] Step 1707: In response to the natural language command, the NPC is controlled to perform a virtual activity according to the first intention corresponding to the first classification label, or the NPC is controlled to perform a virtual activity associated with the target entity, or the NPC is controlled to perform a virtual activity associated with the target entity according to the first intention corresponding to the first classification label;

[0634] Exemplarily, the NPC is controlled to perform a virtual activity according to the first classification tag and / or the instruction of the target entity.

[0635] Step 1708: reporting the NPC's feedback information;

[0636] Exemplarily, the game application can generate corresponding feedback information in real time according to the environmental perception information of the non-player character and make a broadcast. For example, when the environmental perception information of the non-player character triggers the broadcast condition, the game application can generate corresponding feedback information in real time according to the environmental perception information that triggers the broadcast condition; or, when the non-player character or the main control virtual character triggers the broadcast condition, the game application can generate corresponding feedback information in real time according to the environmental perception information of the non-player character.

[0637] The preprocessing stage of step 1701 can be implemented as follows:

[0638] Sub-step 1: Obtaining attribute text of scene objects in the virtual scene;

[0639] Exemplarily, attribute text is used to introduce the inherent attributes of scene objects in a virtual scene; on the one hand, attribute text realizes the description of scene objects in a textual mode, and while describing scene objects, it provides semantic information for the prediction of the visual labels of scene objects. On the other hand, attribute text is the description of scene objects in a virtual scene. In the face of a large number of scene objects in a virtual scene and the existence of reused object models, it can more accurately describe the inherent attributes of scene objects in a virtual scene.

[0640] In an optional implementation, the attribute text includes at least one of a name of the scene object in the virtual scene and a size of the scene object in the virtual scene.

[0641] Sub-step 2: obtaining an appearance image of a scene object in a virtual scene;

[0642] Exemplarily, the appearance image is used to describe the style of the scene object; the appearance image carries the appearance style of the scene object such as color, texture, shape, etc., and the relative position relationship between each sub-part in the picture mode, and can fully describe the scene object from the picture mode (or visual mode). In an optional implementation, the appearance image of the scene object includes images obtained by observing the scene object from at least two perspectives.

[0643] Sub-step 3: Call the multimodal model to perform prediction on the attribute text and appearance image of the scene object to obtain the visual label of the scene object;

[0644] Exemplarily, the multimodal model has the ability to perform model prediction on text information and image information of different modalities. In this embodiment, the input parameters of the multimodal model are the attribute text and appearance image of the scene object; the multimodal model predicts the visual label of the scene object from the two modalities of text modality and image modality, and the visual label is used to describe the visual features of the scene object in at least one dimension.

[0645] Optionally, the multimodal model includes a visual question answering model; the question statement carries attribute text of the scene objects; on the one hand, the question statement is used to guide the visual question answering model to convert the appearance image into the visual label of the scene object; on the other hand, while guiding the conversion of the visual label of the scene object, the question statement provides supplementary information of the scene object in text form.

[0646] Get the expected information of scene objects;

[0647] Exemplarily, the expected information is used to indicate the description dimension of the scene object expected in the visual tag, and / or the expected format of the visual tag; in one example, the expected information is used to indicate that the description dimension of the scene object expected in the visual tag includes but is not limited to at least one of the type description, material, transparency, color, surface features, and shape of the scene object. In another example, the expected information is used to indicate that the expected format of the visual tag is at least one of the following: Comma Separated Values ​​(CSV), JavaScript Object Notation (JSON), and eXtensible Markup Language (XML).

[0648] Construct a question statement of the appearance image based on the expected information and attribute text;

[0649] Exemplarily, the first subpart of the question sentence is supplementary introduction information of the scene object, carrying attribute text of the scene object; the second subpart of the question sentence is an answer guidance sentence for the visual question answering model, carrying expected information.

[0650] Optionally, the spatial data set includes matching labels of scene objects in the virtual scene; correspondingly: performing pseudo-spoken rewriting on the visual labels of the scene objects to obtain matching labels that conform to spoken expressions of natural language.

[0651] Exemplarily, as described above, visual labels are used to describe the visual features of scene objects in at least one dimension, and the appearance image of the scene objects presents rich visual features of the scene objects. In the spoken expression of natural language, however, the description of scene objects cannot cover the visual features of scene objects in all dimensions. The purpose of performing a pseudo-colloquial rewrite on the visual labels of scene objects is to obtain a matching label that is closer to the spoken expression; it can be understood that performing a pseudo-colloquial rewrite can be to delete part of the content in the visual label, or to change the visual label to a label with the same semantics but different textual expression.

[0652] In an optional implementation, the pseudo-colloquial rewriting is performed by calling a natural language model; the visual labels of the scene objects are input into the natural language model, and matching labels that conform to the spoken expressions of the natural language are predicted. The pseudo-colloquial rewriting of the visual labels is implemented based on calling the natural language model. Exemplarily, the natural language model carries prior knowledge of the spoken expressions of the natural language. For example, the natural language model is implemented as a large language model (Large Language Model, LLM).

[0653] Optionally, the spatial dataset also includes spatial information of scene objects, and accordingly, also includes:

[0654] The spatial position of the scene object in the virtual scene is obtained, and the spatial position of the scene object is determined as auxiliary information of the visual label of the scene object.

[0655] Exemplarily, the spatial position of the scene object in the virtual scene is used to indicate the deployment of the scene object in the virtual scene, and the spatial information is used to indicate the position, size and other information of the scene object in the virtual scene after the scene object is deployed in the virtual scene.

[0656] Further, obtaining at least one of the coordinate position, orientation information, bounding box information, and cover point information of the scene object in the virtual scene;

[0657] Among them, the coordinate position (Location) is used to indicate the position of the scene object in the virtual scene, such as the coordinate information of the center point or preset point of the scene object in the virtual scene; the orientation (Rotation) is used to indicate the direction facing the scene object in the virtual scene, such as the direction facing the front of the scene object in the virtual scene; the bounding box (BoundingBox) is used to indicate the size of the scene object in the virtual scene; the cover points (Cover points) are used to indicate the recommended virtual character position points when the virtual character approaches the scene object, so that the scene object can mask the virtual character.

[0658] The intention recognition phase of step 1705 can be implemented as follows:

[0659] Sub-step 4: calling at least one hierarchical prediction network to perform intent recognition on the natural language command, and obtaining a first classification label of the natural language command intent to command the NPC;

[0660] Exemplarily, the hierarchical prediction network has the ability to predict the classification label corresponding to the command text. Exemplarily, the hierarchical prediction network includes at least two layers of sub-networks, the upper network and the lower network are cascaded, and the lower network further performs the prediction of the classification label according to the prediction result output by the upper network. At least two layers of sub-networks are constructed based on the tree structure of multiple classification labels. The hierarchical prediction network constructs at least two layers of sub-networks corresponding to the hierarchical structure for the tree structure of the classification labels, and splits the prediction task of a wide variety of classification labels into prediction sub-tasks performed by at least two layers of sub-networks, so as to reduce the complexity of classification prediction of each layer of sub-network.

[0661] In an optional implementation, each hierarchical prediction network in at least one hierarchical prediction network is used to predict the command intention of the command text for a non-player character on an intention dimension; the intention dimension includes: at least one of the subject dimension, the semantic dimension, and the behavior dimension. Exemplarily, in at least one hierarchical prediction network, the i-th layer subnetwork in each hierarchical prediction network is used to predict the first-level behavior label of the command text on an intention dimension, and the i+1-th layer subnetwork in the hierarchical prediction network is used to predict the second-level behavior label corresponding to the command text in the first-level behavior label, i is a positive integer; the high-level behavior label (such as the first-level behavior label) predicted by the high-level subnetwork (such as the i-th layer subnetwork) in the hierarchical prediction network includes multiple sublabels, or subordinate lower-level labels; the corresponding low-level subnetwork (such as the i+1-th layer subnetwork) needs to be called to further perform classification prediction (such as predicting the second-level behavior label). The prediction task of a wide variety of classification labels is split into prediction subtasks performed by at least two layers of subnetworks to reduce the complexity of classification prediction of each layer of subnetwork.

[0662] Furthermore, the intent dimension includes a subject dimension, the hierarchical prediction network includes a hierarchical subject prediction network, and the subject prediction network has the ability to predict the subject type in a natural language command; the subject type is used to indicate the identity of the NPC commanded by the natural language command; exemplarily, the subject prediction network is used to predict which of a plurality of non-player characters the non-player character commanded by the command text is, and the number of non-player characters commanded by the command text can be one or more.

[0663] And / or, the intent dimension includes a semantic dimension, the hierarchical prediction network includes a hierarchical semantic prediction network, the semantic prediction network has the ability to predict the semantic type in a natural language command; the semantic type is used to indicate the NPC's control method for initiating a virtual attack; exemplarily, the subject prediction network is used to predict whether the virtual activity performed by a non-player character commanded by the command text is related to initiating a virtual attack.

[0664] And / or, the intention dimension includes a behavior dimension, the hierarchical prediction network includes a hierarchical behavior prediction network, the behavior prediction network has the ability to predict the behavior type in the natural language command; the behavior type is used to indicate the behavior mode of the NPC for performing the virtual activity. Exemplarily, the behavior prediction network is used to predict what virtual activity the non-player character performs.

[0665] Accordingly, in step 1707, in response to the natural language command, the NPC is controlled to perform a virtual activity according to the first intention corresponding to the first classification label;

[0666] Exemplarily, the first intention corresponding to the first classification tag indicates a command intention for a non-player character in an intention dimension, such as at least one of the following: indicating which non-player character is commanded by the command text among a plurality of non-player characters, indicating whether the virtual activity performed by the non-player character is related to launching a virtual attack, and indicating what virtual activity the non-player character performs. According to the instruction of the first classification tag, the NPC is controlled to perform the virtual activity.

[0667] The target entity identification process of step 1706 can be implemented as follows:

[0668] Sub-step 5: querying and obtaining the target entity from the entity set of the virtual environment according to the target entity information of the target entity indicated by the natural language command and the environment perception information;

[0669] The target entity is determined from the virtual environment in combination with environmental perception information, and the environmental perception information includes information perceived by the main virtual character and / or the non-player character from the virtual environment.

[0670] Since the target entity information in natural language commands is usually vague, for example, a natural language command may be "move to the red truck", where the target entity information is "red truck", and there may be many red trucks in the virtual environment. It is impossible to accurately determine the target entity in the virtual environment based solely on the target entity information in the natural language command.

[0671] Therefore, the embodiment of the present application provides a method for determining the target entity by combining the target entity information and the environmental perception information. For the previous example, the "red truck" expressed by the player in the natural language command should be a red truck that the player can see. Therefore, combined with the field of view of the main control virtual character, the red truck within the field of view of the main control system character can be screened out from the multiple red trucks in the virtual environment. The red truck is the target entity indicated by the player in the natural language command.

[0672] Since natural language commands are issued by players based on their perception of the virtual environment, in order to accurately identify the target entity indicated in the natural language command, the game application will combine the environmental perception information when the player issues the natural language command to identify the target entity in the natural language command. Based on the player's perception of the virtual environment, the entity that the player can perceive in the virtual environment and is closest to the target entity information is inferred as the target entity.

[0673] Exemplarily, the virtual environment includes multiple candidate entities that match the natural language command, and the target entity is an entity selected from the multiple candidate entities based on the environmental perception information of the main virtual character or the non-player character. For example, the game application first selects multiple candidate entities that match the target entity information in the natural language command from the entity set, and then selects the entity that can be perceived by the main virtual character or the non-player character from the multiple candidate entities as the target entity based on the environmental perception information.

[0674] For example, the environmental perception information of the main virtual character may include: the pictures or entities that the main virtual character can see when observing the virtual environment, the environmental sound effects that the main virtual character can hear, the source direction of the environmental sound effects, the type and volume of the environmental sound effects, the perception information obtained by the main virtual character by using perception skills (for example, visual information, auditory information, sound wave information, light reflection information, etc.), and the perception information obtained by the main virtual character by using sensing props (for example, sensor signals from sensors, positioning signals from positioning props, etc.).

[0675] The environmental perception information of the non-player character may include: the images or entities that the non-player character can see when observing the virtual environment, the environmental sounds that the non-player character can hear, the source direction of the environmental sounds, the type and volume of the environmental sounds, the perception information obtained by the non-player character by using perception skills (for example, visual information, auditory information, sound wave information, light reflection information, etc.), and the perception information obtained by the non-player character by using sensory props (for example, sensor signals of sensors, positioning signals of positioning props, etc.).

[0676] It should be noted that the environmental perception information may include the information perceived by the controlling virtual character and the non-player character in real time when receiving the natural language command, and may also include the information perceived by the controlling virtual character and the non-player character in history before receiving the natural language command. That is, the target entity may be the entity currently seen by the player, or it may be the entity that the player has seen in the past. Therefore, the game application needs to combine the real-time environmental perception information and the historical environmental perception information of the controlling virtual character and / or the non-player character to identify the target entity.

[0677] For example, when the natural language command is "Let's go back to the hotel we just passed by", the game application needs to query the hotels that the main virtual character has passed by based on the historical environmental perception information of the main virtual character.

[0678] In an optional embodiment, the client queries and obtains the target entity from the entity set of the virtual environment according to the target entity information and the environment perception information indicated by the natural language command. In another optional embodiment, the client reports the natural language command to the server, and the server queries and obtains the target entity from the entity set of the virtual environment according to the target entity information and the environment perception information indicated by the natural language command.

[0679] Optionally, the game application infers the target entity based on the target entity information, the environment perception information and the pre-processed virtual environment data, wherein the pre-processed virtual environment data includes a set of entities in the virtual environment.

[0680] The entity set includes entity information of each entity in the virtual environment. The entity information includes at least one of the following: name, type, location, feature, text label, embedding vector of text label, image, embedding vector of image.

[0681] The text label may be at least one word obtained after segmenting a feature (eg, a descriptive text of an entity appearance), and the embedding model is called to obtain an embedding vector corresponding to each text label.

[0682] The image may include an image obtained by observing a three-dimensional model of the entity from at least one direction, for example, the image may include three views of the entity. The embedding vector of the image is the embedding vector of the image feature label. The image feature label is obtained by calling the multimodal model to perform feature recognition based on the image and text label of the entity.

[0683] Exemplarily, the entity set includes a first type of entity and a second type of entity, the first type of entity includes at least one of an area and a building, and the second type of entity includes a physical object. The text labels of the first type of entity are obtained by manual labeling. For example, the text label of a room of a building is manually labeled as: an area, a building, a floor, and a room. The text labels of the second type of entity are automatically generated by calling a large language model.

[0684] The method for generating text labels for the second type of entity may include: obtaining at least one image of the entity, inputting the at least one image into a large language model, and obtaining a description text of the entity; segmenting the description text to obtain at least one text label of the entity, each text label including a word obtained after the segmentation process; performing a vectorization operation on the text label to obtain an embedding vector corresponding to the text label.

[0685] The embedding vector is the high-dimensional vector data of the text label. By pre-generating the embedding vector of the text label, there is no need to repeatedly generate vectors for the text label of each entity when matching the target entity from the entity set based on the target entity information, which can improve the matching efficiency of the target entity information and the entity information.

[0686] It should be noted that this method does not use the description text directly as the text label, but uses the word segmentation result of the description text as the text label. This is because the description text is usually of different lengths. For longer description texts, their embedding vectors are less effective in actual search and similarity comparison. For example, when searching for "a small blue truck with rusty body" using "truck", "a small blue truck with rusty body" is likely to be ranked after "a car" in the recall results. In fact, for the searched "truck", no matter how complex its additional description is, "truck" should be searched first. Therefore, when performing similarity search, the comparison should be one feature word at a time, not the description text.

[0687] In addition, in order to further extract the image features of the entity, the method provided in the embodiment of the present application can also further extract hidden visual features in the image of the entity based on the text label and image of the entity. Taking the first entity as an example, the method includes: obtaining at least one perspective image of the first entity; obtaining at least one text label of the first entity; calling the multimodal model to extract the image features of the first entity based on at least one perspective image and at least one text label of the first entity, and obtain the image embedding vector of the first entity.

[0688] Since text labels are manually annotated, or generated by a large language model based on human language characteristics; and text labels extracted based on human language habits may ignore some features of the entity. For example, when manually annotating a text label for an oil drum, only more eye-catching text labels such as "metal" and "rusty" may be annotated, while detailed features such as "rusty", "blue", "yellow", and "paint peeling in the lower right corner" are ignored. Therefore, the method provided in the embodiment of the present application also uses a multimodal model based on the text label and entity image of the entity, so that the multimodal model extracts more image features from it; performs image similarity matching with the target entity based on the image features to improve the recognition accuracy of the target entity.

[0689] Among them, the training method of the multimodal model can be: inputting sample entity images and sample labels into the pre-trained multimodal model, fine-tuning the pre-trained multimodal model according to the loss of the predicted labels and sample labels output by the pre-trained multimodal model, and obtainin...

Claims

1. A method for controlling a virtual character, characterized in that: The method comprises: Displaying at least one of a main control virtual character and an NPC in a virtual environment; Acquire a natural language command in speech form, wherein the natural language command includes natural semantics for instructing the NPC; In response to the behavioral intention indicated by the natural language command, controlling the NPC to perform the virtual activity; The behavioral intention is identified from the natural language command based on environmental perception information, and the environmental perception information includes information perceived by at least one of the main virtual character and the NPC about the virtual environment.

2. The method according to claim 1, characterized in that The step of controlling the NPC to perform the virtual activity in response to the behavioral intention indicated by the natural language command includes: Convert the natural language command into a command analysis result in text form, wherein the command analysis result is a result of converting the natural language command speech into text based on the environmental perception information; wherein the command analysis result is used to obtain the behavior intention through behavior intention recognition, and the behavior intention is the intention information expressed in the command analysis result; The NPC is controlled to perform the virtual activity based on the behavioral intention.

3. The method according to claim 2, characterized in that The command analysis result of converting the natural language command into text form includes: Based on the first virtual scene where the main control virtual character and / or the NPC is located, a plurality of first scene hot words are obtained, where the plurality of first scene hot words are scene-related words of the first virtual scene; The command analysis result converts the natural language command into text form based on the multiple first scene hot words.

4. The method according to claim 3, characterized in that The acquiring of a plurality of first scene hot words based on the first virtual scene where the main control virtual character and / or the NPC is located includes: Obtaining a first scene type corresponding to the first virtual scene where the master virtual character and / or the NPC is located; obtaining a hot word set corresponding to the first scene type, and using the words in the hot word set as the multiple first scene hot words, wherein the hot word set is a set of scene-related words collected based on the first scene type; or, Acquire the object perception range of the master virtual character and / or the NPC in the first virtual scene, where the object perception range is the three-dimensional space range in which the master virtual character and / or the NPC perceives the existence of other scene elements; acquire the multiple first scene hot words based on the environmental perception information within the object perception range; or, Obtain a first scene type corresponding to the first virtual scene; obtain a hot word set corresponding to the first scene type; obtain an object perception range of the master virtual character and / or the NPC in the first virtual scene; and obtain at least two words from the hot word set as the multiple first scene hot words based on environmental perception information within the object perception range.

5. The method according to claim 4, characterized in that The object perception range includes at least one of the following: The visual perception range of the master virtual character and / or the NPC; The auditory perception range of the master virtual character and / or the NPC; The olfactory perception range of the master virtual character and / or the NPC; The range of perception of the perception skills or perception tools possessed by the main virtual character and / or the NPC.

6. The method according to claim 5, characterized in that The obtaining of the object perception range of the master virtual character and / or the NPC in the first virtual scene includes: In the case where the object perception range includes a visual perception range, a preset visual cone is generated as the visual perception range with the position of the main virtual character and / or the NPC as the starting point and the direction of the main virtual character and / or the NPC as the center line of the cone. The preset visual cone is a three-dimensional area that measures the visual perception range.

7. The method according to claim 3, characterized in that The environmental perception information includes virtual elements; The acquiring of a plurality of first scene hot words based on the first virtual scene where the main control virtual character and / or the NPC is located includes: Highlighting the virtual elements in the first virtual scene that are within the perception range of the object; The element name of the virtual element is used as the first scene hot word; or the action name of the interactive action corresponding to the virtual element is used as the first scene hot word.

8. The method according to claim 3, characterized in that The first virtual scene is one of multiple virtual scenes in the virtual environment; The method further comprises: Based on the position of the master virtual character and / or the NPC in the virtual environment, the first virtual scene where the master virtual character and / or the NPC is located is determined from the multiple virtual scenes.

9. The method according to claim 8, characterized in that The method further comprises: In the case where the command analysis result is not obtained, in response to the main control virtual character and / or the NPC moving from the first virtual scene to the second virtual scene, obtaining a plurality of second scene hot words corresponding to the second virtual scene; The command analysis result converts the natural language command into text form based on the multiple second scene hot words.

10. The method according to claim 2, characterized in that After converting the natural language command into the command analysis result in text form, the method further includes: Display the command analysis results in text form; The method further comprises: The reply sentence of the NPC in reply to the command analysis result is displayed, and the reply sentence is a text sentence generated after semantic analysis is performed based on the command analysis result.

11. The method according to claim 3, characterized in that The acquiring of a plurality of first scene hot words based on the first virtual scene where the main control virtual character and / or the NPC is located includes: Acquire a plurality of scene hot words, where the plurality of scene hot words correspond to at least two virtual scenes; Based on the first virtual scene where the main control virtual character and / or the NPC is located, the multiple first scene hot words corresponding to the first virtual scene are obtained from the multiple scene hot words.

12. The method according to claim 11, characterized in that The multiple scene hot words correspond to scene identifiers, and the scene identifiers are used to represent virtual scenes when collecting scene hot words; The acquiring, based on the first virtual scene in which the master virtual character and / or the NPC is located, the plurality of first scene hot words corresponding to the first virtual scene from the plurality of scene hot words comprises: Based on the first virtual scene where the main control virtual character and / or the NPC is located, a plurality of candidate scene hot words with a first scene identifier are obtained from the plurality of scene hot words, where the first scene identifier is a scene identifier corresponding to the first virtual scene; Based on the object perception range of the main control virtual character and / or the NPC in the first virtual scene, at least two candidate scene hot words are selected from the multiple candidate scene hot words as the first scene hot words.

13. The method according to claim 3, characterized in that The command analysis result of converting the natural language command into text form based on the multiple first scene hot words includes: Acquire a pre-trained natural language analysis model, wherein the pre-trained natural language analysis model includes an acoustic network, a language network, and a preset dictionary, wherein the preset dictionary includes the plurality of first scene hot words, other scene hot words, and common vocabulary; Acquiring a plurality of speech units corresponding to the natural language command through the acoustic network, wherein the speech unit is a basic unit of vocabulary pronunciation; The command analysis result in text form is obtained by analyzing the sequence matching relationship between the multiple voice units and the selected words in the preset dictionary through the language network, and the selected words at least include the multiple first scene hot words and the common words.

14. The method according to claim 13, characterized in that The selected vocabulary also includes the other scene hot words; Among them, the vocabulary in the preset dictionary includes analysis weights, the first analysis weights of the multiple first-scene hot words are higher than the second analysis weights of the other scene hot words, and the analysis weight refers to the degree of attention paid to the vocabulary when it participates in sequence matching.

15. The method according to claim 13, characterized in that The step of analyzing the sequence matching relationship between the plurality of speech units and the selected words in the preset dictionary through the language network to obtain the command analysis result in text form includes: Analyzing the matching relationship between the multiple speech units and the selected words in the preset dictionary to obtain multiple candidate word sequences, wherein the multiple candidate word sequences include at least one word from the multiple selected words, and the candidate word sequences are word sequences obtained based on the change relationship between the multiple speech units; The sequence semantics of the plurality of candidate vocabulary sequences are analyzed by the language network, and at least one candidate vocabulary sequence is obtained from the plurality of candidate vocabulary sequences as the command analysis result in text form.

16. The method according to claim 14, characterized in that The language network includes language sub-networks corresponding to at least two virtual scenes respectively; The step of analyzing the sequence semantics of the plurality of candidate vocabulary sequences by using the language network and obtaining at least one candidate vocabulary sequence from the plurality of candidate vocabulary sequences as the command analysis result in text form includes: Based on the first virtual scene where the main control virtual character and / or the NPC is located, determining a first language sub-network corresponding to the first virtual scene from at least two language sub-networks; The sequence semantics of the plurality of candidate vocabulary sequences are analyzed by the first language sub-network, and at least one candidate vocabulary sequence is obtained from the plurality of candidate vocabulary sequences as the command analysis result in text form.

17. The method according to claim 2, characterized in that The NPC has multiple behavioral capabilities in the virtual scene, and the multiple behavioral capabilities correspond to multiple classification labels; The method further comprises: Calling at least one hierarchical prediction network to perform behavioral intention recognition on the command analysis result, and obtaining a first classification label of the command analysis result that intends to command the NPC as the behavioral intention; the hierarchical prediction network includes at least two layers of sub-networks, and the at least two layers of sub-networks are constructed based on the tree structure of the multiple classification labels; The controlling the NPC to perform the virtual activity based on the behavioral intention includes: The NPC is controlled to perform the virtual activity according to the first intention corresponding to the first classification tag.

18. The method according to claim 17, characterized in that The behavior intention includes at least one intention dimension; each hierarchical prediction network in the at least one hierarchical prediction network is used to predict the command intention of the command analysis result for the NPC on one intention dimension; The i-th layer sub-network in each hierarchical prediction network is used to predict the first-level behavior label of the command analysis result on the one intention dimension, and the i+1-th layer sub-network in the hierarchical prediction network is used to predict the second-level behavior label corresponding to the command analysis result in the first-level behavior label, where i is a positive integer; The intention dimension includes at least one of a subject dimension, a semantic dimension, and a behavior dimension.

19. The method according to claim 18, characterized in that The intention dimension includes the subject dimension, the hierarchical prediction network includes a hierarchical subject prediction network, and the subject prediction network has the ability to predict the subject type in the command analysis result; the subject type is used to indicate the identity of the NPC commanded by the command analysis result; And / or, the intention dimension includes the semantic dimension, the hierarchical prediction network includes a hierarchical semantic prediction network, the semantic prediction network has the ability to predict the semantic type in the command analysis result; the semantic type is used to indicate the control method of the NPC for launching a virtual attack; And / or, the intention dimension includes the behavior dimension, the hierarchical prediction network includes a hierarchical behavior prediction network, and the behavior prediction network has the ability to predict the behavior type in the command analysis results; the behavior type is used to indicate the behavior mode of the NPC in performing virtual activities.

20. A method for controlling a virtual character, characterized in that: The method is executed by a server, and comprises: Acquire a natural language command in speech form, wherein the natural language command includes natural semantics for instructing the NPC; Convert the natural language command into a command analysis result in text form, wherein the command analysis result is a result of converting the natural language command voice into text based on environmental perception information, wherein the environmental perception information includes information perceived by at least one of the main control virtual character and the NPC about the virtual environment; Performing behavior intention recognition on the command analysis result to obtain behavior intention, wherein the behavior intention is intention information expressed in the command analysis result, and the behavior intention is used to control the NPC to perform virtual activities.

21. The method according to claim 20, characterized in that The command analysis result of converting the natural language command into text form includes: Based on the first virtual scene where the main control virtual character and / or the NPC is located, a plurality of first scene hot words are obtained, where the plurality of first scene hot words are scene-related words of the first virtual scene; The command analysis result converts the natural language command into text form based on the multiple first scene hot words.

22. The method according to claim 21, characterized in that The acquiring of a plurality of first scene hot words based on the first virtual scene where the main control virtual character and / or the NPC is located includes: Obtaining a first scene type corresponding to the first virtual scene where the master virtual character and / or the NPC is located; obtaining a hot word set corresponding to the first scene type, and using the words in the hot word set as the multiple first scene hot words, wherein the hot word set is a set of scene-related words collected based on the first scene type; or, Acquire the object perception range of the master virtual character and / or the NPC in the first virtual scene, where the object perception range is the three-dimensional space range in which the master virtual character and / or the NPC perceives the existence of other scene elements; acquire the multiple first scene hot words based on the environmental perception information within the object perception range; or, Obtain a first scene type corresponding to the first virtual scene; obtain a hot word set corresponding to the first scene type; obtain an object perception range of the master virtual character and / or the NPC in the first virtual scene; and obtain at least two words from the hot word set as the multiple first scene hot words based on environmental perception information within the object perception range.

23. The method according to claim 22, characterized in that The object perception range includes at least one of the following: The visual perception range of the master virtual character and / or the NPC; The auditory perception range of the master virtual character and / or the NPC; The olfactory perception range of the master virtual character and / or the NPC; The range of perception of the perception skills or perception tools possessed by the main virtual character and / or the NPC.

24. The method according to claim 23, characterized in that The obtaining of the object perception range of the master virtual character and / or the NPC in the first virtual scene includes: In the case where the object perception range includes a visual perception range, a preset visual cone is generated as the visual perception range with the position of the main virtual character and / or the NPC as the starting point and the direction of the main virtual character and / or the NPC as the center line of the cone. The preset visual cone is a three-dimensional area that measures the visual perception range.

25. The method according to claim 21, characterized in that The environmental perception information includes virtual elements; The acquiring of a plurality of first scene hot words based on the first virtual scene where the main control virtual character and / or the NPC is located includes: Determining the virtual elements in the first virtual scene that are within the perception range of the object; The element name of the virtual element is used as the first scene hot word; or the action name of the interactive action corresponding to the virtual element is used as the first scene hot word.

26. The method according to claim 21, characterized in that The first virtual scene is one of multiple virtual scenes in the virtual environment; The method further comprises: Based on the position of the master virtual character and / or the NPC in the virtual environment, the first virtual scene where the master virtual character and / or the NPC is located is determined from the multiple virtual scenes.

27. The method according to claim 26, characterized in that The method further comprises: In the case where the command analysis result is not obtained, in response to the main control virtual character and / or the NPC moving from the first virtual scene to the second virtual scene, obtaining a plurality of second scene hot words corresponding to the second virtual scene; The command analysis result converts the natural language command into text form based on the multiple second scene hot words.

28. The method according to claim 21, characterized in that The acquiring of a plurality of first scene hot words based on the first virtual scene where the main control virtual character and / or the NPC is located includes: Acquire a plurality of scene hot words, where the plurality of scene hot words correspond to at least two virtual scenes; Based on the first virtual scene where the main control virtual character and / or the NPC is located, the multiple first scene hot words corresponding to the first virtual scene are obtained from the multiple scene hot words.

29. The method according to claim 28, characterized in that The multiple scene hot words correspond to scene identifiers, and the scene identifiers are used to represent virtual scenes when collecting scene hot words; The acquiring, based on the first virtual scene in which the master virtual character and / or the NPC is located, the plurality of first scene hot words corresponding to the first virtual scene from the plurality of scene hot words comprises: Based on the first virtual scene where the main control virtual character and / or the NPC is located, a plurality of candidate scene hot words with a first scene identifier are obtained from the plurality of scene hot words, where the first scene identifier is a scene identifier corresponding to the first virtual scene; Based on the object perception range of the main control virtual character and / or the NPC in the first virtual scene, at least two candidate scene hot words are selected from the multiple candidate scene hot words as the first scene hot words.

30. The method according to claim 21, characterized in that The command analysis result of converting the natural language command into text form based on the multiple first scene hot words includes: Acquire a pre-trained natural language analysis model, wherein the pre-trained natural language analysis model includes an acoustic network, a language network, and a preset dictionary, wherein the preset dictionary includes the plurality of first scene hot words, other scene hot words, and common vocabulary; Acquiring a plurality of speech units corresponding to the natural language command through the acoustic network, wherein the speech unit is a basic unit of vocabulary pronunciation; The command analysis result in text form is obtained by analyzing the sequence matching relationship between the multiple voice units and the selected words in the preset dictionary through the language network, and the selected words at least include the multiple first scene hot words and the common words.

31. The method according to claim 30, characterized in that The selected vocabulary also includes the other scene hot words; Among them, the vocabulary in the preset dictionary includes analysis weights, the first analysis weights of the multiple first-scene hot words are higher than the second analysis weights of the other scene hot words, and the analysis weight refers to the degree of attention paid to the vocabulary when it participates in sequence matching.

32. The method according to claim 30, characterized in that The step of analyzing the sequence matching relationship between the plurality of speech units and the selected words in the preset dictionary through the language network to obtain the command analysis result in text form includes: Analyzing the matching relationship between the multiple speech units and the selected words in the preset dictionary to obtain multiple candidate word sequences, wherein the multiple candidate word sequences include at least one word from the multiple selected words, and the candidate word sequences are word sequences obtained based on the change relationship between the multiple speech units; The sequence semantics of the plurality of candidate vocabulary sequences are analyzed by the language network, and at least one candidate vocabulary sequence is obtained from the plurality of candidate vocabulary sequences as the command analysis result in text form.

33. The method according to claim 32, characterized in that The language network includes language sub-networks corresponding to at least two virtual scenes respectively; The step of analyzing the sequence semantics of the plurality of candidate vocabulary sequences by using the language network and obtaining at least one candidate vocabulary sequence from the plurality of candidate vocabulary sequences as the command analysis result in text form includes: Based on the first virtual scene where the main control virtual character and / or the NPC is located, determining a first language sub-network corresponding to the first virtual scene from at least two language sub-networks; The sequence semantics of the plurality of candidate vocabulary sequences are analyzed by the first language sub-network, and at least one candidate vocabulary sequence is obtained from the plurality of candidate vocabulary sequences as the command analysis result in text form.

34. The method according to claim 20, characterized in that The NPC has multiple behavioral capabilities in the virtual scene, and the multiple behavioral capabilities correspond to multiple classification labels; The performing behavior intention recognition on the command analysis result to obtain the behavior intention includes: Calling at least one hierarchical prediction network to perform behavioral intention recognition on the command analysis result, and obtaining a first classification label of the command analysis result that intends to command the NPC as the behavioral intention; The hierarchical prediction network includes at least two layers of sub-networks, and the at least two layers of sub-networks are constructed based on the tree structure of the multiple classification labels.

35. The method according to claim 34, characterized in that The behavior intention includes at least one intention dimension; each hierarchical prediction network in the at least one hierarchical prediction network is used to predict the command intention of the command analysis result for the NPC on one intention dimension; The i-th layer sub-network in each hierarchical prediction network is used to predict the first-level behavior label of the command analysis result on the one intention dimension, and the i+1-th layer sub-network in the hierarchical prediction network is used to predict the second-level behavior label corresponding to the command analysis result in the first-level behavior label, where i is a positive integer; The intention dimension includes at least one of a subject dimension, a semantic dimension, and a behavior dimension.

36. The method according to claim 35, characterized in that The intention dimension includes the subject dimension, the hierarchical prediction network includes a hierarchical subject prediction network, and the subject prediction network has the ability to predict the subject type in the command analysis result; the subject type is used to indicate the identity of the NPC commanded by the command analysis result; And / or, the intention dimension includes the semantic dimension, the hierarchical prediction network includes a hierarchical semantic prediction network, the semantic prediction network has the ability to predict the semantic type in the command analysis result; the semantic type is used to indicate the control method of the NPC for launching a virtual attack; And / or, the intention dimension includes the behavior dimension, the hierarchical prediction network includes a hierarchical behavior prediction network, and the behavior prediction network has the ability to predict the behavior type in the command analysis results; the behavior type is used to indicate the behavior mode of the NPC in performing virtual activities.

37. A control device for a virtual character, characterized in that: The device comprises: A display module, used to display at least one of a main control virtual character and an NPC in a virtual environment; An acquisition module, used for acquiring a natural language command in a voice form, wherein the natural language command includes natural semantics for indicating the NPC; A control module is used to control the NPC to perform the virtual activity in response to the behavioral intention indicated by the natural language command; wherein the behavioral intention is identified from the natural language command based on environmental perception information, and the environmental perception information includes information perceived by at least one of the main virtual character and the NPC about the virtual environment.

38. A control device for a virtual character, characterized in that: The device comprises: An acquisition module, used for acquiring a natural language command in a voice form, wherein the natural language command includes natural semantics for instructing the NPC; A conversion module, configured to convert the natural language command into a command analysis result in text form, wherein the command analysis result is a result of converting the natural language command voice into text based on environment perception information, wherein the environment perception information includes information perceived by at least one of the main control virtual character and the NPC about the virtual environment; The recognition module is used to perform behavioral intention recognition on the command analysis result to obtain the behavioral intention, wherein the behavioral intention is the intention information expressed in the command analysis result, and the behavioral intention is used to control the NPC to perform the virtual activity.

39. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the control method of the virtual character as described in any one of claims 1 to 36.

40. A computer-readable storage medium, characterized in that The storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the control method of the virtual character as described in any one of claims 1 to 36.

41. A computer program product, characterized in that It comprises computer instructions, which, when executed by a processor, implement the control method of a virtual character as described in any one of claims 1 to 36.

Images