Non-player character broadcasting method and device, equipment, medium and product
By using natural language commands and environment-aware information to generate feedback in the game, the problem of low interaction efficiency of non-player characters in complex scenarios is solved, the feedback content and interaction flexibility of NPCs are improved, and the immersion and interactive experience of the game is enhanced. .
Patent Information
- Application Number
- CN202411095466.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-22
AI Technical Summary
The human-computer interaction of non-player characters in existing games is inefficient and cannot provide effective feedback in multiple complex scenarios, resulting in the inability to interact with NPCs by players and NPCs.
By realizing the reception and processing of natural language commands in a three-dimensional virtual environment, non-player characters generate non-fixed text feedback based on environment perception information, providing voice broadcast based on environment perception, and enhancing NPC feedback content and interactive flexibility.
It improves the human-computer interaction efficiency between non-player characters and players, enhances the immersion and interactive experience of the game, and makes NPC feedback more realistic and timely, adapting to different environments and combat situations.
Smart Images

Figure CN118987618A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of human-computer interaction, and more particularly to a method, device, equipment, medium and product for broadcasting a non-player character. Background Art
[0002] In the field of game design and development, AI (Artificial Intelligence) controlling NPC (Non-Player Character) has become a key technology to enhance game immersion and interactivity.
[0003] In the related art, in order to provide players with feedback on the activity status of non-player characters, game applications provide voice broadcast capabilities. The game application packages pre-recorded voice broadcasts into the client and triggers the voice broadcast when receiving a fixed command. For example, when a non-player character receives a command from a player, it broadcasts "received".
[0004] The methods in the related art can only perform simple voice broadcasts when receiving fixed instructions. In more complex scenarios, players cannot obtain effective feedback from NPCs, and the human-computer interaction efficiency between players and NPCs is too low. Summary of the invention
[0005] The embodiment of the present application provides a method, device, equipment, medium and product for broadcasting a non-player character. The technical solution is as follows:
[0006] In one aspect, a method for broadcasting a non-player character is provided, the method comprising:
[0007] Displaying at least one of a main control virtual character and a non-player character in a three-dimensional virtual environment, wherein the non-player character follows the main control virtual character to move in the three-dimensional virtual environment;
[0008] receiving a natural language command directed to the non-player character;
[0009] In response to the natural language command, feedback information of the non-player character is broadcasted, wherein the feedback text corresponding to the feedback information is a non-fixed text generated based on environmental perception information, and the environmental perception information includes information perceived by at least one of the main virtual character and the non-player character about the three-dimensional virtual environment.
[0010] On the other hand, a method for broadcasting a non-player character is provided, the method comprising:
[0011] Receiving a natural language command for a non-player character sent by a client, the client having control authority over a master virtual character, the non-player character following the master virtual character in the three-dimensional virtual environment;
[0012] Acquire environmental perception information of at least one of the main control virtual character and the non-player character;
[0013] Determining, according to the natural language command and the environmental perception information, a feedback text of the non-player character, wherein the feedback text is a non-fixed text generated based on the environmental perception information of the non-player character;
[0014] Based on the feedback text, relevant information for playing feedback information is sent to the client, where the feedback information is human voice audio corresponding to the feedback text.
[0015] On the other hand, a broadcasting device for a non-player character is provided, the device comprising:
[0016] A display module, for displaying at least one of a main control virtual character and a non-player character in a three-dimensional virtual environment, wherein the non-player character follows the main control virtual character to move in the three-dimensional virtual environment;
[0017] A receiving module, configured to receive a natural language command directed to the non-player character;
[0018] The broadcast module is used to broadcast the feedback information of the non-player character in response to the natural language command, wherein the feedback text corresponding to the feedback information is a non-fixed text generated based on the environmental perception information, and the environmental perception information includes the information perceived by at least one of the main virtual character and the non-player character about the three-dimensional virtual environment.
[0019] On the other hand, a broadcasting device for a non-player character is provided, the device comprising:
[0020] A receiving module, configured to receive a natural language command sent by a client for a non-player character, wherein the client has control authority over a master virtual character, and the non-player character follows the master virtual character to move in the three-dimensional virtual environment;
[0021] An acquisition module, used for acquiring environmental perception information of at least one of the main virtual character and the non-player character;
[0022] a determination module, configured to determine a feedback text of the non-player character according to the natural language command and the environmental perception information, wherein the feedback text is a non-fixed text generated based on the environmental perception information of the non-player character;
[0023] A sending module is used to send relevant information for playing feedback information to the client based on the feedback text, where the feedback information is human voice audio corresponding to the feedback text.
[0024] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the broadcasting method of the non-player character as described above.
[0025] On the other hand, a computer-readable storage medium is provided, in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the broadcasting method of a non-player character as described above.
[0026] On the other hand, an embodiment of the present application provides a computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the broadcasting method of the non-player character provided in the above optional implementation.
[0027] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:
[0028] When the user uses natural language commands to feedback relevant information of the three-dimensional virtual environment to the non-player character in the three-dimensional virtual environment, feedback information corresponding to the non-player character is generated and broadcasted according to the information perceived by the non-player character from the three-dimensional virtual environment, so that the feedback information can cover a variety of scenarios. When the natural language command instructs to ask the non-player character whether there is an enemy in the three-dimensional virtual environment, when the non-player character perceives that the virtual characters included in the three-dimensional virtual environment include hostile virtual characters, the main control virtual character is reminded of the existence of enemies by broadcasting feedback information; when the natural language command instructs the non-player character to provide virtual props to the main control virtual character, when the non-player character has virtual props required by the user, the non-player character submits the virtual props to the main control virtual character, and reminds the main control virtual character that the virtual props have been submitted by broadcasting feedback information. When controlling non-player characters through natural language commands, feedback information can be broadcast based on the environmental perception information of the non-player characters. This can enrich the content of the feedback information and make the broadcast not limited to fixed trigger scenarios. Game applications can flexibly broadcast feedback information based on environmental perception information during game matches, enrich the feedback information broadcast corresponding to non-player characters, and improve the efficiency of human-computer interaction between non-player characters and players. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0030] Figure 1 is a structural block diagram of a computer system provided by an exemplary embodiment of the present application;
[0031] Figure 2 is a method flow chart of a method for broadcasting a non-player character provided by an exemplary embodiment of the present application;
[0032] Figure 3 is a schematic diagram of feedback information corresponding to a non-player character provided by an exemplary embodiment of the present application;
[0033] Figure 4 is a method flow chart of a method for broadcasting a non-player character provided by an exemplary embodiment of the present application;
[0034] Figure 5 is a method flow chart of a method for broadcasting a non-player character provided by an exemplary embodiment of the present application;
[0035] Figure 6 is a method flow chart of a method for broadcasting a non-player character provided by an exemplary embodiment of the present application;
[0036] Figure 7 is a schematic diagram of a three-dimensional virtual environment provided by an exemplary embodiment of the present application;
[0037] Figure 8 is a schematic diagram of a three-dimensional virtual environment provided by an exemplary embodiment of the present application;
[0038] Fig. 9 is a schematic diagram of the position and orientation of a master virtual character provided by an exemplary embodiment of the present application;
[0039] Fig.10 is a schematic diagram of spatial classification of target entities provided by an exemplary embodiment of the present application;
[0040] Fig.11 is a method flow chart of a method for broadcasting a non-player character provided by an exemplary embodiment of the present application;
[0041] Fig.12 is a method flow chart of a method for broadcasting a non-player character provided by an exemplary embodiment of the present application;
[0042] Fig.13 is a method flow chart of a method for broadcasting a non-player character provided by an exemplary embodiment of the present application;
[0043] Fig.14 is a schematic diagram of a grid structure provided by an exemplary embodiment of the present application;
[0044] Fig.15 is a schematic diagram of a reverberant sound wave provided by an exemplary embodiment of the present application;
[0045] Fig.16 is a method flow chart of a method for broadcasting a non-player character provided by an exemplary embodiment of the present application;
[0046] Fig.17 is a schematic diagram of a speech generation model provided by an exemplary embodiment of the present application;
[0047] Fig.18 is a method flow chart of a method for broadcasting a non-player character provided by an exemplary embodiment of the present application;
[0048] Fig.19 is a block diagram of a broadcasting device for a non-player character provided by an exemplary embodiment of the present application;
[0049] Fig. 20 is a block diagram of a broadcasting device for a non-player character provided by an exemplary embodiment of the present application;
[0050] Fig.21It is a structural block diagram of a terminal provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION
[0051] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.
[0052] First, the nouns involved in the embodiments of the present application are briefly introduced:
[0053] 3D virtual environment: is a 3D virtual environment displayed (or provided) when an application is running on a terminal. The 3D virtual environment can be a simulation of the real world, a semi-simulation and semi-fictitious environment, or a purely fictitious environment. The 3D virtual environment can be any one of a 2D 3D virtual environment, a 2.5D 3D virtual environment, and a 3D virtual environment, which is not limited in the embodiments of the present application.
[0054] Virtual character: A virtual character refers to an active object in a three-dimensional virtual environment. The active object can be a virtual person, a virtual animal, an animated character, etc., such as a person displayed in a three-dimensional virtual environment. Optionally, the virtual character is a three-dimensional model created based on animation skeleton technology. Each virtual character has its own shape and volume in the three-dimensional virtual world and occupies a part of the space in the three-dimensional virtual world.
[0055] User Interface (UI) controls: refers to any visual controls or elements that can be seen on the user interface of an application, such as images, input boxes, text boxes, buttons, labels, etc. Some UI controls respond to user operations.
[0056] 3D (three-dimensional) model: In computer graphics and video game design, a 3D model is a three-dimensional object defined by three-dimensional geometric data. This data can be created manually or generated by a 3D scanner. A complete 3D model allows the user to view the object from any angle and may contain additional information such as texture, color, and surface material properties. In video games, this type of model is often used to create the image of the player or NPC (non-player character).
[0057] First-person shooting game (First-person Shooting, FPS): refers to a shooting game that users can play from a first-person perspective. The screen of the three-dimensional virtual environment in the game is a screen that observes the three-dimensional virtual environment from the perspective of the virtual character controlled by the user. In the game, virtual characters of at least two camps play a single-game battle mode in a three-dimensional virtual environment. The virtual characters achieve the purpose of surviving in the three-dimensional virtual environment by avoiding attacks launched by other virtual characters and dangers in the three-dimensional virtual environment (such as gas circles, swamps, etc.). When the life value of the virtual character in the three-dimensional virtual environment is zero, the life of the virtual character in the three-dimensional virtual environment ends, and the virtual character that survives in the three-dimensional virtual environment is the winner. Optionally, the competitive mode of the battle (game) can include a single-player battle mode, a two-player team battle mode, or a multiplayer large group battle mode, and the battle mode in the embodiment of the present application is not limited.
[0058] Natural Language Processing (NLP): It is a subfield of artificial intelligence and computer science, which studies how to enable computers to understand, interpret and generate human natural language and realize natural language communication between humans and computers.
[0059] Text-to-Speech (TTS) is a technology that converts text information into audible voice signals, enabling computers or other devices to read text information using human voice pronunciation.
[0060] Entity Query: is a method of querying specific entities (such as characters, objects, etc.) in a game or virtual world. It is usually used to obtain a list of entities that meet certain conditions for further processing.
[0061] Spatial Query: A method for querying a corresponding space based on spatial relationships (such as distance, direction, intersection, etc.) to determine the specific space where the target entity is currently located. Spatial query can return a location description of the target entity's location based on the spatial position relationship between the master virtual character and the target entity.
[0062] Intent Recognition: It is a task in the field of natural language processing, which aims to identify the user's intention from the natural language text input by the user so as to perform corresponding processing and response.
[0063] Large Language Model (LLM): A natural language processing model trained on a large amount of text data. It usually has strong language understanding and generation capabilities and can be applied to tasks such as machine translation, text summarization, and question-answering systems.
[0064] NPC (Non-Player Character): refers to a character controlled by a computer in the game, which usually has certain behavior patterns and tasks, can interact with players, provide tasks or act as an enemy, etc.
[0065] Figure 1 A structural block diagram of a computer system provided by an exemplary embodiment of the present application is given. The computer system includes: a terminal 110 and a server 120.
[0066] The terminal 110 is installed and runs a client 111 that supports a three-dimensional virtual environment, and the client 111 is a client of an application. When the terminal runs the client 111, the user interface of the client 111 is displayed on the screen of the terminal 110. The application can be any one of a battle royale shooting game, a virtual reality (VR) application, an augmented reality (AR) program, a three-dimensional map program, a virtual reality game, an augmented reality game, a first-person shooting game (FPS), a third-person shooting game (TPS), a multiplayer online tactical competitive game (MOBA), and a simulation game (SLG).
[0067] In this embodiment, the client is a MOBA game. Terminal 110 is a terminal used by a first user 112. In the game, the first user 112 uses terminal 110 to control a first virtual character in a three-dimensional virtual environment to perform activities. The first virtual character can be referred to as the main virtual character of the first user 112 in the game. The activities of the first virtual character include, but are not limited to, at least one of adjusting body posture, crawling, walking, running, riding, flying, jumping, driving, picking up, shooting, attacking, and throwing.
[0068] Figure 1Only one terminal is shown in the figure, but in different embodiments, there are multiple other terminals that can access the server 120. Optionally, there are one or more terminals corresponding to the developer, and a development and editing platform for the client supporting the three-dimensional virtual environment is installed on the terminal. The developer can edit and update the client on the terminal, and transmit the updated client installation package to the server 120 through a wired or wireless network. The terminal 110 can download the client installation package from the server 120 to update the client.
[0069] The terminal 110 and other terminals are connected to the server 120 via a wireless network or a wired network.
[0070] The server 120 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. The server 120 is used to provide background services for the client supporting the three-dimensional virtual environment. Optionally, the server 120 undertakes the main computing work, and the terminal undertakes the secondary computing work; or, the server 120 undertakes the secondary computing work, and the terminal undertakes the main computing work; or, the server 120 and the terminal adopt a distributed computing architecture for collaborative computing.
[0071] In an illustrative example, the server 120 includes a processor, a user account database, a battle service module, and a user-oriented input / output interface (I / O interface). The processor is used to load instructions stored in the server 120 and process data in the user account database and the battle service module; the user account database is used to store data of user accounts used by the terminal 110 and other terminals, such as the user account's avatar, the user account's nickname, the user account's combat power index, and the service area where the user account is located; the battle service module is used to provide multiple battle rooms for users to fight, such as 1V1 battle, 3V3 battle, 5V5 battle, etc.; the user-oriented I / O interface is used to establish communication and exchange data with the terminal 110 through a wireless network or a wired network.
[0072] The embodiments of the present application provide a solution for users to use natural language commands in the form of voice to command non-player characters (NPC) in a virtual environment. In this solution, in addition to the main virtual character controlled by the user, the user usually has one or more NPCs as teammates. The user can use a relatively casual and non-mechanical human conversation method to command the NPC to give the desired feedback to the scene objects in the virtual environment, thereby commanding the NPC to work with the main virtual character to complete the task. Exemplary reference Figure 1, the user controls the main virtual character 10 to play the game, and the main virtual character 10 has an NPC teammate 20. There is a car 30 in the field of vision of the main virtual character 10. The user speaks a natural voice command in the form of speech: "No. 2, you go to ambush behind the car 30", and then the NPC teammate 20 will move to the back of the car 30 to ambush. It should be noted that Figure 1 There may be multiple cars in the scene shown, and the NPC teammate 20 will accurately understand that the car the user is talking about is the car 30 located within the user's field of vision. That is, the NPC teammate 20 has a relatively intelligent natural language understanding ability.
[0073] On the one hand, compared with the traditional technology of using mechanical fixed instructions to command NPC, the embodiment of the present application uses relatively random natural language commands to command NPC, which can provide users with more natural, more complex and more flexible language command capabilities.
[0074] On the other hand, the intelligence of the natural language understanding capability provided by the embodiment of the present application is reflected in that when understanding natural language commands, NPC teammates not only consider the literal information of natural language commands. NPC teammates will also combine the main virtual character and / or the NPC's own environmental perception information in the virtual environment to assist in understanding the semantics in natural language commands. That is, NPC not only considers the information of the natural language command mode, but also considers the perception information of other modes such as vision, hearing, radar, etc. to assist in the understanding and execution of natural language commands. Since natural language commands in the form of voice are spoken commands rather than written commands, there may be multiple candidate understanding methods or unclear places when considering the literal information of natural language commands alone. NPC teammates can determine a reasonable understanding method from multiple candidate understanding methods, or eliminate doubts about unclear places, by combining the main virtual character and / or the NPC's own environmental perception information in the virtual environment, thereby achieving a more intelligent natural language understanding capability. The above-mentioned environmental perception information includes, but is not limited to, at least one of the following:
[0075] Visual information perceived within the field of vision of the controlling virtual character;
[0076] Auditory information perceived within the hearing range of the controlling virtual character;
[0077] The information perceived by the master virtual character’s perception skills or perception tools;
[0078] Visual information perceived by the NPC within its field of vision;
[0079] Auditory information perceived within the NPC's hearing range;
[0080] Information perceived by the NPC's perception skills or perception tools.
[0081] Combined with reference Figure 1 The program includes at least one of the following five stages:
[0082] Phase 1: Preprocessing of spatial data;
[0083] The server will pre-process the scene objects in the virtual scene and construct the spatial data of the virtual scene.
[0084] The spatial data of the virtual scene includes the visual labels of the scene objects and the spatial positions of the scene objects. In some examples, the spatial data also includes the appearance images of the scene objects. The scene objects are any objects that appear in the virtual scene, such as Figure 1 Cars, walls, boxes, etc.
[0085] First, the pre-acquisition process of the visual labels of the scene objects is introduced. The attribute information of the scene objects in the virtual scene is obtained, such as at least one of the name and size of the scene objects; and the appearance image of the scene objects is obtained. The appearance image includes an image obtained by observing the scene objects from at least two perspectives. Observing the scene objects from multiple perspectives can ensure that the appearance image carries comprehensive appearance information of the scene objects.
[0086] The server calls the multimodal model to predict the attribute information and appearance image of the scene object. Based on the multimodal model's ability to process text dimension and image dimension information, it extracts hidden features from the attribute information of the text dimension and the appearance image of the image dimension, and decodes the hidden features to predict the visual label of the scene object. The visual label is used to describe the scene object in at least one dimension in a textual manner; for example, the scene object is described in dimensions such as material, transparency, color, and shape. The natural language model is called to optimize the visual label to obtain the visual label of the scene object; the natural language model has the ability to generate text, and rewrites the visual label input to the natural language model in a manner that conforms to the spoken expression of natural language, thereby optimizing the visual label and obtaining the optimized visual label of the scene object. The optimized visual label conforms to the spoken expression of natural language, and describes the scene object in at least one dimension in a manner close to the spoken expression.
[0087] For example, when describing a virtual bed in a virtual scene in spoken language, people usually pay attention to the color, material, and placement of the virtual bed, but usually neglect to pay attention to the craftsmanship used for the headboard and the side panels of the bed, such as carving, embossing, etc. The purpose of performing pseudo-spoken rewriting on the visual labels of scene objects is to obtain matching labels that are closer to spoken expressions; for example, using multiple words with the same semantics to enrich the labels of dimensions such as color, material, and placement that are paid attention to in spoken expressions, and deleting the labels of the surface feature dimensions of the headboard and the side panels of the bed that are neglected in spoken expressions.
[0088] Next, the spatial position of the scene objects is introduced. The spatial position includes at least one of the following of the scene objects: coordinate position (Location), such as the coordinate information of the center point or preset point of the scene object in the virtual scene; orientation (Rotation), such as the direction the front of the scene object faces in the virtual scene; bounding box (Bounding Box), used to indicate the size of the scene object in the virtual scene; cover points (Cover points), used to indicate the recommended virtual character position points when the virtual character approaches the scene object, so that the scene object can cover the virtual character. The bounding box (UE4 BoundingBox) is a rectangular box used to represent the boundary of a three-dimensional object in the Unreal Engine4 (UE4) game engine, and is usually used for fast collision detection and spatial query operations.
[0089] Next, the process of pre-acquiring the appearance image of the scene object is introduced. The appearance image is acquired in the process of predicting the visual label of the scene object, and the appearance image includes images obtained by observing the scene object from at least two perspectives.
[0090] Optionally, the spatial data of each scene object in the entire virtual environment may be pre-acquired and organized into a data set for use in a subsequent query process.
[0091] Phase 2: Speech recognition and intent recognition;
[0092] When the user controls the main virtual character, the terminal receives the natural language command input by the user in the form of voice; the server performs voice recognition and intent recognition on the natural language command to analyze the user's input instruction;
[0093] Speech recognition is to perform text conversion on the natural language commands input by the user through voice, obtain the instruction text corresponding to the natural language command, and convert the speech information into text information. Call the encoding network to perform feature encoding on the instruction text to obtain the feature representation of the natural language command in the latent space; call the text segmentation network to perform segmentation on the feature representation to obtain multiple clauses in the instruction text, and then perform intent recognition on each of the multiple clauses. In the case that the natural language command input by the user through voice is a long sentence, the segmentation of the instruction text corresponding to the natural language command can be achieved to meet the needs of analyzing the user's input instructions in the long sentence scenario.
[0094] The following introduces the intent recognition of each clause; the user's input instruction is obtained through intent recognition, and the user's input instruction includes an entity, a semantic type, a subject type, and an intent type in at least one clause.
[0095] On the one hand, the Conditional Random Field (CRF) method is used to identify entities in clauses, such as identifying entities in clauses as scene objects such as buildings, virtual items, and virtual vegetation in virtual scenes. The entities in the clauses are used to indicate the scene objects targeted by the virtual activities. For example, if the entity in the clause is a virtual building, and the clause carries the semantics of instructing the NPC to launch a virtual attack, then the virtual attack is executed against the virtual building, that is, a virtual attack is launched against the virtual building. On the other hand, the prediction network is called to predict the semantic type, subject type, and intention type in the clause respectively. Among them, the semantic type includes whether to instruct the NPC to launch a virtual attack; the subject type is used to indicate that the subject of the clause is one or more NPCs, or a master virtual character controlled by the user; the intention type is the expected virtual activity indicated by the clause, such as moving, launching a virtual attack, etc.
[0096] Phase 3: Query of spatial data;
[0097] For the entity in the input command, it is necessary to determine the spatial position of the entity in the virtual environment. In the case where there are multiple candidate entities in the virtual environment that match the entity in the input command, the target entity that matches the input command can be uniquely determined from the multiple candidate entities in combination with the environmental perception information of the main virtual character and / or NPC.
[0098] For example, the environment perception information is the field of view information of the main virtual character, the spatial data in the virtual environment is used as the query range, and the real-time position and orientation of the main virtual character is used as the query reference condition to query the spatial position of the target entity corresponding to the input instruction in the virtual environment, so as to subsequently control the NPC to move to the vicinity of the target entity, or to perform the behavior indicated by the input instruction on the target entity.
[0099] However, since the main virtual character and / or NPC may have multiple environmental perception information, multiple environmental perception information can be integrated to assist the determination process of the above-mentioned target entity, or multiple environmental perception information can be used according to priority to assist the determination process of the above-mentioned target entity, and there is no limitation on this.
[0100] Stage 4: NPC voice feedback;
[0101] NPC will give voice feedback to the user. The types of voice feedback include: at least one of instant feedback, command execution feedback, and dynamic feedback. Among them, instant feedback means that after the user issues a natural language command to the NPC, the NPC immediately gives voice feedback on the natural language command, such as "received", "start execution", "OK", etc.; execution command feedback means that after the NPC executes the control command indicated by the intention of the natural language command, the NPC gives voice feedback on the execution result of the control command, such as "successful execution", "failed to execute successfully", etc.; dynamic feedback means that when the user does not issue a natural language command to the NPC, the NPC gives voice feedback based on real-time environmental perception.
[0102] The above-mentioned voice feedback is obtained by converting text content into speech. The text content is inferred by a large language model combined with the above-mentioned spatial data and dynamic environmental perception information. Using a large language model to dynamically generate the text content of voice feedback can, on the one hand, break through the monotonous, mechanical, and easily repetitive voice feedback of fixed templated voice feedback; on the other hand, it does not need to save too many pre-made template voices in advance, which can avoid the problem of too large a total data volume on the client.
[0103] Stage 5: NPC behavior control;
[0104] Taking the user's input as input, and the spatial data of the virtual scene and the runtime information corresponding to the master virtual character as reference, the AI model is used to determine the target entity in the virtual space and the control instructions for the NPC;
[0105] Control instructions are instructions that can be used for NPC behavior tree execution; the subsequent actions that the control instruction indicates the NPC needs to perform can be a single action or a sequence of actions. In the case where the control instruction is a multi-action sequence instruction, the multi-action sequence instruction is used to instruct the NPC to perform a sequence of actions. The data structure storing the multi-action sequence instruction is in the form of a list, and the instructions to be executed in the sequence instruction are cached. When the NPC's behavior tree executes the current instruction, the cache is called back, and the instructions to be executed in the cache are continued to be sent for execution.
[0106] Hot word update function:
[0107] For phase 2, this solution also introduces a hot word system to improve the accuracy of speech recognition.
[0108] The hot word system is used to provide scene hot words related to the scenes involved in the natural language commands during the speech recognition process, so that when a natural language command input by voice is received, the hot word library can be preferentially searched for words as the command analysis result based on the currently running virtual scene, so that the match between the command analysis result and the virtual scene is higher.
[0109] The virtual scene includes at least one of a plurality of virtual scenes such as a virtual battle scene, a virtual transaction scene, a virtual office scene, and a virtual kitchen scene. Frequently used words in a plurality of virtual scenes, specific virtual elements existing in a virtual scene, etc. (such as virtual buildings and virtual characters that only exist in a specific scene type, etc.) are pre-analyzed as scene hot words, and a scene hot word library consisting of a plurality of scene hot words is obtained, and the plurality of scene hot words correspond to virtual scenes respectively. In addition, different scene language models are pre-trained for different virtual scenes, such as a scene language model in a battle scene that pays more attention to words related to battle, and a scene language model in a transaction scene that pays more attention to words related to transaction; the scene language model can perform a more targeted analysis of natural language commands received in a virtual scene.
[0110] For example: when a user controls a non-player character, a natural language command is received. Based on the game state, object position, and current task completion status, the first virtual scene in which the target virtual character (at least one of the main virtual character and the non-player character) is currently located is determined to be a virtual battle scene, and a battle scene language model corresponding to the virtual battle scene is obtained. In addition, based on the field of view of the target virtual character, the first scene hot words in the virtual battle scene are obtained, including: "truck", "virtual grass", "virtual house", "enter", "open truck", "defense", "desert", "attack", "oak tree", "stable", etc. Under the constraints of the first scene hot words, the natural language command is decoded and processed through a pre-trained natural language analysis model (including an acoustic network, a language network, a preset dictionary, etc.), and the instruction analysis result is output to accurately control the non-player character. The instruction analysis result is illustrated as follows.
[0111] (1) The command analysis result is "No. 1, go to the front and defend." This command analysis result can specifically control No. 1 to move forward and take a defensive posture to prevent the enemy virtual character from attacking the main control virtual character first. If there is no constraint of the hot words in the first scene, the natural language command is easily recognized as "No. 1, go to the front and speak," thereby affecting the player's virtual combat plan.
[0112] (2) The command analysis result is “No. 2, go explore the desert.” This command analysis result can specifically control No. 2 to move to the desert to explore whether there are virtual elements such as enemy virtual characters or virtual treasure chests. If there is no constraint of the hot words in the first scene, the natural language command is easily recognized as “No. 2, go explore the mountains,” causing No. 2 to move to the wrong location, resulting in low human-computer interaction efficiency.
[0113] (3) The command analysis result is “No. 1, No. 2, attack me”. This command analysis result can be used to specifically control No. 1 and No. 2 to carry out virtual attacks on nearby attackable virtual characters. If there is no constraint of the hot words in the first scene, the natural language command is easily recognized as “No. 1, No. 2, supply me”, which causes No. 1 and No. 2 to make incorrect behaviors that do not meet the player’s expectations, and they can neither protect the main virtual character nor easily allow the enemy virtual character to win the virtual game.
[0114] (4) The command analysis result is “No. 1, go find a nearby oak tree”. This command analysis result can be used to specifically control No. 1 to search for oak trees in the nearby area so that the game task can be completed or an oak tree that can be used to avoid attacks can be found. If there is no constraint of the hot words in the first scene, the natural language command is easily recognized as “No. 1, go find a nearby project book”, which causes No. 1 to search nearby for virtual elements that do not meet the player’s expectations and fail to meet the player’s virtual combat needs.
[0115] (5) The command analysis result is “No. 1, No. 2, go to the stable over there”. This command analysis result can specifically control No. 1 and No. 2 to search for the nearby stable and move to the location of the stable. If there is no constraint of the hot words in the first scene, it is easy to recognize the natural language command as “No. 1, No. 2, go over there, it will be ready soon”, which will cause No. 1 and No. 2 to mistakenly believe that the main virtual character currently hopes to complete the virtual game by themselves, and therefore cannot provide good game assistance to the main virtual character.
[0116] (6) The command analysis result is "No. 2, pick up the props in front". This command analysis result can specifically control No. 2 to search for props in front and pick up the props. If there is no constraint of the hot words in the first scene, it is easy to identify the natural language command as "No. 2, pick up the crab in front", which makes No. 2 unable to accurately find the virtual element that the main virtual character in front wants to pick up. It is possible that the player is prompted that "the crab cannot be found", or the player may pick up the wrong virtual element "crab", which interferes with the player's virtual combat process.
[0117] Ambient sound function:
[0118] In addition, the solution also provides a spatial audio enhancement solution for the ambient sound effects of the entire virtual scene.
[0119] The audio played by the terminal includes environmental audio and NPC audio. Environmental audio is audio generated based on the scene characteristics of the virtual scene where the main virtual object is currently located, so that the user can feel immersive; NPC audio is audio generated based on the characteristics of the NPC, so that the user can intuitively feel the character's emotions and physical state through the sound heard.
[0120] Environmental audio: Identify the scene elements in the virtual scene where the master virtual object is located, generate appropriate element sound effects in real time based on the scene elements / select appropriate element sound effects from the sound effect library, and synthesize the element sound effects to obtain environmental audio. For example, if the virtual scene is a forest at night, the scene elements include trees, owls, insects, etc., and the environmental audio includes the rustling of leaves, the calls of owls, the chirping of insects, etc.
[0121] NPC audio: Determine the NPCs included in the virtual scene, and generate corresponding character voices based on the NPC's character type, current emotion, and current behavior. For example, if the NPC is a middle-aged man running, the generated character voice is a deep male voice with a running panting sound effect.
[0122] When the terminal plays the ambient audio and / or NPC audio to the user, the terminal performs audio enhancement processing on the ambient audio and / or NPC audio to improve the realism of the audio.
[0123] Combined with the above description of the implementation environment, the broadcasting method of the non-player character provided in the embodiment of the present application is described, and the execution subject of the method is Figure 1 The terminal shown is used as an example.
[0124] Figure 2 A flowchart of a method for broadcasting a non-player character provided by an exemplary embodiment of the present application is shown. The method can be performed by the above Figure 1 The method comprises:
[0125] Step 210 , displaying at least one of a main control virtual character and a non-player character in a three-dimensional virtual environment, wherein the non-player character moves in the three-dimensional virtual environment following the main control virtual character.
[0126] Exemplarily, the method may be executed by a client installed on the terminal, the client being a client of an application, and the application being an application supporting a three-dimensional virtual environment, for example, the application may be an application for a shooting game.
[0127] Optionally, the terminal displays a game interface, which includes a three-dimensional virtual environment screen, in which a control virtual character and non-player characters are displayed. The three-dimensional virtual environment screen is a screen obtained by observing the three-dimensional virtual environment from the perspective of the control virtual character.
[0128] The main control virtual character is a virtual character directly controlled by the terminal. The terminal can directly control the behavior of the main control virtual character according to the received control instructions based on the preset control logic. For example, when receiving the forward instruction, the main control virtual character is controlled to move forward; when receiving the jump instruction, the main control virtual character is controlled to jump upward.
[0129] For example, the terminal displays control controls corresponding to the main virtual character (e.g., moving controls, shooting controls, throwing controls, jumping controls, etc.), and the user controls the main virtual character to move in the three-dimensional virtual environment by triggering the control controls. Alternatively, the terminal is connected to an input device (e.g., a mouse, keyboard, camera, sensor, etc.), and the user issues control instructions through the input device, and controls the main virtual character to move in the three-dimensional virtual environment according to the control instructions.
[0130] The non-player character is a virtual character controlled by a game application, which is controlled by a server or an artificial intelligence program in a client. In the embodiment of the present application, the game application can direct the behavior of the non-player character according to the received natural language command. The game application can understand the intention of the natural language command, generate control instructions according to the intention, and direct the non-player character to execute the intention indicated by the natural language command.
[0131] It should be noted that the non-player characters have self-help behavior capabilities, and users can only command the non-player characters but cannot directly control the non-player characters. The non-player characters understand and execute the natural language commands based on their own self-help behavior capabilities.
[0132] In an optional embodiment, the main virtual character is a virtual character controlled by the control instructions generated by the triggering UI control; the non-player character is a virtual character controlled by the AI according to the natural language command. When no natural language command is received, the AI can control the non-player character to move in the three-dimensional virtual environment; when a natural language command is received, the AI can control the non-player character to move in the three-dimensional virtual environment according to the intention of the natural language command.
[0133] In some embodiments, the main virtual character and the non-player character belong to the same camp. For example, the non-player character is a teammate of the main virtual character, or the non-player character is a pet of the main virtual character, or the non-player character is a servant of the main virtual character, or the non-player character is an intelligent robot controlled by the main virtual character, or the non-player character is an avatar of the main virtual character.
[0134] For example, in a team-based game, a player needs to team up with at least one other player to play the game. When a player lacks teammates, the game application can provide a non-player character as the player's teammate to play the game with the player. During the game, the player can command the activities of the non-player character by issuing voice commands (natural language commands) and make the non-player character broadcast feedback information based on environmental perception information.
[0135] For example, during a game, a player can bring a pet with him to explore the 3D virtual environment. The player can command the pet to move in the 3D virtual environment through voice commands or text commands to explore unknown areas in the 3D virtual environment, and the pet can broadcast feedback information in real time according to the environmental perception information during the exploration process, so as to facilitate the main virtual character to make strategic decisions and deployments.
[0136] Alternatively, in a MOBA game, the main virtual character teams up with the virtual characters controlled by the other four players to play against each other. When one of the teammates is offline and cannot control his virtual character, for example, the player of the first virtual character is offline and cannot control the first virtual character, the player can use voice commands to command the first virtual character during the teammate's offline period, so that the first virtual character assists other teammates in participating in the team battle, or commands the first virtual character to take accurate actions to respond to changes in the battle situation; the first virtual character can broadcast feedback information based on environmental perception information to facilitate the player's combat decision-making.
[0137] In some embodiments, the user controls the main virtual character to move in the three-dimensional virtual environment through the client. Schematically, the client receives a control operation on the main virtual character, and controls the main virtual character to move in the three-dimensional virtual environment according to the control operation.
[0138] Step 220, receiving a natural language command for a non-player character.
[0139] In an embodiment of the present application, the client receives a natural language command from a user directed to a non-player character, and directs the non-player character to move in a three-dimensional virtual environment according to the intention indicated by the natural language command.
[0140] In an embodiment of the present application, the above-mentioned natural language command is a command inputted by a user through voice received by the client. In some embodiments, a voice receiving control is displayed in the client, and in response to the voice receiving control receiving a trigger operation, the client starts recording the user's voice audio, and processes the recorded voice audio as a natural language command.
[0141] Step 230, in response to the natural language command, broadcast the feedback information of the non-player character.
[0142] Illustratively, the feedback text corresponding to the feedback information is a non-fixed text generated based on the environment perception information, and the environment perception information includes information perceived by at least one of the main control virtual character and the non-player character about the three-dimensional virtual environment.
[0143] Exemplarily, after receiving the natural language command, the game application can generate corresponding feedback information in real time according to the environmental perception information of the non-player character and make a broadcast. For example, when the environmental perception information of the non-player character triggers the broadcast condition, the game application can generate corresponding feedback information in real time according to the environmental perception information that triggers the broadcast condition; or, when the non-player character or the main virtual character triggers the broadcast condition, the game application can generate corresponding feedback information in real time according to the environmental perception information of the non-player character.
[0144] For example, the feedback text is obtained by inferring the large language model based on the static entity data of the three-dimensional virtual environment and the dynamic environment perception information of the non-player character. The large language model will generate different feedback texts according to the different perception situations of the non-player character.
[0145] For example, the feedback text can also be obtained by inferring the static entity data of the three-dimensional virtual environment and the dynamic environment perception information of the non-player character through the language model, and the feedback text can be in line with the personality characteristics of the non-player character. For example, when the non-player character is a strong uncle, the feedback text can adopt a heroic tone; when the non-player character is a reporter, the feedback text can adopt a news report tone.
[0146] It should be noted that due to the randomness of the results generated by the large language model, in the same scene, when the environmental perception information of the non-player character is the same, the generated feedback text may also be different.
[0147] Exemplarily, the environmental perception information includes at least one of the following:
[0148] Visual information perceived within the field of vision of the controlling virtual character;
[0149] Auditory information perceived within the hearing range of the controlling virtual character;
[0150] The information perceived by the master virtual character’s perception skills or perception tools;
[0151] Visual information perceived within the field of view of non-player characters;
[0152] Auditory information perceived within the hearing range of non-player characters;
[0153] Information perceived by non-player characters using their perception skills or perception tools.
[0154] For example, the environmental perception information of the main virtual character may include: the pictures or entities that the main virtual character can see when observing the three-dimensional virtual environment, the environmental sound effects that the main virtual character can hear, the source direction of the environmental sound effects, the type and volume of the environmental sound effects, the perception information obtained by the main virtual character by using perception skills (for example, visual information, auditory information, sound wave information, light reflection information, etc.), and the perception information obtained by the main virtual character by using sensing props (for example, sensor signals from sensors, position detection signals from position detection props, etc.).
[0155] The environmental perception information of the non-player character may include: the images or entities that the non-player character can see when observing the three-dimensional virtual environment, the environmental sound effects that the non-player character can hear, the source direction of the environmental sound effects, the type and volume of the environmental sound effects, the perception information obtained by the non-player character by using perception skills (for example, visual information, auditory information, sound wave information, light reflection information, etc.), and the perception information obtained by the non-player character by using sensing props (for example, sensor signals of sensors, position detection signals of position detection props, etc.).
[0156] It should be noted that the environmental perception information may include information perceived by the non-player character in real time, and may also include information perceived by the non-player character historically.
[0157] Exemplarily, the feedback information is inferred by the game application based on the environmental perception information of the non-player character and the static entity data of the three-dimensional virtual environment. The feedback information can be a voice broadcast or a text broadcast. The static entity data includes data of relatively unchanged entities in the three-dimensional virtual environment, for example, model information of various buildings, terrains, and vehicles in the three-dimensional virtual environment.
[0158] Optionally, the feedback information may include at least one of the following: immediate feedback and execution feedback, wherein both the immediate feedback and execution feedback are feedback information generated for the natural language command.
[0159] 1. Instant feedback is feedback information generated immediately when a natural language command is received. Instant feedback can include reply broadcast and instant broadcast. Reply broadcast is used to reply to inquiries in natural language commands; instant broadcast is used to feedback the reception of natural language commands.
[0160] For example, the client can command the non-player character to move in the three-dimensional virtual environment according to the intention of the natural language command. Then, when the client receives the natural language command, it can generate instant feedback corresponding to the natural language command, and the instant feedback is used to indicate that the natural language command has been received, or the instant feedback is used to instantly reply to the natural language command.
[0161] 2. Execution feedback is the feedback generated when a task is executed according to the intent indicated by the natural language command after receiving the natural language command. Execution feedback can also be called execution broadcast.
[0162] For example, when a natural language command is used to instruct a non-player character to perform a target task, the client can generate execution feedback during the non-player character's execution of the target task, and the execution feedback is used to indicate the execution status of the target task; or, the client can generate execution feedback after the non-player character completes the target task, and the execution feedback is used to indicate the execution result of the target task.
[0163] In some other embodiments, the feedback information may also include dynamic feedback, which is feedback information spontaneously generated by a non-player character.
[0164] Indicatively, dynamic feedback is feedback information that is spontaneously generated based on the environmental perception information of non-player characters. Dynamic feedback can also be called dynamic broadcast.
[0165] For example, the client can spontaneously generate dynamic feedback based on the perception of the three-dimensional virtual environment without receiving natural language commands, and the dynamic feedback is used to indicate abnormal situations found by the non-player character in the three-dimensional virtual environment. For example, abnormal situations may include: finding hostile virtual characters, finding changes in the status of friendly virtual characters, finding dangerous situations, finding traces of battle or looting, etc.
[0166] In an optional embodiment, the feedback information is implemented as spatial human voice audio of the non-player character, that is, the feedback information played by the client is spatial human voice audio after spatial sound effect processing based on the generated human voice audio. In an illustrative manner, the human voice audio corresponding to the feedback text is obtained, and based on the relative position relationship between the non-player character and the main virtual character in the three-dimensional virtual environment, the sound effect parameters corresponding to the non-player character are obtained, the human voice audio is adjusted based on the sound effect parameters, and the spatial human voice audio corresponding to the non-player character is generated and broadcast. Among them, the spatial human voice audio is used to express the perceptual effect of the audio generated by the main virtual character on the non-player character under the relative position relationship.
[0167] Illustratively, the relative position relationship between the non-player character and the controlling virtual character in the three-dimensional virtual environment is used to indicate the relative relationship between a first position corresponding to the non-player character and a second position corresponding to the controlling virtual character in the three-dimensional virtual environment.
[0168] Optionally, the above-mentioned relative orientation relationship includes at least one of position distance, position direction, and spatial surrounding situation between the non-player character and the main virtual character, wherein the position distance is used to indicate the distance relationship between the non-player character and the main virtual character, the position direction is used to indicate the angular relationship between the first position of the non-player character and the orientation direction of the main virtual character, and the spatial surrounding situation is used to indicate the surround situation of the main virtual character by the spatial elements forming the virtual space in the three-dimensional virtual environment, and the propagation effect of the space on the audio when the non-player character emits audio in the virtual space.
[0169] In some embodiments, the determination of the relative position relationship between the non-player character and the main virtual character can be implemented as follows: obtaining first position information of the non-player character in the three-dimensional virtual environment, obtaining second position information of the main virtual character in the three-dimensional virtual environment, and determining the relative position relationship between the non-player character and the main virtual character based on the first position information and the second position information.
[0170] In some embodiments, the first position information of the non-player character and the second position information of the controlling virtual character are position information determined based on the same preset coordinate system. Optionally, the preset coordinate system can be implemented as a world coordinate system corresponding to the three-dimensional virtual environment, or the preset coordinate system can be implemented as a coordinate system established with the controlling virtual character as the origin.
[0171] In some embodiments, when the sound effect parameters corresponding to the non-player characters are pre-generated and stored in a database, the sound effect parameters corresponding to the non-player characters are obtained from the database. In other embodiments, the sound effect parameters corresponding to the non-player characters may also be generated in real time, that is, the sound effect parameters corresponding to the non-player characters are generated based on the relative position relationship.
[0172] Optionally, the sound effect parameters corresponding to the non-player character may include at least one of the following types:
[0173] The first type is distance type
[0174] Illustratively, the distance-type sound effect parameter is used to indicate that the sound effect of the human voice audio is adjusted based on the distance between the non-player character and the main virtual character. In some embodiments, the distance-type sound effect parameter can adjust the audio volume of the human voice audio to simulate the effect of distance on the sound volume; and / or, the distance-type sound effect parameter can adjust the audio delay of the human voice audio to simulate the effect of distance on the sound propagation time.
[0175] Second, direction type
[0176] Illustratively, the direction-type sound effect parameter is used to indicate the adjustment of the sound effect of the human voice audio based on the direction of the non-player character relative to the main virtual character. In some embodiments, the direction-type sound effect parameter can adjust the audio volume of the human voice audio to simulate whether the main virtual character is facing the scene audio element; and / or, the direction-type sound effect parameter can adjust the volume parameters of the human voice audio in different channels to simulate the direction of the audio emitted by the scene sound source element.
[0177] The third type is space effects
[0178] Illustratively, the sound effect parameters of the spatial special effect type are used to indicate that the virtual space formed based on the spatial elements adjusts the sound effects in the virtual space, that is, the sound effect parameters of the spatial special effect type are used to indicate the effect of the virtual space on the sound performance of the audio emitted by the non-player characters. In some embodiments, the sound effect parameters of the spatial special effect type can add echo sound effects and reverberation sound effects to simulate the effect of sound in the space.
[0179] Optionally, the sound effect parameters of the spatial special effect type include at least one of a reverberation time parameter, a pre-delay parameter, a wetness (Wet / DryMix) parameter, a space width parameter, and a distance effect parameter. Among them, the reverberation time parameter is used to indicate the speed at which the audio decays in the virtual space; the pre-delay parameter is used to indicate the time difference between the audio directly reaching the main virtual character and the audio reaching the main virtual character after the first reflection; the wetness parameter is used to indicate the ratio between the audio directly transmitted and the audio transmitted through reflection; the space width parameter is used to indicate the degree of diffusion of the audio on the horizontal plane in the virtual space; and the distance effect parameter is used to indicate the attenuation of the audio with the propagation distance during the propagation process.
[0180] The fourth type is custom type
[0181] Schematically, the custom type of sound effect parameters are user-defined parameters for sound effect adjustment. Optionally, the user can customize the overall volume of the scene audio, whether to add background music, customize the volume of different types of audio, etc.
[0182] In some embodiments, the client provides a customization interface for sound effect parameters to the user, and the user configures the customized sound effect parameters through the customization interface.
[0183] In some embodiments, a parameter prediction model for personalized learning of the custom sound effect parameters is configured in the custom sound effect parameter server for the user. Schematically, the custom sound effect parameters of multiple candidate accounts are obtained, the multiple custom sound effect parameters are input into the parameter prediction model to be trained, the parameter prediction model to be trained is iteratively trained to obtain the parameter prediction model, and the sound effect parameters generated by the system (for example, the sound effect parameters of the above-mentioned distance type, direction type, and spatial special effect type) are optimized through the parameter prediction model.
[0184] In an embodiment of the present application, the human voice audio is adjusted according to the sound effect parameters corresponding to the non-player character, thereby generating the spatial human voice audio corresponding to the non-player character. That is, the spatial human voice audio is the audio data corresponding to the non-player character obtained after the human voice audio is adjusted by the sound effect parameters. In some embodiments, the spatial human voice audio includes the audio after the sound effect parameters are adjusted, as well as at least one of the audio duration information, the audio playback start timestamp, the audio playback end timestamp, the audio playback condition information, and the element identification information corresponding to the audio.
[0185] Optionally, when the sound effect parameters indicate that the volume of the non-player character's vocal audio is to be adjusted, the volume of the non-player character's vocal audio on at least one channel is adjusted based on the sound effect parameters. Optionally, when the sound effect parameters indicate that the playback delay of the non-player character's vocal audio is to be adjusted, the audio playback start time corresponding to the non-player character's vocal audio is adjusted based on the sound effect parameters. Optionally, when the sound effect parameters indicate that the speed of the non-player character's vocal audio is to be adjusted, the playback speed of the non-player character's vocal audio is adjusted based on the sound effect parameters. Optionally, when the sound effect parameters indicate the pitch of the non-player character's vocal audio, the frequency in the spectrum of the non-player character's vocal audio is adjusted based on the sound effect parameters. When the sound effect parameters indicate the timbre of the non-player character's vocal audio, a filter corresponding to the non-player character is determined based on the sound effect parameters, and the timbre corresponding to the vocal audio is adjusted through the filter.
[0186] In summary, when a user uses a natural language command to provide feedback to a non-player character in a three-dimensional virtual environment about the relevant information of the three-dimensional virtual environment, feedback information corresponding to the non-player character is generated and broadcasted based on the information perceived by the non-player character from the three-dimensional virtual environment, so that the feedback information can cover a variety of scenarios. When a natural language command instructs the non-player character to ask whether there is an enemy in the three-dimensional virtual environment, when the non-player character perceives that the virtual characters included in the three-dimensional virtual environment include hostile virtual characters, the main virtual character is reminded of the existence of enemies by broadcasting feedback information; when a natural language command instructs the non-player character to provide a virtual prop to the main virtual character, when the non-player character has a virtual prop that the user requires, the non-player character submits the virtual prop to the main virtual character, and reminds the main virtual character that the virtual prop has been submitted by broadcasting feedback information. When controlling non-player characters through natural language commands, feedback information can be broadcast based on the environmental perception information of the non-player characters. This can enrich the content of the feedback information and make the broadcast not limited to fixed trigger scenarios. Game applications can flexibly broadcast feedback information based on environmental perception information during game matches, enrich the feedback information broadcast corresponding to non-player characters, and improve the efficiency of human-computer interaction between non-player characters and players.
[0187] The method provided in this embodiment can enhance the player's sense of immersion during the game, increase the fluency of the player-NPC interaction process, and make the NPC feedback closer to the real player experience. In the face of different environments, different battle situations, and different self-states in the game, the NPC can provide players with instant and effective voice feedback to help players obtain key information in real time. It can also respond quickly to the instructions issued by the players, allowing players to understand the current execution progress of the instructions, and whether there are new changes in the surrounding environment of the game when the instructions are executed, so that the NPC is no longer in the interpretation mode of silently executing the behavior tree logic and playing fixed audio. To a greater extent, it provides players with an anthropomorphic teammate NPC interaction experience and the design of derivative gameplay.
[0188] In an optional embodiment, if Figure 3 As shown, the feedback information corresponding to the non-player character includes at least one of dynamic feedback 301 , instant feedback 302 , and execution feedback 303 .
[0189] 1. Dynamic feedback 301 includes at least one of the following: firefight scene feedback and environment perception feedback.
[0190] The firefight scene feedback includes at least one of the following: attack feedback when the non-player character or the main virtual character is attacked; escape feedback when the hostile virtual character escapes; danger warning feedback when the non-player character finds danger; kill feedback when the non-player character kills the hostile virtual character; death feedback when the friendly virtual character is killed; reload feedback when the non-player character changes bullets; projectile attack feedback when the non-player character is attacked by projectiles. Environmental perception feedback includes at least one of the following: battle trace feedback when the non-player character finds battle traces in the three-dimensional virtual environment; looting trace feedback when the non-player character finds looting traces in the three-dimensional virtual environment.
[0191] Firefight scenes are areas in the game where players engage in fierce battles with other players or NPCs. They usually have rich terrain, cover, and tactical elements to increase the strategic and interesting nature of the battle. Throwing refers to the tactical behavior of players using throwing objects such as grenades and flash bombs to attack or control the enemy in battle. Searching for traces means that in the game, players can obtain information about the enemy's whereabouts, combat situation, etc. through searching for traces, so as to better formulate tactics and deal with the enemy. Combat traces are traces left by combat behavior in the game, such as bullet holes, broken objects, etc., which can help players judge the enemy's movement trajectory and combat situation, and are mainly reflected in the reward box after defeating the enemy.
[0192] For example, when the player does not give instructions to the NPC, the NPC will automatically provide feedback to the player through its own perception of the game scene, helping the player understand the battle information in real time and improving the gaming experience.
[0193] For the feedback of the firefight scene, it can be divided into three types according to the benchmarks: enemies, NPCs themselves, and teammates. The firefight scene refers to the situation where there may be hostile virtual characters nearby or potential dangers.
[0194] The benchmark is the enemy: when a non-player character perceives a hostile virtual character, a spatial query will be performed based on the location of the hostile virtual character to obtain a location description of the hostile virtual character, and then template splicing will be performed to obtain fire scene feedback such as "XXX was attacked" and "XXX discovered the enemy"; when the enemy is killed or the enemy escapes, a spatial query with the same logic will be performed, and fire scene feedback such as "killed", "enemy fell to the ground", "enemy escaped" will be fed back according to the probability; or the location description obtained by splicing the spatial query will be used to feedback the fire scene feedback such as "XXX killed the enemy", "XXX fell to the ground", and "XXX enemy disappeared".
[0195] The benchmark is the NPC itself: when a non-player character is changing ammunition, the feedback is "changing ammunition"; when a non-player character is attacked by projectiles, the feedback is "projectiles incoming" (grenades, flash bombs, etc.); when a non-player character senses a dangerous situation, the feedback is "danger nearby".
[0196] Benchmark is teammates: When a non-player character senses that a friendly virtual character has died, a report of the friendly virtual character's death is fed back, for example, "No. 3 has died." The purpose of this report is to clearly inform players which other NPCs can receive natural language commands, to prevent players from issuing natural language commands to dead NPCs.
[0197] Regarding environmental perception feedback, environmental perception feedback covers the automatic feedback logic of NPCs outside of firefighting battle scenes, including feedback broadcasts of two types of intelligence: battle traces and looting traces. These two types of broadcasts can provide players with information about historical situations that have been released in the three-dimensional virtual environment. For example, a battle has occurred in a certain area and someone was defeated, or someone else has arrived in the area and looted. At this time, non-player characters need to use environmental perception feedback to remind players to pay attention to their surroundings and avoid being attacked.
[0198] Battle trace broadcast: Reward boxes will appear randomly in the three-dimensional virtual environment. Reward boxes are item containers that provide players with additional rewards in the game. They usually contain valuable props, equipment or resources. Players can obtain them by completing tasks, defeating enemies, etc. Since the reward boxes are dynamically generated after a kill occurs during the game, they cannot be found through the static entity data of the three-dimensional virtual environment. Therefore, the game application will control the NPC to periodically search for the reward boxes around it, and after finding the reward boxes, it will determine whether it was killed by its own team or whether it was caused by the defeat of teammates. If neither of the above two situations is met and the non-player character finds the reward box for the first time, the non-player character needs to broadcast it. For example, directly call the voice package produced offline to feedback "Battle traces found here". At the same time, the game application will add the reward box to the cache to avoid subsequent repeated broadcasts.
[0199] Looting traces report: Since looting traces are also generated by other virtual characters during the game, they also need to be acquired in a state. Therefore, the game application will also regularly search for lootable items in the "opened" state around the NPC, and determine whether the past interaction records of the lootable items include the main virtual character. If not, and the non-player character discovers the lootable item for the first time, the offline voice package can be directly called to feedback "locating traces are found here". At the same time, the looted items are added to the cache to avoid subsequent repeated reports.
[0200] 2. Instant feedback 302 includes at least one of the following: instant feedback on command reception and reply broadcast in response to natural language commands.
[0201] Among them, the inquiry natural language command may include: at least one of battlefield intelligence inquiry, target location inquiry, and chat inquiry. According to different inquiry questions, corresponding reply broadcasts can be generated.
[0202] For example, the instant feedback of receiving a natural language command can be: the instant feedback of the NPC to the natural language command issued by the player can ensure that the player feels that "the natural language command just now has been successfully transmitted to the NPC". Instant feedback is required regardless of whether the natural language command is executable.
[0203] For example, the instant feedback for fine-tuning commands and movement commands can be "Received", "Understood", or "No. 3 received"; the instant feedback for purposeful movements such as viewing and attacking commands can be "Advancing"; the instant feedback for assembly protection movement commands can be "On the way" or "Returning"; the instant feedback for meaningless commands such as player chat can be "I don't understand" or "Please don't chat on the battlefield", etc.
[0204] For another example, the instant feedback includes instant feedback in response to player inquiries. Player inquiries in the game may include at least one of the following: intelligence inquiries, location inquiries, and chat inquiries.
[0205] For intelligence inquiries: players can ask NPCs about the current battle situation, whether a certain location is safe, etc. At this time, the game application needs to query the area based on the location of the NPC accepting the question using spatial query logic, and then generate response feedback and broadcast it based on whether the NPC perception system perceives hostile virtual characters. For example, "There are enemies on the second floor of the stable, be careful to hide" or "I am safe here."
[0206] Regarding location inquiries: players can ask NPCs about the current location of NPCs, hostile virtual characters, attack sounds, etc. At this time, spatial query logic will be used to generate reply feedback text, and online text-to-speech services will be used to generate reply feedback voice for broadcast.
[0207] For chat inquiries: players can chat with NPCs at will, such as "How is the current weather?", "Have you eaten?", and the large language model's inference service will generate the corresponding reply feedback text. When encountering questions that are not covered by the inference service, a reply will be selected from the preset rejection template, such as "Don't chat, please fight seriously" and "The conversation has nothing to do with the game". After receiving the reply feedback text, the game application calls the online text-to-speech service to generate the reply feedback voice and then broadcasts it.
[0208] 3. Execution feedback 303 includes at least one of the following: entity query feedback, virtual prop usage feedback, movement feedback, door hinge feedback, and item interaction feedback.
[0209] When the intention of the natural language command is to command the non-player character to perform a task, the task may be a task related to the target entity. The game application needs to first query the target entity referred to in the natural language command from the three-dimensional virtual environment, and then command the non-player character to perform the task related to the target entity. At this time, entity query feedback can be generated based on the entity query result to indicate the result of the entity query. If the intention of the task natural language command is to command the non-player character to use a virtual prop, virtual prop usage feedback can be generated based on the usage of the virtual prop. If the intention of the task natural language command is to command the non-player character to move, movement feedback can be generated based on the movement result, for example, the movement feedback can include successful movement to the target entity or target position, or the movement feedback can include movement failure. If the intention of the task natural language command is to command the non-player character to interact with the door (for example, open or close the door), door hinge feedback can be generated based on the interaction result between the non-player character and the door. If the intention of the task natural language command is to command the non-player character to perform item interaction, for example, to command the non-player character to exchange virtual props with the main virtual character, item interaction feedback can be generated based on the item interaction result.
[0210] For example, for entity search feedback, entity search distinguishes between "can be found" and "cannot be found". The large language model will directly return feedback text based on the search results, such as "folder found" and "cannot find safe nearby". Game applications can directly convert text to speech to get the feedback corresponding to the feedback text.
[0211] For example, for feedback on thrown objects (virtual props), thrown objects include grenades, smoke bombs, incendiary bombs, stun grenades, decoy bombs, etc. When the non-player character performs an action, it will broadcast "XXX is ready" in the preparation stage, and "XXX has been thrown" after the throwing is completed. Since the feedback text related to the thrown objects here is small and limited, the game application can directly use the offline packaged audio resources for playback.
[0212] For example, for movement feedback, movement instructions are divided into two categories according to the moving target point: "moving to the physical position" and "moving to the spatial position". Movement feedback is divided into two situations: "moving successfully" and "moving failed" according to the movement execution results.
[0213] When the movement is successful, the feedback text of reaching the entity position is spliced by the large language model, such as "reached XXX position" and "reached near XXX", where XXX is the target entity extracted from the large language model. When reaching the building space position, the feedback text is designed to provide feedback based on the combat information. If there is a hostile virtual character in the building space position, it will broadcast "XXX has an enemy, pay attention to concealment", and if there is no hostile virtual character, it will broadcast "XXX is safe here", where XXX is the name of the area.
[0214] If the move fails, the message "Cannot move closer to the target location" will be displayed. Non-player characters cannot reach the target location when the center of a large target location is too far from the NPC's navigation grid, causing the NPC to move to its edge. It may also be because the target location marked by the player on the minimap fails to be mapped to the navigation grid.
[0215] For example, for door interaction feedback, the execution feedback of interacting with the door in the scene is divided into two cases: "there is an interactive door" and "there is no interactive door".
[0216] "An interactive door" means that the NPC can find the "door" entity through entity search around it, and the current state of the door is opposite to the player's natural language command; for example, if the player issues an order to open the door, and the current door is closed, the NPC's feedback content is "opening / closing the door".
[0217] "No interactive doors" means that the query of entities around the NPC fails, or the status of the query result is consistent with the player's expectations; for example, when the player issues an order to open the door, the door itself is already open, at which point the NPC can give feedback that "there are no doors nearby that need to be opened or closed."
[0218] For example, item interaction feedback refers to the player's request to the NPC to deliver items, such as "Give me a medical kit", "I'm out of bullets, give me bullets", etc. When the NPC can deliver the corresponding item, it will directly feedback "Give you XXX" or "I put XXX here". When the NPC cannot deliver due to business logic restrictions or the non-existence of the item, it will directly fill in the entity word "I don't have XXX" or "I can't provide XXX" for feedback.
[0219] In summary, the method provided in this embodiment has three types of feedback: immediate feedback, instruction execution feedback, and dynamic feedback. The text content of immediate feedback and instruction execution feedback will make corresponding feedback based on the instructions inferred by the language model combined with static and dynamic game environment data, covering "immediate response", "start execution", "successful execution", "failed to execute successfully", etc.; the text content of dynamic feedback is when the player does not issue an instruction during the game, the NPC will provide necessary feedback based on real-time environmental perception.
[0220] In an optional embodiment, the feedback information includes a reply report. The reply report includes a reply to the query natural language command generated in response to the query natural language command raised by the player. For example, when the query natural language command includes querying the location of the target entity, the corresponding reply report should include the query result of the location of the target entity.
[0221] Figure 4A flowchart of a method for broadcasting a non-player character provided by an exemplary embodiment of the present application is shown. The method can be performed by the above Figure 1 Based on the terminal in Figure 2 In the illustrated embodiment, step 230 of the method may include step 231 .
[0222] Step 231, when the behavioral intention of the natural language command is to inquire, broadcast the reply broadcast of the non-player character; the reply broadcast includes the reply content for the natural language command generated according to the environmental perception information of the non-player character on the three-dimensional virtual environment.
[0223] The natural language command is used to inquire about information from the non-player character. The natural language command may be referred to as an inquiry natural language command. The inquiry natural language command is a natural language command intended to be an inquiry. The inquiry natural language command contains at least one question.
[0224] Natural language commands are command instructions conveyed using human natural language. The terminal receives the natural language commands issued by the player and commands the non-player characters according to the intention corresponding to the natural language commands. For example, the terminal receives the voice audio of the player, converts the voice audio into text, and obtains the natural language command. Alternatively, the terminal receives the text of the natural language command input by the player.
[0225] Players can use spoken or written language to issue natural language commands. The game application will use the large language model to understand the natural language commands, extract the intentions expressed in the natural language commands, generate control instructions based on the intentions, and direct the activities of non-player characters.
[0226] For example, a natural language command may be "Come to me", and the game application may parse out that the intention of the natural language command is "control the non-player character to move to the location of the controlling virtual character". The game application then obtains the location of the controlling virtual character, generates a navigation route for the non-player character to move to the location of the controlling virtual character, and controls the non-player character to move to the location of the controlling virtual character along the navigation route.
[0227] For another example, a natural language command may be "go pick up a treasure chest", then the game application may parse out that the intention of the natural language command is "control the non-player character to move to the location of the treasure chest and pick up the treasure chest", then the game application may obtain the location of the treasure chest, control the non-player character to move to the location of the treasure chest, and perform the operation of picking up the treasure chest after arriving at the location of the treasure chest.
[0228] For example, when the natural language command includes an intelligence inquiry about the target entity, a first reply report is broadcast based on the non-player character's perception of the target entity; the first reply report includes the intelligence perception result of the non-player character about the target entity.
[0229] Alternatively, when the natural language command includes an inquiry about the location of the target entity, a second reply broadcast is broadcast based on the non-player character's perception of the target entity, and the second reply broadcast includes a description of the location of the target entity.
[0230] Exemplarily, a large language model is called to parse the received natural language command to obtain the intention of the natural language command. When the intention is of an inquiry type, the natural language command can be determined to be an inquiry natural language command. When the natural language command is an inquiry natural language command, the large language model can be inferred according to the intention of the natural language command, based on static entity data and environmental perception information, to obtain a reply feedback text. That is, the large language model is called to parse the received natural language command, and the intention of the natural language command is output as an inquiry, as well as the reply feedback text of the natural language command. The game application calls the text-to-speech service according to the processing logic of the inquiry intention, and converts the reply feedback text into a reply broadcast, which is an audio broadcast, and sends the reply audio to the client for broadcast.
[0231] For example, when the large language model recognizes that the intention of the natural language command is to ask a question, since the user may ask hundreds of questions and it is impossible to store the answer to each question locally on the client, the game application will call the text-to-speech service to generate a reply broadcast in real time based on the reply feedback text returned by the large language model, so that the game application can instantly generate a corresponding reply broadcast to respond to the player's question.
[0232] Exemplarily, a game application (client or server) calls a large language model to parse natural language commands, and obtains reply feedback text based on static environmental data and environmental perception information of the three-dimensional virtual environment; and generates human voice audio based on the reply feedback text to obtain a reply broadcast.
[0233] For example, the client reports the received natural language query command to the server, the server calls the large language model to parse the natural language query command, and infers the reply feedback text based on the static environment data and environmental perception information of the three-dimensional virtual environment; generates human voice audio based on the reply feedback text to obtain a reply broadcast; the server sends the reply broadcast to the client; and the client broadcasts the reply broadcast.
[0234] Static environment data is used to describe relatively unchanged environmental features in a three-dimensional virtual environment. Static environment data may include model data of a three-dimensional model that is fixed in the three-dimensional virtual environment. Static environment data includes at least one of the following information: the name of the three-dimensional model, introduction text, location information, orientation information, and spatial relationship (for example, a cabin includes a kitchen and a bedroom). Exemplarily, the static environment data includes entity information of all entities in the three-dimensional virtual environment.
[0235] Dynamic reasoning can be performed based on static environmental data that does not change, and dynamically changing environmental perception information. For example, when the environmental perception information includes the sound effect of flowing water emitted by a received river, the river in the direction of the sound effect emitted by the NPC can be found from the static environmental data based on the NPC's current position, the NPC's current orientation, the direction of the sound effect emitted by the flowing water, and the type of entity that can emit the sound effect (river or stream), and the position of the river in the three-dimensional virtual environment can be determined.
[0236] In summary, the method provided in this embodiment, when receiving an inquiry natural language command, the game application can combine the environmental perception information of the non-player character, infer the reply feedback text corresponding to the inquiry natural language command, and generate reply feedback. By replying to the player's question through reply feedback, the non-player character can interact with the player in the game in real time, and provide the player with richer in-game information based on the perception of the non-player character.
[0237] In an optional embodiment, the feedback information includes a feedback report. The feedback report includes a report on the task execution status and task execution result generated in response to the task natural language command proposed by the player. For example, when the task natural language command includes moving to a target entity, the corresponding feedback report may include: moving to the target entity, or having reached the target entity.
[0238] Figure 5 A flowchart of a method for broadcasting a non-player character provided by an exemplary embodiment of the present application is shown. The method can be performed by the above Figure 1 Based on the terminal in Figure 2 In the illustrated embodiment, step 230 of the method may include step 232 .
[0239] Step 232, when the behavioral intention of the natural language command is to execute the task, a feedback report is broadcast; the feedback report includes the execution status after the natural language command is executed according to the environmental perception information of the non-player character on the three-dimensional virtual environment.
[0240] The natural language command is used to instruct the non-player character to perform a task. The natural language command may be referred to as a task natural language command. The task natural language command includes a description text of the task, for example, the description text of the task may include at least one of the following: the task name, the action that the non-player character needs to perform, the method used to perform the task, the task target, and the location of the task target.
[0241] For example, when the intention of the task natural language command includes searching for a target entity, a first feedback announcement is broadcast, and the first feedback announcement includes the search result of the non-player character for the target entity.
[0242] Alternatively, when the intention of the task natural language command includes using a virtual prop, a second feedback announcement is announced, and the second feedback announcement is used to indicate that the virtual prop is ready.
[0243] Alternatively, when the intention of the task natural language command includes the use of virtual props, a third feedback announcement is broadcast, and the third feedback announcement is used to indicate the result of the use of the virtual props.
[0244] Alternatively, when the intention of the task natural language command includes controlling the movement of the non-player character, a fourth feedback announcement is announced, and the fourth feedback announcement is used to indicate the movement result of the movement.
[0245] Alternatively, when the intention of the task natural language command includes performing an interaction with a target entity, a fifth feedback announcement is broadcast, and the fifth feedback announcement is used to indicate at least one of a search result of the non-player character for the target entity and an interaction result between the non-player character and the target entity.
[0246] Alternatively, when the intention of the task natural language command includes item interaction, a sixth feedback broadcast is broadcast, and the sixth feedback broadcast is used to indicate the item interaction result of the non-player character performing the item interaction.
[0247] Exemplarily, a large language model is called to parse the received natural language command to obtain the intention of the natural language command. When the intention is a task, the natural language command can be determined to be a task natural language command. When the natural language command is a task natural language command, the large language model can be inferred according to the intention of the natural language command, based on static entity data and environmental perception information, to obtain a task instruction sequence, and the task instruction sequence is used to control the non-player character to perform the task. When the game application receives the task instruction sequence, it controls the non-player character activity to perform the task according to the task instructions in the task instruction sequence. During the control of the non-player character according to the task instruction, the corresponding execution status can be obtained for feedback broadcast according to the feedback broadcast logic corresponding to different task instructions.
[0248] For example, since the instructions that can be executed by non-player characters in a three-dimensional virtual environment are finite and traversable, the client can locally store feedback broadcasts corresponding to each character's instructions. When the non-player character executes the corresponding instruction, the client can read the local feedback broadcast for voice broadcast.
[0249] Alternatively, when there is variable content in the feedback broadcast corresponding to a task instruction, for example, the target entity in the feedback broadcast is variable, or the position information in the feedback broadcast is variable; the client can also generate a broadcast voice with variable content based on the target entity or position information returned by the large language model, and splice it with the broadcast voice with immutable content stored locally to obtain the final feedback broadcast.
[0250] For example, when the task natural language command is "find a nearby treasure chest", calling the large language model to parse the task natural language command can get the intention of the task, the task goal is to find the target entity, and the target entity is a treasure chest; based on the static entity data and environmental perception information, reasoning can be obtained that there is a treasure chest in the left front of the main control virtual character. In the case that the target entity can be found, the feedback broadcast corresponding to the task goal is "found aaa at xxx", where xxx is the location of the target entity and aaa is the name of the target entity. The game application can call the online text-to-speech technology to generate the location voice according to the location information "left front" in the reasoning result; according to the name of the target entity "treasure chest", call the online text-to-speech technology to generate the name voice, and then splice the location voice and name voice into the feedback broadcast to get the final feedback broadcast.
[0251] Exemplarily, a game application (client or server) calls a large language model to parse natural language commands for a task, and infers task execution instructions based on static environmental data and environmental perception information of a three-dimensional virtual environment; controls a non-player character to perform the task according to the task execution instructions; generates status feedback text based on the execution status of the non-player character performing the task; and generates human voice audio based on the status feedback text to obtain a feedback broadcast.
[0252] For example, the client reports a task natural language command to the server; the server parses the task natural language command and infers the task execution instructions based on the static environment data and environment perception information of the three-dimensional virtual environment; the non-player character is controlled to perform the task according to the task execution instructions; status feedback text is generated according to the execution status of the non-player character performing the task; human voice audio is generated based on the status feedback text to obtain a feedback broadcast; the server sends a feedback broadcast to the client; the client receives and broadcasts the feedback broadcast.
[0253] In summary, the method provided in this embodiment, when receiving a task natural language command, can generate execution feedback in real time based on the environmental perception information of the non-player character during the process of directing the non-player character to perform the task, and report the status of the non-player character performing the task to the player. It can also generate execution feedback based on the task completion result after the task is completed, and report the result of the non-player character performing the task to the player. This enables the player to grasp the status and progress of the non-player character performing the task in real time, and provides a smooth interactive experience for the player.
[0254] In an optional embodiment, the feedback information includes dynamic reporting. The dynamic reporting includes reporting of abnormal situations perceived by non-player characters. For example, when a non-player character perceives being attacked, a report is made that the character is being attacked; when a non-player character perceives a dangerous situation ahead, a report is made that there is danger ahead.
[0255] Figure 6 A flowchart of a method for broadcasting a non-player character provided by an exemplary embodiment of the present application is shown. The method can be performed by the above Figure 1 Based on the terminal in Figure 2 In the illustrated embodiment, the method may further include step 240 , which is a parallel step of steps 220 to 230 .
[0256] Step 240 : When the environmental perception information of the non-player character about the three-dimensional virtual environment meets the dynamic broadcasting conditions, a dynamic broadcast is broadcasted.
[0257] Dynamic reporting is a report generated by the game application based on the environmental perception information of non-player characters when it determines that there is an abnormal situation that needs to be reported. The dynamic report includes reminder information of the abnormal situation. For example, the abnormal situation may include at least one of the following: discovering a hostile virtual character, discovering a change in the status of a friendly virtual character, and discovering a new trace.
[0258] Dynamic broadcast conditions are used to trigger dynamic broadcasts. Dynamic broadcast conditions are trigger conditions set based on environmental perception information. Dynamic broadcast conditions can be set according to abnormal situations. For example, dynamic broadcast conditions can include at least one of the following: discovering hostile virtual characters according to the NPC's environmental perception information (for example, a three-dimensional model of a hostile virtual character appears within the NPC's field of view, the NPC receives the combat sound effects of the hostile virtual character, etc.), discovering changes in the status of friendly virtual characters according to the NPC's environmental perception information (for example, a friendly virtual character is killed within the NPC's field of view, the NPC receives a notification message that a friendly virtual character is killed, and the NPC receives the attack sound effects of a friendly virtual character), and discovering new traces according to the NPC's environmental perception information (for example, new combat traces or looting traces appear within the NPC's field of view).
[0259] For example, in the case where the non-player character perceives a hostile virtual character, a first dynamic broadcast is broadcast; the first dynamic broadcast includes the position of the hostile virtual character perceived by the non-player character.
[0260] Alternatively, when the non-player character senses a dangerous situation, a second dynamic broadcast is broadcast; the second dynamic broadcast is used to prompt the dangerous situation.
[0261] Alternatively, when the non-player character senses a change in the state of the friendly virtual character, a third dynamic broadcast is broadcast; the third dynamic broadcast is used to prompt that the state of the friendly virtual character has changed.
[0262] Alternatively, when the non-player character finds new traces in the three-dimensional virtual environment, a fourth dynamic broadcast is broadcast; the fourth dynamic broadcast is used to prompt the new traces. The new traces may be combat traces and / or looting traces.
[0263] For example, since the number of dynamic announcements that can be triggered by non-player characters in a three-dimensional virtual environment is limited and traversable, the client can store dynamic announcement voices locally, and the client can read the dynamic announcement voices locally for announcement when the corresponding dynamic announcement voices are triggered. Similarly, some dynamic announcement voices can contain variable content, and the client can generate variable content in real time through text-to-speech technology, and splice it with the dynamic announcement template to obtain the final dynamic announcement.
[0264] In summary, the method provided in this embodiment can spontaneously generate dynamic feedback based on environmental perception information when a non-player character perceives an abnormal situation, and remind the player of the abnormal situation nearby. For example, when a non-player character finds a hostile virtual character, the position of the hostile virtual character is broadcast according to the environmental perception information; when a non-player character finds a dangerous situation, the player is reminded of the danger; when a friendly virtual character's status changes, the player is reminded of the status of teammates, etc. This allows the player to learn about the non-player character's perception of the environment and make better combat decisions based on the non-player character's perception.
[0265] In an optional embodiment, the feedback information includes a description of the location of the target entity. Schematically, when generating the feedback information, it may be necessary to query the location of the target entity in the three-dimensional virtual environment, or to determine the target entity indicated in the natural language command from the three-dimensional virtual environment. In this case, an entity query method needs to be used to query the target entity.
[0266] Before executing the target entity query, it is necessary to pre-build the static entity data of the 3D virtual environment. The static entity data construction can serve both the reasoning of the large language model and the real-time feedback text generation on the game application side after the reasoning results are returned.
[0267] In an optional embodiment, entities in the three-dimensional virtual environment are traversed, and entity information of each entity is exported from a game engine to obtain static entity data.
[0268] For example, for in-game scenes, editor tools have been developed to perform StaticMesh traversal of the entire scene, special Actor traversal (such as interactive doors, which do not belong to the StaticMesh category), vegetation traversal, and export the entity's position, orientation, and bounding box size as static entity data for subsequent labeling and reasoning by the large language model. Figure 7 The vehicle 701, box 702, tree 703, interactive door of the building, etc. can all be used to collect static entity data using this method.
[0269] Among them, StaticMesh is a static geometry resource type in the Unreal Engine 4 (UE4) game engine, which is used to represent immutable 3D models, such as buildings, props, etc. Actor is a basic object class in the Unreal Engine 4 (UE4) game engine, representing an entity in the game world, such as a character, object, light source, etc. Actor can contain multiple components to achieve different functions.
[0270] Exemplarily, static entity data can also include spatial data, which is obtained by manual labeling. Spatial data is more complex than static entities because there are multiple floors in the space and multiple layers of nesting within the floors (such as the motel area includes the second floor, and the second floor area includes room 201). For this type of data, manual calibration is used to construct it. Add Volume to the scene in the editor, divide the area required for the entire map and label it accordingly. Volume is a special Actor class in the UnrealEngine 4 (UE4) game engine, which represents a three-dimensional area with specific functions, such as a trigger area, audio area, etc. For example, Figure 8 Volume area 704 in the map is labeled "Farm Map - Recycling Station Tower - Second Floor Area".
[0271] After obtaining the static entity data, you can query entities or spaces based on the static entity data during the game.
[0272] For example, when a player issues a natural language command, the natural language command will be accompanied by the constructed static entity data and the real-time captured runtime data (including the environmental perception information of non-player characters and / or the main virtual character) to make an inference request to the large language model. In addition, when the NPC needs to provide dynamic feedback based on the runtime data, an entity query is also performed to generate feedback text based on the current state.
[0273] When a natural language command includes a target entity, the game application will query the target entity based on the position and orientation of the current controlling virtual character, the position and orientation of teammates or enemies, and based on the target entity information of the target entity described in the natural language command and the environmental perception information of the controlling virtual character and / or the non-player character, filter out a target entity that matches the target entity information from at least one candidate entity within the perception range of the controlling virtual character and / or the non-player character.
[0274] Optionally, the natural language command is used to instruct the non-player character to perform an activity related to the target entity. For example, the target entity may be a destination for the non-player character to move to, an object that the non-player character needs to observe, a target that the non-player character needs to attack, or an object that the non-player character needs to interact with.
[0275] The target entity is a virtual character existing in a three-dimensional virtual environment. The target entity is the entity described in a natural language command. An entity may refer to a virtual character that is stationary in a three-dimensional virtual environment. For example, a target entity may be a virtual building, a virtual terrain, a virtual vehicle, a virtual plant, a virtual prop, a virtual item, etc. An entity may also refer to a virtual character that can interact with a virtual character in a three-dimensional virtual environment. For example, a target entity may be an interaction point in a three-dimensional virtual environment (e.g., a door, a window, a cabinet, a cellar, etc.), a virtual prop, a virtual character, a virtual light source, etc.
[0276] For example, a natural language command may be "Please help me open the door", then the target entity may be "door", and the non-player character needs to perform a door-related activity "open the door". Alternatively, a natural language command may be "Is the kitchen safe", then the target entity may be "kitchen", and the non-player character needs to perform a kitchen-related activity "check if there are enemy avatars or other dangerous situations in the kitchen".
[0277] Optionally, the natural language command includes target entity information of the target entity. The target entity information may be a description text of the target entity. For example, the target entity information may include at least one of the following contents: the name, type, location, and features of the target entity. The game application may find the target entity from among the many entities in the three-dimensional virtual environment based on the target entity information in the natural language.
[0278] The game application parses the natural language command, extracts the target entity information therein, determines the target entity from the entity set of the three-dimensional virtual environment based on the target entity information, and then controls the non-player character to perform activities related to the target entity according to the intention of the natural language command.
[0279] The target entity is determined from the three-dimensional virtual environment in combination with environmental perception information, and the environmental perception information includes information perceived by at least one of the main virtual character and the non-player character from the three-dimensional virtual environment.
[0280] Since the target entity information in natural language commands is usually vague, for example, a natural language command may be "move to the red truck", where the target entity information is "red truck", and there may be many red trucks in the three-dimensional virtual environment. It is impossible to accurately determine the target entity in the three-dimensional virtual environment based solely on the target entity information in the natural language command.
[0281] Therefore, the embodiment of the present application provides a method for determining the target entity by combining the target entity information and the environmental perception information. For the previous example, the "red truck" expressed by the player in the natural language command should be a red truck that the player can see. Therefore, combined with the field of view of the main control virtual character, the red truck within the field of view of the main control system character can be screened out from multiple red trucks in the three-dimensional virtual environment. The red truck is the target entity indicated by the player in the natural language command.
[0282] Since natural language commands are issued by players based on their perception of the 3D virtual environment, in order to accurately identify the target entity indicated in the natural language command, the game application will combine the environmental perception information when the player issues the natural language command to identify the target entity in the natural language command. Based on the player's perception of the 3D virtual environment, the entity that the player can perceive in the 3D virtual environment and that is closest to the target entity information is inferred as the target entity.
[0283] Exemplarily, the three-dimensional virtual environment includes multiple candidate entities that match the natural language command, and the target entity is an entity selected from the multiple candidate entities based on the environmental perception information of the main virtual character or the non-player character. For example, the game application first selects multiple candidate entities that match the target entity information in the natural language command from the entity set, and then selects the entity that can be perceived by the main virtual character or the non-player character from the multiple candidate entities as the target entity based on the environmental perception information.
[0284] For example, Fig. 9As shown, the perception range of the main control virtual character is determined according to the position and orientation 916 of the main control virtual character. For example, the perception range is a sector area of a certain visual field size in front of the main control virtual character; entities that are not within the sector range are filtered out. Then, the target entity information and the entity information of the candidate entities are sorted according to the similarity, and entities with higher similarity are selected as target entities. When there are two entities with the highest and equal similarity scores within the perception range, for example, the similarity scores of entity 2 and entity 3 are both 2 points, then entity 2, which is closer to the main control virtual character, is selected as the target entity.
[0285] When it is necessary to generate feedback information based on the position description of the target entity, the game application can perform a spatial query to generate a position description based on the position relationship between the master virtual character and the target entity.
[0286] The main application scenario of spatial query is to obtain the location of the main virtual character, non-player characters, hostile virtual characters, gunshots, etc. For example, a report containing spatial information such as "enemies found on the second floor of the motel" can be made.
[0287] The spatial query logic defines the concepts of parent area and child area. The parent area refers to a large area in a 3D virtual environment that covers multiple buildings or multiple isolated spaces, such as a motel with front and back yards and multiple rooms and basements. The child area refers to an independent space in the parent area, which can be a closed space or a special function space, such as a motel guest room or kitchen.
[0288] In order to make the location description returned by the spatial query closer to human expression, when the main virtual character and the queried location are in different parent areas, the query result returns complete text information (for example, the main virtual character is outside the motel and the queried location is on the second floor of the motel, then "the enemy is on the second floor of the motel" is returned). When the main virtual character and the queried location are in the same parent area but different child areas, the query result only returns the child area information (for example, the main virtual character is on the first floor of the motel and the queried location is on the second floor, then "the enemy is on the second floor" is returned). When the main virtual character and the queried location are in the same parent area and the same child area, the query result only returns the direction relative to the main perspective player (for example, the main virtual character and the queried location are both on the second floor of the motel, then "the enemy is in the right front position" is returned).
[0289] For example, Fig.10 As shown in (1), when the master virtual character 1011 and the target entity 1010 are located in different parent spaces, the position description includes the parent space and the child space where the target entity is located.
[0290] like Fig.10As shown in (2), when the master virtual character 1011 and the target entity 1010 are located in different child spaces of the same parent space, the position description includes the child space where the target entity is located.
[0291] like Fig.10 As shown in (3), when the controlling virtual character 1011 and the target entity 1010 are located in the same child space of the same parent space, the position description includes the relative position of the target entity and the controlling virtual character; wherein a parent space includes at least one child space.
[0292] In some embodiments, the identification process of the target entity in the three-dimensional virtual environment can be determined according to the target entity information indicated by the natural language command and the environmental perception information. In an exemplary manner, according to the target entity information environmental perception information indicated by the natural language command, the target entity is queried from the entity set of the three-dimensional virtual environment.
[0293] Optionally, the client infers the target entity based on the target entity information, the environment perception information and the preprocessed three-dimensional virtual environment data. The preprocessed three-dimensional virtual environment data includes a set of entities in the three-dimensional virtual environment. The entity set includes entity information of each entity in the three-dimensional virtual environment. The entity information includes at least one of the following: name, type, location, feature, text label, embedding vector of the text label, image, and embedding vector of the image. The text label can be at least one word obtained after word segmentation of the feature (for example, a descriptive text of the entity appearance), and the embedding model is called to obtain the embedding vector corresponding to each text label. The image may include an image obtained by observing the three-dimensional model of the entity from at least one direction, for example, the image may include three views of the entity. The embedding vector of the image is the embedding vector of the image feature label. The image feature label is obtained by calling the multimodal model to perform feature recognition based on the image and text label of the entity.
[0294] In some embodiments, natural language commands are parsed to obtain target entity information, wherein the target entity information includes at least one of the following: entity type, entity name, entity location, and entity features; similarity between the target entity information and each entity information in an entity set is calculated; and the target entity is determined from the entity set based on the similarity and environmental perception information.
[0295] Optionally, the client calls the large language model to parse the natural language command to obtain target entity information of the target entity, and calls the embedding model to vectorize the target entity information to obtain a target embedding vector of the target entity information.
[0296] For example, named entity recognition technology in the field of natural language processing can also be used to extract target entity information from natural language commands. Named entity recognition technology is used to identify entities with specific meanings in texts, such as names of people, places, locative words, adjectives, etc.
[0297] Furthermore, the similarity includes text similarity, and the entity set includes entity information of a first entity. Schematically, the similarity calculation process is implemented as follows: segmenting the target entity information to obtain at least one target entity tag; converting at least one target entity tag into at least one target embedding vector; obtaining entity information of the first entity, the entity information including a text embedding vector, which is an embedding vector obtained based on the text tag conversion of the first entity; respectively calculating the text parent similarity of at least one target embedding vector and the text embedding vector to obtain at least one text parent similarity corresponding to at least one target embedding vector; and determining the sum of at least one text parent similarity as the text similarity between the target entity information and the entity information of the first entity.
[0298] In some embodiments, the client first performs a similarity search based on the target entity information to obtain at least one candidate entity with a high similarity, and sorts them according to the similarity; then, based on the position, location, orientation and other information of the main virtual character and / or the non-player character, the client performs perceptual filtering and sorting, and finally obtains the target entity.
[0299] Exemplarily, the game application determines a target perception range based on target entity information and environmental perception information; and obtains the target entity by filtering entities perceived within the target perception range based on similarity.
[0300] Alternatively, the game application determines x entities with the highest similarity from the entity set according to the similarity to form a candidate entity list; determines the target perception range according to the target entity information and the environmental perception information, and determines the entity in the target perception range among the x entities as the target entity. x is a positive integer.
[0301] Exemplarily, the game application determines the entity with the highest similarity in the perception range as the target entity.
[0302] It should be noted that when there are multiple entities with equal similarity within the target perception range, the multiple entities can be sorted according to the distance between the entity and the controlling virtual character. For example, when there are at least two entities with the highest similarity in the perception range, the entity with the highest similarity in the perception range and the closest distance to the controlling virtual character is determined as the target entity.
[0303] For example, the search range is determined according to the position and orientation of the main virtual character. For example, if the target entity information indicates that the target entity is located in front of the main virtual character, the search range is a fan-shaped area of a certain visual field size in front of the main virtual character; entities that are not within the fan-shaped range are filtered out. Then, the entities are sorted according to the similarity. When there are two entities with the highest and equal similarity scores within the search range, that is, the similarity scores of entity 2 and entity 3 are both 2 points, entity 2, which is closer to the main virtual character, is selected as the target entity.
[0304] In an optional embodiment, the game application determines the entity with the highest similarity within the perception range and above the threshold as the target entity. Exemplarily, the number of the perception range is at least one. For example, the perception range includes at least one of the following: the visual range of the main virtual character, the hearing range of the main virtual character, the perception range of the main virtual character's perception of virtual props, the visual range of the non-player character, the hearing range of the non-player character, and the perception range of the non-player character's perception of virtual props.
[0305] Exemplarily, the game application may sequentially traverse at least one of the above-mentioned perception ranges, or the game application may sequentially traverse at least one perception range determined according to the natural language command. For example, if the natural language command is "Can you see the truck in front? Move to the truck", the game application may first traverse the visual range of the main virtual character, and then traverse the visual range of the non-player character, and match the entity with the highest similarity and above the threshold.
[0306] Exemplarily, when there are at least two perception ranges, the traversal order of the at least two perception ranges may be preset, or the traversal order of the at least two perception ranges may be determined according to the intention indicated in the natural language command. For example, when the intention is to search for an item, the visual range is traversed first. When the intention is to find a combat scene, the auditory range is traversed first.
[0307] If there are multiple levels of nested instructions in the natural language command, the first target entity will be searched first, and then the next target entity will be searched based on the position of the previous target entity. In the case where the natural language command includes a second target entity determined based on the position of the first target entity, the first target entity is queried from the entity set of the three-dimensional virtual environment based on the first target entity information and environmental perception information of the first target entity indicated by the natural language command; then the second target entity is queried from the entity set of the three-dimensional virtual environment based on the second target entity information, the first target entity and environmental perception information of the second target entity.
[0308] It is worth noting that the above target entity identification process can also be implemented by the server, which will not be described in detail below.
[0309] In summary, the method provided in this embodiment can enable the game application to generate feedback information that is closer to human expression. In response to the user's current perception, the game application can generate a location description text that is more in line with the current human perception. When the main virtual character and the target entity are in different buildings, the complete location information of the target entity is broadcast to the player; when the main virtual character and the target entity are in different spaces in the same building, the spatial location information is broadcast to the player; when the main virtual character and the target entity are in the same space in the same building, the orientation information is broadcast to the player, so that the player can intuitively know the spatial position relationship with the target entity through the broadcast content, thereby improving the efficiency of information transmission of the broadcast.
[0310] The method provided in this embodiment proposes a solution for NPC to provide real-time voice feedback to players during the game based on static offline data and runtime data of the game, combined with a large language model and text-to-speech technology, to bring players an immersive experience during the game.
[0311] In an optional embodiment, the processing of natural language commands includes intent recognition of natural language commands. Schematically, the non-player character has multiple behavioral capabilities in the virtual environment, and the multiple behavioral capabilities correspond to multiple classification labels. The command text corresponding to the natural language command is obtained, and at least one hierarchical prediction network is called to perform intent recognition on the command text to obtain a classification label of the command text's intent to command the non-player character, wherein the hierarchical prediction network includes at least two layers of sub-networks, and the at least two layers of sub-networks are constructed based on a tree structure of multiple classification labels, and the non-player character is controlled to perform a virtual activity according to the intention corresponding to the classification label, and feedback information of the non-player character is broadcast, and the feedback text corresponding to the feedback information is a non-fixed text generated based on the environmental perception information during the virtual activity.
[0312] Exemplarily, the hierarchical prediction network has the ability to predict the classification label corresponding to the command text. Exemplarily, the hierarchical prediction network includes at least two layers of sub-networks, the upper network and the lower network are cascaded, and the lower network further performs the prediction of the classification label according to the prediction results output by the upper network. At least two layers of sub-networks are constructed based on the tree structure of multiple classification labels. The hierarchical prediction network constructs at least two layers of sub-networks corresponding to the hierarchical structure for the tree structure of the classification labels, and splits the prediction tasks of a wide variety of classification labels into prediction sub-tasks performed by at least two layers of sub-networks, so as to reduce the complexity of classification prediction of each layer of sub-network. Exemplarily, due to the richness of natural language expression, there are many types of classification labels corresponding to natural language. By summarizing semantically similar classification labels into a high-level label, it is achieved that a wide variety of classification labels are constructed into a tree structure to clarify the similarity between classification labels.
[0313] In an optional implementation, each hierarchical prediction network in at least one hierarchical prediction network is used to predict the command intention of the command text for non-player characters on an intention dimension; the intention dimension includes: at least one of the subject dimension, the semantic dimension, and the behavior dimension. In one example, at least one hierarchical prediction network includes a subject prediction network, a behavior prediction network, and a semantic prediction network to realize the tactical instruction understanding of the command text in parallel; the command intention for non-player characters is described from three dimensions of the subject dimension, the semantic dimension, and the behavior dimension attack. It can be understood that performing intention recognition on the command text is also called performing tactical instruction understanding on the command text; by calling at least one hierarchical prediction network to perform tactical instruction understanding on the command text, a classification label of the command text intention to command the non-player character is obtained. Exemplarily, natural language has rich semantics, and there are rich semantic elements in the command text in natural language form. Each hierarchical prediction network predicts the command intention in the command text from different dimensions, which reduces the number of label types that need to be classified in the hierarchical prediction network prediction task on the one hand, and reduces the complexity of the classification prediction task; on the other hand, the hierarchical prediction network only needs to understand the natural language semantics in the command text on one dimension, which reduces the difficulty of extracting the natural language semantics in the command text.
[0314] Exemplarily, in at least one hierarchical prediction network, the i-th layer subnetwork in each hierarchical prediction network is used to predict the first-level behavior label of the command text on an intent dimension, and the i+1-th layer subnetwork in the hierarchical prediction network is used to predict the second-level behavior label corresponding to the command text in the first-level behavior label, where i is a positive integer; the high-level behavior label (such as the first-level behavior label) predicted by the high-level subnetwork (such as the i-th layer subnetwork) in the hierarchical prediction network includes multiple sublabels, or subordinate lower-level labels; the corresponding low-level subnetwork (such as the i+1-th layer subnetwork) needs to be called to further perform classification prediction (such as predicting the second-level behavior label). The prediction task of a wide variety of classification labels is split into prediction subtasks performed by at least two layers of subnetworks to reduce the complexity of classification prediction in each layer of subnetwork.
[0315] In some embodiments, hot word recognition is integrated into the recognition process of command text. Schematically, multiple scene hot words corresponding to the three-dimensional virtual environment are obtained, where the three-dimensional virtual environment is a virtual environment in which at least one of the main control virtual character or the non-player character is located, and the multiple scene hot words are scene-related vocabulary of the three-dimensional virtual environment; based on the multiple scene hot words, the natural language command is converted into command text.
[0316] Among them, a plurality of scene hot words are scene-related words of the three-dimensional virtual environment, that is, there is an association relationship between the scene hot words and the three-dimensional virtual environment, and the scene hot words are words that describe the three-dimensional virtual environment.
[0317] Optionally, the scene hot words include the scene status of the three-dimensional virtual environment, such as: the three-dimensional virtual environment is a virtual restaurant, and the first scene hot words include business status, rest status, closed status, etc. Optionally, the scene hot words include element names of virtual elements in the three-dimensional virtual environment, such as: the three-dimensional virtual environment is a virtual restaurant, and the scene hot words include virtual tables, virtual chairs, virtual kitchen utensils, virtual plates, etc. Optionally, the scene hot words include interactive words that interact with virtual elements in the three-dimensional virtual environment, such as: the three-dimensional virtual environment is a virtual battlefield, and the scene hot words include attack, attack virtual characters, defense, and build barriers, etc.
[0318] In some embodiments, after receiving a natural language command, multiple scene hot words are collected based on the three-dimensional virtual environment; or, after receiving a natural language command, multiple scene hot words corresponding to the three-dimensional virtual environment are filtered out from multiple scene hot words acquired in advance based on the three-dimensional virtual environment, etc.
[0319] In schematic form, since scene hot words are words that are highly associated with the three-dimensional virtual environment, when it is desired to use natural language commands to direct non-player characters to better cooperate with the activities of the main virtual character, the scene hot words can be used as constraints based on the three-dimensional virtual environment. This allows the command text to be more closely aligned with the first virtual scene during the process of converting the natural language commands into command text, thereby avoiding obtaining command text that is incompatible with the three-dimensional virtual environment, thereby reducing the error rate of command analysis.
[0320] In some embodiments, a natural language command is analyzed by a pre-trained natural language analysis model, and the natural language command and multiple scene hot words are used as inputs of the model. The instruction feature representation of the natural language command and the hot word feature representation of multiple scene hot words are extracted by the natural language analysis model, the instruction feature representation and the hot word feature representation are fused to obtain a fused feature representation, and the command text is output based on the fused feature representation.
[0321] In some embodiments, natural language commands are analyzed by a pre-trained natural language analysis model, and the natural language commands and multiple scene hot words are used as inputs of the model. In the process of analyzing the natural language commands by the natural language analysis model, an attention mechanism is used to determine the weights of the scene hot words in the instruction analysis process, thereby achieving the purpose of using multiple scene hot words as analysis constraints, and finally using the output of the natural language analysis model as the command text.
[0322] Optionally, the natural language analysis model includes Gaussian Mixture Model—Hidden Markov Model (GMM-HMM), deep learning model—Recurrent Neural Networks (RNN), deep learning model—Transformer, end-to-end model—Deep Learning Speech Recognition (End-to-End Deep Learning ASR), etc., among which the end-to-end model—Deep Learning Speech Recognition includes Connectionist Temporal Classification, Sequence-to-Sequence, etc. The selection of natural language analysis model is not limited here.
[0323] In some embodiments, the process of obtaining multiple scene hot words corresponding to the three-dimensional virtual environment can be implemented as at least one of the following: obtaining the first scene type corresponding to the three-dimensional virtual environment; obtaining a hot word set corresponding to the first scene type, and using the words in the hot word set as multiple scene hot words, the hot word set being a set of scene-related words collected based on the first scene type; or, obtaining multiple scene hot words based on environmental perception information; or, obtaining the first scene type corresponding to the first virtual scene; obtaining a hot word set corresponding to the first scene type; and obtaining at least two words from the hot word set as multiple scene hot words based on environmental perception information.
[0324] It is worth noting that the above-mentioned intent recognition and hot word recognition process can also be implemented by the server, which will not be elaborated below.
[0325] It should be noted that the calculations in the above method embodiments can be completely executed by the client, for example, the game application is a stand-alone game application; it can also be completely executed by the server, for example, in a cloud game scenario, all calculations of the application are completed by the server, and the client is only responsible for displaying and collecting user operation behaviors; it can also be completed by the client and server in collaboration, for example, the client is responsible for part of the calculation and the server is responsible for another part of the calculation.
[0326] Fig.11 A flowchart of a method for broadcasting a non-player character provided by an exemplary embodiment of the present application is shown. The method can be performed by the above Figure 1 The method comprises:
[0327] Step 510: receiving a natural language command sent by the client to a non-player character.
[0328] Schematically, the client has control authority over the master virtual character, and the non-player characters follow the master virtual character to move in the three-dimensional virtual environment.
[0329] Exemplarily, the method may be executed by a server installed on a server, the server being a server of an application program, and the application program is an application program supporting a three-dimensional virtual environment, for example, an application program for a shooting game.
[0330] Exemplarily, the server controls the non-player characters to move in a three-dimensional virtual environment.
[0331] Step 520: Obtain environmental perception information of at least one of the main virtual character and the non-player character.
[0332] Exemplarily, the environmental perception information includes at least one of the following:
[0333] Visual information perceived within the field of vision of the controlling virtual character;
[0334] Auditory information perceived within the hearing range of the controlling virtual character;
[0335] The information perceived by the master virtual character’s perception skills or perception tools;
[0336] Visual information perceived within the field of view of non-player characters;
[0337] Auditory information perceived within the hearing range of non-player characters;
[0338] Information perceived by non-player characters using their perception skills or perception tools.
[0339] For example, the environmental perception information of the main virtual character may include: the pictures or entities that the main virtual character can see when observing the three-dimensional virtual environment, the environmental sound effects that the main virtual character can hear, the source direction of the environmental sound effects, the type and volume of the environmental sound effects, the perception information obtained by the main virtual character by using perception skills (for example, visual information, auditory information, sound wave information, light reflection information, etc.), and the perception information obtained by the main virtual character by using sensing props (for example, sensor signals from sensors, position detection signals from position detection props, etc.).
[0340] The environmental perception information of the non-player character may include: the images or entities that the non-player character can see when observing the three-dimensional virtual environment, the environmental sound effects that the non-player character can hear, the source direction of the environmental sound effects, the type and volume of the environmental sound effects, the perception information obtained by the non-player character by using perception skills (for example, visual information, auditory information, sound wave information, light reflection information, etc.), and the perception information obtained by the non-player character by using sensing props (for example, sensor signals of sensors, position detection signals of position detection props, etc.).
[0341] It should be noted that the environmental perception information may include information perceived by the non-player character in real time, and may also include information perceived by the non-player character historically.
[0342] Step 530 , determining feedback text of the non-player character based on the natural language command and the environmental perception information.
[0343] The feedback text is a non-fixed text generated based on the environmental perception information of the non-player character.
[0344] In some embodiments, the feedback text can be generated by calling a large language model. Schematically, the large language model is called to infer the natural language command based on the static environment data and environmental perception information of the three-dimensional virtual environment to obtain the inference result of the natural language command; based on the inference result, the feedback text is determined.
[0345] In some embodiments, static environment data includes data of environmental elements that exist in the three-dimensional virtual environment, such as virtual buildings, virtual terrain (e.g., grasslands, caves, mountains, canyons, etc.), virtual animals, virtual plants, virtual streams, etc. in the three-dimensional virtual environment.
[0346] In some embodiments, the feedback text includes reply feedback text. When the behavioral intention of the natural language command is to inquire, the reply feedback text of the non-player character is determined according to the environmental perception information; the reply feedback text includes the reply content to the natural language command generated according to the environmental perception information of the non-player character of the three-dimensional virtual environment.
[0347] Illustratively, in a case where the natural language command includes an intelligence inquiry about the target entity, based on the non-player character's perception of the target entity, a first reply feedback text is determined according to environmental perception information; the first reply feedback text includes the non-player character's intelligence perception result of the target entity; in a case where the natural language command includes a location inquiry about the target entity, based on the non-player character's perception of the target entity, a second reply feedback text is determined according to environmental perception information, and the second reply feedback text includes a location description of the target entity.
[0348] In other embodiments, the feedback text includes status feedback text. When the behavioral intention of the natural language command is to execute a task, the status feedback text is determined based on environmental perception information; the status feedback text includes the execution status after the natural language command is executed based on the environmental perception information of the three-dimensional virtual environment by the non-player character.
[0349] Illustratively, when the behavioral intention of the natural language command includes searching for a target entity, a first state feedback text is determined according to the environmental perception information, and the state feedback text includes the search result of the non-player character for the target entity; when the behavioral intention of the natural language command includes using a virtual prop, a second state feedback text is determined according to the environmental perception information, and the second state feedback text is used to indicate that the virtual prop is ready; when the behavioral intention of the task natural language command includes using a virtual prop, a third state feedback text is determined according to the environmental perception information, and the third state feedback text is used to indicate the result of using the virtual prop; when the behavioral intention of the natural language command includes controlling the movement of the non-player character, a fourth state feedback text is determined according to the environmental perception information, and the fourth state feedback text is used to indicate the movement result of the movement; when the behavioral intention of the natural language command includes performing an interaction with the target entity, a fifth state feedback text is determined according to the environmental perception information, and the fifth state feedback text is used to indicate at least one of the search result of the non-player character for the target entity and the interaction result between the non-player character and the target entity; when the behavioral intention of the natural language command includes item interaction, a sixth state feedback text is determined according to the environmental perception information, and the sixth state feedback text is used to indicate the item interaction result of the non-player character performing the item interaction.
[0350] Illustratively, when the behavioral intention of a natural language command is to execute a task, a large language model is called to parse the natural language command, and based on the static environmental data and environmental perception information of the three-dimensional virtual environment, task execution instructions are inferred; the non-player character is controlled to execute the task according to the task execution instructions; and the status feedback text is determined according to the execution status of the non-player character executing the task.
[0351] Step 540: Send relevant information for playing the feedback information to the client based on the feedback text.
[0352] In the embodiment of the present application, the feedback information is the human voice audio corresponding to the feedback text, that is, the client broadcasts the feedback information corresponding to the feedback text.
[0353] Exemplarily, the feedback information can be determined by at least one of the following methods: directly broadcasting the offline voice stored locally on the client as feedback information, concatenating the offline voice stored locally on the client with the entity voice of the dynamically generated target entity to obtain feedback information, or converting the inference result of the large language model into voice to obtain feedback information.
[0354] In combination with the above feedback information determination method, the relevant information sent by the server to the client includes at least one of the following:
[0355] The first type is feedback text
[0356] In an illustrative manner, when the feedback information is implemented as offline voice stored locally on the client or a combination of offline voice and physical voice, the server directly sends the generated feedback text to the client, and the client generates feedback information based on the feedback text and then broadcasts the feedback information.
[0357] Second, feedback information
[0358] Illustratively, when the feedback information is obtained by converting the inference result of the large language model into speech, the server generates feedback information based on the feedback text, sends the generated feedback information to the client, and the client broadcasts the received feedback information.
[0359] The third type is the voice identifier corresponding to the feedback information
[0360] Illustratively, when the feedback information is implemented as offline voice stored locally on the client or a combination of offline voice and physical voice, the server determines the feedback information that the client needs to broadcast based on the feedback text, and sends the voice identifier corresponding to the feedback information to the client. The client reads the corresponding feedback information locally according to the voice identifier and broadcasts it.
[0361] In some embodiments, when the client stores the first offline voice corresponding to the inference result, the server determines the feedback information according to the first offline voice. In an exemplary manner, a feedback text corresponding to the first offline voice or a first voice identifier corresponding to the first offline voice is sent to the client, and the first voice identifier is used to instruct the client to generate feedback information based on the first offline voice, that is, when the client stores the first offline voice, the client generates the feedback text by querying the corresponding first offline voice according to the received feedback text, or generates the feedback text by querying the corresponding first offline voice according to the received first voice identifier.
[0362] Optionally, after receiving the feedback text, the client determines at least one corresponding first offline voice based on the feedback text. When the number of determined first offline voices is one, the first offline voice is directly broadcast as feedback information; when the number of determined first offline voices is multiple, the multiple first offline voices are spliced to obtain feedback information, and the feedback information is broadcast.
[0363] Optionally, when the client receives a first voice identifier, the corresponding first offline voice is queried according to the first voice identifier and broadcast as feedback information; when the client receives multiple first voice identifiers, the first offline voices corresponding to the multiple first voice identifiers are spliced to obtain feedback information, and the feedback information is broadcast.
[0364] In some embodiments, when the inference result includes entity information of the target entity, the server calls a text-to-speech service to generate an entity voice corresponding to the entity information, and sends feedback text corresponding to the entity voice and the first offline voice to the client; or, sends the entity voice and the first voice identifier corresponding to the first offline voice to the client; the client is used to splice the entity voice and the first offline voice to obtain feedback information; wherein the entity information includes at least one of the following: the entity name of the target entity, the location description of the target entity.
[0365] In other embodiments, when the client does not store the offline speech corresponding to the behavioral intention and the inference result, the text-to-speech service is called to generate feedback information based on the feedback text, and the feedback information is sent to the client.
[0366] In some embodiments, when the behavioral intention of the natural language command is to inquire, the reply feedback text of the non-player character is determined based on the environmental perception information; the reply feedback text includes the reply content for the natural language command generated based on the environmental perception information of the non-player character of the three-dimensional virtual environment.
[0367] For example, if the natural language command is "Where are you?" and the reply feedback text returned by the large language model is "I'm in the cabin," then the reply broadcast (feedback information) is "I'm in the cabin."
[0368] Alternatively, when the behavioral intention is an inquiry and the inference results provided by the large language model include failure information such as failure to generate reply feedback text, failure to search, and failure to identify behavioral intention, the first offline voice corresponding to the inquiry failure stored locally on the client is determined as feedback information.
[0369] For example, if the natural language command is "Help me find a nearby reward box", and the inference result fed back by the large language model is "No reward box was found near the NPC", then the reply broadcast can be the offline voice broadcast "Not found" stored locally on the client.
[0370] Alternatively, when the behavior intention is an inquiry and the inference result provided by the large language model includes confirmation information such as yes, confirm, no problem, etc., the first offline voice corresponding to the confirmation stored locally on the client is determined as feedback information.
[0371] For example, if the natural language command is "Are you on the second floor?", and the inference result fed back by the large language model is "Yes, the NPC is on the second floor", the reply broadcast can be the offline voice broadcast "Yes" stored locally on the client.
[0372] Alternatively, when the behavioral intention is an inquiry and the inference result provided by the large language model includes entity information of the target entity, the text-to-speech service is called to generate entity speech based on the entity information; according to the inquiry behavioral intention and the specific content provided in the inference result, the corresponding first offline speech is determined, and the entity speech is filled into the position reserved for the entity information in the first offline speech to obtain feedback information.
[0373] For example, if the natural language command is "Is there any danger ahead?" and the inference result returned by the large language model is "There is a first enemy virtual character at the first position", the feedback information of "An enemy is found at the first position" can be obtained by splicing the offline voice template "xxx found an enemy" stored locally on the client and the physical voice of the "first position" obtained using text-to-speech technology.
[0374] In other embodiments, when the behavioral intention of the natural language command is to execute a task, the state feedback text is determined according to the environmental perception information; the state feedback text includes the execution state after the natural language command is executed according to the environmental perception information of the three-dimensional virtual environment by the non-player character. Schematically, when the behavioral intention of the natural language command is to execute a task, the large language model is called to parse the natural language command, and the task execution instruction is inferred based on the static environmental data and environmental perception information of the three-dimensional virtual environment; the non-player character is controlled to execute the task according to the task execution instruction; and the state feedback text is determined according to the execution state of the non-player character executing the task.
[0375] For example, if the natural language command is "throw the first virtual prop at the first virtual character", the inference result may include a control instruction to control the NPC to throw the first virtual prop at the first virtual character; when the NPC fails to throw the first virtual prop at the first virtual character, the server inputs the execution results of the NPC's execution of the control instruction into the large language model to generate feedback text corresponding to the execution result. For example, the execution result may be that the NPC does not have the first virtual prop, then the feedback text may be "the first virtual prop is not currently held"; or, the execution result may be that the first virtual character is not hit, then the feedback text may be "sorry, I threw it crookedly"; the text-to-speech service is called to generate the human voice audio corresponding to the feedback text to obtain feedback information.
[0376] Alternatively, when the behavior intention is a task, according to the execution state of the task, the first offline voice corresponding to the execution state stored locally in the client is determined as the feedback information.
[0377] For example, if the natural language command is "throw the first virtual prop at the first virtual character", the inference result may include a control instruction to control the NPC to throw the first virtual prop at the first virtual character; the server controls the NPC according to the control instruction, and when the execution result of the control instruction includes successfully hitting the first virtual character, the corresponding first offline voice "hit" is determined as feedback information.
[0378] Alternatively, when the behavioral intention is a task, and the task is related to the target entity, and the first offline voice corresponding to the task execution status includes entity information of the target entity, the text-to-speech service is called to generate entity voice based on the entity information; the entity voice is filled into the position reserved for the entity information in the first offline voice to obtain feedback information.
[0379] Optionally, the server can also directly determine the feedback information based on the behavior intention. For example, when the behavior intention is the first type of behavior intention, the text-to-speech service is called to generate feedback information based on the inference results output by the large language model. When the behavior intention is the second type of behavior intention, the offline voice stored in the client corresponding to the second type of behavior intention is used as feedback information.
[0380] In other embodiments, the client will also broadcast dynamic broadcast information. Schematically, when the environmental perception information of the non-player character about the three-dimensional virtual environment meets the dynamic broadcast conditions, the server determines the dynamic broadcast text according to the environmental perception information; the dynamic broadcast text includes abnormal conditions identified according to the environmental perception information of the non-player character; and based on the dynamic broadcast text, relevant information for playing the dynamic broadcast information is sent to the client, where the dynamic broadcast information is the human voice audio corresponding to the dynamic broadcast text.
[0381] Illustratively, when a non-player character perceives a hostile virtual character, a first dynamic broadcast text is generated; the first dynamic broadcast text includes the position of the hostile virtual character perceived by the non-player character; when the non-player character perceives a dangerous situation, a second dynamic broadcast text is generated; the second dynamic broadcast text is used to prompt the dangerous situation; when the non-player character perceives a change in the state of a friendly virtual character, a third dynamic broadcast text is generated; the third dynamic broadcast text is used to prompt the change in the state of the friendly virtual character; when the non-player character discovers new traces in the three-dimensional virtual environment, a fourth dynamic broadcast text is generated; the fourth dynamic broadcast text is used to prompt the new traces.
[0382] In some embodiments, when the environmental perception information of the non-player character meets the dynamic broadcast condition, the server determines the dynamic broadcast information according to the second offline voice corresponding to the dynamic broadcast condition.
[0383] Exemplarily, the server may determine the second offline voice corresponding to the dynamic broadcast condition as the dynamic broadcast information. The second offline voice is an offline voice broadcast stored locally on the client.
[0384] For example, when an enemy virtual character appears within the field of vision of a non-player character, the server determines the offline voice "Danger ahead" as feedback information.
[0385] Exemplarily, when the environmental perception information includes the target entity, the server queries the entity information of the target entity; calls the text-to-speech service to generate the entity voice corresponding to the entity information; splices the entity voice and the second offline voice to obtain dynamic broadcast information; wherein the entity information includes at least one of the following: the entity name of the target entity, the location description of the target entity.
[0386] For example, when an enemy virtual character appears within the field of vision of a non-player character, the server generates a physical voice based on the nickname of the enemy virtual character "Robot No. 1", and splices the physical voice with the offline voice template "xxx is found ahead", to obtain the feedback information "Robot No. 1 is found ahead".
[0387] In summary, the method provided in this embodiment, when a user uses a natural language command to feedback relevant information of the three-dimensional virtual environment to a non-player character in the three-dimensional virtual environment, feedback information corresponding to the non-player character is generated and broadcasted according to the information perceived by the non-player character from the three-dimensional virtual environment, so that the feedback information can cover a variety of scenarios. When the natural language command instructs to ask the non-player character whether there is an enemy in the three-dimensional virtual environment, when the non-player character perceives that the virtual characters included in the three-dimensional virtual environment include hostile virtual characters, the main virtual character is reminded of the existence of enemies by broadcasting feedback information; when the natural language command instructs the non-player character to provide virtual props to the main virtual character, when the non-player character has virtual props required by the user, the non-player character submits the virtual props to the main virtual character, and reminds the main virtual character that the virtual props have been submitted by broadcasting feedback information. When controlling non-player characters through natural language commands, feedback information can be broadcast based on the environmental perception information of the non-player characters. This can enrich the content of the feedback information and make the broadcast not limited to fixed trigger scenarios. Game applications can flexibly broadcast feedback information based on environmental perception information during game matches, enrich the feedback information broadcast corresponding to non-player characters, and improve the efficiency of human-computer interaction between non-player characters and players.
[0388] In an optional embodiment, the server performs spatial sound processing on the vocal audio of the non-player character to obtain spatial vocal audio as feedback information.
[0389] Fig.12A flowchart of a method for broadcasting a non-player character provided by an exemplary embodiment of the present application is shown. The method can be performed by the above Figure 1 Based on the server in Fig.11 In the illustrated embodiment, the method may further include steps 541 to 544 , where steps 541 to 544 are subordinate steps of step 540 .
[0390] Step 541, obtaining the human voice audio corresponding to the feedback text.
[0391] Step 542: based on the relative position relationship between the non-player character and the main control virtual character in the three-dimensional virtual environment, obtain the sound effect parameters corresponding to the non-player character.
[0392] Illustratively, the relative position relationship between the non-player character and the controlling virtual character in the three-dimensional virtual environment is used to indicate the relative relationship between a first position corresponding to the non-player character and a second position corresponding to the controlling virtual character in the three-dimensional virtual environment.
[0393] Optionally, the above-mentioned relative orientation relationship includes at least one of position distance, position direction, and spatial surrounding situation between the non-player character and the main virtual character, wherein the position distance is used to indicate the distance relationship between the non-player character and the main virtual character, the position direction is used to indicate the angular relationship between the first position of the non-player character and the orientation direction of the main virtual character, and the spatial surrounding situation is used to indicate the surround situation of the main virtual character by the spatial elements forming the virtual space in the three-dimensional virtual environment, and the propagation effect of the space on the audio when the non-player character emits audio in the virtual space.
[0394] In some embodiments, the determination of the relative position relationship between the non-player character and the main virtual character can be implemented as follows: obtaining first position information of the non-player character in the three-dimensional virtual environment, obtaining second position information of the main virtual character in the three-dimensional virtual environment, and determining the relative position relationship between the non-player character and the main virtual character based on the first position information and the second position information.
[0395] In some embodiments, the first position information of the non-player character and the second position information of the controlling virtual character are position information determined based on the same preset coordinate system. Optionally, the preset coordinate system can be implemented as a world coordinate system corresponding to the three-dimensional virtual environment, or the preset coordinate system can be implemented as a coordinate system established with the controlling virtual character as the origin.
[0396] In some embodiments, when the sound effect parameters corresponding to the non-player characters are pre-generated and stored in a database, the sound effect parameters corresponding to the non-player characters are obtained from the database. In other embodiments, the sound effect parameters corresponding to the non-player characters may also be generated in real time, that is, the sound effect parameters corresponding to the non-player characters are generated based on the relative position relationship.
[0397] Optionally, the sound effect parameters corresponding to the non-player character may include at least one of the following types:
[0398] The first type is distance type
[0399] Illustratively, the distance-type sound effect parameter is used to indicate that the sound effect of the human voice audio is adjusted based on the distance between the non-player character and the main virtual character. In some embodiments, the distance-type sound effect parameter can adjust the audio volume of the human voice audio to simulate the effect of distance on the sound volume; and / or, the distance-type sound effect parameter can adjust the audio delay of the human voice audio to simulate the effect of distance on the sound propagation time.
[0400] Second, direction type
[0401] Illustratively, the direction-type sound effect parameter is used to indicate the adjustment of the sound effect of the human voice audio based on the direction of the non-player character relative to the main virtual character. In some embodiments, the direction-type sound effect parameter can adjust the audio volume of the human voice audio to simulate whether the main virtual character is facing the scene audio element; and / or, the direction-type sound effect parameter can adjust the volume parameters of the human voice audio in different channels to simulate the direction of the audio emitted by the scene sound source element.
[0402] The third type is space effects
[0403] Illustratively, the sound effect parameters of the spatial special effect type are used to indicate that the virtual space formed based on the spatial elements adjusts the sound effects in the virtual space, that is, the sound effect parameters of the spatial special effect type are used to indicate the effect of the virtual space on the sound performance of the audio emitted by the non-player characters. In some embodiments, the sound effect parameters of the spatial special effect type can add echo sound effects and reverberation sound effects to simulate the effect of sound in the space.
[0404] Optionally, the sound effect parameters of the spatial special effect type include at least one of a reverberation time parameter, a pre-delay parameter, a wetness (Wet / DryMix) parameter, a space width parameter, and a distance effect parameter. Among them, the reverberation time parameter is used to indicate the speed at which the audio decays in the virtual space; the pre-delay parameter is used to indicate the time difference between the audio directly reaching the main virtual character and the audio reaching the main virtual character after the first reflection; the wetness parameter is used to indicate the ratio between the audio directly transmitted and the audio transmitted through reflection; the space width parameter is used to indicate the degree of diffusion of the audio on the horizontal plane in the virtual space; and the distance effect parameter is used to indicate the attenuation of the audio with the propagation distance during the propagation process.
[0405] The fourth type is custom type
[0406] Schematically, the custom type of sound effect parameters are user-defined parameters for sound effect adjustment. Optionally, the user can customize the overall volume of the scene audio, whether to add background music, customize the volume of different types of audio, etc.
[0407] In some embodiments, the client provides a customization interface for sound effect parameters to the user, and the user configures the customized sound effect parameters through the customization interface.
[0408] In some embodiments, a parameter prediction model for personalized learning of the custom sound effect parameters is configured in the custom sound effect parameter server for the user. Schematically, the custom sound effect parameters of multiple candidate accounts are obtained, the multiple custom sound effect parameters are input into the parameter prediction model to be trained, the parameter prediction model to be trained is iteratively trained to obtain the parameter prediction model, and the sound effect parameters generated by the system (for example, the sound effect parameters of the above-mentioned distance type, direction type, and spatial special effect type) are optimized through the parameter prediction model.
[0409] Step 543: Adjust the vocal audio based on the sound effect parameters to generate spatial vocal audio corresponding to the non-player character.
[0410] Among them, the spatial human voice audio is used to express the perceptual effect of the audio produced by the main virtual character on the non-player character under the relative position relationship.
[0411] In an embodiment of the present application, the human voice audio is adjusted according to the sound effect parameters corresponding to the non-player character, thereby generating the spatial human voice audio corresponding to the non-player character. That is, the spatial human voice audio is the audio data corresponding to the non-player character obtained after the human voice audio is adjusted by the sound effect parameters. In some embodiments, the spatial human voice audio includes the audio after the sound effect parameters are adjusted, as well as at least one of the audio duration information, the audio playback start timestamp, the audio playback end timestamp, the audio playback condition information, and the element identification information corresponding to the audio.
[0412] Optionally, when the sound effect parameters indicate that the volume of the non-player character's vocal audio is to be adjusted, the volume of the non-player character's vocal audio on at least one channel is adjusted based on the sound effect parameters. Optionally, when the sound effect parameters indicate that the playback delay of the non-player character's vocal audio is to be adjusted, the audio playback start time corresponding to the non-player character's vocal audio is adjusted based on the sound effect parameters. Optionally, when the sound effect parameters indicate that the speed of the non-player character's vocal audio is to be adjusted, the playback speed of the non-player character's vocal audio is adjusted based on the sound effect parameters. Optionally, when the sound effect parameters indicate the pitch of the non-player character's vocal audio, the frequency in the spectrum of the non-player character's vocal audio is adjusted based on the sound effect parameters. When the sound effect parameters indicate the timbre of the non-player character's vocal audio, a filter corresponding to the non-player character is determined based on the sound effect parameters, and the timbre corresponding to the vocal audio is adjusted through the filter.
[0413] Step 544: Send the spatial human voice audio to the client.
[0414] Illustratively, the server sends the obtained spatial human voice audio to the client, and the client broadcasts the spatial human voice audio as feedback information in response to the natural language command.
[0415] In summary, the method provided in this embodiment, when a player seeks voice feedback from a non-player character through a natural language command, obtains the sound effect parameters of the non-player character according to the relative position relationship between the main virtual character and the non-player character, and adjusts the human voice audio of the non-player character through the sound effect parameters to obtain spatial human voice audio. Since the spatial human voice audio is generated in real time based on the relative position relationship between the main virtual character and the non-player character, the application that provides a three-dimensional virtual environment does not need to pre-configure too many application resources in order to improve the diversity of sound effect performance effects, thereby reducing the waste of storage space of the computer device running the above application.
[0416] At the same time, after the sound effect parameters are adjusted, the human voice audio corresponding to the non-player characters can make the spatial human voice audio played by the client simulate the perception effect of the audio produced by the main virtual character on the non-player characters in the current relative position relationship, so that the spatial human voice audio played by the client can enable the player to replace the main virtual character to perceive the human voice produced by the non-player characters in the three-dimensional virtual environment, thereby enhancing the diversity of sound effects felt by the player during the game, thereby enhancing the player's sense of substitution in the three-dimensional virtual environment and improving the user experience.
[0417] In an optional embodiment, when generating sound effect parameters based on the relative position relationship between the non-player character and the controlling virtual character, it is necessary to combine the first position of the non-player character in the three-dimensional virtual environment and the second position of the controlling virtual character in the three-dimensional virtual environment.
[0418] Fig.13 A flowchart of a method for broadcasting a non-player character provided by an exemplary embodiment of the present application is shown. The method can be performed by the above Figure 1 Based on the server in Fig.12 In the illustrated embodiment, the method may further include steps 5421 to 5423 , where steps 5421 to 5423 are subordinate steps of step 542 .
[0419] Step 5421, obtaining first position information of the non-player character in the three-dimensional virtual environment; and obtaining second position information of the master virtual character in the three-dimensional virtual environment.
[0420] In the embodiment of the present application, the non-player character is a virtual character that follows the main virtual character in the three-dimensional virtual environment, and the position of the non-player character in the three-dimensional virtual environment is determined by real-time monitoring of the position of the non-player character.
[0421] In some embodiments, the first position information is coordinate point data determined based on a preset coordinate system.
[0422] In some other embodiments, the three-dimensional virtual environment is divided into a grid structure, and the grid points in the grid structure are used to indicate the position information of the scene elements in the three-dimensional virtual environment. In an exemplary manner, a first grid point where the non-player character is located in the grid structure is obtained, and the position of the first grid point in the grid is determined as the first position information.
[0423] like Fig.14 As shown, it shows a schematic diagram of a grid structure provided by an exemplary embodiment of the present application, in which a three-dimensional virtual environment 1400 is divided into a grid structure composed of multiple grids, and a virtual character 1401 in the three-dimensional virtual environment 1400 occupies a certain number of grid points in the grid structure.
[0424] The user can control the main virtual character to move in the three-dimensional virtual environment. Therefore, the position of the main virtual character in the three-dimensional virtual environment is not fixed. Schematically, the position of the main virtual character in the three-dimensional virtual environment is detected to obtain the second position information.
[0425] In some embodiments, the second position information is coordinate point data determined based on a preset coordinate system.
[0426] In other embodiments, the three-dimensional virtual environment is divided into a grid structure. Schematically, a second grid point corresponding to the position of the master virtual character in the three-dimensional virtual environment in the grid structure is determined, and the position of the second grid point in the grid is determined as the second position information.
[0427] Step 5422: Determine the relative position relationship between the non-player character and the main virtual character based on the first position information and the second position information.
[0428] In some embodiments, when the first position information and the second position information are coordinate point data determined based on a preset coordinate system, the relative orientation relationship between the non-player character and the master virtual character is determined based on the first coordinate point indicated by the first position information and the second coordinate point indicated by the second position information.
[0429] In other embodiments, when the first position information and the second position information are based on grid points indicated by a grid structure, the relative orientation relationship between the non-player character and the master virtual character is determined based on the first grid point corresponding to the first position information and the second grid point indicated by the second position information.
[0430] Optionally, the above-mentioned relative orientation relationship includes at least one of position distance, position direction, and spatial surrounding situation between the non-player character and the main virtual character, wherein the position distance is used to indicate the distance relationship between the non-player character and the main virtual character, the position direction is used to indicate the angular relationship between the first position of the non-player character and the orientation direction of the main virtual character, and the spatial surrounding situation is used to indicate the surround situation of the main virtual character by the spatial elements forming the virtual space in the three-dimensional virtual environment, and the propagation effect of the space on the audio when the non-player character emits audio in the virtual space.
[0431] Step 5423, based on the relative position relationship, generate sound effect parameters corresponding to the non-player character.
[0432] Optionally, the sound effect parameters corresponding to the non-player characters may be generated in at least one of the following ways:
[0433] The first method generates corresponding sound effect parameters based on the sound propagation path corresponding to the audio generated by the non-player character under the relative position relationship.
[0434] The above-mentioned sound propagation path is used to indicate the propagation path of the sound wave when the non-player character is used as the sound source. It is worth noting that one non-player character can correspond to multiple sound propagation paths.
[0435] In an illustrative manner, based on the relative position relationship, the sound propagation path of the audio generated by the non-player character between the non-player character and the main virtual character is determined, and the sound effect parameters corresponding to the non-player character are generated based on the sound propagation path. That is, since the non-player character is a sound source in the three-dimensional virtual environment, the audio emitted by it propagates in the three-dimensional virtual environment, and the path pointing to the main virtual character is the sound propagation path of the audio between the non-player character and the main virtual character.
[0436] In some embodiments, the sound propagation path between the non-player character and the main virtual character can be implemented as a ray path pointing from the non-player character to the main virtual character, that is, the sound is directly propagated from the non-player character to the main virtual character.
[0437] In some embodiments, since the audio emitted by the non-player characters has propagation paths in other directions, and after being reflected by other scene elements in the three-dimensional virtual environment, there will also be a sound propagation path for the main virtual character to receive the audio, that is, the sound propagation path is the path of the audio emitted by the non-player characters after being reflected in the three-dimensional virtual environment.
[0438] In one example, taking the example that the sound propagation path of the audio received by the main virtual character includes a direct propagation path and a propagation path after reflection, schematically, based on the relative orientation relationship, a first propagation path of the audio generated by the non-player character is determined to be directly propagated from the non-player character to the main virtual character, and based on the relative orientation relationship, a second propagation path of the audio generated by the non-player character is determined to be propagated to the main virtual character after reflection in the three-dimensional virtual environment, and based on the first propagation path and the second propagation path, the sound effect parameters corresponding to the non-player character are determined.
[0439] Specifically, please refer to Formula 1 and Formula 2, which show the radiation process of sound propagation in a three-dimensional virtual environment. Formula 1 and Formula 2 jointly describe the change in sound energy in the process of audio being emitted from a non-player character and propagating to the main virtual character, where Formula 1 is the change in total energy, and Formula 2 is the energy change of audio during transmission due to the inhomogeneity of the medium or the scattering effect caused by other sound sources.
[0440] Formula 1:
[0441] Formula 2:
[0442] Where t is time, x′ is the first position of the non-player character, x is the second position of the main control virtual character, L(x′→x,t) represents the sound field contribution from x′ to x at time t, L0(x′→x,t) represents the primary sound field directly radiated from x′ to x, and L s (x′→x,t) represents the scattering effect of the sound wave due to the inhomogeneity of the medium or other sound sources during the propagation from x′ to x. is the scattering operator corresponding to the sound source / medium at x′′, L(x″→x′,t) represents the contribution of the sound field from x″ to x′ at time t, is the acoustic geometry term of the field of view object for the propagation of sound waves between x″ and x′, and ρ(x″′→x′→x) is used to indicate the two-way propagation function of sound on the surface of the object on the scattering path.
[0443] In some embodiments, the detection of the sound propagation path can be achieved by simulating the direct propagation or reflected propagation of sound through ray tracing or image source method.
[0444] In some embodiments, when adding the sound effects generated by the sound during the reflection propagation process in the process of determining the sound effect parameters, reverberation calculation can also be incorporated. Specifically, the early reflections of the audio emitted by the non-player characters in the three-dimensional virtual environment are calculated. These reflections usually reach the main virtual character within a specified time length (for example, 50 milliseconds), and then a first-in-first-out (FIFO) sound queue and Schroeder algorithm are used to simulate the environmental tail sound, thereby adding the audio effect of the reverberation tail. Fig.15 As shown, it shows a schematic diagram of a reverberation sound wave provided by an exemplary embodiment of the present application. In the reverberation sound wave schematic diagram chart 1500, the horizontal axis is used to represent the time when the sound reaches the master virtual character, and the vertical axis is used to represent the volume of the received sound, wherein the volume of the first sound 1501 received at time t1 and transmitted to the master virtual character through the direct propagation path is the largest, the volume of the second sound 1502 received at the master virtual character through the early reflection propagation path in the time period t2 is reduced to a certain extent, and the volume of the third sound 1503 as the reverberation tail received in the time period t3 is relatively low.
[0445] That is, by simulating the propagation path of sound in a three-dimensional virtual environment to generate corresponding sound effect parameters, the scene audio can reflect the result of the audio emitted by each non-player character propagating in the three-dimensional virtual environment and reaching the main virtual character, thereby improving the authenticity of the scene audio, making the user feel more involved, and improving the user experience. Furthermore, when simulating the propagation path of sound, in addition to directly broadcasting the propagation path to the main virtual character, the result of the sound after reflection in the three-dimensional virtual environment is also simulated, which can improve the spatial sense of the spatial audio corresponding to the non-player character, that is, the reflected sound can increase the depth and width of the space, making the sound sound richer and more three-dimensional, thereby improving the audio effect of the scene audio and improving the user experience.
[0446] The second method is to determine the corresponding sound effect parameters based on whether the non-player character is within the field of vision of the main control virtual character.
[0447] In an illustrative manner, the visual field corresponding to the controlling virtual character is determined based on the orientation of the controlling virtual character in the three-dimensional virtual environment. When the non-player character is within the visual field, the sound effect parameter is determined based on the first parameter. When the non-player character is outside the visual field, the sound effect parameter is determined based on the second parameter, wherein the audio volume corresponding to the first parameter is greater than the audio volume corresponding to the second parameter. That is, when the controlling virtual character faces the non-player character, the audio volume corresponding to the non-player character is increased to let the user know that the sound source is in front of the controlling virtual character.
[0448] In some embodiments, the field of view of the controlling virtual character is the range of the three-dimensional virtual environment displayed by the client when providing the three-dimensional virtual environment screen to the user, that is, the three-dimensional virtual environment screen is a screen in which the three-dimensional virtual environment is observed from the perspective of the controlling virtual character, and the range of the three-dimensional virtual environment corresponding to the screen is the above-mentioned field of view.
[0449] In some embodiments, the field of view of the main virtual character is determined by a virtual camera (Camera), wherein the virtual camera is a crucial component in the three-dimensional virtual environment, which determines the content and perspective that the user sees on the screen. The position of the virtual camera in three-dimensional space determines the observation point of the three-dimensional virtual environment, the orientation of the virtual camera in three-dimensional space determines the observation direction of the three-dimensional virtual environment, and the field of view (FOV) of the virtual camera determines the size of the range that can display the three-dimensional virtual environment on the screen. Usually, the user cannot directly observe the virtual camera in the three-dimensional virtual environment.
[0450] Optionally, when the main virtual character controlled by the user observes the three-dimensional virtual environment from a first-person perspective, the virtual camera is bound to the head position of the main virtual character; when the main virtual character controlled by the user observes the three-dimensional virtual environment from a third-person perspective, the virtual camera is bound to the rear position of the main virtual character.
[0451] In one example, the field of view is determined by the frustum of the virtual camera, where the frustum is a spatial region defined by the virtual camera position and FOV. Schematically, after determining the loading range corresponding to the three-dimensional virtual environment, frustum culling is performed to calculate which scene elements are within the frustum, thereby determining which scene elements need to be rendered, and then determining the non-player characters that emit audio from the rendered scene elements, that is, as non-player characters within the field of view, and correspondingly configuring the sound effect parameters of the first parameter.
[0452] Specifically, six planes (upper, lower, left, right, front, and back) are extracted from the camera's view frustum. For each scene element, its bounding box is tested with the view frustum plane. If the bounding box is completely outside the view frustum, the scene element is removed from the field of view of the master virtual character.
[0453] When configuring sound effect parameters of different volumes according to the visual range of the main virtual character, users can distinguish the positions of different non-player characters in the three-dimensional virtual environment through scene audio, determine whether the non-player characters are within their own visual range, and enhance the sense of reality while allowing users to more accurately determine the direction of the sound source and improve the user's ability to determine the position of the non-player characters. At the same time, when the user rotates the viewing angle of the main virtual character, the volume of the audio emitted by non-player characters in different positions will change, thereby realizing interactive feedback of the change of viewing angle at the sound experience level.
[0454] The third method is to determine the corresponding sound effect parameters based on the space type of the virtual space formed by the space elements.
[0455] In some embodiments, the three-dimensional virtual environment includes space elements that form a virtual space in the three-dimensional virtual environment, the main virtual character is in the virtual space, based on the relative position relationship between the space elements and the main virtual character, the surrounding state information of the space elements to the main virtual character is determined, based on the surrounding state information, the space type of the virtual space where the main virtual character is located is determined, the space sound effect parameters matching the space type are obtained, and the sound effect parameters are determined based on the space audio parameters. The surrounding state information is used to indicate the relative relationship between the location of the main virtual character and the distribution of the space elements in the three-dimensional virtual environment.
[0456] Optionally, the space type can be implemented as at least one of an open type, a closed type, and a semi-closed type. When the space type is an open type, the three-dimensional virtual environment can be implemented as scenes such as grasslands, plains, and sea surfaces; when the space type is a closed type, the three-dimensional virtual environment can be implemented as scenes such as caves and rooms; when the space type is a semi-closed type, the three-dimensional virtual environment can be implemented as scenes such as canyons and forests.
[0457] In some embodiments, different space types correspond to different sound effect parameters. For example, when the space type is an open type, the corresponding sound effect parameters will make the audio more transparent and clear, with corresponding characteristics such as lower reverberation and faster sound attenuation. Indicatively, the sound effect parameters corresponding to different space types can be pre-configured, that is, the database pre-stores pre-configured sound effect parameter strategies corresponding to different space types. After the space type of the virtual space where the master virtual character is located is determined based on the surround state information, the sound effect parameter strategy corresponding to the space type is found from the database based on the type identifier of the space type, and the sound effect parameters matching the space type are obtained from the sound effect parameter strategy.
[0458] That is, by obtaining sound effect parameters that match the space type formed by the space elements in the three-dimensional virtual environment, the user can feel the virtual space of the three-dimensional virtual environment from the auditory perception level, enhance the user's sense of immersion, and make the user feel more immersed in the three-dimensional virtual environment.
[0459] Fourthly, the corresponding sound effect parameters are determined based on the volume attenuation and delay of the sound propagation based on the distance between the main virtual character and the non-player character.
[0460] In some embodiments, since there is a certain distance between the main virtual character and the non-player character, the sound effect parameters are also used to adjust the attenuation effect and / or delay effect of the sound. In an exemplary manner, based on the relative position relationship, the distance information between the main virtual character and the non-player character is determined, based on the distance information, the attenuation parameter and delay parameter corresponding to the audio generated by the non-player character are determined, and based on the attenuation parameter and delay parameter, the sound effect parameters are determined.
[0461] In some embodiments, the attenuation parameter is a preset sound attenuation function. In some embodiments, the delay parameter is a delay duration determined based on distance information and sound propagation speed.
[0462] That is, the attenuation effect of sound after propagation in a three-dimensional virtual environment is simulated by attenuation parameters, and the delay effect of sound in a three-dimensional virtual environment is simulated by delay parameters, thereby improving the authenticity of scene audio.
[0463] In summary, the method provided in this embodiment, when a player seeks voice feedback from a non-player character through a natural language command, obtains the sound effect parameters of the non-player character according to the relative position relationship between the main virtual character and the non-player character, and adjusts the human voice audio of the non-player character through the sound effect parameters to obtain spatial human voice audio. Since the spatial human voice audio is generated in real time based on the relative position relationship between the main virtual character and the non-player character, the application that provides a three-dimensional virtual environment does not need to pre-configure too many application resources in order to improve the diversity of sound effect performance effects, thereby reducing the waste of storage space of the computer device running the above application.
[0464] At the same time, after the sound effect parameters are adjusted, the human voice audio corresponding to the non-player characters can make the spatial human voice audio played by the client simulate the perception effect of the audio produced by the main virtual character on the non-player characters in the current relative position relationship, so that the spatial human voice audio played by the client can enable the player to replace the main virtual character to perceive the human voice produced by the non-player characters in the three-dimensional virtual environment, thereby enhancing the diversity of sound effects felt by the player during the game, thereby enhancing the player's sense of substitution in the three-dimensional virtual environment and improving the user experience.
[0465] In an embodiment of the present application, based on the first position information of the non-player character in the three-dimensional virtual environment and the second position information of the main virtual character in the three-dimensional virtual environment, sound effect parameters that conform to the relative position relationship between the two are generated in real time, so that the spatial human voice audio finally played by the client can truly simulate the main virtual character's perception of the non-player character's voice at the current position in the three-dimensional virtual environment, thereby improving the user experience.
[0466] In an optional embodiment, character feature information of the non-player character may be incorporated into the process of generating the vocal audio of the non-player character.
[0467] Fig.16 A flowchart of a method for broadcasting a non-player character provided by an exemplary embodiment of the present application is shown. The method can be performed by the above Figure 1 Based on the server in Figure 1 In the illustrated embodiment, the method may further include steps 5411 to 5412 , where steps 5411 to 5412 are subordinate steps of step 541 .
[0468] Step 5411, obtaining feedback text and character feature information corresponding to the non-player character.
[0469] In the embodiment of the present application, the character characteristic information is used to describe the attributes of the non-player character, that is, the character characteristic information is information used to describe the current state and characteristics of the non-player character.
[0470] In some embodiments, the attributes of the non-player character include at least one of the basic attributes of the non-player character and the situational performance attributes. The basic attributes are pre-configured attributes for the non-player character, that is, the attributes of the non-player character that do not change due to the three-dimensional virtual environment or the current situation, such as the age information of the non-player character, the male / female information of the non-player character, the martial arts information of the non-player character, etc. The situational performance attributes are attributes determined in real time by the non-player character in the situation of the three-dimensional virtual environment, that is, the attributes associated with the current situation, such as the emotional information and behavioral information of the non-player character in the current situation.
[0471] Optionally, the character feature information includes at least one of basic character information, character emotion information, and character behavior information. The basic character information is used to indicate the basic situation of the non-player character, and the basic character information is information about the basic attributes of the non-player character, such as the age information of the non-player character, the male / female information of the non-player character, and the martial arts information of the non-player character; the character emotion information is used to indicate the emotional state of the non-player character in the dialogue scenario corresponding to the feedback text, and the character emotion information is the situational performance attribute of the non-player character, such as the character emotion information indicates that the non-player character is in an excited state, an angry state, a sad state, etc.; the character behavior information is used to indicate the actions performed by the non-player character in the dialogue scenario, and the character behavior information is the situational performance attribute of the non-player character, such as being in a running state, an attacking state, an injured treatment state, etc.
[0472] It is worth noting that in the embodiment of the present application, different character feature information corresponds to different human voice audio performances. In one example, when the character feature information indicates that the non-player character is a young man, currently in a combat state, and emotionally excited, the timbre of the corresponding human voice audio is the timbre of a young man. Because it is in a combat state, the human voice audio is mixed with the panting sound of the combat process, and because the non-player character is emotionally excited when speaking, the pitch of the human voice audio is higher. That is, generating human voice audio with different performances according to different character feature information can enhance the authenticity of the human voice audio of the non-player character, so that the user has a greater sense of immersion when interacting with the non-player character.
[0473] Step 5412, generating human voice audio matching the character feature information based on the feedback text.
[0474] In an embodiment of the present application, vocal audio of a non-player character is generated based on feedback text and character feature information, wherein the voice content of the vocal audio is determined by the feedback text, and the voice performance of the vocal audio is determined by the character feature information.
[0475] In some embodiments, the generation of human voice audio is achieved through a pre-trained speech generation model, and the speech generation model is used to generate speech that matches the character feature information of the non-player character. In an illustrative example, the feedback text and the character feature information are input into the pre-trained speech generation model to obtain human voice audio as the human voice audio.
[0476] Optionally, the above-mentioned speech generation model can be implemented by a neural network model such as a convolutional neural network, a feedforward neural network, a residual network, a Transformer, a multimodal large language model (Multimodal Large Language Model, MLLM), etc., which is not specifically limited here.
[0477] In some embodiments, the speech generation model includes a text encoder, a role encoder and a decoder. The text encoder is used to perform text encoding on the input feedback text, that is, the feedback text is input into the text encoder to obtain a text encoding representation. The role encoder is used to perform feature encoding on the input role feature information, that is, the role feature information is input into the role encoder to obtain a role encoding representation.
[0478] Illustratively, after obtaining the text coding representation and the role coding representation, the text coding representation and the role coding representation are fused to obtain the coding representation input to the decoder, that is, the text coding representation and the role coding representation are fused to obtain a joint coding representation, and the joint coding representation is input into the decoder to generate human voice audio.
[0479] In some embodiments, the character encoder includes at least one of a first sub-encoder, a second sub-encoder, and a third sub-encoder. The first sub-encoder is used to perform feature encoding based on the basic attributes of the non-player character, that is, when the character feature information includes the character basic information, the character basic information is input into the first sub-encoder to obtain a first character encoding representation; the second sub-encoder is used to perform feature encoding based on the emotional state of the non-player character, that is, when the character feature information includes the character emotional information, the character emotional information is input into the second sub-encoder to obtain a second character encoding representation; the third sub-encoder is used to perform feature encoding based on the behavior state of the non-player character, that is, when the character feature information includes the character behavioral information, the character behavioral information is input into the third sub-encoder to obtain a third character encoding representation.
[0480] like Fig.17As shown, it shows a schematic diagram of a speech generation model 1700 provided by an exemplary embodiment of the present application, in which the speech generation model 1700 includes an encoder part 1710 and a decoder part 1720, wherein the encoder part 1710 includes a text encoder 1711, a first sub-encoder 1712, a second sub-encoder 1713, and a third sub-encoder 1714, the feedback text 1701 is input into the text encoder 1711, and the output is a text encoding representation, the role basic information 1702 of the non-player character is input into the first sub-encoder 1712, and the output is a first role encoding representation, the role emotion information 1703 of the non-player character is input into the second sub-encoder 1713, and the output is a second role encoding representation, and the role behavior information 1704 of the non-player character is input into the third sub-encoder 1714, and the output is a third role encoding representation. The text encoding representation, the first role encoding representation, the second role encoding representation, and the third role encoding representation are feature fused through the fusion layer 1730 to obtain a joint encoding representation, which is input into the decoder part 1720, and the human voice audio is output through the decoder.
[0481] In the embodiment of the present application, the method for generating sound effect parameters for human voice audio may be implemented as at least one of the following methods in addition to the method provided in the aforementioned embodiment:
[0482] First, an adjustment parameter is generated based on the distance between the non-player character and the main control virtual character to determine the sound effect parameter.
[0483] Illustratively, first position information of a non-player character in a three-dimensional virtual environment is obtained; and second position information of a master virtual character in the three-dimensional virtual environment is obtained; when the position distance between the first position information and the second position information reaches a first threshold, a voice adjustment parameter is generated, and the voice adjustment parameter is used to adjust the voice of the human voice audio; and the sound effect parameter is determined based on the voice adjustment parameter.
[0484] In some embodiments, the first position information of the non-player character and the second position information of the controlling virtual character are position information determined based on the same preset coordinate system. Optionally, the preset coordinate system can be implemented as a world coordinate system corresponding to the three-dimensional virtual environment, or the preset coordinate system can be implemented as a coordinate system established with the controlling virtual character as the origin.
[0485] In some embodiments, when the position distance between the first position information and the second position information reaches a first threshold, a first intonation adjustment parameter is generated; when the position distance between the first position information and the second position information does not reach the first threshold, a second intonation adjustment parameter is generated, wherein the pitch of the intonation corresponding to the first intonation adjustment parameter is greater than the pitch of the intonation corresponding to the second intonation adjustment parameter.
[0486] In some embodiments, the first intonation adjustment parameter and the second intonation adjustment parameter may be pre-set parameters; in other embodiments, the first intonation adjustment parameter may be determined based on a first control function, and the second intonation adjustment parameter may be determined based on a second control function. Optionally, the first control function and the second control function may be implemented as the same function or as different functions.
[0487] In other embodiments, the above-mentioned adjustment parameters can also be implemented as volume control parameters. Schematically, first position information of a non-player character in a three-dimensional virtual environment is obtained, and second position information of a master virtual character in the three-dimensional virtual environment is obtained. When the position distance between the first position information and the second position information reaches a first threshold, a volume control parameter is generated, and the sound effect parameter is determined based on the volume control parameter, wherein the volume control parameter is used to adjust the volume of the human voice audio so that the volume of the human voice audio is positively correlated with the position distance.
[0488] In some embodiments, when the position distance between the first position information and the second position information reaches a first threshold, a first volume control parameter is generated; when the position distance between the first position information and the second position information does not reach the first threshold, a second volume control parameter is generated, wherein the volume level corresponding to the first volume control parameter is greater than the volume level corresponding to the second volume control parameter.
[0489] In some embodiments, the first volume control parameter and the second volume control parameter may be pre-set parameters; in other embodiments, the first volume control parameter may be determined based on a third control function, and the second volume control parameter may be determined based on a fourth control function. Optionally, the third control function and the fourth control function may be implemented as the same function or as different functions.
[0490] In one example, when the main virtual character is outside the virtual building and the non-player character is on the second floor of the virtual building, the voice tone and / or voice volume of the non-player character's vocal audio is increased through volume control parameters, so that the user can receive clearer feedback voice from the non-player character.
[0491] The second method is to detect scene changes of the three-dimensional virtual environment during the movement when the non-player character is a continuously moving virtual character, and generate corresponding scene sound effect parameters based on the scene changes to determine the sound effect parameters.
[0492] Illustratively, a movement path of a non-player character in a three-dimensional virtual environment is obtained; when the movement path indicates that the non-player character moves between multiple virtual sub-scenes, scene sound effect parameters corresponding to the multiple virtual sub-scenes are obtained; and sound effect parameters are determined based on the scene sound effect parameters.
[0493] In some embodiments, the three-dimensional virtual environment is a scene composed of multiple virtual sub-scenes, and the virtual sub-scenes are connected to obtain the three-dimensional virtual environment. For example, a virtual character gets off a virtual vehicle, walks through a path and enters a virtual building, then the multiple virtual sub-scenes include a scene inside the vehicle, a road scene and a scene inside the building.
[0494] In some embodiments, different virtual sub-scenes are pre-configured with different scene sound effect parameters. For example, when the virtual sub-scene is a scene inside a vehicle, since the vehicle is a relatively small enclosed space, the corresponding scene sound effect parameters will process the human voice audio into an effect with a lower volume and a relatively low tone.
[0495] In some embodiments, after obtaining scene sound effect parameters corresponding to different virtual sub-scenes, the scene change nodes in the moving path are determined based on the moving path, the time points of the scene change in the voice timeline are determined based on the voice timeline of the human voice audio, the moving timeline corresponding to the moving path, and the scene change nodes, the scene sound effect parameters corresponding to the corresponding virtual sub-scenes are allocated to different time periods in the human voice audio based on the time points of the scene change, and in the process of determining the sound effect parameters based on the scene sound effect parameters, the parameter effective timestamps of the different scene sound effect parameters are recorded.
[0496] The third method is to generate intonation adjustment parameters based on the ambient sound information corresponding to the natural language command to determine the sound effect parameters.
[0497] Illustratively, ambient sound information corresponding to a natural language command is obtained, where the ambient sound information is used to indicate information related to background ambient sound in speech when collecting speech including natural language commands; intonation adjustment parameters of human voice audio are generated based on the ambient sound information; and sound effect parameters are determined based on the intonation adjustment parameters.
[0498] Among them, the above-mentioned natural language command is the user voice recorded by the terminal. After receiving the above-mentioned natural language command, the server detects the ambient sound information corresponding to the natural language command. Schematically, the user's voice content is filtered out from the sound spectrum corresponding to the natural language command to obtain ambient sound data, and the ambient sound data is detected to determine the corresponding ambient sound information.
[0499] In some embodiments, the ambient sound information includes the ambient volume, and the pitch height corresponding to the pitch adjustment parameter is positively correlated with the ambient volume. In an exemplary embodiment, the ambient volume corresponding to the ambient sound data is detected, and when the ambient volume is less than a preset volume threshold, a third pitch adjustment parameter is generated, and when the ambient sound volume is greater than or equal to the preset volume threshold, a fourth pitch adjustment parameter is generated, wherein the pitch of the third pitch adjustment parameter is lower than the pitch of the fourth pitch adjustment parameter.
[0500] In some other embodiments, the ambient sound information includes an environment recognition type corresponding to the background ambient sound, and the fifth intonation adjustment parameter is generated based on the environment recognition type. Schematically, different environment recognition types are pre-configured with different intonation adjustment parameters.
[0501] In one example, when the environment recognition type is an Internet cafe type, the tone adjustment parameter corresponds to the first tone, when the environment recognition type is a day bedroom, the tone adjustment parameter corresponds to the second tone, and when the environment recognition type is a night bedroom, the tone adjustment parameter corresponds to the third tone, wherein the first tone is higher than the second tone, and the second tone is higher than the third tone. That is, in a noisy environment, the tone of the dialogue voice replied by the non-player character is higher, and in a quiet environment, the tone of the dialogue voice replied by the non-player character is lower.
[0502] In other embodiments, the ambient sound information also affects the volume of the human voice audio. Schematically, the ambient sound information corresponding to the natural language command is obtained, and the volume adjustment parameters of the human voice audio are generated based on the ambient sound information; the sound effect parameters are determined based on the volume adjustment parameters. Schematically, the ambient volume corresponding to the ambient sound data is detected, and when the ambient volume is less than a preset volume threshold, it is determined that the natural language command corresponds to a first ambient sound state, and when the ambient sound volume is greater than or equal to the preset volume threshold, it is determined that the natural language command corresponds to a second ambient sound state, and a first volume adjustment parameter is obtained based on the first ambient sound state, and a second volume adjustment parameter is obtained based on the second ambient sound state, wherein the volume corresponding to the first volume adjustment parameter is less than the volume of the second volume adjustment parameter.
[0503] That is, when the ambient sound state corresponding to the natural language command indicates that the ambient sound is relatively noisy, the volume adjustment parameter is increased to increase the volume corresponding to the human voice audio, so that the user can better receive the reply result of the non-player character.
[0504] In some embodiments, in order to ensure the accuracy of the output human voice audio, the voice duration of the human voice audio is synchronized by static synchronization. In an illustrative manner, before generating the human voice audio, the speech rate setting information corresponding to the non-player character is obtained, wherein the speech rate setting information is used to indicate the speech rate setting of the non-player character, the estimated playback duration corresponding to the feedback text is determined based on the speech rate setting information, the speech duration corresponding to the human voice audio is obtained, and the playback speed of the human voice audio is adjusted based on the difference between the speech duration and the estimated duration.
[0505] That is, during the static synchronization execution process, before generating the human voice audio, the time length of each text segment in the feedback text is pre-calculated, and the generated voice segment is ensured to match it. Specifically, the input segment text is analyzed, and the voice length of each segment text is estimated. In one example, the estimation can be based on a preset speech rate (such as how many words per second), so that an estimated time length is assigned to each segment text. After generating the human voice audio, the difference between the voice length of the actually generated human voice audio and the estimated time length is corrected to ensure accurate synchronization of voice and text, so that the final human voice audio has a more natural voice performance, while ensuring the readability of the human voice audio that can maintain synchronization with the text.
[0506] In some embodiments, when the human voice audio is a response voice generated based on the natural language command of the first account to the non-player character, it is necessary to ensure the real-time and continuity of the human voice audio when playing the generated human voice audio to feedback the natural language command. Therefore, it is necessary to ensure the synchronization of dynamic voice data.
[0507] Illustratively, a natural language command for a non-player character is received from a first account, wherein the first account is an account for controlling a main virtual character, and multiple text segments of the non-player character are continuously generated based on the natural language command to form a feedback text, and a first start timestamp and a first end timestamp when each text segment is generated are recorded. When the i-th text segment is generated, an i-th character segment voice matching the character feature information is generated according to the i-th text segment as human voice audio, and a second start timestamp and a second end timestamp when the i-th character segment voice is generated are recorded, where i is a positive integer, a first difference between the first start timestamp and the second start timestamp is determined, and a second difference between the first end timestamp and the second end timestamp is determined.
[0508] Optionally, based on the difference between the first difference and the second difference, the first generation speed of the i+1th text segment is adjusted; or, based on the difference between the first difference and the second difference, the second generation speed of the i+1th character segment voice is adjusted; or, based on the difference between the first difference and the second difference, the transmission speed of sending the scene audio data corresponding to the i-th text segment to the terminal is adjusted.
[0509] That is, dynamic synchronization is achieved by adjusting the speech generation speed and text processing speed in real time during the process of generating human voice audio. During the speech generation process, the length of the generated speech and the progress of text processing are detected in real time. Based on the real-time detected data, the speech generation speed and / or text processing speed can be dynamically adjusted to ensure that the speech generation and text processing are consistent in time through the feedback mechanism.
[0510] Specifically, a timer or clock is used to record the time of speech generation and text processing in real time, and the speech generation speed or text processing speed is dynamically adjusted according to the real-time detected data. The speech generation speed is adjusted by changing the speech rate parameter of the text-to-speech (TTS) engine.
[0511] In other embodiments, the buffer can also be managed to control the transmission speed of the generated human voice audio, wherein the buffer is used to store the generated character segment voice and text segment to ensure that they can be output on time. The generated character segment voice and text segment are stored in the buffer. The character segment voice and text segment are output on time from the buffer. In one example, a circular buffer is used to manage data storage and output to ensure that the data is output on time. In the generation process of the text segment and the character segment voice, an accurate timestamp is added to each text segment and the character segment voice, and its start and end time is recorded, and the timestamp is recorded in the data structure. The text processing module for generating the text segment and the voice generation module for generating the character segment voice exchange time information in real time. The server detects the status of text processing and voice generation in real time during communication interaction to ensure that they are consistent in time, and detects the time error of text processing and voice generation in real time, calculates the error through the timestamp information in the circular buffer area and adjusts the system parameters in real time according to the error to control the transmission speed of the scene audio data corresponding to the human voice audio to the client by the circular buffer, ensuring time synchronization.
[0512] By dynamically adjusting the text / voice generation speed and the transmission speed of the generated voice by recording the timestamps corresponding to the text generation and the voice generation, it is possible to ensure the real-time and continuity of the output voice when the non-player character gives feedback on the natural language commands issued by the user, avoid the problem of stuttering in the output human voice audio, and improve the time accuracy of data output.
[0513] In summary, the method provided in this embodiment, when a player seeks voice feedback from a non-player character through a natural language command, obtains the sound effect parameters of the non-player character according to the relative position relationship between the main virtual character and the non-player character, and adjusts the human voice audio of the non-player character through the sound effect parameters to obtain spatial human voice audio. Since the spatial human voice audio is generated in real time based on the relative position relationship between the main virtual character and the non-player character, the application that provides a three-dimensional virtual environment does not need to pre-configure too many application resources in order to improve the diversity of sound effect performance effects, thereby reducing the waste of storage space of the computer device running the above application.
[0514] At the same time, after the sound effect parameters are adjusted, the human voice audio corresponding to the non-player characters can make the spatial human voice audio played by the client simulate the perception effect of the audio produced by the main virtual character on the non-player characters in the current relative position relationship, so that the spatial human voice audio played by the client can enable the player to replace the main virtual character to perceive the human voice produced by the non-player characters in the three-dimensional virtual environment, thereby enhancing the diversity of sound effects felt by the player during the game, thereby enhancing the player's sense of substitution in the three-dimensional virtual environment and improving the user experience.
[0515] In an embodiment of the present application, sound effect parameters are applied to the human voice audio of non-player characters in the three-dimensional virtual environment, so that the human voice audio received by the user also has the effect after it is transmitted in the three-dimensional virtual environment, thereby improving the sound effect authenticity of the non-player human voice audio when the non-player character talks with the main virtual character, and improving the user experience of the user when perceiving the human voice audio of the non-player character.
[0516] An exemplary embodiment of the present application provides a method for controlling a virtual character. The method comprises:
[0517] Displaying at least one of a main control virtual character and an NPC in a virtual environment;
[0518] Exemplarily, the main virtual character is a virtual character that the user directly controls in the virtual environment. There are also one or more non-player characters (NPC) in the virtual environment; the NPC and the main virtual character belong to the same virtual camp and are teammates of the main virtual character. The NPC follows the main virtual character in the virtual environment. In addition, the NPC can also be a follower character, a pet character, etc. controlled by the main virtual character. In addition, the NPC can also be a neutral character, and only cooperates with the main virtual character when certain conditions are met.
[0519] Obtain natural language commands;
[0520] Exemplarily, the natural language command includes the natural semantics of commanding the NPC; the NPC is commanded in the virtual scene through the natural language command, and the natural language command can be directly input by the user or extracted from the voice information input by the user. The present application does not limit the method of obtaining the natural language command.
[0521] The natural language command includes at least one of a behavior intention and a scene entity; the behavior intention is used to indicate the type of virtual activity performed by the NPC, such as indicating what type of virtual activity is performed in the virtual environment; the scene entity includes a first scene object, such as indicating what scene object in the virtual environment the virtual activity is performed on. The behavior intention and scene entity included in the natural language command are partial words in the natural language command.
[0522] In one example, the natural language command controls the NPC in terms of what type of virtual activity to perform and for what scene object in the virtual environment to perform the virtual activity; it is a complex instruction to control the NPC.
[0523] Controlling NPCs to perform virtual activities in response to natural language commands;
[0524] Exemplarily, controlling the NPC to perform virtual activities is determined based on the environmental perception information of the main virtual character and / or NPC. In some examples, the main virtual character and / or NPC obtains environmental perception information within the perception range of vision, hearing, movement trajectory and other perception methods. Exemplarily, the virtual activity is close to the current perception of the virtual environment by the main virtual character and / or NPC, and the NPC is controlled with reference to the current perception of the virtual character and / or NPC of the virtual environment. Exemplarily, the NPC has autonomous behavior capabilities, and the user only directs the NPC. The user gives commands, and the NPC understands and executes the commands based on its own autonomous behavior capabilities.
[0525] To sum up, the method provided in this embodiment determines the virtual activities that need to be performed by facing complex instructions for controlling the NPC in terms of what type of virtual activities to perform and for what scene objects in the virtual environment to perform the virtual activities, with reference to the environmental perception information of the master virtual character and / or the NPC; and ensures that the NPC and the master virtual character perform virtual activities in collaboration with each other by using the environmental perception information of the master virtual character and / or the NPC as a reference.
[0526] In an optional embodiment, the server also provides a command analysis function for natural language commands, where the command analysis function includes entity recognition, intent recognition, and hot word detection in natural language commands.
[0527] Fig.18A flowchart of a method for controlling a virtual character provided by an exemplary embodiment of the present application is shown. The method comprises:
[0528] Step 201: Acquire a spatial data set;
[0529] The spatial dataset of the virtual environment includes visual labels of scene objects, and the visual labels are used to describe the visual characteristics of the scene objects in at least one dimension, such as describing the scene objects in dimensions such as material, transparency, color, and shape.
[0530] Step 202: displaying at least one of a main control virtual character and an NPC in a virtual environment;
[0531] Exemplarily, the main control virtual character is a virtual character that the user directly controls in the virtual environment. There are also one or more NPCs in the virtual environment; the NPCs and the main control virtual character belong to the same virtual camp and are teammates of the main control virtual character.
[0532] Step 203: Acquire a natural language command in voice form;
[0533] The natural language command includes natural semantics for commanding the NPC, and the command information includes behavioral intention and scene entity; the behavioral intention is used to indicate the type of virtual activity performed by the NPC, and the scene entity is used to indicate the target entity of the virtual activity;
[0534] Step 204: converting the natural language command in voice form into the natural language command in text form;
[0535] Exemplarily, speech-to-text (Automatic Speech Recognition, ASR) processing is performed on the natural language command in the form of speech to determine the natural language command in the form of text. Speech-to-text processing generally includes calling components such as acoustic models and language models to recognize the pronunciation, vocabulary, grammatical structure and other information of the natural language command in the form of speech, and convert it into a natural language command in the form of text.
[0536] Step 205: performing intent recognition on the natural language command to obtain a first classification label;
[0537] Exemplarily, the first intention indication corresponding to the first classification label includes a command intention for a non-player character on an intention dimension; for example, at least one of the following: indicating which of a plurality of non-player characters the non-player character commanded by the command text is, indicating whether the virtual activity performed by the non-player character is related to initiating a virtual attack, and indicating what kind of virtual activity the non-player character performs.
[0538] Step 206: Perform entity recognition on the natural language command to obtain a target entity;
[0539] The target entity is determined from the virtual environment in combination with the environmental perception information, which includes information perceived by at least one of the main virtual character and the non-player character from the virtual environment. The target entity may be the entity currently seen by the player, or may be an entity that the player has seen in the past.
[0540] Step 207: In response to the natural language command, the NPC is controlled to perform a virtual activity according to the first intention corresponding to the first classification label, or the NPC is controlled to perform a virtual activity associated with the target entity, or the NPC is controlled to perform a virtual activity associated with the target entity according to the first intention corresponding to the first classification label;
[0541] Exemplarily, the NPC is controlled to perform a virtual activity according to the first classification tag and / or the instruction of the target entity.
[0542] Step 208: reporting the NPC's feedback information;
[0543] Exemplarily, the game application can generate corresponding feedback information in real time according to the environmental perception information of the non-player character and make a broadcast. For example, when the environmental perception information of the non-player character triggers the broadcast condition, the game application can generate corresponding feedback information in real time according to the environmental perception information that triggers the broadcast condition; or, when the non-player character or the main control virtual character triggers the broadcast condition, the game application can generate corresponding feedback information in real time according to the environmental perception information of the non-player character.
[0544] The preprocessing stage of step 201 can be implemented as follows:
[0545] Sub-step 1: Obtaining attribute text of scene objects in the virtual scene;
[0546] Exemplarily, attribute text is used to introduce the inherent attributes of scene objects in a virtual scene; on the one hand, attribute text realizes the description of scene objects in a textual mode, and while describing scene objects, it provides semantic information for the prediction of the visual labels of scene objects. On the other hand, attribute text is the description of scene objects in a virtual scene. In the face of a large number of scene objects in a virtual scene and the existence of reused object models, it can more accurately describe the inherent attributes of scene objects in a virtual scene.
[0547] In an optional implementation, the attribute text includes at least one of a name of the scene object in the virtual scene and a size of the scene object in the virtual scene.
[0548] Sub-step 2: obtaining an appearance image of a scene object in a virtual scene;
[0549] Exemplarily, the appearance image is used to describe the style of the scene object; the appearance image carries the appearance style of the scene object such as color, texture, shape, etc., and the relative position relationship between each sub-part in the picture mode, and can fully describe the scene object from the picture mode (or visual mode). In an optional implementation, the appearance image of the scene object includes images obtained by observing the scene object from at least two perspectives.
[0550] Sub-step 3: Call the multimodal model to perform prediction on the attribute text and appearance image of the scene object to obtain the visual label of the scene object;
[0551] Exemplarily, the multimodal model has the ability to perform model prediction on text information and image information of different modalities. In this embodiment, the input parameters of the multimodal model are the attribute text and appearance image of the scene object; the multimodal model predicts the visual label of the scene object from the two modalities of text modality and image modality, and the visual label is used to describe the visual features of the scene object in at least one dimension.
[0552] Optionally, the multimodal model includes a visual question answering model; the question statement carries attribute text of the scene objects; on the one hand, the question statement is used to guide the visual question answering model to convert the appearance image into the visual label of the scene object; on the other hand, while guiding the conversion of the visual label of the scene object, the question statement provides supplementary information of the scene object in text form.
[0553] Get the expected information of scene objects;
[0554] Exemplarily, the expected information is used to indicate the description dimension of the scene object expected in the visual tag, and / or the expected format of the visual tag; in one example, the expected information is used to indicate that the description dimension of the scene object expected in the visual tag includes but is not limited to at least one of the type description, material, transparency, color, surface features, and shape of the scene object. In another example, the expected information is used to indicate that the expected format of the visual tag is at least one of the following: Comma Separated Values (CSV), JavaScript Object Notation (JSON), and eXtensible Markup Language (XML).
[0555] Construct a question statement of the appearance image based on the expected information and attribute text;
[0556] Exemplarily, the first subpart of the question sentence is supplementary introduction information of the scene object, carrying attribute text of the scene object; the second subpart of the question sentence is an answer guidance sentence for the visual question answering model, carrying expected information.
[0557] Optionally, the spatial data set includes matching labels of scene objects in the virtual scene; correspondingly: performing pseudo-spoken rewriting on the visual labels of the scene objects to obtain matching labels that conform to spoken expressions of natural language.
[0558] Exemplarily, as described above, visual labels are used to describe the visual features of scene objects in at least one dimension, and the appearance image of the scene objects presents rich visual features of the scene objects. In the spoken expression of natural language, however, the description of scene objects cannot cover the visual features of scene objects in all dimensions. The purpose of performing a pseudo-colloquial rewrite on the visual labels of scene objects is to obtain a matching label that is closer to the spoken expression; it can be understood that performing a pseudo-colloquial rewrite can be to delete part of the content in the visual label, or to change the visual label to a label with the same semantics but different textual expression.
[0559] In an optional implementation, the pseudo-colloquial rewriting is performed by calling a natural language model; the visual labels of the scene objects are input into the natural language model, and matching labels that conform to the spoken expressions of the natural language are predicted. The pseudo-colloquial rewriting of the visual labels is implemented based on calling the natural language model. Exemplarily, the natural language model carries prior knowledge of the spoken expressions of the natural language. For example, the natural language model is implemented as a large language model (Large Language Model, LLM).
[0560] Optionally, the spatial dataset also includes spatial information of scene objects, and accordingly, also includes:
[0561] The spatial position of the scene object in the virtual scene is obtained, and the spatial position of the scene object is determined as auxiliary information of the visual label of the scene object.
[0562] Exemplarily, the spatial position of the scene object in the virtual scene is used to indicate the deployment of the scene object in the virtual scene, and the spatial information is used to indicate the position, size and other information of the scene object in the virtual scene after the scene object is deployed in the virtual scene.
[0563] Further, obtaining at least one of the coordinate position, orientation information, bounding box information, and cover point information of the scene object in the virtual scene;
[0564] Among them, the coordinate position (Location) is used to indicate the position of the scene object in the virtual scene, such as the coordinate information of the center point or preset point of the scene object in the virtual scene; the orientation (Rotation) is used to indicate the direction facing the scene object in the virtual scene, such as the direction facing the front of the scene object in the virtual scene; the bounding box (BoundingBox) is used to indicate the size of the scene object in the virtual scene; the cover points (Cover points) are used to indicate the recommended virtual character position points when the virtual character approaches the scene object, so that the scene object can mask the virtual character.
[0565] The intention recognition phase of step 205 can be implemented as follows:
[0566] Sub-step 4: calling at least one hierarchical prediction network to perform intent recognition on the natural language command, and obtaining a first classification label of the natural language command intent to command the NPC;
[0567] Exemplarily, the hierarchical prediction network has the ability to predict the classification label corresponding to the command text. Exemplarily, the hierarchical prediction network includes at least two layers of sub-networks, the upper network and the lower network are cascaded, and the lower network further performs the prediction of the classification label according to the prediction result output by the upper network. At least two layers of sub-networks are constructed based on the tree structure of multiple classification labels. The hierarchical prediction network constructs at least two layers of sub-networks corresponding to the hierarchical structure for the tree structure of the classification labels, and splits the prediction task of a wide variety of classification labels into prediction sub-tasks performed by at least two layers of sub-networks, so as to reduce the complexity of classification prediction of each layer of sub-network.
[0568] In an optional implementation, each hierarchical prediction network in at least one hierarchical prediction network is used to predict the command intention of the command text for a non-player character on an intention dimension; the intention dimension includes: at least one of the subject dimension, the semantic dimension, and the behavior dimension. Exemplarily, in at least one hierarchical prediction network, the i-th layer subnetwork in each hierarchical prediction network is used to predict the first-level behavior label of the command text on an intention dimension, and the i+1-th layer subnetwork in the hierarchical prediction network is used to predict the second-level behavior label corresponding to the command text in the first-level behavior label, i is a positive integer; the high-level behavior label (such as the first-level behavior label) predicted by the high-level subnetwork (such as the i-th layer subnetwork) in the hierarchical prediction network includes multiple sublabels, or subordinate lower-level labels; the corresponding low-level subnetwork (such as the i+1-th layer subnetwork) needs to be called to further perform classification prediction (such as predicting the second-level behavior label). The prediction task of a wide variety of classification labels is split into prediction subtasks performed by at least two layers of subnetworks to reduce the complexity of classification prediction of each layer of subnetwork.
[0569] Furthermore, the intent dimension includes a subject dimension, the hierarchical prediction network includes a hierarchical subject prediction network, and the subject prediction network has the ability to predict the subject type in a natural language command; the subject type is used to indicate the identity of the NPC commanded by the natural language command; exemplarily, the subject prediction network is used to predict which of a plurality of non-player characters the non-player character commanded by the command text is, and the number of non-player characters commanded by the command text can be one or more.
[0570] And / or, the intent dimension includes a semantic dimension, the hierarchical prediction network includes a hierarchical semantic prediction network, the semantic prediction network has the ability to predict the semantic type in a natural language command; the semantic type is used to indicate the NPC's control method for initiating a virtual attack; exemplarily, the subject prediction network is used to predict whether the virtual activity performed by a non-player character commanded by the command text is related to initiating a virtual attack.
[0571] And / or, the intention dimension includes a behavior dimension, the hierarchical prediction network includes a hierarchical behavior prediction network, the behavior prediction network has the ability to predict the behavior intention type in the natural language command; the behavior intention type is used to indicate the behavior mode of the NPC for performing the virtual activity. Exemplarily, the behavior prediction network is used to predict what virtual activity the non-player character performs.
[0572] Accordingly, in step 207, in response to the natural language command, the NPC is controlled to perform a virtual activity according to the first intention corresponding to the first classification label;
[0573] Exemplarily, the first intention corresponding to the first classification tag indicates a command intention for a non-player character in an intention dimension, such as at least one of the following: indicating which non-player character is commanded by the command text among a plurality of non-player characters, indicating whether the virtual activity performed by the non-player character is related to launching a virtual attack, and indicating what virtual activity the non-player character performs. According to the instruction of the first classification tag, the NPC is controlled to perform the virtual activity.
[0574] The target entity identification process of step 206 can be implemented as follows:
[0575] Sub-step 5: querying and obtaining the target entity from the entity set of the virtual environment according to the target entity information of the target entity indicated by the natural language command and the environment perception information;
[0576] The target entity is determined from the virtual environment in combination with environmental perception information, and the environmental perception information includes information perceived by the main virtual character and / or the non-player character from the virtual environment.
[0577] Since the target entity information in natural language commands is usually vague, for example, a natural language command may be "move to the red truck", where the target entity information is "red truck", and there may be many red trucks in the virtual environment. It is impossible to accurately determine the target entity in the virtual environment based solely on the target entity information in the natural language command.
[0578] Therefore, the embodiment of the present application provides a method for determining the target entity by combining the target entity information and the environmental perception information. For the previous example, the "red truck" expressed by the player in the natural language command should be a red truck that the player can see. Therefore, combined with the field of view of the main control virtual character, the red truck within the field of view of the main control system character can be screened out from the multiple red trucks in the virtual environment. The red truck is the target entity indicated by the player in the natural language command.
[0579] Since natural language commands are issued by players based on their perception of the virtual environment, in order to accurately identify the target entity indicated in the natural language command, the game application will combine the environmental perception information when the player issues the natural language command to identify the target entity in the natural language command. Based on the player's perception of the virtual environment, the entity that the player can perceive in the virtual environment and is closest to the target entity information is inferred as the target entity.
[0580] Exemplarily, the virtual environment includes multiple candidate entities that match the natural language command, and the target entity is an entity selected from the multiple candidate entities based on the environmental perception information of the main virtual character or the non-player character. For example, the game application first selects multiple candidate entities that match the target entity information in the natural language command from the entity set, and then selects the entity that can be perceived by the main virtual character or the non-player character from the multiple candidate entities as the target entity based on the environmental perception information.
[0581] For example, the environmental perception information of the main virtual character may include: the pictures or entities that the main virtual character can see when observing the virtual environment, the environmental sound effects that the main virtual character can hear, the source direction of the environmental sound effects, the type and volume of the environmental sound effects, the perception information obtained by the main virtual character by using perception skills (for example, visual information, auditory information, sound wave information, light reflection information, etc.), and the perception information obtained by the main virtual character by using sensing props (for example, sensor signals from sensors, position detection signals from position detection props, etc.).
[0582] The environmental perception information of the non-player character may include: the images or entities that the non-player character can see when observing the virtual environment, the environmental sounds that the non-player character can hear, the source direction of the environmental sounds, the type and volume of the environmental sounds, the perception information obtained by the non-player character by using perception skills (for example, visual information, auditory information, sound wave information, light reflection information, etc.), and the perception information obtained by the non-player character by using sensing props (for example, sensor signals from sensors, position detection signals from position detection props, etc.).
[0583] It should be noted that the environmental perception information may include the information perceived by the controlling virtual character and the non-player character in real time when receiving the natural language command, and may also include the information perceived by the controlling virtual character and the non-player character in history before receiving the natural language command. That is, the target entity may be the entity currently seen by the player, or it may be the entity that the player has seen in the past. Therefore, the game application needs to combine the real-time environmental perception information and the historical environmental perception information of the controlling virtual character and / or the non-player character to identify the target entity.
[0584] For example, when the natural language command is "Let's go back to the hotel we just passed by", the game application needs to query the hotels that the main virtual character has passed by based on the historical environmental perception information of the main virtual character.
[0585] In an optional embodiment, the client queries and obtains the target entity from the entity set of the virtual environment according to the target entity information and the environment perception information indicated by the natural language command. In another optional embodiment, the client reports the natural language command to the server, and the server queries and obtains the target entity from the entity set of the virtual environment according to the target entity information and the environment perception information indicated by the natural language command.
[0586] Optionally, the game application infers the target entity based on the target entity information, the environment perception information and the pre-processed virtual environment data, wherein the pre-processed virtual environment data includes a set of entities in the virtual environment.
[0587] The entity set includes entity information of each entity in the virtual environment. The entity information includes at least one of the following: name, type, location, feature, text label, embedding vector of text label, image, embedding vector of image.
[0588] The text label may be at least one word obtained after segmenting a feature (for example, a descriptive text of an entity appearance), and the embedding model is called to obtain an embedding vector corresponding to each text label.
[0589] The image may include an image obtained by observing a three-dimensional model of the entity from at least one direction, for example, the image may include three views of the entity. The embedding vector of the image is the embedding vector of the image feature label. The image feature label is obtained by calling the multimodal model to perform feature recognition based on the image and text label of the entity.
[0590] Exemplarily, the entity set includes a first type of entity and a second type of entity, the first type of entity includes at least one of an area and a building, and the second type of entity includes a physical object. The text labels of the first type of entity are obtained by manual labeling. For example, the text label of a room of a building is manually labeled as: an area, a building, a floor, and a room. The text labels of the second type of entity are automatically generated by calling a large language model.
[0591] The method for generating text labels for the second type of entity may include: obtaining at least one image of the entity, inputting the at least one image into a large language model, and obtaining a description text of the entity; segmenting the description text to obtain at least one text label of the entity, each text label including a word obtained after the segmentation process; performing a vectorization operation on the text label to obtain an embedding vector corresponding to the text label.
[0592] The embedding vector is the high-dimensional vector data of the text label. By pre-generating the embedding vector of the text label, there is no need to repeatedly generate vectors for the text label of each entity when matching the target entity from the entity set based on the target entity information, which can improve the matching efficiency of the target entity information and the entity information.
[0593] It should be noted that this method does not use the description text directly as the text label, but uses the word segmentation result of the description text as the text label. This is because the description text is usually of different lengths. For longer description texts, their embedding vectors are less effective in actual search and similarity comparison. For example, when searching for "a small blue truck with rusty body" using "truck", "a small blue truck with rusty body" is likely to be ranked after "a car" in the recall results. In fact, for the searched "truck", no matter how complex its additional description is, "truck" should be searched first. Therefore, when performing similarity search, the comparison should be one feature word at a time, not the description text.
[0594] In addition, in order to further extract the image features of the entity, the method provided in the embodiment of the present application can also further extract hidden visual features in the image of the entity based on the text label and image of the entity. Taking the first entity as an example, the method includes: obtaining at least one perspective image of the first entity; obtaining at least one text label of the first entity; calling the multimodal model to extract the image features of the first entity based on at least one perspective image and at least one text label of the first entity, and obtain the image embedding vector of the first entity.
[0595] Since text labels are manually annotated, or generated by a large language model based on human language characteristics; and text labels extracted based on human language habits may ignore some features of the entity. For example, when manually annotating a text label for an oil drum, only more eye-catching text labels such as "metal" and "rust" may be annotated, while detailed features such as "rust", "blue", "yellow", and "paint peeling in the lower right corner" are ignored. Therefore, the method provided in the embodiment of the present application also uses a multimodal model based on the text label and entity image of the entity, so that the multimodal model extracts more image features from it; performs image similarity matching with the target entity based on the image features to improve the recognition accuracy of the target entity.
[0596] Among them, the training method of the multimodal model can be: inputting sample entity images and sample labels into the pre-trained multimodal model, fine-tuning the pre-trained multimodal model according to the loss of the predicted labels and sample labels output by the pre-trained multimodal model, and obtaining a multimodal model that can output image feature labels based on the input text labels and images, and the image feature labels include more detailed description texts proposed from the image. For example, the text description of "a metal oil barrel" can only be associated with the two text labels "metal" and "oil barrel", while the Clip (multimodal) model can also identify the hidden visual information in the image from the image of the oil barrel, such as "rusty", "blue", and "yellow" image feature labels, which are features that do not appear in the text labels. Therefore, combining the Clip model for image feature search can further improve the accuracy of entity search.
[0597] In an optional embodiment, the game application may query the target entity using the following method.
[0598] (1) Parse the natural language command to obtain target entity information; the target entity information includes at least one of the following: entity type, entity name, entity location, and entity feature.
[0599] Optionally, the game application calls the large language model to parse the natural language command to obtain target entity information of the target entity, and calls the embedding model to vectorize the target entity information to obtain a target embedding vector of the target entity information.
[0600] For example, if the natural language command is "Come to the blue truck in front", the target entity information that can be parsed by the large language model includes: the direction information "front" and the entity name "blue truck".
[0601] For example, named entity recognition technology in the field of natural language processing can also be used to extract target entity information from natural language commands. Named entity recognition technology is used to identify entities with specific meanings in texts, such as names of people, places, locative words, adjectives, etc.
[0602] For example, if the natural language command is "Go to the first floor of the motel and find a paper box behind the red sofa", the named entity recognition technology can identify: the architectural noun "the first floor of the motel"; the object nouns "sofa" and "paper box"; the locative word "behind"; the adjective "red". After logical construction, a hierarchical scene query call form with search type, content, locative word, and constraint information can be formed. Among them, the search type is used to narrow the data retrieval scope based on similarity matching, the search content is a specific entity description, and the adjective will be merged into the description text for query together, and the locative word is used to form a constraint on the search scope. The floor constraint is determined by the player's location. In indoor scenes, the upper and lower ranges need to be narrowed to avoid finding invisible entities across floors.
[0603] (2) Calculate the similarity between the target entity information and each entity information in the entity set.
[0604] Exemplarily, the entity set includes entity information of at least one entity, and the entity information of each entity may include at least one of a text embedding vector of a text label and an embedding vector of an image feature. Then the target entity information may calculate similarity with the text embedding vector and the image embedding vector respectively, and the entity with higher similarity is determined as the target entity.
[0605] The following are methods for similarity matching with text embedding vectors and similarity matching with image embedding vectors.
[0606] 1) Similarity includes the text similarity between the target entity information and the text embedding vector.
[0607] Taking the first entity in the entity set as an example, the game application (client or server) segments the target entity information to obtain at least one target entity tag; converts the at least one target entity tag into at least one target embedding vector; obtains the entity information of the first entity, the entity information includes a text embedding vector, and the text embedding vector is an embedding vector obtained based on the text tag conversion of the first entity; calculates the text parent similarity between at least one target embedding vector and the text embedding vector respectively, and obtains at least one text parent similarity corresponding to at least one target embedding vector; and determines the sum of the at least one text parent similarity as the text similarity between the target entity information and the entity information of the first entity.
[0608] For example, the number of target entity tags is two, target entity tag 1 and target entity tag 2, and the number of entities in the entity set is also two, entity 1 and entity 2. Entity 1 corresponds to text embedding vector 1, and entity 2 corresponds to text embedding vector 2. Then similarity 1 between target entity tag 1 and text embedding vector 1 is calculated, and similarity 2 between target entity tag 2 and text embedding vector 1 is calculated, and the sum of similarity 1 and similarity 2 is determined as the text similarity between the target entity information and entity 1. Similarity 3 between target entity tag 1 and text embedding vector 2 is calculated, and similarity 4 between target entity tag 2 and text embedding vector 2 is calculated, and the sum of similarity 3 and similarity 4 is determined as the text similarity between the target entity information and entity 2.
[0609] Exemplarily, the entity information of the first entity includes at least one text embedding vector; the at least one target embedding vector includes a first target embedding vector; the game application calculates the text sub-similarity between the first target embedding vector and the at least one text embedding vector respectively to obtain at least one text sub-similarity; the highest value of the at least one text sub-similarity is determined as the text parent similarity corresponding to the first target embedding vector.
[0610] For example, the number of target entity tags is 1: target entity tag 1. The number of entities in the entity set is also 1: entity 1. Entity 1 corresponds to text embedding vector 1 and text embedding vector 3. Then the similarity 1 between target entity tag 1 and text embedding vector 1 is calculated, and the similarity 5 between target entity tag 1 and text embedding vector 3 is calculated, and the larger value of similarity 1 and similarity 5 is determined as the similarity between the target entity information (target entity tag 1) and entity 1.
[0611] For example, for the input target entity information "blue car", the target entity information is first segmented into target entity labels "blue" and "car", which means that the queried target entity has two features. Then each feature is calculated with each entity in the entity set for similarity, and each feature takes the maximum similarity. Finally, the sum of the similarity scores corresponding to all query features is taken, which represents the text similarity between the target entity information and the queried entity.
[0612] For example, the target entity information includes two target entity tags "blue" and "car", and the two target entity tags are converted into two target embedding vectors. Then the entity information in the entity set is obtained. For example, the entity set includes entity 1 and entity 2. The text tags of entity 1 include "metal", "old", "car", "truck", "blue", and "damaged", and the text tags of entity 2 include "metal", "scratches", "car", "damaged", "money transporter", and "black".
[0613] Calculate the text similarity between the target entity information and entity 1: first calculate the similarity between the target entity label "blue" and each text label of entity 1, and take the maximum value. For example, the similarity between the target entity label "blue" and the text label "blue" of entity 1 is 1 (1 is the maximum value); then calculate the similarity between the target entity label "car" and each text label of entity 1, and take the maximum value. For example, the similarity between the target entity label "car" and the text label "car" of entity 1 is 1. Then sum the two similarities of the target entity labels "car" and "blue" to obtain the final similarity between the target entity information and entity 1, which is 2.
[0614] Similarly, calculate the text similarity between the target entity information and entity 2: first calculate the similarity between the target entity label "blue" and each text label of entity 2, and take the maximum value. For example, the similarity between the target entity label "blue" and the text label "black" of entity 1 is 0.91; then calculate the similarity between the target entity label "car" and each text label of entity 1, and take the maximum value. For example, the similarity between the target entity label "car" and the text label "car" of entity 2 is 1 (1 is the maximum value). Then sum the two similarities of the target entity labels "car" and "blue" to obtain the final similarity between the target entity information and entity 2, which is 1.91.
[0615] It can be seen that the similarity between the target entity information and entity 1 is 2, which is higher than the similarity between the target entity information and entity 2, which is 1.91.
[0616] 2) Similarity includes the image similarity between the target entity information and the image embedding vector.
[0617] Taking the first entity in the entity set as an example, the game application (client or server) segments the target entity information to obtain at least one target entity tag; converts the at least one target entity tag into at least one target embedding vector; obtains the entity information of the first entity, the entity information includes an image embedding vector, and the image embedding vector is an embedding vector obtained based on the image extraction of the first entity; calculates the image parent similarity of at least one target embedding vector and the text embedding vector respectively, and obtains at least one image parent similarity corresponding to the at least one target embedding vector; and determines the sum of the at least one image parent similarity as the image similarity between the target entity information and the entity information of the first entity.
[0618] For example, the number of target entity tags is two, target entity tag 1 and target entity tag 2, and the number of entities in the entity set is also two, entity 1 and entity 2. Entity 1 corresponds to image embedding vector 1, and entity 2 corresponds to image embedding vector 2. Then similarity 1 between target entity tag 1 and image embedding vector 1 is calculated, and similarity 2 between target entity tag 2 and image embedding vector 1 is calculated, and the sum of similarity 1 and similarity 2 is determined as the image similarity between the target entity information and entity 1. Similarity 3 between target entity tag 1 and image embedding vector 2 is calculated, and similarity 4 between target entity tag 2 and image embedding vector 2 is calculated, and the sum of similarity 3 and similarity 4 is determined as the image similarity between the target entity information and entity 2.
[0619] Exemplarily, the entity information of the first entity includes at least one image embedding vector; at least one target embedding vector includes a first target embedding vector; the game application calculates the image sub-similarity between the first target embedding vector and the at least one image embedding vector respectively to obtain at least one image sub-similarity; and the highest value of the at least one image sub-similarity is determined as the image parent similarity corresponding to the first target embedding vector.
[0620] For example, the number of target entity tags is 1: target entity tag 1. The number of entities in the entity set is also 1: entity 1. Entity 1 corresponds to image embedding vector 1 and image embedding vector 3. Then the similarity 1 between target entity tag 1 and image embedding vector 1 is calculated, and the similarity 5 between target entity tag 1 and image embedding vector 3 is calculated, and the larger value of similarity 1 and similarity 5 is determined as the similarity between the target entity information (target entity tag 1) and entity 1.
[0621] In an optional embodiment, the target image embedding vector may be used for calculating the image similarity, that is, the target image embedding vector is used instead of the target embedding vector in the above method. The target image embedding vector may be obtained by: segmenting the target entity information to obtain at least one target entity tag; inputting at least one target entity tag into a multimodal model to obtain a target image embedding vector.
[0622] 3) Similarity includes text similarity and image similarity.
[0623] When the first entity and the target entity have corresponding text similarity and image similarity, the game application determines an average value of the text similarity and the image similarity as the similarity between the first entity and the target entity.
[0624] Alternatively, when the first entity and the target entity have corresponding text similarity and image similarity, the larger value of the text similarity and the image similarity is determined as the similarity between the first entity and the target entity.
[0625] Alternatively, when the first entity and the target entity have corresponding text similarity and image similarity, the sum of the text similarity and the image similarity is determined as the similarity between the first entity and the target entity.
[0626] (3) Determine the target entity from the entity set based on similarity and environmental perception information.
[0627] In an optional embodiment, the game application first performs a similarity search based on the target entity information to obtain at least one candidate entity with a high similarity, and sorts them according to the similarity; then, based on the position, location, orientation and other information of the main virtual character and / or the non-player character, the game application performs perceptual filtering and sorting, and finally screens out the target entity.
[0628] Exemplarily, the game application determines a target perception range based on target entity information and environmental perception information; and obtains the target entity by filtering entities perceived within the target perception range based on similarity.
[0629] Alternatively, the game application determines x entities with the highest similarity from the entity set according to the similarity to form a candidate entity list; determines the target perception range according to the target entity information and the environmental perception information, and determines the entity in the target perception range among the x entities as the target entity. x is a positive integer.
[0630] Exemplarily, the game application determines the entity with the highest similarity in the perception range as the target entity.
[0631] It should be noted that when there are multiple entities with equal similarity within the target perception range, the multiple entities can be sorted according to the distance between the entity and the controlling virtual character. For example, when there are at least two entities with the highest similarity in the perception range, the entity with the highest similarity in the perception range and the closest distance to the controlling virtual character is determined as the target entity.
[0632] In an optional embodiment, the game application determines the entity with the highest similarity within the perception range and above the threshold as the target entity. Exemplarily, the number of the perception range is at least one. For example, the perception range includes at least one of the following: the visual range of the main virtual character, the hearing range of the main virtual character, the perception range of the main virtual character's perception of virtual props, the visual range of the non-player character, the hearing range of the non-player character, and the perception range of the non-player character's perception of virtual props.
[0633] Exemplarily, the game application may sequentially traverse at least one of the above-mentioned perception ranges, or the game application may sequentially traverse at least one perception range determined according to the natural language command. For example, if the natural language command is "Can you see the truck in front? Move to the truck", the game application may first traverse the visual range of the main virtual character, and then traverse the visual range of the non-player character, and match the entity with the highest similarity and above the threshold.
[0634] Exemplarily, when there are at least two perception ranges, the traversal order of the at least two perception ranges may be preset, or the traversal order of the at least two perception ranges may be determined according to the intention indicated in the natural language command. For example, when the intention is to search for an item, the visual range is traversed first. When the intention is to find a combat scene, the auditory range is traversed first.
[0635] If there are multiple levels of nested instructions in the natural language command, the first target entity will be searched first, and then the next target entity will be searched based on the position of the previous target entity. In the case where the natural language command includes a second target entity determined based on the position of the first target entity, the first target entity is queried from the entity set of the virtual environment based on the first target entity information and environmental perception information of the first target entity indicated by the natural language command; then the second target entity is queried from the entity set of the virtual environment based on the second target entity information, the first target entity and environmental perception information of the second target entity.
[0636] In response to natural language commands, the NPC is controlled to perform a virtual activity associated with a target entity.
[0637] Exemplarily, a large language model is called to identify the behavioral intent of natural language commands, and a control instruction or a control instruction sequence is generated according to the behavioral intent. According to the control instruction or the control instruction sequence, a non-player character is controlled to complete an activity based on a target entity.
[0638] The broadcasting of the NPC feedback information in step 208 may be implemented as follows:
[0639] Sub-step 6: reporting feedback information of the non-player character, wherein the feedback text corresponding to the feedback information is a non-fixed text generated based on the environmental perception information of the non-player character;
[0640] Exemplarily, the game application can generate corresponding feedback information in real time according to the environmental perception information of the non-player character and make a broadcast. For example, when the environmental perception information of the non-player character triggers the broadcast condition, the game application can generate corresponding feedback information in real time according to the environmental perception information that triggers the broadcast condition; or, when the non-player character or the main control virtual character triggers the broadcast condition, the game application can generate corresponding feedback information in real time according to the environmental perception information of the non-player character.
[0641] For example, the feedback text is obtained by inferring the large language model based on the static entity data of the three-dimensional virtual environment and the dynamic environment perception information of the non-player characters. The large language model will generate different feedback texts according to the different perception situations of the non-player characters.
[0642] For example, the feedback text can also be obtained by inferring the static entity data of the three-dimensional virtual environment and the dynamic environment perception information of the non-player character through the language model, and the feedback text can be in line with the personality characteristics of the non-player character. For example, when the non-player character is a strong uncle, the feedback text can adopt a heroic tone; when the non-player character is a reporter, the feedback text can adopt a news report tone.
[0643] It should be noted that due to the randomness of the results generated by the large language model, in the same scene, when the environmental perception information of the non-player character is the same, the generated feedback text may also be different.
[0644] The environmental perception information may include information perceived by the non-player character in real time, or may include information perceived by the non-player character historically.
[0645] Exemplarily, the feedback information is inferred by the game application based on the environmental perception information of the non-player character and the static entity data of the three-dimensional virtual environment. The feedback information can be a voice broadcast or a text broadcast. The static entity data includes data of relatively unchanged entities in the three-dimensional virtual environment, for example, model information of various buildings, terrains, and vehicles in the three-dimensional virtual environment.
[0646] Exemplarily, the feedback information may include at least one of the following: instant feedback, execution feedback, and dynamic feedback. Instant feedback and execution feedback are both feedback information generated for natural language commands, and dynamic feedback is feedback information spontaneously generated by non-player characters.
[0647] 1. Instant feedback is feedback information generated immediately when a natural language command is received. Instant feedback can include reply broadcast and instant broadcast. Reply broadcast is used to resume the inquiry in the natural language command; instant broadcast is used to feedback the reception of the natural language command.
[0648] For example, the client can command the non-player character to move in the three-dimensional virtual environment according to the intention of the natural language command. Then, when the client receives the natural language command, it can generate instant feedback corresponding to the natural language command, and the instant feedback is used to indicate that the natural language command has been received, or the instant feedback is used to instantly reply to the natural language command.
[0649] In an optional embodiment, the feedback information includes a reply report. The reply report includes a reply to the query natural language command generated in response to the query natural language command raised by the player. For example, when the query natural language command includes querying the location of the target entity, the corresponding reply report should include the query result of the location of the target entity.
[0650] When the behavioral intention of the received natural language command is an inquiry, the terminal broadcasts a reply message of the non-player character; the reply message includes reply content for the natural language command generated according to the environmental perception information of the non-player character on the three-dimensional virtual environment.
[0651] The natural language command is used to inquire about information from the non-player charact...
Claims
1. A method for broadcasting a non-player character, characterized in that: The method comprises: Displaying at least one of a main control virtual character and a non-player character in a three-dimensional virtual environment, wherein the non-player character follows the main control virtual character to move in the three-dimensional virtual environment; receiving a natural language command directed to the non-player character; In response to the natural language command, feedback information of the non-player character is broadcasted, wherein the feedback text corresponding to the feedback information is a non-fixed text generated based on environmental perception information, and the environmental perception information includes information perceived by at least one of the main virtual character and the non-player character about the three-dimensional virtual environment.
2. The method according to claim 1, characterized in that The feedback information includes a reply broadcast; The reporting of the feedback information of the non-player character in response to the natural language command includes: In the case where the behavioral intention of the natural language command is an inquiry, a reply report of the non-player character is broadcasted; the reply report includes reply content for the natural language command generated according to the environmental perception information of the non-player character about the three-dimensional virtual environment; The natural language command is used to inquire about information from the non-player character.
3. The method according to claim 2, characterized in that When the behavioral intention of the natural language command is an inquiry, reporting the reply of the non-player character includes at least one of the following: In the case where the natural language command includes an intelligence inquiry about the target entity, based on the perception of the non-player character about the target entity, a first reply broadcast is broadcast; the first reply broadcast includes the intelligence perception result of the non-player character about the target entity; In the case where the natural language command includes a query about the location of the target entity, based on the non-player character's perception of the target entity, a second reply broadcast is broadcast, wherein the second reply broadcast includes a description of the location of the target entity.
4. The method according to claim 2, characterized in that: The method further comprises: Calling a large language model to parse the natural language command, and inferring a reply feedback text based on the static environment data of the three-dimensional virtual environment and the environment perception information; Human voice audio is generated based on the reply feedback text to obtain the reply broadcast.
5. The method according to claim 1, characterized in that The feedback information includes feedback broadcast; The reporting of the feedback information of the non-player character in response to the natural language command includes: In the case where the behavioral intention of the natural language command is to execute a task, broadcasting the feedback broadcast; the feedback broadcast includes an execution state after the natural language command is executed according to the environmental perception information of the three-dimensional virtual environment by the non-player character; The natural language command is used to instruct the non-player character to perform a task.
6. The method according to claim 5, characterized in that When the behavior intention of the natural language command is to execute a task, broadcasting the feedback broadcast includes at least one of the following: In a case where the behavioral intention of the natural language command includes searching for a target entity, broadcasting a first feedback broadcast, wherein the first feedback broadcast includes a search result of the non-player character for the target entity; When the behavioral intention of the natural language command includes using a virtual prop, broadcasting a second feedback broadcast, wherein the second feedback broadcast is used to indicate that the virtual prop is ready; In a case where the behavioral intention of the task natural language command includes using a virtual prop, broadcasting a third feedback broadcast, wherein the third feedback broadcast is used to indicate a result of using the virtual prop; When the behavioral intention of the natural language command includes controlling the movement of the non-player character, broadcasting a fourth feedback broadcast, wherein the fourth feedback broadcast is used to indicate a movement result of the movement; In a case where the behavioral intention of the natural language command includes performing an interaction with a target entity, broadcasting a fifth feedback broadcast, wherein the fifth feedback broadcast is used to indicate at least one of a search result of the non-player character for the target entity and an interaction result between the non-player character and the target entity; In a case where the behavioral intention of the natural language command includes item interaction, a sixth feedback broadcast is broadcast, wherein the sixth feedback broadcast is used to indicate an item interaction result of the non-player character performing the item interaction.
7. The method according to claim 5, characterized in that The method further comprises: Calling a large language model to parse the natural language command, and inferring a task execution instruction based on the static environment data of the three-dimensional virtual environment and the environment perception information; Controlling the non-player character to perform the task according to the task execution instruction; Generate status feedback text according to the execution status of the task performed by the non-player character; Human voice audio is generated based on the status feedback text to obtain the feedback broadcast.
8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: When the environmental perception information of the non-player character about the three-dimensional virtual environment meets the dynamic broadcasting condition, a dynamic broadcast is broadcasted; the dynamic broadcast includes an abnormal situation identified according to the environmental perception information of the non-player character.
9. The method according to claim 8, characterized in that When the environmental perception information of the non-player character about the three-dimensional virtual environment meets the dynamic broadcasting condition, broadcasting the dynamic broadcasting includes at least one of the following: In the case where the non-player character perceives a hostile virtual character, broadcasting a first dynamic broadcast; the first dynamic broadcast includes a position of the hostile virtual character perceived by the non-player character; When the non-player character senses a dangerous situation, a second dynamic broadcast is broadcast; the second dynamic broadcast is used to prompt the dangerous situation; When the non-player character senses a change in the state of the friendly virtual character, a third dynamic broadcast is broadcast; the third dynamic broadcast is used to prompt that the state of the friendly virtual character has changed; When the non-player character finds a new trace in the three-dimensional virtual environment, a fourth dynamic broadcast is broadcast; the fourth dynamic broadcast is used to prompt the new trace.
10. The method according to any one of claims 1 to 7, characterized in that: The feedback information includes a location description of the target entity; the method further includes: The position description is generated according to the position relationship between the master virtual character and the target entity.
11. The method according to claim 10, characterized in that Before generating the position description according to the position relationship between the master virtual character and the target entity, the method further includes: According to the target entity information indicated by the natural language command and the environment perception information, the target entity is queried from the entity set of the three-dimensional virtual environment.
12. The method according to any one of claims 1 to 7, characterized in that: The feedback information is the spatial vocal audio of the non-player character; The reporting of the feedback information of the non-player character includes: Obtaining human voice audio corresponding to the feedback text; Based on the relative position relationship between the non-player character and the master virtual character in the three-dimensional virtual environment, obtaining sound effect parameters corresponding to the non-player character; The human voice audio is adjusted based on the sound effect parameters, and the spatial human voice audio corresponding to the non-player character is generated and broadcasted, wherein the spatial human voice audio is used to express the perceptual effect of the audio generated by the main control virtual character on the non-player character under the relative position relationship.
13. A method for broadcasting a non-player character, characterized in that: The method comprises: Receiving a natural language command for a non-player character sent by a client, the client having control authority over a master virtual character, the non-player character following the master virtual character in the three-dimensional virtual environment; Acquire environmental perception information of at least one of the main control virtual character and the non-player character; Determining, according to the natural language command and the environmental perception information, a feedback text of the non-player character, wherein the feedback text is a non-fixed text generated based on the environmental perception information of the non-player character; Based on the feedback text, relevant information for playing feedback information is sent to the client, where the feedback information is human voice audio corresponding to the feedback text.
14. The method according to claim 13, characterized in that The step of determining the feedback text of the non-player character according to the natural language command and the environmental perception information includes: Calling a large language model to infer the natural language command based on the static environment data of the three-dimensional virtual environment and the environment perception information to obtain an inference result of the natural language command; The feedback text is determined according to the inference result.
15. The method according to claim 14, characterized in that The sending, based on the feedback text, the relevant information for playing the feedback information to the client includes: In a case where the client stores a first offline voice corresponding to the inference result, sending the feedback text corresponding to the first offline voice, or a first voice identifier corresponding to the first offline voice to the client, where the first voice identifier is used to instruct the client to generate the feedback information based on the first offline voice; In the case that the client does not store the offline speech corresponding to the inference result, a text-to-speech service is called to generate the feedback information based on the feedback text; and the feedback information is sent to the client.
16. The method according to any one of claims 13 to 15, characterized in that: The feedback text includes reply feedback text; The step of determining the feedback text of the non-player character according to the natural language command and the environmental perception information includes: In a case where the behavioral intention of the natural language command is an inquiry, determining a reply feedback text of the non-player character according to the environmental perception information; The reply feedback text includes reply content for the natural language command generated according to the environmental perception information of the non-player character about the three-dimensional virtual environment.
17. The method according to claim 16, characterized in that When the behavioral intention of the natural language command is an inquiry, determining the reply feedback text of the non-player character according to the environmental perception information includes at least one of the following: In the case where the natural language command includes an intelligence inquiry about a target entity, based on the perception of the target entity by the non-player character, a first reply feedback text is determined according to the environmental perception information; the first reply feedback text includes the intelligence perception result of the non-player character about the target entity; In the case where the natural language command includes a query about the location of the target entity, based on the non-player character's perception of the target entity, a second reply feedback text is determined according to the environmental perception information, and the second reply feedback text includes a description of the location of the target entity.
18. The method according to any one of claims 13 to 15, characterized in that: The feedback text includes status feedback text; The step of determining the feedback text of the non-player character according to the natural language command and the environmental perception information includes: In a case where the behavioral intention of the natural language command is to execute a task, determining the status feedback text according to the environmental perception information; The status feedback text includes an execution status after the non-player character executes the natural language command according to the environmental perception information of the three-dimensional virtual environment.
19. The method according to claim 18, characterized in that When the behavior intention of the natural language command is to execute a task, determining the status feedback text according to the environmental perception information includes at least one of the following: In a case where the behavioral intention of the natural language command includes searching for a target entity, determining a first state feedback text according to the environmental perception information, the state feedback text including a search result of the non-player character for the target entity; In a case where the behavioral intention of the natural language command includes using a virtual prop, determining a second state feedback text according to the environmental perception information, wherein the second state feedback text is used to indicate that the virtual prop is ready; In a case where the behavioral intention of the task natural language command includes using a virtual prop, determining a third state feedback text according to the environmental perception information, wherein the third state feedback text is used to indicate a result of using the virtual prop; When the behavioral intention of the natural language command includes controlling the movement of the non-player character, determining a fourth state feedback text according to the environmental perception information, wherein the fourth state feedback text is used to indicate a movement result of the movement; In a case where the behavioral intention of the natural language command includes performing an interaction with a target entity, determining a fifth state feedback text according to the environmental perception information, the fifth state feedback text being used to indicate at least one of a search result of the non-player character for the target entity and an interaction result between the non-player character and the target entity; In a case where the behavioral intention of the natural language command includes item interaction, a sixth state feedback text is determined according to the environmental perception information, and the sixth state feedback text is used to indicate an item interaction result of the non-player character performing the item interaction.
20. The method according to claim 18, characterized in that When the behavioral intention of the natural language command is to execute a task, determining the status feedback text according to the environmental perception information includes: In the case where the behavioral intention of the natural language command is to execute a task, calling a large language model to parse the natural language command, and inferring a task execution instruction based on the static environment data of the three-dimensional virtual environment and the environment perception information; Controlling the non-player character to perform the task according to the task execution instruction; The status feedback text is determined according to the execution status of the task performed by the non-player character.
21. The method according to any one of claims 13 to 15, characterized in that: The method further comprises: In the case where the environmental perception information of the non-player character about the three-dimensional virtual environment meets the dynamic broadcast condition, a dynamic broadcast text is determined according to the environmental perception information; the dynamic broadcast text includes an abnormal situation identified according to the environmental perception information of the non-player character; Based on the dynamic broadcast text, relevant information for playing dynamic broadcast information is sent to the client, where the dynamic broadcast information is human voice audio corresponding to the dynamic broadcast text.
22. The method according to claim 21, characterized in that When the environmental perception information of the non-player character about the three-dimensional virtual environment meets the dynamic broadcast condition, determining the dynamic broadcast text according to the environmental perception information includes at least one of the following: In the case where the non-player character perceives a hostile virtual character, generating a first dynamic broadcast text; the first dynamic broadcast text includes a position of the hostile virtual character perceived by the non-player character; When the non-player character senses a dangerous situation, generating a second dynamic broadcast text; the second dynamic broadcast text is used to prompt the dangerous situation; When the non-player character senses a change in the state of the friendly virtual character, a third dynamic broadcast text is generated; the third dynamic broadcast text is used to prompt that the state of the friendly virtual character has changed; When the non-player character finds a new trace in the three-dimensional virtual environment, a fourth dynamic broadcast text is generated; the fourth dynamic broadcast text is used to prompt the new trace.
23. The method according to any one of claims 13 to 15, characterized in that: The feedback text includes a location description of the target entity; the method further includes: The position description is generated according to the position relationship between the master virtual character and the target entity.
24. The method according to claim 23, characterized in that Before generating the position description according to the position relationship between the master virtual character and the target entity, the method further includes: According to the target entity information indicated by the natural language command and the environment perception information, the target entity is queried from the entity set of the three-dimensional virtual environment.
25. The method according to any one of claims 13 to 15, characterized in that: The feedback information is the spatial vocal audio of the non-player character; The method further comprises: Obtaining human voice audio corresponding to the feedback text; Based on the relative position relationship between the non-player character and the master virtual character in the three-dimensional virtual environment, obtaining sound effect parameters corresponding to the non-player character; The human voice audio is adjusted based on the sound effect parameters to generate the spatial human voice audio corresponding to the non-player character, and the spatial human voice audio is used to express the perceptual effect of the audio generated by the main control virtual character on the non-player character in the relative orientation relationship.
26. The method according to claim 25, characterized in that The step of obtaining the sound effect parameters corresponding to the non-player character based on the relative position relationship between the non-player character and the master virtual character in the three-dimensional virtual environment includes: Acquire first position information of the non-player character in the three-dimensional virtual environment; and acquire second position information of the master virtual character in the three-dimensional virtual environment; Determining the relative position relationship between the non-player character and the master virtual character based on the first position information and the second position information; Based on the relative position relationship, the sound effect parameters corresponding to the non-player character are generated.
27. The method according to claim 26, characterized in that The generating the sound effect parameter corresponding to the non-player character based on the relative position relationship includes at least one of the following: Based on the relative position relationship, determining a sound propagation path of the audio generated by the non-player character between the non-player character and the master virtual character; Generating sound effect parameters corresponding to the non-player character based on the sound propagation path; or, Determining a field of view corresponding to the master virtual character based on the orientation of the master virtual character in the three-dimensional virtual environment; When the non-player character is within the field of view, determining the sound effect parameter based on a first parameter; When the non-player character is outside the field of view, determining the sound effect parameter based on a second parameter; wherein the audio volume corresponding to the first parameter is greater than the audio volume corresponding to the second parameter; or, Based on the relative position relationship, determine the distance information between the main control virtual character and the non-player character; based on the distance information, determine the attenuation parameter and the delay parameter corresponding to the audio generated by the non-player character; based on the attenuation parameter and the delay parameter, determine the sound effect parameter; or, Based on the relative orientation relationship between the spatial element and the master virtual character, the surrounding state information of the spatial element to the master virtual character is determined, the three-dimensional virtual environment includes the spatial elements that form a virtual space in the three-dimensional virtual environment, and the master virtual character is in the virtual space; based on the surrounding state information, the space type of the virtual space where the master virtual character is located is determined; the spatial sound effect parameters matching the space type are obtained; and the sound effect parameters are determined based on the spatial audio parameters.
28. The method according to claim 25, characterized in that The obtaining of human voice audio corresponding to the feedback text includes: Acquiring the feedback text and character feature information corresponding to the non-player character, wherein the character feature information is used to describe the attributes of the non-player character, wherein the attributes of the non-player character include at least one of basic attributes and situational performance attributes of the non-player character, wherein the basic attributes are attributes pre-configured for the non-player character, and the situational performance attributes are attributes determined in real time by the non-player character in the situation of the three-dimensional virtual environment; The human voice audio matching the character feature information is generated according to the feedback text.
29. The method according to any one of claims 13 to 15, characterized in that: The non-player character has multiple behavioral capabilities in the virtual environment, and the multiple behavioral capabilities correspond to multiple classification tags; The step of determining the feedback text of the non-player character according to the natural language command and the environmental perception information includes: Obtaining a command text corresponding to the natural language command; Calling at least one hierarchical prediction network to perform intent recognition on the command text to obtain a classification label of the command text's intent to command the non-player character; wherein the hierarchical prediction network includes at least two layers of sub-networks, and the at least two layers of sub-networks are constructed based on a tree structure of the multiple classification labels; Determining, based on the intention indicated by the classification tag, environmental perception information during the process in which the non-player character performs the virtual activity corresponding to the intention; The feedback text is generated based on the environmental perception information.
30. The method according to claim 29, characterized in that The obtaining of the command text corresponding to the natural language command includes: Acquire a plurality of scene hot words corresponding to the three-dimensional virtual environment, wherein the three-dimensional virtual environment is a virtual environment in which at least one of the main control virtual character or the non-player character is located, and the plurality of scene hot words are scene-related words of the three-dimensional virtual environment; The natural language command is converted into the command text based on the multiple scene hot words.
31. A broadcasting device for a non-player character, characterized in that: The device comprises: A display module, for displaying at least one of a main control virtual character and a non-player character in a three-dimensional virtual environment, wherein the non-player character follows the main control virtual character to move in the three-dimensional virtual environment; A receiving module, configured to receive a natural language command directed to the non-player character; The broadcast module is used to broadcast the feedback information of the non-player character in response to the natural language command, wherein the feedback text corresponding to the feedback information is a non-fixed text generated based on the environmental perception information, and the environmental perception information includes the information perceived by at least one of the main virtual character and the non-player character about the three-dimensional virtual environment.
32. A broadcasting device for a non-player character, characterized in that: The device comprises: A receiving module, configured to receive a natural language command sent by a client for a non-player character, wherein the client has control authority over a master virtual character, and the non-player character follows the master virtual character to move in the three-dimensional virtual environment; An acquisition module, used for acquiring environmental perception information of at least one of the main virtual character and the non-player character; a determination module, configured to determine a feedback text of the non-player character according to the natural language command and the environmental perception information, wherein the feedback text is a non-fixed text generated based on the environmental perception information of the non-player character; A sending module is used to send relevant information for playing feedback information to the client based on the feedback text, where the feedback information is human voice audio corresponding to the feedback text.
33. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the broadcasting method of the non-player character as described in any one of claims 1 to 30.
34. A computer-readable storage medium, characterized in that: The readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the broadcasting method of the non-player character as described in any one of claims 1 to 30.
35. A computer program product or a computer program, characterized in that The computer program product or computer program includes computer instructions, which are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes to implement the broadcasting method of a non-player character as described in any one of claims 1 to 30.
Citation Information
Cited By
Broadcasting method and apparatus for non-player character, device, medium, and product
WO2026031852A1
