Voice conversion method and device based on virtual scene, equipment, medium and product
By obtaining scene hot words in virtual scenes to convert natural language commands, the problem of inaccurate virtual character control caused by poor quality of natural language commands is solved, and the accuracy and operation efficiency of command analysis are improved.
Patent Information
- Application Number
- CN202411098483.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-08-09
AI Technical Summary
In the prior art, poor quality of natural language commands (such as fuzzy pronunciation, too fast or too slow speech) makes it difficult for speech recognition systems to accurately recognize, which in turn affects the accuracy of virtual character control.
By obtaining the scene hot words in the virtual scene where the target virtual character is located, converting natural language commands into command analysis results in text form, and using these hot words to constrain the analysis process to improve the accuracy and efficiency of command analysis.
It improves the analysis accuracy of natural language commands, reduces invalid or erroneous commands for non-player characters, provides a more efficient and comfortable operating environment, and improves human-computer interaction efficiency.
Smart Images

Figure CN119049463A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of machine learning, and in particular to a method, device, equipment, medium and product for voice conversion based on a virtual scene. Background Art
[0002] With the improvement of cultural and entertainment living standards, people's life experience and requirements for the virtual world are also getting higher and higher. Virtual scenes are provided in terminal games, virtual reality (VR), augmented reality (AR), etc., and players can control the activities of virtual characters in the virtual scenes manually or by voice.
[0003] In the related art, if a player controls a virtual character by voice, after receiving a natural language command in audio form, a pre-trained speech recognition system will perform text analysis on the natural language command, so as to control the virtual character based on the text content obtained by the analysis; wherein, the model in the speech recognition system is obtained by training with a large number of text vocabularies, and the text content is the content obtained by combining some text vocabularies in the large number of text vocabularies.
[0004] However, natural language commands may have poor quality problems, such as unclear pronunciation, too fast or too slow speaking speed, etc., which may make the speech recognition system unable to accurately recognize natural language commands, resulting in low accuracy of text content. Summary of the invention
[0005] The embodiment of the present application provides a method, device, equipment, medium and product for voice conversion based on a virtual scene, which can make the command analysis result more closely match the first virtual scene and improve the accuracy of the command analysis result. The technical solution is as follows.
[0006] On the one hand, a method for voice conversion based on a virtual scene is provided, the method comprising:
[0007] Acquire a natural language command in voice form, the natural language command being used to instruct a non-player character, wherein a target virtual character is in a first virtual scene, and the target virtual character includes at least one of the non-player character and a master virtual character;
[0008] Acquire a plurality of first scene hot words corresponding to the first virtual scene where the target virtual character is located, wherein the plurality of first scene hot words are scene-related words of the first virtual scene;
[0009] A command analysis result is obtained by converting the natural language command into text based on the multiple first scene hot words.
[0010] On the other hand, a virtual scene-based voice conversion device is provided, the device comprising:
[0011] An acquisition module, configured to acquire a natural language command in a voice form, wherein the natural language command is used to instruct a non-player character, wherein a target virtual character is in a first virtual scene, and wherein the target virtual character includes at least one of the non-player character and a master virtual character;
[0012] The acquisition module is further used to acquire a plurality of first scene hot words corresponding to the first virtual scene where the target virtual character is located, wherein the plurality of first scene hot words are scene-related words of the first virtual scene;
[0013] An analysis module is used to convert the natural language command into a command analysis result in text form based on the multiple first scene hot words.
[0014] In an optional embodiment, the acquisition module is also used to acquire the first scene type corresponding to the first virtual scene in which the target virtual character is located; acquire a hot word set corresponding to the first scene type, and use the words in the hot word set as the multiple first scene hot words, wherein the hot word set is a set of scene-related words collected based on the first scene type; or, acquire the object perception range of the target virtual character in the first virtual scene, wherein the object perception range is the three-dimensional spatial range in which the target virtual character perceives the existence of other scene elements; acquire the multiple first scene hot words based on environmental perception information within the object perception range; or, acquire the first scene type corresponding to the first virtual scene; acquire the hot word set corresponding to the first scene type; acquire the object perception range of the target virtual character in the first virtual scene; and acquire at least two words from the hot word set as the multiple first scene hot words based on the environmental perception information within the object perception range.
[0015] In an optional embodiment, the object perception range includes at least one of the following:
[0016] The visual perception range of the target virtual character;
[0017] The auditory perception range of the target virtual character;
[0018] The olfactory perception range of the target virtual character;
[0019] The target virtual character's perception skill or the range of perception of the perception tool.
[0020] In an optional embodiment, the acquisition module is also used to generate a preset visual cone as the visual perception range, taking the position of the target virtual character as the starting point and the orientation of the target virtual character as the midline of the cone, when the object perception range includes the visual perception range. The preset visual cone is a three-dimensional area that measures the visual perception range.
[0021] In an optional embodiment, the environmental perception information includes virtual elements;
[0022] The acquisition module is also used to determine the virtual element in the first virtual scene that is within the perception range of the object; use the element name of the virtual element as the first scene hot word; or use the action name of the interactive action corresponding to the virtual element as the first scene hot word.
[0023] In an optional embodiment, the first virtual scene is one of a plurality of virtual scenes in a virtual environment;
[0024] In an optional embodiment, the acquisition module is further used to determine the first virtual scene where the target virtual character is located from the multiple virtual scenes based on the position of the target virtual character in the virtual environment.
[0025] In an optional embodiment, the acquisition module is also used to, in response to the target virtual character moving from the first virtual scene to the second virtual scene, acquire a plurality of second scene hot words corresponding to the second virtual scene when the command analysis result is not generated; and convert the natural language command into the command analysis result in text form based on the plurality of second scene hot words.
[0026] In an optional embodiment, the acquisition module is also used to determine the first virtual scene corresponding to the first virtual game based on the first virtual game in which the target virtual character participates; wherein, multiple virtual games correspond to one virtual scene respectively, and the multiple virtual games are simulated battle environments provided for the target virtual character.
[0027] In an optional embodiment, the acquisition module is also used to acquire multiple scene hot words, and the multiple scene hot words correspond to at least two virtual scenes; based on the first virtual scene where the target virtual character is located, the multiple first scene hot words corresponding to the first virtual scene are acquired from the multiple scene hot words.
[0028] In an optional embodiment, the multiple scene hot words correspond to scene identifiers, and the scene identifiers are used to represent virtual scenes when the scene hot words are collected;
[0029] The acquisition module is also used to acquire a plurality of candidate scene hot words with a first scene identifier from the plurality of scene hot words based on the first virtual scene in which the target virtual character is located, wherein the first scene identifier is a scene identifier corresponding to the first virtual scene; and based on the object perception range of the target virtual character in the first virtual scene, select at least two candidate scene hot words from the plurality of candidate scene hot words as the first scene hot words.
[0030] In an optional embodiment, the analysis module is also used to obtain a pre-trained natural language analysis model, the pre-trained natural language analysis model includes an acoustic network, a language network and a preset dictionary, the preset dictionary includes the multiple first scene hot words, other scene hot words and common vocabulary; obtain multiple speech units corresponding to the natural language command through the acoustic network, the speech unit is the basic building block of vocabulary pronunciation; analyze the sequence matching relationship between the multiple speech units and the selected vocabulary in the preset dictionary through the language network to obtain the command analysis result in text form, the selected vocabulary at least includes the multiple first scene hot words and the common vocabulary.
[0031] In an optional embodiment, the selected vocabulary also includes the other scene hot words;
[0032] Among them, the vocabulary in the preset dictionary includes analysis weights, the first analysis weights of the multiple first-scene hot words are higher than the second analysis weights of the other scene hot words, and the analysis weight refers to the degree of attention paid to the vocabulary when it participates in sequence matching.
[0033] In an optional embodiment, the analysis module is further used to analyze the matching relationship between the multiple speech units and the selected words in the preset dictionary, and obtain multiple candidate vocabulary sequences, wherein the multiple candidate vocabulary sequences include at least one word from the multiple selected words, and the candidate vocabulary sequences are vocabulary sequences obtained based on the change relationship between the multiple speech units; analyze the sequence semantics of the multiple candidate vocabulary sequences through the language network, and obtain at least one candidate vocabulary sequence from the multiple candidate vocabulary sequences as the command analysis result in text form.
[0034] In an optional embodiment, the analysis module is also used to determine, based on the first virtual scene in which the target virtual character is located, a first language subnetwork corresponding to the first virtual scene from at least two language subnetworks; analyze the sequence semantics of the multiple candidate vocabulary sequences through the first language subnetwork, and obtain at least one candidate vocabulary sequence from the multiple candidate vocabulary sequences as the command analysis result in text form.
[0035] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the virtual scene-based speech conversion method as described in any of the above-mentioned embodiments of the present application.
[0036] On the other hand, a computer-readable storage medium is provided, in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the virtual scene-based speech conversion method as described in any of the above-mentioned embodiments of the present application.
[0037] On the other hand, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the virtual scene-based voice conversion method described in any of the above embodiments.
[0038] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:
[0039] When directing a non-player character through a natural language command, a plurality of first scene hot words are obtained according to the first virtual scene where the target virtual character is located, so as to analyze the natural language command under the constraint of the first scene hot words, so as to direct the non-player character to act according to the natural language command through the command analysis result that is easier for the device to understand. Taking the first virtual scene where the target virtual character is located as a condition for analyzing the natural language command, more targeted text information can be provided for the analysis process through the first scene hot words, reducing the analysis ambiguity caused by the poor quality of the natural language command, which is not only conducive to a more accurate understanding of the natural language command, but also enables the command analysis result to be more matched with the first virtual scene, improves the accuracy of the command analysis result, greatly avoids invalid or erroneous commands to non-player characters, and can provide players with a more efficient and comfortable operating environment, improving the efficiency of human-computer interaction; a small number of first scene hot words can also reduce the amount of data processing during command analysis, and improve the efficiency of command analysis. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0041] Figure 1 is a flow chart of using a voice command in a natural language form (natural language command) to command a non-player character in a virtual environment provided by an exemplary embodiment of the present application;
[0042] Figure 2 is a structural block diagram of a conversion system provided by an exemplary embodiment of the present application;
[0043] Figure 3 is a flow chart of a virtual scene-based voice conversion method provided by an exemplary embodiment of the present application;
[0044] Figure 4 is a flow chart of a virtual scene-based voice conversion method provided by another exemplary embodiment of the present application;
[0045] Figure 5 is a schematic diagram of an interface for determining a perception range of an object provided by an exemplary embodiment of the present application;
[0046] Figure 6 is a flowchart of a voice conversion method based on a virtual scene provided by another exemplary embodiment of the present application;
[0047] Figure 7 is a flow chart of analyzing a natural language command by a natural language analysis model provided by an exemplary embodiment of the present application;
[0048] Figure 8 is a technical flow chart of a virtual scene-based voice conversion method provided by an exemplary embodiment of the present application;
[0049] Fig. 9 It is a schematic diagram of an interface of a virtual scene-based voice conversion method provided by an exemplary embodiment of the present application;
[0050] Fig.10 is an overall flow chart of a virtual scene-based voice conversion method provided by an exemplary embodiment of the present application;
[0051] Fig.11 is a flow chart of a method for controlling a virtual character provided by an exemplary embodiment of the present application;
[0052] Fig.12 It is a structural block diagram of a speech conversion device based on a virtual scene provided by an exemplary embodiment of the present application;
[0053] Fig.13 It is a schematic diagram of the structure of a server provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION
[0054] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.
[0055] First, a brief introduction is given to the terms involved in the embodiments of the present application.
[0056] Virtual scene: a virtual scene displayed (or provided) when an application is running on a terminal. The virtual scene can be a simulation of a real scene, a semi-simulation and semi-fictitious scene, or a purely fictitious scene. The virtual scene can be any one of a two-dimensional virtual scene, a 2.5-dimensional virtual scene, and a three-dimensional virtual scene, and this application does not limit this. The following embodiments are illustrated by taking a three-dimensional virtual scene as an example.
[0057] Virtual model: a model used to simulate a real scene in a virtual scene. Exemplarily, a virtual model occupies a certain volume in a virtual scene. Exemplarily, virtual models include: terrain models, building models, animal and plant models, virtual prop models, virtual vehicle models, and virtual object models. For example, terrain models include: ground, mountains, water, stones, steps, etc.; building models include: houses, walls, containers, and fixed facilities inside buildings: tables, chairs, cabinets, beds, etc.; animal and plant models include: trees, flowers, birds, etc.; virtual prop models include: virtual attack props, medicine boxes, airdrops, etc.; virtual vehicle models include: cars, ships, helicopters, etc.; virtual object models include: people, animals, cartoon characters, etc.
[0058] Virtual character / virtual object: refers to an active object in a virtual scene. The active object can be a virtual object, a virtual animal, an animated character, etc., such as a person, an animal, a plant, an oil drum, a wall, a stone, etc. displayed in a three-dimensional virtual scene. Optionally, the virtual object is a three-dimensional stereo model created based on animation skeleton technology. Each virtual object has its own shape and volume in a three-dimensional virtual scene and occupies a part of the space in the three-dimensional virtual scene.
[0059] In the related art, if a player controls a virtual character by voice, after receiving a natural language command in the form of audio, a pre-trained voice recognition system will perform text analysis on the natural language command, so as to control the virtual character based on the text content obtained by the analysis; wherein, the model in the voice recognition system is obtained by training a large number of text words, and the text content is the content obtained by combining some text words in a large number of text words. However, natural language commands may have poor quality problems, such as unclear pronunciation, too fast or too slow speech speed, etc., which may make the voice recognition system unable to accurately recognize natural language commands, resulting in a low accuracy rate of text content.
[0060] In an embodiment of the present application, a method for voice conversion based on a virtual scene is introduced, which can use the first virtual scene in which the target virtual character is located as a condition for analyzing natural language commands, and provide more targeted text information for the analysis process through the first scene hot words, so that the command analysis results in text form are more matched with the first virtual scene, thereby improving the accuracy of the command analysis results, and thus improving the accuracy of the command to non-player characters. The method for voice conversion based on a virtual scene provided in an embodiment of the present application can be applied to virtual scenes such as terminal game scenes, virtual reality scenes, augmented reality scenes, virtual tourism scenes, virtual meeting scenes, virtual shopping scenes, virtual fitness scenes, etc. that require a voice conversion process, and the embodiment of the present application does not limit this.
[0061] In an optional embodiment, the application of the virtual scene-based voice conversion method to a terminal game is used as an example for description.
[0062] Schematically, a game application is installed in the terminal. When the game application is running, a virtual scene in which players play a virtual game is displayed. Non-player characters (NPC) in the virtual scene can be commanded by voice control. Taking the target virtual character in the first virtual scene as an example, the target virtual character includes at least one of a non-player character and a main virtual character. That is, when commanding a non-player character, the first virtual scene can be determined based on the non-player character, the first virtual scene can be determined based on the main virtual character, and the first virtual scene can be determined based on the location where the non-player character and the main virtual character are located. Optionally, multiple first scene hot words corresponding to the first virtual scene where the target virtual character is located are obtained. The first scene hot words are scene-related words of the first virtual scene. For example, the main virtual character manually operated by the player through the terminal screen, the first virtual scene is a virtual scene determined based on the main virtual character, and multiple first scene hot words include element names of virtual elements in the first virtual scene. For example, the first virtual scene is a virtual kitchen, and multiple first scene hot words include virtual kitchen knives, virtual plates, etc. Optionally, a command analysis result is converted into text form based on multiple first-scene hot words, that is, a speech-to-text process is constrained by multiple first-scene hot words. This facilitates converting natural language commands in speech form into an instruction analysis text in text form that is more closely aligned with the first virtual scene. For example, an instruction analysis text in text form obtained based on the constraints of multiple first-scene hot words is "use a kitchen knife to defend", which avoids being recognized as "use a cutting knife to defend", thereby improving the accuracy of the instruction analysis text and facilitating better command of non-player characters to use virtual kitchen knives to defend against attacks from other virtual characters, thereby improving the efficiency and accuracy of character command.
[0063] In an optional embodiment, the application of the virtual scene-based speech conversion method to a virtual reality scene is taken as an example for description.
[0064] Schematically, a virtual scene received from a computer device can be transmitted and presented through a VR head-mounted display. Players observe the virtual scene by wearing a VR head-mounted display. Players can also move to other locations in the virtual scene to explore the virtual scene by moving, adjusting changes in body parts, etc. Taking the case where the player himself is the main virtual character when wearing a VR head-mounted display as an example, there can also be virtual characters and non-player characters controlled and / or directed by other players in the virtual scene presented by the VR head-mounted display. Optionally, the player can command the non-player character through natural language commands in the form of voice; the VR head-mounted display can receive natural language commands in the form of voice through a deployed microphone array, autonomously analyze the natural language commands, or send the natural language commands to a computer device connected thereto for analysis. The natural language command is used to command the non-player character, and the target virtual character is in the first virtual scene, and the target virtual character includes at least one of the non-player character and the main virtual character. For example: the first virtual scene is a virtual scene where the main virtual character and the non-player character to be commanded are located together. Optionally, multiple first scene hot words corresponding to the first virtual scene where the target virtual character is located are obtained. For example, the target virtual character is the main control virtual character. When the player wears a VR head mounted display to explore the first virtual scene as a virtual battlefield, it is necessary to conduct a virtual battle with the enemy virtual character A. The multiple first scene hot words obtained include virtual sword, attack, attack enemy virtual character A, etc. Optionally, the natural language command is converted into a textual command analysis result based on multiple first scene hot words, and the command analysis result obtained is "give me an attack", avoiding recognition as "give me supplies", improving the accuracy of the command analysis text, and also facilitating players to better command non-player characters to use effective attacks on enemy virtual characters, improving the accuracy of character command, enriching the fun of virtual reality scenes, and enabling players to get a more realistic and in-depth virtual experience.
[0065] It should be noted that before collecting relevant data of users and during the process of collecting relevant data of users, this application can display a prompt interface, pop-up window or output voice prompt information, and the prompt interface, pop-up window or voice prompt information is used to prompt the user that its relevant data is currently being collected, so that this application only starts to execute the relevant steps of obtaining relevant data of users after obtaining the confirmation operation issued by the user to the prompt interface or pop-up window, otherwise (that is, when the confirmation operation issued by the user to the prompt interface or pop-up window is not obtained), the relevant steps of obtaining relevant data of users are terminated, that is, the relevant data of users are not obtained. In other words, all user data collected by this application are collected with the consent and authorization of the user, and the collection, use and processing of relevant user data need to comply with the relevant laws, regulations and standards of the relevant regions.
[0066] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant regions. For example, the natural language commands, scene hot words and other contents involved in this application are all obtained with full authorization.
[0067] The embodiments of the present application provide a solution for users to use natural language voice commands in the form of natural language speech to command and control non-player characters (NPC) in a virtual environment. In this solution, in addition to the main virtual character controlled by the user, the user usually has one or more NPCs as teammates. The user can use a relatively casual and non-mechanical human conversation method to command the NPC to make the desired feedback to the scene objects in the virtual environment, thereby commanding the NPC to work with the main virtual character to complete the task. Exemplary reference Figure 1 , the user controls the main virtual character 10 to play the game, and the main virtual character 10 has an NPC teammate 20. There is a car 30 in the field of vision of the main virtual character 10. The user speaks a natural voice command in the form of natural language: "No. 2, you go to ambush behind the car 30", and then the NPC teammate 20 will move to the back of the car 30 to ambush. It should be noted that Figure 1 There may be multiple cars in the scene shown, and the NPC teammate 20 will accurately understand that the car the user is talking about is the car 30 located within the user's field of vision. That is, the NPC teammate 20 has a relatively intelligent natural language understanding ability.
[0068] On the one hand, compared with the traditional technology of using mechanical fixed instructions to command NPC, the embodiment of the present application uses relatively random natural language commands to command NPC, which can provide users with more natural, more complex and more flexible language command capabilities.
[0069] On the other hand, the intelligence of the natural language understanding capability provided by the embodiment of the present application is reflected in that when understanding natural language commands, NPC teammates not only consider the literal information of natural language commands. NPC teammates will also combine the main virtual character and / or the NPC's own environmental perception information in the virtual environment to assist in understanding the semantics in natural language commands. That is, NPC not only considers the information of the natural language command mode, but also considers the perception information of other modes such as vision, hearing, radar, etc. to assist in the understanding and execution of natural language commands. Since natural language commands in the form of voice are spoken commands rather than written commands, there may be multiple candidate understanding methods or unclear places when considering the literal information of natural language commands alone. NPC teammates can determine a reasonable understanding method from multiple candidate understanding methods, or eliminate doubts about unclear places, by combining the main virtual character and / or the NPC's own environmental perception information in the virtual environment, thereby achieving a more intelligent natural language understanding capability. The above-mentioned environmental perception information includes, but is not limited to, at least one of the following:
[0070] Visual information perceived within the field of vision of the controlling virtual character;
[0071] Auditory information perceived within the hearing range of the controlling virtual character;
[0072] The information perceived by the master virtual character’s perception skills or perception tools;
[0073] Visual information perceived by the NPC within its field of vision;
[0074] Auditory information perceived within the NPC's hearing range;
[0075] Information perceived by the NPC's perception skills or perception tools.
[0076] Combined with reference Figure 1 The program includes at least one of the following five stages:
[0077] Phase 1: Preprocessing of spatial data;
[0078] The server will pre-process the scene objects in the virtual scene and construct the spatial data of the virtual scene.
[0079] The spatial data of the virtual scene includes the matching labels of the scene objects, visual labels and the spatial positions of the scene objects. In some examples, the spatial data also includes the appearance images of the scene objects. The scene objects are any objects that appear in the virtual scene, such as Figure 1 Cars, walls, boxes, etc.
[0080] First, the pre-acquisition process of matching label visual labels for scene objects is introduced. First, the attribute information of the scene objects in the virtual scene is obtained, such as at least one of the name, size and other information of the scene objects; and the appearance image of the scene objects is obtained. The appearance image includes an image obtained by observing the scene objects from at least two perspectives. Observing the scene objects from multiple perspectives can ensure that the appearance image carries comprehensive appearance information of the scene objects.
[0081] The server calls the multimodal model to predict the attribute information and appearance image of the scene object. Based on the multimodal model's ability to process text dimension and image dimension information, it extracts hidden features from the attribute information of the text dimension and the appearance image of the image dimension, and decodes the hidden features to predict the visual object label of the scene object. The visual object label is used to describe the scene object in at least one dimension in a textual manner; for example, the scene object is described in dimensions such as material, transparency, color, and shape. The natural language model is called to perform label optimization on the visual object label to obtain the matching label visual label of the scene object; the natural language model has the ability to generate text, and rewrites the visual object label input into the natural language model in a manner that conforms to the spoken expression of natural language, realizes label optimization of the visual object label, and obtains the optimized matching label visual label of the scene object. The optimized visual label conforms to the spoken expression of natural language, and describes the scene object in at least one dimension in a manner close to the spoken expression.
[0082] For example, when describing a virtual bed in a virtual scene in spoken language, people usually pay attention to the color, material, and placement of the virtual bed, but usually neglect to pay attention to the craftsmanship used for the headboard and the side panels of the bed, such as carving, embossing, etc. The purpose of performing pseudo-spoken rewriting on the visual labels of scene objects is to obtain matching labels that are closer to spoken expressions; for example, using multiple words with the same semantics to enrich the labels of dimensions such as color, material, and placement that are paid attention to in spoken expressions, and deleting the labels of the surface feature dimensions of the headboard and the side panels of the bed that are neglected in spoken expressions.
[0083] Next, the spatial position of the scene object is introduced. The spatial position includes at least one of the following of the scene object: coordinate position (Location), such as the coordinate information of the center point or preset point of the scene object in the virtual scene; orientation (Rotation), such as the direction the front of the scene object faces in the virtual scene; bounding box (Bounding Box), used to indicate the size of the scene object in the virtual scene; cover points (Cover points), used to indicate the recommended virtual character position points when the virtual character approaches the scene object, so that the scene object can cover the virtual character.
[0084] Next, the process of pre-acquiring the appearance image of the scene object is introduced. The appearance image is acquired in the process of predicting the matching label visual label of the scene object, and the appearance image includes images obtained by observing the scene object from at least two perspectives.
[0085] Optionally, the spatial data of each scene object in the entire virtual environment may be pre-acquired and organized into a data set for use in a subsequent query process.
[0086] Phase 2: Speech recognition and intent recognition;
[0087] When the user controls the main virtual character, the terminal receives the natural language command input by the user in the form of voice; the server performs voice recognition and intent recognition on the natural language command to analyze the user's input instruction;
[0088] Speech recognition is to perform text conversion on the natural language commands input by the user through voice, obtain the instruction text corresponding to the natural language command, and convert the speech information into text information. Call the encoding network to perform feature encoding on the instruction text to obtain the feature representation of the natural language command in the latent space; call the text segmentation network to perform segmentation on the feature representation to obtain multiple clauses in the instruction text, and then perform intent recognition on each of the multiple clauses. In the case that the natural language command input by the user through voice is a long sentence, the segmentation of the instruction text corresponding to the natural language command can be achieved to meet the needs of analyzing the user's input instructions in the long sentence scenario.
[0089] The following introduces the intent recognition of each clause; the user's input instruction is obtained through intent recognition, and the user's input instruction includes an entity, a semantic type, a subject type, and an intent type in at least one clause.
[0090] On the one hand, the Conditional Random Field (CRF) method is used to identify entities in clauses, such as identifying entities in clauses as scene objects such as buildings, virtual items, and virtual vegetation in virtual scenes. The entities in the clauses are used to indicate the scene objects targeted by the virtual activities. For example, if the entity in the clause is a virtual building, and the clause carries the semantics of instructing the NPC to launch a virtual attack, then the virtual attack is executed against the virtual building, that is, a virtual attack is launched against the virtual building. On the other hand, the prediction network is called to predict the semantic type, subject type, and intention type in the clause respectively. Among them, the semantic type includes whether to instruct the NPC to launch a virtual attack; the subject type is used to indicate that the subject of the clause is one or more NPCs, or a master virtual character controlled by the user; the intention type is the expected virtual activity indicated by the clause, such as moving, launching a virtual attack, etc.
[0091] Phase 3: Query of spatial data;
[0092] Taking the user's input as input, and the spatial data of the virtual scene and the runtime information corresponding to the master virtual character as reference, the AI model is used to determine the target entity in the virtual space and the control instructions for the NPC;
[0093] For the entity in the input instruction, it is necessary to determine the spatial position of the entity in the virtual environment. For example, the text input instruction is first parsed to parse the entity in the text instruction. In the case where there are multiple candidate entities in the virtual environment that match the entity in the input instruction, the target entity that matches the input instruction can be uniquely determined from the multiple candidate entities in combination with the environmental perception information of the main virtual character and / or NPC.
[0094] For example, the environment perception information is the field of view information of the main virtual character, and then the spatial data of the scene in the virtual environment is used as the query range, and the real-time position and orientation of the main virtual character is used as the query reference condition to query the spatial position of the target entity corresponding to the input instruction in the virtual environment, so as to subsequently control the NPC to move to the vicinity of the target entity position, or to perform the behavior indicated by the input instruction on the target entity position.
[0095] However, since the main virtual character and / or NPC may have multiple environmental perception information, multiple environmental perception information can be integrated to assist the determination process of the above-mentioned target entity, or multiple environmental perception information can be used according to priority to assist the determination process of the above-mentioned target entity, and there is no limitation on this.
[0096] Stage 4: NPC voice feedback;
[0097] NPC will give voice feedback to the user. The types of voice feedback include: at least one of instant feedback, command execution feedback, and dynamic feedback. Among them, instant feedback means that after the user issues a natural language command to the NPC, the NPC immediately gives voice feedback on the natural language command, such as "received", "start execution", "OK", etc.; execution command feedback means that after the NPC executes the control command indicated by the intention of the natural language command, the NPC gives voice feedback on the execution result of the control command, such as "successful execution", "failed to execute successfully", etc.; dynamic feedback means that when the user does not issue a natural language command to the NPC, the NPC gives voice feedback based on real-time environmental perception.
[0098] The above-mentioned voice feedback is obtained by converting text content into speech. The text content is inferred by a large language model combined with the above-mentioned spatial data and dynamic runtime information environment perception information. Using a large language model to dynamically generate the text content of voice feedback can, on the one hand, break through the monotonous, mechanical, and easily repetitive voice feedback of fixed templated voice feedback; on the other hand, it does not need to save too many pre-made template voices in advance, which can avoid the problem of too large a total data volume on the client.
[0099] Stage 5: NPC behavior control;
[0100] Taking the user's input as input, and the spatial data of the virtual scene and the runtime information corresponding to the master virtual character as reference, the AI model is used to determine the target entity in the virtual space and the control instructions for the NPC;
[0101] The above natural language commands usually also include control instructions, which are instructions that can be used for the NPC's behavior tree to execute; the control instruction indicates that the NPC needs to perform a subsequent action, which can be a single action or a sequence of actions. In the case where the control instruction is a multi-action sequence instruction, the multi-action sequence instruction is used to instruct the NPC to perform an action sequence, and the data structure storing the multi-action sequence instruction is in the form of a list, and the instructions to be executed in the sequence instruction are cached. When the NPC's behavior tree executes the current instruction, the cache is called back, and the instructions to be executed in the cache are continued to be sent for execution.
[0102] Hot word update function:
[0103] For phase 2, this solution also introduces a hot word system to help improve the accuracy of sound recognition.
[0104] The hot word system is used to provide scene hot words related to the scene involved in the natural language command during the speech recognition process, so that when a natural language command input by voice is received, the hot word library can be preferentially searched for words as the speech recognition result command analysis result based on the currently running virtual scene, so that the command analysis result and the speech recognition result are more matched with the virtual scene.
[0105] The virtual scene includes at least one of a plurality of scene types, such as a virtual battle scene type, a virtual transaction scene type, a virtual office scene type, a virtual kitchen scene type, and the like. Frequently used words in a plurality of virtual scene types, specific virtual elements existing in a virtual scene type, etc. (such as virtual buildings and virtual character objects that only exist in a specific scene type, etc.) are pre-analyzed as scene hot words, and a scene hot word library composed of a plurality of scene hot words is obtained, and the plurality of scene hot words correspond to virtual scene types respectively. In addition, different scene language models are pre-trained for different virtual scene types, such as a scene language model in a battle scene that pays more attention to words related to battle, and a scene language model in a transaction scene that pays more attention to words related to transaction; the scene language model can perform a more targeted analysis of natural language commands received in a virtual scene type.
[0106] For example: when a natural language command is received during the process of a user controlling a main non-player controlled virtual object character, based on the game state, object position, and current task completion status, it is determined that the first virtual scene in which the target virtual character (at least one of the main virtual character and the non-player character) is currently located is a virtual battle scene, the target scene type is a battle scene type, and a battle scene language model corresponding to the virtual battle scene type is obtained, and based on the field of view of the main control target virtual object character, the target first scene hot words under the virtual battle scene type are obtained, including: "truck", "virtual grass", "virtual house", "enter", "open truck", "defense", "desert", "attack", "oak tree", "stable", etc., "enemy A", "attack", etc. within the field of view, and the natural language command is decoded and processed by a pre-trained natural language analysis model (including an acoustic model network, a target scene language network model, a preset dictionary, etc.) decoding network under the constraint of the first target scene hot words, and the output text content is "give me attack", rather than the result obtained by voice recognition under normal circumstances-"give me supply", thereby improving the accuracy of the user's input instructions, and outputting the command analysis result to accurately control the non-player character. The command analysis result is illustrated as follows.
[0107] (1) The command analysis result is "No. 1, go to the front and defend." This command analysis result can specifically control No. 1 to move forward and take a defensive posture to prevent the enemy virtual character from attacking the main virtual character first. If there is no constraint of the hot words in the first scene, the natural language command is easily recognized as "No. 1, go to the front and counterattack and make a sound," thereby affecting the player's virtual combat plan.
[0108] (2) The command analysis result is “No. 2, go explore the desert.” This command analysis result can specifically control No. 2 to move to the desert to explore whether there are virtual elements such as enemy virtual characters or virtual treasure chests. If there is no constraint of the hot words in the first scene, the natural language command is easily recognized as “No. 2, go explore the mountains,” causing No. 2 to move to the wrong location, resulting in low human-computer interaction efficiency.
[0109] (3) The command analysis result is “No. 1, No. 2, attack me”. This command analysis result can be used to specifically control No. 1 and No. 2 to carry out virtual attacks on nearby attackable virtual characters. If there is no constraint of the hot words in the first scene, the natural language command is easily recognized as “No. 1, No. 2, supply me”, which causes No. 1 and No. 2 to make incorrect behaviors that do not meet the player’s expectations, and they can neither protect the main virtual character nor easily allow the enemy virtual character to win the virtual game.
[0110] (4) The command analysis result is “No. 1, go find a nearby oak tree”. This command analysis result can be used to specifically control No. 1 to search for oak trees in the nearby area so that the game task can be completed or an oak tree that can be used to avoid attacks can be found. If there is no constraint of the hot words in the first scene, the natural language command is easily recognized as “No. 1, go find a nearby project book”, which causes No. 1 to search nearby for virtual elements that do not meet the player’s expectations and fail to meet the player’s virtual combat needs.
[0111] (5) The command analysis result is “No. 1, No. 2, go to the stable over there.” This command analysis result can specifically control No. 1 and No. 2 to search for a nearby stable and move to the location of the stable. If there is no constraint of the hot words in the first scene, the natural language command is easily recognized as “No. 1, No. 2, go over there, it will be ready soon.” As a result, No. 1 and No. 2 mistakenly believe that the main virtual character currently wants to complete the virtual game by themselves, and therefore cannot provide good game assistance to the main virtual character.
[0112] Ambient sound function:
[0113] In addition, the solution also provides a spatial audio enhancement solution for the ambient sound effects of the entire virtual scene.
[0114] The audio played by the terminal includes environmental audio and NPC audio. Environmental audio is audio generated based on the scene characteristics of the virtual scene where the main virtual object is currently located, so that the user can feel immersive; NPC audio is audio generated based on the characteristics of the NPC, so that the user can intuitively feel the character's emotions and physical state through the sound heard.
[0115] Environmental audio: Identify the scene elements in the virtual scene where the master virtual object is located, generate appropriate element sound effects in real time based on the scene elements / select appropriate element sound effects from the sound effect library, and synthesize the element sound effects to obtain environmental audio. For example, if the virtual scene is a forest at night, the scene elements include trees, owls, insects, etc., and the environmental audio includes the rustling of leaves, the calls of owls, the chirping of insects, etc.
[0116] NPC audio: Determine the NPCs included in the virtual scene, and generate corresponding character voices based on the NPC's character type, current emotion, and current behavior. For example, if the NPC is a middle-aged man running, the generated character voice is a deep male voice with a running panting sound effect.
[0117] When the terminal plays the ambient audio and / or NPC audio to the user, the terminal performs audio enhancement processing on the ambient audio and / or NPC audio to improve the realism of the audio.
[0118] In an optional embodiment, the conversion system involved in the embodiment of the present application is described. The virtual scene-based voice conversion method provided in the embodiment of the present application can be implemented by a terminal alone, or by a server, or by a terminal and a server through data interaction, which is not limited in the embodiment of the present application. Optionally, the method of voice conversion based on a virtual scene is described by taking the interaction between a terminal and a server as an example.
[0119] For illustration, please refer to Figure 2 The conversion system involves a terminal 210 and a server 220 , and the terminal 210 and the server 220 are connected via a communication network 230 .
[0120] In some embodiments, a game application is installed in the terminal 210. A virtual scene will be displayed during the operation of the game application, and the player can control the virtual character in the virtual scene through manual control or voice control.
[0121] Optionally, taking the example of a player commanding a non-player character by voice, the terminal 210 is deployed with a microphone array, which can obtain natural language commands in the form of voice, and the natural language commands are used to command the non-player characters. The non-player characters are NPC characters used to assist players in virtual games, such as: virtual soldiers, virtual heroes, virtual monsters, virtual mounts, etc.
[0122] The target virtual character is in the first virtual scene, and the target virtual character includes at least one of a non-player character and a master virtual character.
[0123] Indicatively, the main virtual character is also a virtual character controlled by the player. Optionally, the main virtual character is a virtual character mainly controlled by the player, such as the main virtual character is a virtual character manually controlled by the player, and the non-player character is a virtual character controlled by the player in the form of voice; or, the main virtual character is a virtual character selected by the player, and the non-player character is a virtual character configured by the system by default; or, the main virtual character and the non-player character are both virtual characters selected by the player, wherein the main virtual character is the virtual character with the highest priority, and the non-player character is a virtual character with a lower priority, etc.
[0124] Optionally, the virtual scene where the non-player characters are located is used as the first virtual scene; or the virtual scene where the controlling virtual character is located is used as the first virtual scene; or the virtual scene where the controlling virtual character and the non-player characters are located together is used as the first virtual scene, etc. For example, the virtual scene where the players participate in the virtual game can also be called a virtual environment. The virtual environment is large and includes multiple virtual scenes. The non-player characters and the controlling virtual character may be in the same virtual scene or in different virtual scenes, which is not limited here.
[0125] In some embodiments, when the terminal 210 analyzes the natural language command by itself, the terminal 210 obtains multiple first scene hot words corresponding to the first virtual scene where the target virtual character is located after receiving the natural language command; when the terminal 210 analyzes the natural language command with the help of the server 220, the terminal 210 sends the natural language command in voice form to the server 220 through the communication network 230, thereby obtaining multiple first scene hot words corresponding to the first virtual scene where the target virtual character is located through the server 220.
[0126] Among them, the multiple first scene hot words are scene-related words of the first virtual scene.
[0127] Illustratively, the first scene hot words are words that describe the first virtual scene, or the first virtual scene is a word that describes the virtual elements in the first virtual scene, or the first scene hot words are words that describe the interaction with the virtual elements, etc. For example: the first virtual scene is a virtual battle scene, and the first scene hot words include attack, defense, virtual sword, virtual bow and arrow, etc.; or the first virtual scene is a virtual kitchen, and the first scene hot words are virtual kitchen knife, virtual scissors, virtual plate, etc.
[0128] In some embodiments, taking the analysis of natural language commands by the server 220 as an example, the server 220 converts the natural language commands into command analysis results in text form based on a plurality of first scenario hot words.
[0129] Illustratively, the server 220 uses a plurality of first-scene hot words as constraints during command analysis, and converts natural language commands into command analysis results in text form through speech-to-text technology; due to the constraints of the first-scene hot words, the first-scene hot words are preferentially used as the vocabulary required for text conversion during the command analysis process, thereby establishing a strong correlation between the command analysis results and the first virtual scene, making the command analysis results more in line with the first virtual scene, and improving the accuracy of the command analysis results.
[0130] Optionally, the server 220 commands the non-player characters to move more accurately based on the command analysis results in text form, and generates activity animation data for commanding the character's activities and sends it to the terminal 210 via the communication network 230, so that the activity animation corresponding to the activity animation data can be rendered and displayed on the screen interface of the terminal 210.
[0131] Optionally, the server 220 sends the command analysis results to the terminal 210 via the communication network 230. The terminal 210 can command the non-player characters to move more accurately based on the command analysis results in text form, and render the non-player characters' activity animations on the screen interface of the terminal 210, thereby improving the accuracy and efficiency of the character commands.
[0132] It is worth noting that the above-mentioned terminals include but are not limited to mobile terminals such as mobile phones, tablet computers, portable laptops, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, etc., and can also be implemented as desktop computers, etc.; the above-mentioned servers can be independent physical servers, or they can be server clusters or distributed systems composed of multiple physical servers, or they can also be cloud servers.
[0133] In combination with the above-mentioned noun introduction and application scenarios, the virtual scene-based voice conversion method provided by the present application is described, and the method is applied to a server as an example. Figure 3 As shown, the method includes the following steps 310 to 330.
[0134] Step 310, obtaining a natural language command in speech form.
[0135] Speech is a form of expression in the form of speech or sound, that is, information or intention is conveyed through speech or sound. In the field of computer science, especially in human-computer interaction and natural language processing, speech usually refers to the way in which users use spoken language to input information.
[0136] Illustratively, natural language commands are usually implemented as audio data as instructions in the form of voice. For example, when a player is operating a terminal, the terminal automatically collects the voice based on a microphone array and obtains audio data as a natural language command in the form of voice; or, when a player is operating a terminal, the terminal triggers an audio collection control and then makes a voice, and the terminal receives the voice and obtains audio data as a natural language command in the form of voice, etc.
[0137] Among them, natural language commands are used to command non-player characters.
[0138] Indicatively, the non-player characters have autonomous action capabilities and can move on their own in the virtual scene. Optionally, when the non-player characters are not commanded by the player (e.g., no natural language commands are received), the non-player characters move in the virtual scene according to a preset trajectory, or the non-player characters attack enemy virtual characters in the virtual scene according to preset game logic, assist the main control virtual character in performing game tasks, etc.; when the non-player characters are commanded by the player (e.g., natural language commands are received), the non-player characters will carry out targeted activities in the virtual scene according to the natural language commands to more purposefully assist the main control virtual character.
[0139] Illustratively, the non-player character is a virtual character displayed in a game, or the non-player character is a virtual character displayed in other virtual environments such as a virtual reality scene, an augmented display scene, etc.
[0140] Optionally, the non-player characters are virtual characters commanded by the player through voice. The non-player characters include at least one of multiple types of virtual characters such as virtual soldiers, virtual heroes, virtual monsters, virtual mounts, etc. Based on the natural language commands issued by the player in the form of voice during the game, the non-player characters in the virtual environment can be commanded to act according to the natural language commands.
[0141] In some embodiments, the process of commanding a non-player character is a process of unlocking a game account after reaching a preset level, and the game account is an account logged in by the player.
[0142] For example: the preset level is level 10. When the game account has not reached the preset level, the non-player character is an uncontrollable virtual character, that is, the player cannot command the non-player character by voice; when the game account reaches the preset level, the non-player character is converted into a virtual character controllable by the player, that is, the player can currently command the non-player character by voice, etc.
[0143] In some embodiments, the non-player characters are virtual characters configured by default, such as virtual soldier A and virtual soldier B configured by default as non-player characters that can be commanded by the player; or, the non-player characters are virtual characters selected by the player, such as providing multiple candidate control characters with different personalities and / or autonomous behavior capabilities, where the personalities can represent the attack strength of the candidate control characters in the virtual game (e.g., the attack strength of the candidate control characters with violent personalities is greater, while the attack strength of the candidate control characters with mild personalities is smaller, etc.); the autonomous behavior capabilities represent the skills of the candidate control characters in the virtual game, such as the candidate control character 1 can use amplification A to attack the enemy virtual characters in the circular area, and the candidate control character 2 can use amplification B to defend against the attacks of nearby enemy virtual characters, etc. The player can select at least one candidate control character from the multiple candidate control characters as a non-player character that can be commanded in the game, etc.
[0144] The target virtual character is in the first virtual scene, and the target virtual character includes at least one of a non-player character and a master virtual character.
[0145] In a schematic diagram, the main virtual character is a virtual character commanded by the player, and the non-player character is a virtual character that assists the main virtual character in a virtual game. Usually, a virtual game includes a main virtual character and at least one non-player character. There is a certain difference between the main virtual character and the non-player character.
[0146] Optionally, the main virtual character is the virtual character mainly controlled by the player, and the non-player character is the virtual character selected by the player to command, such as the main virtual character is the virtual character manually controlled by the player during the game, and the player's manual operations on the terminal interface are used to control the main virtual character; the non-player character is the virtual character commanded by the player in voice form, and the player occasionally issues natural language commands in voice form to command the non-player character.
[0147] Optionally, the main virtual character is a virtual character selected by the player, such as the player selecting virtual hero A as the main virtual character through the hero configuration area before the virtual game starts; the non-player characters are virtual characters configured by system default, such as the game default configuration of virtual soldier 1 and virtual soldier 2 as non-player characters, etc.
[0148] Optionally, both the main virtual character and the non-player character are virtual characters selected by the player, wherein the main virtual character is the virtual character with the highest priority, and the non-player character is the virtual character with a lower priority. For example: the player selects virtual hero A, virtual hero B, and virtual hero C to participate in the virtual game. If virtual hero A is ranked first, or the priority of virtual hero A is set to the highest, it is considered that virtual hero A is the main virtual character, and virtual hero B and virtual hero C are non-player characters; or, if the player chooses to rank virtual hero B first, or sets the priority of virtual hero B to the highest, it is considered that virtual hero B is the main virtual character, and virtual hero A and virtual hero C are non-player characters, etc.
[0149] In some embodiments, the first virtual scene is determined based on the target virtual character, and the target virtual character is at least one of a non-player character and a master virtual character, which means that the process of determining the first virtual scene is implemented as at least one of the following.
[0150] Optionally, if the main control virtual character is selected as the target virtual character, the virtual scene where the main control virtual character is located is used as the first virtual scene.
[0151] Alternatively, if a non-player character is chosen as the target virtual character, the virtual scene in which the non-player virtual character is located will be the first virtual scene; if there are multiple non-player characters, the virtual scene including the most non-player characters can be used as the first virtual scene, the non-player character closest to the main control virtual character can be chosen as the target virtual character, and the virtual scene in which it is located can be used as the first virtual scene, the non-player character with the largest character attribute value (such as at least one of the attribute values such as virtual health, virtual mana, virtual defense value, etc.) can be chosen as the target virtual character, and the virtual scene in which it is located can be used as the first virtual scene, etc.
[0152] Alternatively, if the controlling virtual character and the non-player character are selected as the target virtual character, the virtual scene in which the controlling virtual character and the non-player character are located may be used as the first virtual scene.
[0153] In some embodiments, the target virtual character is determined based on the player's selection; or, the target virtual character is set by system default.
[0154] Optionally, during the virtual game, a role configuration area is displayed on the terminal interface, and the role configuration area is used to adjust and determine the role of the target virtual character of the first virtual scene; the role configuration area includes a first role identifier corresponding to the main virtual character and a second role identifier corresponding to the non-player character. If the first role identifier is selected, it is regarded as taking the main virtual character as the target virtual character; if the second role identifier is selected, it is regarded as taking the non-player character as the target virtual character, etc. In an illustrative manner, if there are multiple non-player characters (such as character 1 and character 2), the corresponding non-player character is determined based on the selected second role identifier (such as the non-player character corresponding to the selected second role identifier is character 2), and the non-player character is taken as the target virtual character (such as taking character 2 as the target virtual character), etc.
[0155] Optionally, the system sets the main virtual character as the target virtual character by default; or, the system sets the non-player character as the target virtual character by default; or, when there are multiple non-player characters, the system sets at least one of the non-player characters as the target virtual character by default, etc.
[0156] Step 320: Acquire multiple first scene hot words corresponding to the first virtual scene where the target virtual character is located.
[0157] Illustratively, after determining a target virtual character and a first virtual scene in which the target virtual character is located, a plurality of first scene hot words are acquired based on the first virtual scene.
[0158] Among them, the plurality of first scene hot words are scene-related words of the first virtual scene, that is, there is an association relationship between the first scene hot words and the first virtual scene, and the first scene hot words are words that describe the first virtual scene.
[0159] Optionally, the first scene hot words include the scene status of the first virtual scene, such as: the first virtual scene is a virtual restaurant, and the first scene hot words include business status, rest status, closed status, etc.
[0160] Optionally, the first scene hot words include element names of virtual elements in the first virtual scene, such as: the first virtual scene is a virtual restaurant, and the first scene hot words include virtual tables, virtual chairs, virtual kitchen utensils, virtual plates, etc.
[0161] Optionally, the first scene hot words include interactive vocabulary for interacting with virtual elements in the first virtual scene, such as: the first virtual scene is a virtual battlefield, and the first scene hot words include attack, attack virtual character, defense, building barriers, etc.
[0162] In some embodiments, after receiving a natural language command, multiple first scene hot words are collected based on the first virtual scene; or, after receiving a natural language command, multiple first scene hot words corresponding to the first virtual scene are filtered from multiple scene hot words pre-acquired based on the first virtual scene, etc.
[0163] Step 330: convert the natural language command into a command analysis result in text form based on the multiple first scenario hot words.
[0164] Indicatively, the first scene hot words are words that are highly associated with the first virtual scene. Therefore, when it is desired to direct non-player characters through natural language commands to better cooperate with the activities of the main virtual character, the first scene hot words can be used as constraints on the basis of the first virtual scene. In this way, in the process of converting the natural language commands into textual command analysis results, the textual content of the command analysis results is more closely aligned with the first virtual scene, thereby avoiding obtaining command analysis results that are incompatible with the first virtual scene, thereby reducing the error rate of the command analysis results.
[0165] In some embodiments, a natural language command is analyzed by a pre-trained natural language analysis model, and the natural language command and multiple first-scene hot words are used as inputs of the model. The instruction feature representation of the natural language command and the hot word feature representation of multiple first-scene hot words are extracted by the natural language analysis model, the instruction feature representation and the hot word feature representation are fused to obtain a fused feature representation, and the command analysis result in text form is output based on the fused feature representation.
[0166] In some embodiments, natural language commands are analyzed through a pre-trained natural language analysis model, and the natural language commands and multiple first-scene hot words are used as inputs to the model. In the process of analyzing the natural language commands through the natural language analysis model, an attention mechanism is used to determine the weights of the first-scene hot words in the instruction analysis process, thereby achieving the purpose of using multiple first-scene hot words as analysis constraints, and finally using the output of the natural language analysis model as the command analysis result, etc.
[0167] Optionally, the natural language analysis model includes Gaussian Mixture Model—Hidden Markov Model (GMM-HMM), deep learning model—Recurrent Neural Networks (RNN), deep learning model—Transformer, end-to-end model—Deep Learning Speech Recognition (End-to-End Deep Learning ASR), etc., among which the end-to-end model—Deep Learning Speech Recognition includes Connectionist Temporal Classification, Sequence-to-Sequence, etc. The selection of natural language analysis model is not limited here.
[0168] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.
[0169] In summary, when directing a non-player character through a natural language command, a plurality of first scene hot words are obtained according to the first virtual scene where the target virtual character is located, so as to analyze the natural language command under the constraint of the first scene hot words, so as to direct the non-player character to act according to the natural language command through a command analysis result that is easier for the device to understand. Taking the first virtual scene where the target virtual character is located as a condition for analyzing the natural language command, more targeted text information can be provided for the analysis process through the first scene hot words, reducing the analysis ambiguity caused by the poor quality of the natural language command, which is not only conducive to a more accurate understanding of the natural language command, but also enables the command analysis results to be more closely matched with the first virtual scene, improving the accuracy of the command analysis results, greatly avoiding invalid or erroneous commands to non-player characters, and providing players with a more efficient and comfortable operating environment, thereby improving the efficiency of human-computer interaction; a small number of first scene hot words can also reduce the amount of data processing during command analysis, thereby improving the efficiency of command analysis.
[0170] In an optional embodiment, the first scene hot word may be obtained according to the first scene type of the first virtual scene, the first scene hot word may be obtained according to the object perception range of the target virtual character in the first virtual scene, or the first scene hot word may be obtained by combining the first scene type and the object perception range. Figure 4 As shown above Figure 3 The illustrated step 320 may also be implemented as the following steps 410 to 411 ; or steps 420 to 421 ; or step 430 or step 432 .
[0171] Step 410: Obtain a first scene type corresponding to a first virtual scene where the target virtual character is located.
[0172] Illustratively, the first scene type is the scene type corresponding to the first virtual scene, and the scene type is used to characterize the scene state of the virtual scene. For example, the scene types include office type, battle type, kitchen type, bedroom type, classroom type, store type, factory type and other types that describe the scene state.
[0173] Optionally, the first virtual scene corresponds to a first scene type, and the first scene type is used to describe the scene state represented by the first virtual scene. For example, if the first virtual scene is a virtual office, the first scene type is an office type; or, if the first virtual scene is a virtual battle area, the first scene type is a battle type; or, if the first virtual scene is a virtual store, the first scene type is a store type, etc.
[0174] Step 411, obtaining a hot word set corresponding to the first scene type, and using the words in the hot word set as a plurality of first scene hot words.
[0175] Schematically, multiple scene types correspond to a hot word set respectively, and the hot word set represents a set of scene-related vocabulary collected based on the scene type, and also represents a set of scene-related vocabulary collected based on at least one virtual scene under the scene type.
[0176] For example: the office type corresponds to hot word set 1, and the hot word set 1 includes multiple scene-related words collected based on at least one virtual office scene under the office type. For example, the hot word set 1 includes scene-related words collected based on virtual office 1, and includes scene-related words collected based on virtual office 2. Virtual office 1 and virtual office 2 are two virtual offices under the office type.
[0177] For example: the battle type corresponds to hot word set 2, and hot word set 2 includes multiple scene-related words collected based on at least one virtual battle scene under the battle type. For example, hot word set 2 includes scene-related words collected based on virtual battle scene 1, and includes scene-related words collected based on virtual battle scene 2. Virtual battle scene 1 and virtual battle scene 2 are two virtual battle scenes under the battle type.
[0178] Optionally, after determining the first scene type corresponding to the first virtual scene, obtain a hot word set corresponding to the first scene type. The hot word set is a set of scene-related words collected based on the first scene type, that is, the hot word set corresponding to the first scene type is a set of scene-related words collected based on the first scene type. For example, the first scene type of the first virtual scene is a battle type, and the hot word set 2 corresponding to the battle type is used as the hot word set corresponding to the first scene type.
[0179] In some embodiments, the words in the hot word set are called scene-related words, or hot word words, or words. After obtaining the hot word set corresponding to the first scene type, the words in the hot word set are used as multiple first scene hot words, that is, when multiple first scene hot words are obtained based on the first virtual scene, the multiple words in the hot word set corresponding to the first scene type to which the first virtual scene belongs are used as multiple first scene hot words corresponding to the first virtual scene.
[0180] Schematically, the first virtual scene is virtual kitchen 1, and the scene type corresponding to virtual kitchen 1 is determined to be kitchen type. A hot word set corresponding to the kitchen type is obtained and the words therein are used as multiple first scene hot words; the hot word set corresponding to the kitchen type includes both words obtained based on virtual kitchen 1 and words obtained based on virtual kitchen 2, so the multiple first scene hot words obtained in this way have a wider range of characteristics.
[0181] Step 420: Obtain the object perception range of the target virtual character in the first virtual scene.
[0182] Illustratively, the target virtual character includes at least one of a main control virtual character and a non-player character. When the object perception range is acquired based on the target virtual character, at least one of the following situations is included.
[0183] Optionally, when the target virtual character is the controlling virtual character, the object perception range of the controlling virtual character in the first virtual scene is obtained; when the target virtual character is a non-player character, the object perception range of the non-player character in the first virtual scene is obtained; or, when the target virtual character is the controlling virtual character and the non-player character, the object perception range of the controlling virtual character in the first virtual scene is obtained and the object perception range of the non-player character in the first virtual scene is obtained.
[0184] Among them, the object perception range is the three-dimensional space range in which the target virtual character perceives the existence of other scene elements.
[0185] Illustratively, the target virtual character is a virtual character in the first virtual scene, and can be regarded as a virtual element in the first virtual scene. The first virtual scene may also include other scene elements besides the target virtual character; the object perception of the target virtual character represents the perception of the target virtual character of other scene elements, and the object perception range is the range of object perception, which is implemented as a three-dimensional space range in the first virtual scene.
[0186] In an optional embodiment, the object perception range includes at least one of the following.
[0187] (1) Object perception range includes the visual perception range of the target virtual character.
[0188] In schematic form, the visual perception range refers to the spatial range that an observer can visually perceive or understand, and is usually used to describe the size of the area that an organism or technical device can visually cover or perceive. The visual perception range of the target virtual character is the three-dimensional spatial range perceived by the target virtual character as an observer through observation.
[0189] Optionally, based on the position and visual observation ability of the target virtual character in the first virtual scene, the visual perception range of the target virtual character is acquired.
[0190] Schematically, the position is used to represent the position information of the target virtual character relative to the first virtual scene, such as represented by the position coordinates; the visual observation ability is used to represent the ability of the target virtual character to observe the first virtual scene, such as the visual observation ability includes the visual distance (i.e., the farthest distance that can be observed and perceived) and the visual angle (i.e., the angle range that can be observed and perceived), etc. Combining the position and the visual observation ability, the visual perception range of the target virtual character can be roughly determined from the first virtual scene.
[0191] Optionally, the visual perception range is a cone range, a spherical range, an irregular three-dimensional space range, etc., and the shape of the visual perception range is not limited here.
[0192] Optionally, corresponding visual observation capabilities are pre-configured for virtual characters during the game configuration phase, and different virtual characters have the same visual perception capabilities; or, different virtual characters have different visual perception capabilities, such as: the main control virtual character has a stronger visual perception capability, and the non-player character has a weaker visual perception capability, etc., which is not limited here.
[0193] (2) The object perception range includes the auditory perception range of the target virtual character.
[0194] In schematic form, the auditory perception range refers to the spatial range that a listener can perceive or understand, and is usually used to describe the range of sounds that an organism or technical device can cover or perceive. The auditory perception range of the target virtual character is the three-dimensional spatial range that the target virtual character can perceive through its auditory perception ability, with the target virtual character as the listener.
[0195] Optionally, based on the position and auditory perception ability of the target virtual character in the first virtual scene, the auditory perception range of the target virtual character is acquired.
[0196] In schematic form, the position is used to represent the position information of the target virtual character relative to the first virtual scene, such as through position coordinates; the auditory perception ability is used to represent the target virtual character's ability to perceive sound changes in the first virtual scene, such as auditory perception ability including auditory distance (i.e., the farthest distance of the sound that can be heard), auditory angle (i.e., the direction of the sound source can be determined or the coverage angle range of the sound heard), frequency range (i.e., the frequency range of the sound that can be perceived), etc. Combining the position and auditory perception ability, the auditory perception range of the target virtual character can be roughly determined from the first virtual scene.
[0197] Optionally, the auditory perception range is a spherical range, a ring range, an irregular three-dimensional space range, etc., and the shape of the auditory perception range is not limited here.
[0198] Optionally, corresponding auditory perception abilities are pre-configured for virtual characters during the game configuration phase; different virtual characters have the same auditory perception abilities; or different virtual characters have different auditory perception abilities, such as the main control virtual character has a stronger auditory perception ability, while the non-player character has a weaker auditory perception ability, etc., which is not limited here.
[0199] (3) The object perception range includes the olfactory perception range of the target virtual character.
[0200] In schematic form, the olfactory perception range refers to the spatial range of the odor perceived or recognized by the sense of smell. The olfactory perception range of the target virtual character is the three-dimensional spatial range that the target virtual character can perceive by the sense of smell.
[0201] Optionally, based on the position and olfactory perception ability of the target virtual character in the first virtual scene, the olfactory perception range of the target virtual character is acquired.
[0202] Schematically, the position is used to represent the position information of the target virtual character relative to the first virtual scene, such as through position coordinates; the olfactory perception range is used to represent the target virtual character's ability to perceive changes in odors in the first virtual scene, such as the olfactory perception range includes olfactory sensitivity (i.e., the ability to respond to odor molecules to different degrees), olfactory perception distance (indirectly representing the source of the odor, such as the target virtual character), etc. Combining the position and olfactory perception ability, the olfactory perception range of the target virtual character can be roughly determined from the first virtual scene.
[0203] Optionally, the olfactory perception range is a spherical range, a ring range, an irregular three-dimensional space range, etc., and the shape of the olfactory perception ability is not limited here.
[0204] Optionally, the corresponding olfactory perception ability is pre-configured for the virtual character in the game configuration stage, and the corresponding odor propagation distance, odor propagation intensity, etc. can also be configured for multiple odor sources. Optionally, different virtual characters have the same olfactory perception ability; or different virtual characters have different olfactory perception abilities, such as: the main virtual character has a stronger olfactory perception ability, and the non-player character has a weaker olfactory perception ability, etc., which is not limited here.
[0205] (4) The object perception range includes the range of perception of the target virtual character’s perception skills or perception tools.
[0206] Illustratively, the perception skill is the ability of the target virtual character to perceive the first virtual scene, and the perception skill includes at least one of a perception acquisition form and a perception enhancement form.
[0207] Optionally, the perception acquisition form is used to represent the process of acquiring a perception skill that is not possessed. After possessing the perception skill through the perception acquisition form, the target virtual character can use the perception skill to perform targeted scene perception of the first virtual scene. For example: the target virtual character has visual perception ability by default, but does not have olfactory perception ability and auditory perception ability; if the target virtual character possesses olfactory perception skills based on completing a specific task, the target virtual character can use the olfactory perception skills to perform olfactory scene perception of the first virtual scene, and of course, can continue to use the original visual perception ability to perform visual perception of the first virtual scene, etc.
[0208] Optionally, the perception enhancement form is used to represent the process of enhancing a certain perception ability by possessing a perception skill. For example, the target virtual character has visual perception ability and auditory perception ability by default; if the target virtual character possesses a visual perception skill based on completing a specific task, the visual perception skill can be used to enhance the current visual perception ability, thereby performing a higher-level visual perception process on the first virtual scene through the enhanced visual perception ability, such as being able to determine a larger visual perception range.
[0209] Illustratively, the sensing prop is a virtual prop used to perceive the first virtual scene, and the sensing prop includes at least one of a sensing acquisition form and a sensing enhancement form; for example, the sensing prop in the sensing acquisition form is a first sensing prop, and the sensing prop in the sensing enhancement form is a second sensing prop.
[0210] Optionally, the target virtual character can obtain a sensory tool based on the process of completing a specific game task or reaching a certain game level. For example, if the target virtual character obtains a first sensory tool, it can obtain additional sensory abilities that it does not currently have, such as additional olfactory sensory abilities; if the target virtual character obtains a second sensory tool, it can enhance the level of at least one sensory ability that it currently has, such as improving the visual sensory ability that it currently has, and / or improving the auditory sensory ability that it currently has.
[0211] Optionally, different perception skills or perception tools each correspond to a preset effective range, which represents the range that can be perceived when the perception skill or perception tool is applied, and the preset effective range represents the object perception range. For example: Perception skill 1 is used to perceive environmental information within 5 meters nearby, and the preset effective range is a circular area with a radius of 5 meters with the target virtual character as the center; or, perception tool 2 is used to perceive environmental information within 10 meters ahead, and the preset effective range is a semicircular area with a radius of 5 meters with the target virtual character as the center, etc.
[0212] It is worth noting that the above object perception range is only an illustrative example and is not limited to this in the embodiments of the present application.
[0213] In an optional embodiment, when the object perception range includes a visual perception range, a preset visual cone is generated as the visual perception range with the position of the target virtual character as the starting point and the direction of the target virtual character as the cone centerline.
[0214] Among them, the preset visual cone is a three-dimensional area that measures the range of visual perception.
[0215] Illustratively, when the object perception range includes the visual perception range, the position and orientation of the target virtual character are determined, with the position as the starting point and the orientation as the cone midline, and the preset visual cone as the visual perception range determined based on the visual perception ability.
[0216] Optionally, the visual cone is preset to be in the shape of a cone, with the virtual eye at the target virtual character's location as the starting point, the direction as the cone's midline, and a preset value as the radius value of the bottom circular area, thereby obtaining the cone as the object perception range.
[0217] Optionally, the preset viewing cone is a polygonal cone shape, with the virtual eye at the target virtual character's position as the starting point, the direction as the cone midline, and a preset value as the area value of the bottom polygonal area, thereby obtaining the polygonal cone as the object perception range, etc.
[0218] In an optional embodiment, taking the target virtual character as the main control virtual character as an example, obtaining the object perception range of the target virtual character in the first virtual scene is to obtain the object perception range of the main control virtual character in the first virtual scene.
[0219] In some embodiments, the object perception range of the target virtual character is determined based on a first position of the master virtual character in the first virtual scene.
[0220] Illustratively, a visual perception range of a first preset visual cone is generated with the first position as the starting point and the first direction as the midline of the cone as the object perception range; and / or an auditory perception range of a spherical area is obtained with the first position as the center and the first value as the auditory perception radius as the object perception range; and / or an olfactory perception range of an irregular three-dimensional space area is obtained with the first position as the center and the second value as the olfactory perception radius as the object perception range, etc.
[0221] In an optional embodiment, taking the target virtual character as a non-player character as an example, obtaining the object perception range of the target virtual character in the first virtual scene is to obtain the object perception range of the non-player character in the first virtual scene.
[0222] In some embodiments, the object perception range of the target virtual character is determined based on a second position of the non-player character in the first virtual scene.
[0223] Illustratively, a visual perception range of a second preset visual cone is generated with the second position as the starting point and the second direction as the midline of the cone as the object perception range; and / or, an auditory perception range of a spherical area is obtained with the second position as the center and the third value as the auditory perception radius as the object perception range; and / or, an olfactory perception range of an irregular three-dimensional space area is obtained with the second position as the center and the fourth value as the olfactory perception radius as the object perception range, etc.
[0224] In an optional embodiment, taking the target virtual character as the controlling virtual character and the non-player character as an example, when the controlling virtual character and the non-player character are in the same first virtual scene, obtaining the object perception range of the target virtual character in the first virtual scene includes: obtaining the first object perception range of the controlling virtual character in the first virtual scene, and obtaining the second object perception range of the non-player character in the first virtual scene, and taking the first object perception range and the second object perception range together as the object perception range.
[0225] Illustratively, if the controlling virtual character and the non-player character are taken as target virtual characters, the first object perception range can be acquired based on the controlling virtual character, and the second object perception range can be acquired based on the non-player character, and the first object perception range and the second object perception range can be taken together as the object perception range for subsequent hot word acquisition.
[0226] Step 421, obtaining a plurality of first scene hot words based on the environmental perception information within the object perception range.
[0227] Schematically, the environmental perception information is used to represent the environmental information perceived and acquired by the target virtual character within the object perception range. Optionally, the environmental perception information includes virtual elements determined by emitting rays, and may also include regional structure information determined by analyzing the geometric structure and layout information of the virtual environment.
[0228] Illustratively, after determining the object perception range corresponding to the target virtual character, based on the range of the object perception range in the first virtual scene and the environmental perception information determined based on perception of the target virtual character within the object perception range, multiple first scene hot words representing the state of the target virtual character are obtained.
[0229] In an optional embodiment, the environmental perception information includes virtual elements; and the virtual elements in the first virtual scene that are within the perception range of the object are determined.
[0230] Schematically, the object perception range is a part of the three-dimensional space range in the first virtual scene; the first virtual scene includes a large number of virtual elements, and the virtual elements are elements that constitute the virtual scene, such as virtual elements including virtual ground, virtual buildings, virtual trees, virtual characters, virtual stones, etc.
[0231] Optionally, when determining the object perception range, the virtual elements within the object perception range are determined from a plurality of virtual elements corresponding to the first virtual scene.
[0232] Indicatively, Figure 5 As shown, taking the visual perception range as the object perception range as an example, if the first virtual scene in which the master virtual character is located is displayed from the first-person perspective, then based on the position and orientation of the master virtual object, it can be determined as follows: Figure 5 The frustum shown is the range of visual perception.
[0233] Optionally, the position and orientation of the master virtual object are represented based on the position and orientation of the virtual camera 510; taking the position of the virtual camera 510 as the starting point and the orientation of the virtual camera as the cone midline, the following is obtained: Figure 5 The tetrahedron frustum shown is used as the object perception range. Alternatively, the two limiting planes in the preset camera rendering are Figure 5 The near clipping plane 521 and the far clipping plane 522 are parallel to the XY plane of the virtual camera, and are separated by a certain distance along the center line. Any virtual element closer to the virtual camera than the near clipping plane 521 and any virtual element farther from the virtual camera than the far clipping plane 522 will not be rendered. Figure 5 The small tetrahedron 532 is truncated from the large tetrahedron 531 shown, and the resulting three-dimensional prism is used as a viewing cone.
[0234] Schematically, the distance between the far clipping plane 522 and the virtual camera 510 is the farthest distance observed by the main virtual object, and the viewing cone is used as the object perception range. The object perception range includes virtual grass, virtual buildings, virtual cars and other virtual elements in the three-dimensional scene, that is, virtual cars, virtual grass, virtual buildings and other virtual elements are virtual elements within the object perception range. Figure 5 As shown, multiple virtual elements are presented by projecting them onto the far clipping plane 522 as an example. In fact, the multiple virtual elements may be displayed dispersedly at multiple positions within the visual perception range (such as within the three-dimensional prism surrounded by the near clipping plane 521 and the far clipping plane 522).
[0235] Optionally, if the visual perception range as the object perception range is determined from a third-person perspective, the virtual camera is located in an area outside the target virtual character, and can not only observe multiple virtual elements, but also observe the target virtual character active in the virtual environment; if the visual perception range is determined at this time, the range captured by the virtual camera can be used as the visual perception range (wherein the virtual camera moves with the target virtual character, and the visual perception range is still related to the target virtual character); or, the visual cone determined by the position and orientation of the target virtual character is used as the visual perception range (refer to Figure 5 As shown, taking the case where the observation angle of the target virtual character is consistent with the observation angle of the virtual camera 510 as an example), the process of determining the visual perception range is not limited here.
[0236] In an optional embodiment, the element name of the virtual element is used as the first scene hot word; or, the action name of the interactive action corresponding to the virtual element is used as the first scene hot word.
[0237] Schematically, a virtual element corresponds to an element name, and the element name is used to represent the virtual element itself. The element name can be one or more, that is, multiple element names can represent and indicate virtual elements, etc. For example, the element name of the virtual element 521 is truck or car; the element name of the virtual element 522 is grass or bush, etc.
[0238] Optionally, after determining the virtual elements within the object perception range, the element names of the virtual elements are summarized and the multiple element names are used as first scene hot words corresponding to the first virtual scene to obtain multiple first scene hot words.
[0239] Indicatively, during the game, the player can command the main virtual character or the non-player character to interact with the virtual elements, that is, there are interactive actions to achieve the interactive process, and the action name of the interactive action is used as the first scene hot word.
[0240] For example, if the element name of virtual element 521 is truck or car, the player can control the main virtual character or direct non-player characters to interact with virtual element 521, such as riding in the car, getting off the car, hitting the truck, climbing onto the truck, etc., which are the action names of the interactive actions; or, if the element name of virtual element 522 is grass or bushes, the player can control the main virtual character or direct non-player characters to interact with virtual element 522, such as hiding in the bushes, trimming the bushes, stepping on the bushes, etc., which are the action names of the interactive actions.
[0241] Optionally, action names that may generate interactive actions with virtual elements are summarized, and at least one action name corresponding to multiple virtual elements is used as a first scene hot word corresponding to the first virtual scene to obtain multiple first scene hot words.
[0242] Step 430, obtaining a first scene type corresponding to the first virtual scene; obtaining a hot word set corresponding to the first scene type.
[0243] Illustratively, the first scene type is the scene type corresponding to the first virtual scene, and the scene type is used to characterize the scene state of the virtual scene. For example, the scene types include office type, battle type, kitchen type, bedroom type, classroom type, store type, factory type and other types that describe the scene state.
[0244] Schematically, multiple scene types correspond to a hot word set respectively, and the hot word set represents a set of scene-related vocabulary collected based on the scene type, and also represents a set of scene-related vocabulary collected based on at least one virtual scene under the scene type.
[0245] Optionally, after determining the first scene type corresponding to the first virtual scene, obtain a hot word set corresponding to the first scene type. The hot word set is a set of scene-related vocabulary collected based on the first scene type. For example, if the first scene type of the first virtual scene is a battle type, the hot word set 2 corresponding to the battle type is used as the hot word set corresponding to the first scene type.
[0246] Step 431: Acquire the object perception range of the target virtual character in the first virtual scene.
[0247] In an exemplary embodiment, the target virtual character includes at least one of a controlling virtual character and a non-player character. When acquiring the object perception range based on the target virtual character, when the target virtual character is the controlling virtual character, the object perception range of the controlling virtual character in the first virtual scene is acquired; when the target virtual character is a non-player character, the object perception range of the non-player character in the first virtual scene is acquired; or, when the target virtual character is the controlling virtual character and the non-player character, the object perception range of the controlling virtual character in the first virtual scene is acquired and the object perception range of the non-player character in the first virtual scene is acquired.
[0248] Optionally, the object perception range includes at least one of a visual perception range, an auditory perception range, an olfactory perception range of the target virtual character, and a range perceived by a perception skill or a perception tool possessed by the target virtual character.
[0249] Step 432: acquiring at least two words from the hot word set as a plurality of first scene hot words based on the environmental perception information within the object perception range.
[0250] Optionally, after determining the object perception range, a virtual element within the object perception range is determined, and words that have an element association relationship with the virtual element are selected from the hot word set as a plurality of first scene hot words.
[0251] Schematically, the element association relationship is used to characterize the relationship that reflects the state of the virtual element. For example, the vocabulary including the element name of the virtual element is obtained from the hot word set as the first scene hot word; or, the vocabulary used to trigger the virtual element is obtained from the hot word set as the first scene hot word; or, the interactive vocabulary used to interact with the virtual element is obtained from the hot word set as the first scene hot word, etc.
[0252] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.
[0253] To sum up, taking the first virtual scene in which the target virtual character is located as a condition for analyzing natural language commands can provide more targeted text information for the analysis process through the first scene hot words, reduce the analysis ambiguity caused by the poor quality of natural language commands, not only help to understand natural language commands more accurately, but also make the command analysis results more closely matched with the first virtual scene, improve the accuracy of the command analysis results, greatly avoid invalid or erroneous commands to non-player characters, provide players with a more efficient and comfortable operating environment, and improve human-computer interaction efficiency; a small number of first scene hot words can also reduce the amount of data processing during command analysis, thereby improving command analysis efficiency.
[0254] In the embodiment of the present application, a method for obtaining hot words of the first scene is introduced. It is possible to obtain more hot words of the first scene through the first scene type corresponding to the first virtual scene, or to obtain more accurate hot words of the first scene through the object perception range of the target virtual character. It is also possible to combine the first scene type and the object perception range, and refine the hot word acquisition range while obtaining the hot words of the first scene as widely as possible, so that the hot words of the first scene are not separated from the first virtual scene, but can also better characterize the situation of the location of the target virtual character, improve the accuracy of the acquisition of the hot words of the first scene, facilitate better constraint of the analysis process of the command analysis results through the hot words of the first scene, improve the correlation between the command analysis results and the first virtual scene, and improve the accuracy of the command analysis results.
[0255] In an optional embodiment, the virtual environment includes multiple virtual scenes, and a first virtual scene can be determined based on the position of the target virtual character in the virtual environment. Then, multiple first scene hot words are determined based on the first virtual scene, so as to convert the natural language command into a command analysis result under the constraints of the multiple first scene hot words. Figure 6 As shown above Figure 3 The illustrated embodiment may also be implemented as the following steps 610 to 640 .
[0256] Step 610, obtaining a natural language command in speech form.
[0257] Among them, natural language commands are used to command non-player characters.
[0258] Illustratively, the terminal collects natural language commands in the form of voice based on the deployed microphone array, and the terminal performs a command analysis process based on the natural language commands; alternatively, the terminal sends the natural language commands to the server, and the server performs a command analysis process based on the natural language commands.
[0259] In some embodiments, during the process of the terminal running the game, whether there are natural language commands in the form of voice for commanding non-player characters in the physical environment is collected in real time; or, during the process of the terminal running the game, when the player triggers the audio collection function independently, whether there are natural language commands in the form of voice for commanding non-player characters in the physical environment will be collected.
[0260] In some embodiments, when there are multiple non-player characters, the natural language command can be an instruction for controlling multiple non-player characters simultaneously, or an instruction for commanding some of the multiple non-player characters. The command situation of the natural language command is not limited here.
[0261] Optionally, if the natural language command is an instruction to command some non-player characters, the non-player characters to be commanded can be selected from multiple non-player characters based on subsequent analysis of the natural language command. For example, if the natural language command is in voice form "No. 1, go check out the room upstairs", even if the non-player characters include No. 1 and No. 2, the natural language command is only used to command No. 1 to perform activities, and will not be elaborated here.
[0262] Step 620 , based on the position of the target virtual character in the virtual environment, determine a first virtual scene where the target virtual character is located from a plurality of virtual scenes.
[0263] In some embodiments, the first virtual scene is a virtual scene among multiple virtual scenes in a virtual environment, and the virtual environment is a three-dimensional simulation environment of a virtual game in which a main virtual character and non-player characters participate together.
[0264] Optionally, multiple virtual characters participating in the virtual game are in a virtual environment, the target virtual character includes at least one of a non-player character and a master virtual character, the virtual environment includes multiple virtual scenes, and the multiple virtual scenes together constitute the virtual environment, for example: the virtual environment includes a virtual kitchen 1, a virtual kitchen 2, a virtual office, and a virtual battle scene, etc.
[0265] Optionally, the target virtual character is determined based on a player selection; or, the target virtual character is determined based on a default configuration, etc.
[0266] Illustratively, if the controlling virtual character is the target virtual character, then based on the position of the controlling virtual character in the virtual environment, a first virtual scene is determined from a plurality of virtual scenes corresponding to the virtual environment.
[0267] Illustratively, if the non-player character is a target virtual character, a first virtual scene is determined from a plurality of virtual scenes corresponding to the virtual environment based on the position of the non-player character in the virtual environment.
[0268] In an illustrative manner, if both the controlling virtual character and the non-player character are used as the target virtual characters, then based on the first position of the controlling virtual character in the virtual environment and the second position of the non-player character in the virtual environment, at least one virtual scene is determined from multiple virtual scenes corresponding to the virtual environment as the first virtual scene. For example, if the first position is in virtual scene 1 and the second position is in virtual scene 2, then virtual scene 1 and virtual scene 2 can be used together as the first virtual scene; or, if the first position is in virtual scene 1 and the second position is also in virtual scene 1, then virtual scene 1 can be used as the first virtual scene, etc.
[0269] That is, the target virtual character is in the first virtual scene.
[0270] In some embodiments, the position coordinates of the target virtual character in the virtual environment are obtained; scene coordinate intervals corresponding to multiple virtual scenes are obtained; and based on the relative position relationship between the position coordinates and the multiple scene coordinate intervals, the first virtual scene where the target virtual character is located is determined from the multiple virtual scenes.
[0271] The scene coordinate interval is used to represent the position range of the virtual scene in the virtual environment.
[0272] Illustratively, the position coordinates of the target virtual character are the content of the position information represented by the coordinate data. Optionally, the position coordinates corresponding to the target virtual character are determined based on the world coordinate system corresponding to the virtual environment.
[0273] Schematically, multiple virtual scenes in the virtual environment correspond to a scene coordinate interval respectively, and the scene coordinate interval is used to limit the three-dimensional space range corresponding to the virtual scene from three dimensions, and the three dimensions represent the horizontal direction, vertical direction and depth direction in the three-dimensional space. The scene coordinate intervals corresponding to different virtual scenes are independent and do not overlap with each other. Optionally, the scene coordinate intervals corresponding to the multiple virtual scenes are determined based on the world coordinate system corresponding to the virtual environment.
[0274] In an illustrative manner, after obtaining the position coordinates, the position coordinates are matched with a plurality of scene coordinate intervals, and the virtual scene corresponding to the scene coordinate interval including the position coordinates is used as the first virtual scene where the target virtual character is located. For example, the scene coordinate interval including the position coordinates is referred to as the first scene coordinate interval, the first virtual scene corresponds to the first scene coordinate interval, and the object coordinates are located within the first scene coordinate interval.
[0275] Optionally, if the controlling virtual character is the target virtual character, the first position coordinates of the controlling virtual character in the virtual environment are obtained; based on the relative position relationship between the first position coordinates and multiple scene coordinate intervals, the first virtual scene where the target virtual character is located is determined from multiple virtual scenes.
[0276] Optionally, if the non-player character is a target virtual character, a first position coordinate of the non-player character in the virtual environment is obtained; based on a relative position relationship between the first position coordinate and multiple scene coordinate intervals, a first virtual scene where the target virtual character is located is determined from multiple virtual scenes.
[0277] In an optional embodiment, based on a first virtual game in which the target virtual character participates, a first virtual scene corresponding to the first virtual game is determined.
[0278] Among them, the multiple virtual games correspond to a virtual scene respectively, and the multiple virtual games are simulated battle environments provided for the target virtual characters.
[0279] Illustratively, in addition to determining the first virtual scene based on the position of the target virtual character in the virtual environment, the virtual scene corresponding to the first virtual game in which the target virtual character participates may be used as the first virtual scene. For example, each virtual game corresponds to a virtual scene, and if the virtual game in which the target virtual character participates is the first virtual game, the virtual scene corresponding to the first virtual game is used as the first virtual scene.
[0280] Optionally, the virtual game includes game tasks (such as task 1 corresponds to virtual scene 1, task 2 corresponds to virtual scene 2, etc.), game levels (such as game level 1 corresponds to virtual scene 1, game level 2 corresponds to virtual scene 2, etc.), etc., which are not limited here.
[0281] Step 630, obtaining a plurality of first scene hot words corresponding to the first virtual scene where the target virtual character is located.
[0282] Among them, the multiple first scene hot words are scene-related words of the first virtual scene.
[0283] Optionally, the first scene hot word includes the scene state of the first virtual scene; or, the first scene hot word includes the element name of the virtual element in the first virtual scene; or, optionally, the first scene hot word includes interactive vocabulary for interacting with the virtual elements in the first virtual scene, etc.
[0284] In an optional embodiment, based on the position and orientation of the target virtual character in the first virtual scene, the object perception range corresponding to the target virtual character is determined from the first virtual scene.
[0285] Illustratively, the position is usually expressed in coordinate form, and determining the position of the target virtual character in the first virtual scene means determining the coordinates of the target virtual character in the first virtual scene, such as through a comprehensive representation of the X-axis, Y-axis, and Z-axis coordinates in three-dimensional space.
[0286] Illustratively, the direction is used to represent the sight direction of the target virtual character in the first virtual scene, and can usually be represented by a direction vector, indicating the front direction of the target virtual character.
[0287] Optionally, the comprehensive position and orientation can determine the object perception range of the target virtual character from the first virtual scene, and the object perception range is the three-dimensional space range in which the target virtual character perceives the existence of other scene elements.
[0288] Indicatively, the object perception range is also affected by the target virtual character's visual perception ability, auditory perception energy, olfactory perception ability, the perception skills acquired by itself, or the perception skills corresponding to the sensing gear. The determination of the object perception range is not elaborated here.
[0289] Optionally, the object perception range may also be affected by environmental factors such as virtual obstacles, virtual buildings, virtual terrain, etc. in the first virtual scene. For example, the above environmental factors may affect the reception of visual perception ability, auditory perception ability and other perception abilities, thereby affecting the determination of the object perception range.
[0290] Optionally, the range that the target virtual character can perceive in a specific direction is calculated based on the position coordinates and the orientation vector of the target virtual character, combined with its visual perception ability and other perception capabilities and environmental factors. Schematically, starting from the position of the target virtual character, a line of sight is projected along its orientation vector within a certain distance to determine where there are no obstacles and define the boundary of the object perception range.
[0291] In some embodiments, if the virtual environment is dynamic, or the position and / or orientation of the target virtual character changes, the calculation of the object perception range may need to be updated in real time.
[0292] Illustratively, real-time changes in object perception can be simulated by using collision detection technology, light detection technology, etc. in a virtual environment, so that a more accurate object perception range can be determined in real time, which helps to simulate the behavior and interaction of the target virtual character in the virtual environment.
[0293] In an optional embodiment, a plurality of first scene hot words are acquired based on environmental perception information within a perception range of the object.
[0294] Illustratively, after the object perception range corresponding to the target virtual character is determined, a plurality of first scene hot words associated with the first virtual scene may be acquired based on the object perception range.
[0295] In an optional embodiment, multiple scene hot words are obtained, wherein the multiple scene hot words correspond to at least two virtual scenes.
[0296] Illustratively, scene hot words corresponding to multiple virtual scenes in the virtual environment are obtained in advance. Optionally, based on the virtual elements in each virtual scene, at least one scene hot word corresponding to the multiple virtual scenes is obtained, and multiple scene hot words are stored.
[0297] Optionally, the element name of the virtual element in the virtual scene is obtained as the scene hot word; or, the action name of the interactive action interacting with the virtual element in the virtual scene is obtained as the scene hot word; or, the state name used to describe the element state of the virtual element in the virtual scene is obtained as the scene hot word, etc.
[0298] In an optional embodiment, based on the first virtual scene where the target virtual character is located, a plurality of first scene hot words corresponding to the first virtual scene are obtained from a plurality of scene hot words.
[0299] In an illustrative manner, multiple scene hot words correspond to scene identifiers, and the scene identifiers are used to represent the virtual scene when the scene hot words are collected. For example, if scene hot words 1 and 2 are obtained based on virtual scene A, scene identifier a corresponding to virtual scene A is marked for scene hot word 1, and scene identifier a is also marked for scene hot word 2; if scene hot word 3 is obtained based on virtual scene B, scene identifier b corresponding to virtual scene B is marked for scene hot word 3, and so on.
[0300] In some embodiments, based on a first virtual scene in which the target virtual character is located, a plurality of candidate scene hot words with a first scene identifier are obtained from a plurality of scene hot words, where the first scene identifier is a scene identifier corresponding to the first virtual scene.
[0301] Illustratively, after determining the first virtual scene in which the target virtual character is located, based on the fact that a plurality of scene hot words are each labeled with a scene identifier, the scene hot words labeled with the first scene identifier can be screened from the plurality of scene hot words as candidate scene hot words.
[0302] In some embodiments, based on the object perception range of the target virtual character in the first virtual scene, at least two candidate scene hot words are selected from a plurality of candidate scene hot words as the first scene hot words.
[0303] Illustratively, after obtaining multiple candidate scene hot words based on the first virtual scene, the object perception range is a partial range in the first virtual scene, so the multiple candidate scene hot words can be further screened based on the object perception range, and at least two candidate scene hot words related to the object perception range are selected as the first scene hot words.
[0304] Optionally, a virtual element within the object's perception range is determined as a perceived virtual element, that is, the perceived virtual element is a virtual element that can be perceived by the target virtual character in the first virtual scene; the element name of the perceived virtual element is filtered from multiple candidate scene hot words as the first scene hot word; or, the action name of the interactive action corresponding to the perceived virtual element is filtered from multiple candidate scene hot words as the first scene hot word, etc.
[0305] Step 640: Convert the natural language command into a command analysis result in text form based on the multiple first scenario hot words.
[0306] In an optional embodiment, when no command analysis result is generated, in response to the target virtual character moving from the first virtual scene to the second virtual scene, a plurality of second scene hot words corresponding to the second virtual scene are acquired.
[0307] Schematically, the virtual environment includes multiple virtual scenes, and the second virtual scene is another virtual scene different from the first virtual scene. As a virtual character participating in the virtual game, the position of the target virtual character may change in the virtual environment, such as moving from location A to location B. If the target virtual character moves from the first virtual scene to the second virtual scene during the process of receiving a natural language command but before obtaining the command analysis result, it is necessary to determine the scene hot words used to constrain the generation of the command analysis result based on the second virtual scene that has been moved to most recently.
[0308] For example: in the process of receiving a natural language command but not yet generating a command analysis result, if the target virtual character moves from the current first virtual scene to the second virtual scene, that is, the position of the target virtual character moves from the first virtual scene to the second virtual scene, then in order to be able to more flexibly analyze the status of the virtual scene after the move, multiple second scene hot words corresponding to the second virtual scene are obtained.
[0309] Optionally, a scene hot word with a second scene identifier is obtained from multiple scene hot words as the second scene hot word; or, a scene hot word with a second scene identifier and associated with the current object perception range is obtained from multiple scene hot words as the second scene hot word.
[0310] The second scene identifier is a scene identifier corresponding to the second virtual scene. The current object perception range is an object perception range determined from the second virtual scene. For example, based on the perception ability (visual perception ability, auditory perception ability, olfactory perception ability, etc.) of the target virtual character, the current object perception range is determined from the second virtual scene.
[0311] In an optional embodiment, a command analysis result is generated by converting a natural language command into text form based on a plurality of second scenario hot words.
[0312] Illustratively, after acquiring multiple second-scenario hot words, the natural language commands are converted into command analysis results in text form through a pre-trained natural language analysis model with the multiple second-scenario hot words as constraints.
[0313] Optionally, a natural language command and multiple second-scenario hot words are used as inputs of the model, and the instruction feature representation of the natural language command and the hot word feature representation of the multiple second-scenario hot words are extracted through the natural language analysis model. The instruction feature representation and the hot word feature representation are fused to obtain a fused feature representation, and the command analysis result in text form is output based on the fused feature representation.
[0314] In an optional embodiment, the non-player character is commanded to act in the virtual environment based on the command analysis result.
[0315] Schematically, the command analysis result is text content in text form. Compared with natural language commands in voice form, the command analysis result is more intuitive and can provide more targeted control over non-player characters.
[0316] Optionally, the form of directing the non-player character's activities based on the command analysis result includes at least one of the following: (1) directing the non-player character to move in the virtual environment; (2) directing the non-player character to search for a specified item; (3) directing the non-player character to cooperate with the main virtual character to defend against the attack of the enemy virtual character; (4) directing the non-player character to attack the enemy virtual character; (5) directing the non-player character to change props, accessories, clothing accessories, etc. The activity form is not limited here.
[0317] For example, if the command analysis result obtained after analyzing the natural language command is "attack me", it means commanding the non-player character to make a virtual attack on a virtual character that can be attacked in the virtual environment.
[0318] In some embodiments, if the server obtains the command analysis result, it can command the non-player character to move in the virtual environment and generate activity animation data based on the command analysis result, and then send the activity animation data to the terminal so that the terminal can render and display the activity animation of the non-player character based on the natural language command.
[0319] In some embodiments, if the server obtains the command analysis result, the command analysis result can be sent to the terminal. The terminal commands the non-player character to move in the virtual environment based on the command analysis result and generates activity animation data, and then renders and displays the activity animation data on the terminal interface, that is, displays the activity animation of the non-player character based on the natural language command.
[0320] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.
[0321] To sum up, taking the first virtual scene in which the target virtual character is located as a condition for analyzing natural language commands can provide more targeted text information for the analysis process through the first scene hot words, reduce the analysis ambiguity caused by the poor quality of natural language commands, not only help to understand natural language commands more accurately, but also make the command analysis results more closely matched with the first virtual scene, improve the accuracy of the command analysis results, greatly avoid invalid or erroneous commands to non-player characters, provide players with a more efficient and comfortable operating environment, and improve human-computer interaction efficiency; a small number of first scene hot words can also reduce the amount of data processing during command analysis, thereby improving command analysis efficiency.
[0322] In the embodiment of the present application, it is introduced that the first virtual scene is determined by the position of the target virtual character, and then the object perception range is determined from the first virtual scene based on the position and orientation and the content of the first scene hot words is obtained. If multiple virtual characters are active in a virtual environment including multiple virtual scenes, the first virtual scene can be specifically determined from the virtual environment according to the position of the target virtual character, so that the object perception range is determined in combination with the orientation of the target virtual character in the first virtual scene. Even if the command analysis result is not generated, the second scene hot words can be obtained based on the process of the target virtual character moving to the second virtual scene to constrain the analysis process of the command analysis result, which not only improves the flexibility of obtaining the scene hot words used to constrain the analysis process, but also helps to further improve the flexibility of obtaining the command analysis results, so as to facilitate more accurate and flexible command of non-player characters through the command analysis results in text form.
[0323] In an optional embodiment, the natural language command is analyzed by a pre-trained natural language analysis model to obtain a command analysis result in text form. Figure 7 As shown above Figure 3 The step 230 shown may also be implemented as the following steps 710 to 730 .
[0324] Step 710, obtaining a pre-trained natural language analysis model.
[0325] Schematically, the natural language analysis model is a pre-trained model for analyzing natural language commands in speech form and outputting command analysis results in text form.
[0326] Among them, the pre-trained natural language analysis model includes acoustic network, language network and preset dictionary.
[0327] Optionally, the natural language analysis model can also be called a natural language analysis system, and the acoustic network, language network and preset dictionary in the natural language analysis model can also be called the acoustic model, language model and preset dictionary in the natural language analysis system. The names of the natural language analysis model and its related network are not limited here.
[0328] Schematically, the acoustic network can also be called an acoustic model, which is used to process input in the form of speech to convert sound signals into potential speech units, such as phonemes or segments. Phonemes are the smallest speech units in speech and are phonemes in language. Phonemes are related to the acoustic features of pronunciation; the cloud is a specific sound unit in the actual speech signal, which describes the pronunciation of sound in actual speech and the acoustic details of pronunciation (such as duration, frequency, sound intensity, etc.). By analyzing natural language commands in the form of speech through the acoustic network, possible sequences of speech units can be identified based on the acoustic features of natural language commands (such as spectrum, tone, speech speed, etc.); the acoustic network can learn the mapping from acoustic features to speech units through machine learning techniques such as deep neural networks (DNN) or recurrent neural networks (RNN).
[0329] Schematically, the language network can also be called a language model, which is used to improve the probability score of the language unit sequence with high probability when understanding text input. The language network predicts the rationality and fluency of a given string of words or speech unit sequences based on the grammatical rules and statistical information of the language; the language network includes n-gram model (n-gram), recurrent neural network language model (RNNLM) or transformer model (Transformer), etc., which are used to model and generate text sequences.
[0330] Schematically, the preset dictionary contains words and their corresponding pronunciation or language feature information that may be encountered in speech recognition or text understanding. It provides a way to map text words to their pronunciation forms or language features; in speech recognition, the preset dictionary helps reduce possible pronunciation ambiguity and improve the recognition accuracy of the system.
[0331] Among them, the preset dictionary includes multiple first-scene hot words, other-scene hot words and common words.
[0332] Illustratively, the first scene hot words are scene hot words corresponding to the first virtual scene, other scene hot words are scene hot words corresponding to other virtual scenes except the first virtual scene, and common words are words frequently used in daily life, such as "we", "I", "here", "there", "big", "small", etc. The preset dictionary is a collection of multiple words collected in advance, including the first scene hot words, other scene hot words, and commonly used common words.
[0333] Step 720: Acquire multiple speech units corresponding to the natural language command through the acoustic network.
[0334] Among them, phonetic units are the basic building blocks of vocabulary pronunciation.
[0335] In some embodiments, instruction feature representations corresponding to natural language commands are extracted.
[0336] Schematically, the feature extraction process is used to indicate that a continuous sound signal - a natural language command is divided into multiple short time periods, and acoustic features such as spectrum, pitch, and speaking rate are calculated within multiple time periods. Each time period corresponds to an acoustic feature. The command feature representation can represent an acoustic feature sequence obtained by arranging multiple acoustic features according to time periods, and can also represent a feature representation after the fusion of multiple acoustic features, which is not limited here.
[0337] In some embodiments, the instruction signature is analyzed via an acoustic network.
[0338] Schematically, taking the instruction feature representation as an example, the goal of the acoustic network is to map a continuous sequence of acoustic features to a sequence of speech units, such as phonemes or segments. It learns to predict possible sequences of speech units (phonemes or segments) given acoustic features.
[0339] Optionally, the acoustic network outputs one or more possible speech unit sequences, wherein the speech unit sequence includes multiple speech units. Different speech unit sequences may contain different speech units, or may just have different orders of speech units. The multiple speech unit sequences represent the arrangement of speech units that the language network considers to be most likely when understanding the input natural language command.
[0340] In some embodiments, in the process of analyzing instruction feature representations through an acoustic network, a context analysis network is additionally introduced. The context analysis network is used to analyze the context and helps to obtain a more reliable speech unit sequence by considering the temporal dependencies between multiple acoustic features.
[0341] Step 730: Analyze the sequence matching relationship between the multiple speech units and the selected words in the preset dictionary through the language network to obtain the command analysis result in text form.
[0342] Illustratively, the preset dictionary includes multiple words, each word corresponds to at least one speech unit. After obtaining at least one speech unit sequence, the at least one speech unit sequence can be matched with the words in the preset dictionary to analyze at least one word represented by the speech unit sequence.
[0343] Optionally, considering the process of obtaining multiple first scene hot words based on the first virtual scene, the first scene hot words are used to constrain the command analysis results. Therefore, when matching words through the speech unit sequence, at least multiple first scene hot words can be used as selected words that need to be applied in the preset dictionary matching process; in addition, considering that some commonly used general words are usually involved in the speech recognition process, general words can also be used as selected words during vocabulary matching, that is, at least one speech unit sequence is matched with the first scene hot words and general words in the selected words to analyze at least one word represented by the speech unit sequence. At least one word is more likely to include the first scene hot words to avoid interference from other scene hot words. That is, the selected words include at least multiple first scene hot words and general words.
[0344] In an optional embodiment, the selected vocabulary also includes other scene hot words.
[0345] Illustratively, if the selected vocabulary is a first scene hot word and a common vocabulary, at least one speech unit sequence can be matched with the first scene hot word and the common vocabulary; if the selected vocabulary is a first scene hot word, other scene hot words and a common vocabulary, at least one speech unit sequence can be matched with the first scene hot word, other scene hot words and a common vocabulary.
[0346] Among them, the vocabulary in the preset dictionary includes analysis weights, and the first analysis weights of multiple first-scene hot words are higher than the second analysis weights of other scene hot words. The analysis weight refers to the degree of attention paid to the vocabulary when it participates in sequence matching.
[0347] Indicatively, each word in the preset dictionary corresponds to an analysis weight. The larger the analysis weight, the more attention the word will receive when it participates in sequence matching, that is, the easier it will be to be paid attention to. The smaller the analysis weight, the more attention the word will receive when it participates in sequence matching, that is, the harder it will be to be paid attention to. The analysis weight can be used to adjust the degree of attention paid to different words in the sequence matching process, so that words with larger analysis weights are matched with the word sequence first, and at the same time, words with larger analysis weights are more likely to be selected as words that constitute the command analysis results.
[0348] Optionally, multiple first-scene hot words are obtained based on the first virtual scene in which the target virtual character is located. The first-scene hot words can better reflect the situation of the first virtual scene and have a greater probability of being associated with the natural language command. Therefore, the first analysis weight of the first scene hot words is set higher than the second analysis weights of other scene hot words, so that when the words participate in sequence matching, more attention can be paid to the first scene hot words, and thus the first scene hot words are more likely to be selected as the words that constitute the command analysis results.
[0349] Optionally, the common vocabulary corresponds to a third analysis weight, the third analysis weight is less than the first analysis weight and the third analysis weight is greater than the second analysis weight; or, the third analysis weight is equal to the first analysis weight and the third analysis weight is greater than the second analysis weight; or, the third analysis weight is less than the first analysis weight and the third analysis weight is equal to the second analysis weight; or, the third analysis weight is less than the first analysis weight and the third analysis weight is less than the second analysis weight, etc. The numerical value of the third analysis weight is not limited here.
[0350] Among them, the first analysis weight, the second analysis weight and the third analysis weight are usually values between 0 and 1. Under the condition that the first analysis weight is greater than the second analysis weight, the specific values of the analysis weights are not limited here.
[0351] In an optional embodiment, the matching relationship between the plurality of speech units and the selected words in the preset dictionary is analyzed to obtain a plurality of candidate word sequences.
[0352] Schematically, the process of analyzing the matching relationship represents the process of analyzing the matching between the selected vocabulary and the speech unit. Schematically, each selected vocabulary corresponds to at least one speech unit, and the conversion probability of the speech unit to the selected vocabulary is determined by analyzing the multiple speech units and the selected vocabulary. The conversion probability represents the probability of matching between the selected vocabulary and the speech unit. The larger the conversion probability, the closer the speech unit is to the selected vocabulary, that is, the greater the possibility that the speech unit may express the selected vocabulary, and the stronger the matching relationship between the selected vocabulary and the speech unit.
[0353] Optionally, a plurality of speech units form at least one speech unit sequence, and based on a matching analysis process between the speech units and the selected vocabulary, at least one candidate vocabulary sequence corresponding to the at least one speech unit sequence is determined, and a plurality of candidate vocabulary sequences are obtained.
[0354] Schematically, after analyzing the natural language command based on the sound network, a speech unit sequence 1 and a speech unit sequence 2 are obtained, wherein the speech unit sequence 1 is composed of speech unit a—speech unit b—speech unit c, and the speech unit sequence 2 is composed of speech unit a—speech unit d—speech unit c, and it can be regarded that the multiple speech units include speech unit a, speech unit b, speech unit c, and speech unit d. After matching and analyzing the multiple speech units and the selected words in the preset dictionary, the candidate vocabulary sequence obtained based on the speech unit sequence 1 includes a candidate vocabulary sequence 11 and a candidate vocabulary sequence 12, wherein the candidate vocabulary sequence 11 is "help me attack this place", and the candidate vocabulary sequence 12 is "help me attack the enemy"; the candidate vocabulary sequence obtained based on the speech unit sequence 2 includes a candidate vocabulary sequence 21 and a candidate vocabulary sequence 22, wherein the candidate vocabulary sequence 21 is "help me supply this place", and the candidate vocabulary sequence 22 is "help me attack the enemy", that is, multiple candidate vocabulary sequences are obtained, including the candidate vocabulary sequence 11, the candidate vocabulary sequence 12, the candidate vocabulary sequence 21, and the candidate vocabulary sequence 22.
[0355] The plurality of candidate vocabulary sequences include at least one of the plurality of selected vocabulary, and the candidate vocabulary sequence is a vocabulary sequence obtained based on a change relationship between a plurality of speech units.
[0356] Illustratively, since the candidate vocabulary sequence is a vocabulary sequence obtained based on selected vocabulary in a preset dictionary, the vocabulary in the candidate vocabulary sequence is the vocabulary in the selected vocabulary; the candidate vocabulary sequence can be a sequence obtained from a single vocabulary, or a sequence obtained from a combination of multiple vocabulary, which is not limited here.
[0357] Since the candidate vocabulary sequence is a sequence obtained after matching analysis between multiple phonetic units and selected vocabulary, the phonetic unit transformation of at least one vocabulary in the candidate vocabulary sequence matches the transformation between multiple phonetic units, that is, the candidate vocabulary sequence is a vocabulary combination sequence predicted based on the changes between multiple phonetic units.
[0358] In an optional embodiment, the sequence semantics of multiple candidate vocabulary sequences are analyzed through a language network, and at least one candidate vocabulary sequence is obtained from the multiple candidate vocabulary sequences as a command analysis result in text form.
[0359] Schematically, sequence semantics is used to represent the semantic information expressed by the candidate vocabulary sequence, which can not only reflect the semantic changes of at least one word in the candidate vocabulary sequence, but also reflect the sentence semantics of the entire candidate vocabulary sequence.
[0360] Schematically, the language network evaluates multiple candidate vocabulary sequences respectively, so as to take at least one candidate vocabulary sequence that is more consistent with the current semantic situation as the command analysis result.
[0361] Optionally, probability evaluation is performed on multiple candidate vocabulary sequences through a language network to obtain prediction probabilities corresponding to the multiple candidate vocabulary sequences respectively; and at least one candidate vocabulary sequence with the largest prediction probability is used as the command analysis result.
[0362] Indicatively, the prediction probability is used to characterize the probability estimation result of the candidate vocabulary sequence being correct under the constraints of multiple speech units. For example, the prediction probability of candidate vocabulary sequence 11 is 0.32, the prediction probability of candidate vocabulary sequence 12 is 0.8, the prediction probability of candidate vocabulary sequence 22 is 0.45; if the candidate vocabulary sequence with the largest prediction probability is used as the command analysis result, the candidate vocabulary sequence 12 - help me attack the enemy is output as the command analysis result.
[0363] In some embodiments, after analyzing the sequence semantics of multiple candidate vocabulary sequences through a language network, probability distributions corresponding to the multiple candidate vocabulary sequences are generated; the multiple probability distributions and multiple speech units are input into a decoding network, and based on the constraints of the multiple speech units, at least one candidate vocabulary sequence is output as a command analysis result.
[0364] Schematically, the probability distribution is used to describe the naturalness between at least one word in a candidate vocabulary sequence; the decoding network (or decoder) is used to combine multiple candidate vocabulary sequences, corresponding probability distributions and multiple language units to select the best candidate vocabulary sequence as the command analysis result.
[0365] Optionally, the decoding network may use beam search or other optimization algorithms to search among multiple candidate vocabulary sequences. Considering the probability distribution of the language network output and the constraints of multiple speech units (phoneme or segment sequences) output by the acoustic network, the beam search can find the most likely candidate vocabulary sequence in the potential sequence space as the command analysis result, thereby improving the accuracy of speech recognition.
[0366] It is worth noting that the decoding network can be configured outside the sound network and the language network, or within the language network; the preset dictionary can be configured within the language network, or outside the language network as an independent neural network layer; and the network structure of the natural language analysis model is not limited here.
[0367] In some embodiments, the language network includes language sub-networks corresponding to at least two virtual scenes respectively.
[0368] Schematically, multiple virtual scenes correspond to a language subnetwork respectively, and different language subnetworks are used to analyze the corresponding virtual scenes in a targeted manner. For example, virtual scene A corresponds to language subnetwork 1, and virtual scene B corresponds to language subnetwork 2. When analysis is required based on virtual scene A, language subnetwork 1 is used for language analysis, and when analysis is required based on virtual scene B, language subnetwork 2 is used for language analysis, and so on.
[0369] Optionally, if the natural language analysis model is called a natural language analysis system, language models corresponding to at least two virtual scenes can be configured, and multiple language models correspond one-to-one to multiple virtual scenes. There is no limitation on the model structure or system architecture here, and the above content is merely a schematic illustration of the analysis process.
[0370] In some embodiments, based on a first virtual scene in which the target virtual character is located, a first language sub-network corresponding to the first virtual scene is determined from at least two language sub-networks.
[0371] Illustratively, the first virtual scene is a virtual scene among at least two virtual scenes. In addition to determining the first scene hot words based on the first virtual scene in which the target virtual character is located, a first language subnetwork corresponding to the first virtual scene is determined from at least two language subnetworks based on the first virtual scene. The first language subnetwork is a subnetwork layer that analyzes a candidate vocabulary sequence based on the first virtual scene.
[0372] In some embodiments, the sequence semantics of multiple candidate vocabulary sequences are analyzed by the first language sub-network, and at least one candidate vocabulary sequence is obtained from the multiple candidate vocabulary sequences as a command analysis result in text form.
[0373] Optionally, different language sub-networks are language sub-networks pre-trained based on corresponding virtual scenes, wherein the first language sub-network is a language sub-network trained based on the first virtual scene.
[0374] In an illustrative example, taking the first language sub-network obtained by training the first virtual scene as an example, the first scene hot words corresponding to the first virtual scene are obtained, and the first scene hot words and common words are used as sample data to train the first language sub-network; or, a plurality of scene description words used to describe the first virtual scene are obtained, and the scene description words are used as sample data to train the first language sub-network, and the scene description words include a plurality of words describing the virtual scene state and describing the virtual elements therein, etc. Other language sub-networks are also trained by corresponding virtual scene-related words (scene hot words, common words, scene description words, etc.), which will not be elaborated here.
[0375] Optionally, after obtaining multiple candidate vocabulary sequences under the constraint of the hot words of the first scene, semantic analysis is performed on the multiple candidate vocabulary sequences respectively through the first language sub-network corresponding to the first virtual scene to obtain prediction probabilities corresponding to the multiple candidate vocabulary sequences, and at least one candidate vocabulary sequence with the largest prediction probability is used as the command analysis result in text form.
[0376] It is worth noting that the above process of determining the corresponding first language sub-network based on the first virtual scene in which the target virtual character is located is only an illustrative example. Multiple virtual scenes can also be configured to correspond to a sound network respectively (such as a sound network is a network trained based on the corresponding virtual scene), and the corresponding first sound sub-network can be selected for analysis based on the first virtual scene, or the corresponding first sound sub-network and first language sub-network can be selected for analysis based on the first virtual scene. This is not limited here.
[0377] To sum up, taking the first virtual scene in which the target virtual character is located as a condition for analyzing natural language commands can provide more targeted text information for the analysis process through the first scene hot words, reduce the analysis ambiguity caused by the poor quality of natural language commands, not only help to understand natural language commands more accurately, but also make the command analysis results more closely matched with the first virtual scene, improve the accuracy of the command analysis results, greatly avoid invalid or erroneous commands to non-player characters, provide players with a more efficient and comfortable operating environment, and improve human-computer interaction efficiency; a small number of first scene hot words can also reduce the amount of data processing during command analysis, thereby improving command analysis efficiency.
[0378] In the embodiment of the present application, in addition to determining the first scene hot words based on the first virtual scene where the target virtual character is located, a process of determining the corresponding first language sub-network for targeted analysis based on the first virtual scene is introduced. The corresponding language sub-networks are trained for multiple virtual scenes respectively, so that the language sub-network can perform a more purposeful analysis of the candidate vocabulary sequence under the constraints of the virtual scene, avoiding the blindness problem when obtaining the command analysis results; the process of obtaining the candidate vocabulary sequence through the first scene hot word constraints improves the accuracy of the candidate vocabulary sequence, and then further constrains the analysis process of the candidate vocabulary sequence in combination with the first language sub-network, which helps to further improve the correlation between the command analysis results and the first virtual scene, improve the accuracy of the analysis of natural language commands, and facilitate the improvement of the command accuracy of non-player characters through more accurate command analysis results.
[0379] In an optional embodiment, the above-mentioned method for voice conversion based on virtual scenes can be applied to a game scene, where a player issues a natural language command in the form of voice during the game, so that the terminal can voice-command a non-player character based on the natural language command. Therefore, the above-mentioned method for voice conversion based on virtual scenes can also be referred to as "an adaptive voice interaction method in a game scene". Schematically, as Figure 8 As shown, it is a technical flow chart of the voice conversion method based on virtual scene, which includes the following steps 810 to 880.
[0380] Step 810, the player enters the game level.
[0381] Schematically, a game level is a specific section or stage in a video game where the player needs to complete specific goals or tasks in order to move on to the next level. Levels are usually designed to gradually increase in difficulty to challenge the player's skills and strategies. Each level may contain different environments, enemies, puzzles, and rewards, designed to provide a diverse gaming experience.
[0382] Optionally, the player can control the main virtual character to participate in the virtual game during the game, and the player can also command other non-player characters to cooperate with the main virtual character to participate in the virtual game. The non-player characters are virtual characters other than the main virtual character, such as virtual soldiers and virtual heroes deployed by the system by default in the virtual game. For example: the player can manually control the main virtual character to fight in the virtual game during the game, and can also command non-player characters to fight in the virtual game through voice.
[0383] Taking the above-mentioned target virtual character as the main control virtual character as an example, when determining the object perception range for obtaining the hot words of the first scene, the visual perception range (or object field of view) is used as the object perception range. Based on the movement of the main control virtual character in the virtual environment, the field of view of the main control virtual character can be updated in real time.
[0384] Step 820 , preparing a hot word list based on virtual elements in the virtual scene of the game level.
[0385] In a illustrative manner, a game usually provides multiple different game levels. A game level corresponds to a virtual environment. A virtual environment includes multiple virtual scenes. A game scene refers to the three-dimensional space in which the target virtual character is located in a specific game level. A virtual scene is not only a visual presentation, but also includes virtual elements, game sound effects and other details, which together constitute the player's gaming experience. A well-designed virtual scene can greatly enhance the immersion and fun of the game. Multiple virtual scenes correspond to a scene type respectively. The scene types of multiple virtual scenes may be the same or different, which is not limited here.
[0386] Optionally, each scene type has its own unique virtual elements and atmosphere, through which the story and interactivity of the game can be enhanced. For example, in a forest scene, the player may need to avoid wild animals or find hidden paths; while in a future city scene, the player may need to use high-tech equipment to complete the task, etc.
[0387] Optionally, each game level may have some specific virtual elements, such as keys for unlocking specific doors or treasure chests, that is, doors leading to new areas or game levels that require specific keys or completion of specific tasks to open. Virtual props obtained in specific game levels may have unique attack effects or capabilities, such as: providing additional defense or special resistance to help players deal with specific hostile virtual characters, etc. There are unique virtual props and consumables, collectibles, gems / gold coins in the game levels; in addition, the game levels also contain puzzle props, specific means of transportation, specific abilities (magic, skills, etc.); and items that interact with the environment, such as virtual explosives, virtual obstacles, virtual buildings, etc. Among them, the virtual elements unique to each specific level in the game level usually have unique functions or effects that can affect the player's gaming experience.
[0388] In some embodiments, based on the virtual elements respectively contained in the multiple virtual scenes, hot word lists corresponding to the multiple virtual scenes are prepared; or, a hot word list corresponding to the multiple virtual scenes is prepared, and the hot word list includes scene hot words corresponding to the multiple virtual scenes respectively.
[0389] Optionally, in order to obtain more abundant scene hot words, this goal can be achieved by designing a hot word management system, which has pre-processed all game assets and can calculate the position and direction of the target virtual character when the player interacts in real time. A hot word library (or hot word list) is generated based on the asset pre-processing, which contains scene hot words for various virtual scenes.
[0390] In some embodiments, virtual elements existing in the object perception range are determined from the first virtual scene where the target virtual character is located based on the object perception range, that is, virtual elements within the object perception range are determined.
[0391] In an illustrative manner, when the virtual scene changes, the hotword management system can automatically adjust the required scene hotwords, and according to the current scene, first select candidate scene hotwords from a predefined hotword library, and then perform a secondary selection according to the player's field of view (object perception range).
[0392] Optionally, taking the field of view as an example of the object perception range, when determining the field of view of the target virtual character, the field of view is determined by the camera's frustum. The frustum is a spatial region defined by the camera position and field of view (FOV). To improve efficiency, spatial segmentation techniques such as quadtree, octree, or grid can be used to organize virtual elements in the virtual scene. These data structures can help quickly determine which virtual elements are within the field of view. After the spatial range is determined, frustum culling will be performed to determine which virtual elements need to be rendered by calculating which virtual elements are within the frustum.
[0393] Indicatively, six planes (upper, lower, left, right, front, and back) are extracted from the camera's view cone to extract the view cone plane. For each virtual element, its bounding box is used to test the view cone plane. If the bounding box is completely outside the view cone, the element name, action name, etc. of the virtual element do not need to be loaded into the first scene hotword.
[0394] Indicatively, based on the result of frustum culling, it is determined which virtual elements' corresponding element names, action names, etc. need to be loaded into the first scene hot words to be used, and which virtual elements' corresponding hot words can be unloaded from the memory. Optionally, thresholds for hot word loading and unloading can also be set to avoid frequent hot word loading and unloading operations.
[0395] Hierarchical Frustum Culling: Combined with the space segmentation technology, the element names and action names of the virtual elements in the large area are first unloaded, and then the element names and action names of the virtual elements in the small area are unloaded.
[0396] Among them, scenario hot words are words frequently used in specific application scenarios. By setting scenario hot words, the system can recognize these words more accurately, thereby improving the overall recognition accuracy. These hot words either only appear in a specific game level of a certain game, or are words frequently used by players in a specific game level. These words are difficult to translate into text based on pronunciation alone, or these words are used very frequently, which affects the user experience.
[0397] Optionally, an initial hot word list is collected, which includes the following contents.
[0398] (1) Scene-specific object terms: such as “Area A”, “virtual stable”, “virtual crash site”, virtual “oil tank truck”, etc.
[0399] (2) Scene-specific character words: such as “virtual hero 1”, “virtual soldier 2”, “virtual monster”, etc.;
[0400] (3) Common commands: such as "attack", "defense", "jump", "run", etc.;
[0401] (4) Specific task words: such as "open the door", "unlock the treasure chest", "dialogue", etc.;
[0402] (5) Environmental interaction words: such as "view map", "use props", "save game", etc.
[0403] Step 830, matching the hot word list.
[0404] Illustratively, after determining the virtual element within the object perception range, the hot word list is matched to determine the scene hot word corresponding to the virtual element as the first scene hot word.
[0405] Optionally, taking the example that multiple virtual scenes correspond to one hot word list respectively, a first hot word list corresponding to the first virtual scene is determined based on the first virtual scene, and multiple first scene hot words are determined from the first hot word list based on the virtual elements within the object perception range. For example, the element name describing the virtual element is used as the first scene hot word, or the action name of the interactive action interacting with the virtual element is used as the first scene hot word, etc.
[0406] Optionally, taking multiple virtual scenes corresponding to one hot word list as an example, multiple first scene hot words are determined from the hot word list based on virtual elements within the object perception range. For example, the element name describing the virtual element is used as the first scene hot word, or the action name of the interactive action interacting with the virtual element is used as the first scene hot word, etc.
[0407] Step 840, ASR system hot word weight.
[0408] Schematically, the ASR system, or Automatic Speech Recognition system, is a commonly used system in speech recognition technology, which aims to convert instructions in the form of human voice into instructions in the form of text. It involves the process of processing, analyzing and understanding sound signals so that computers can "understand" and perform corresponding operations.
[0409] The ASR system is the pre-trained natural language analysis model mentioned above. The ASR system includes the following key components.
[0410] (1) Speech recognition acoustic model (the acoustic network in the above natural language analysis model)
[0411] The acoustic model is used to map the feature vector of the audio signal to the basic speech unit (such as phoneme). It is usually trained through machine learning techniques (such as deep neural network), using a large amount of labeled speech data to learn the relationship between speech signals and phonemes.
[0412] (2) Speech recognition language model (language network in the above natural language analysis model)
[0413] The language model is used to predict the probability of a word sequence, helping the system select the most likely one among multiple possible word sequences. Usually, certain language model information is already included in the end-to-end network, and decoding constraints based on statistical methods (n-gram model) are used to improve accuracy.
[0414] (3) Speech recognition hot words (the first scenario hot words in the preset dictionary in the above natural language analysis model, and can also be selected words including common words)
[0415] Hot words refer to words or phrases that require special attention and priority recognition in a specific virtual scene. For example, in a game scene, "virtual stable" or "virtual crash site" are hot words. Hot words usually require higher recognition accuracy, so there may be special models or algorithms to handle them. These hot words often have scene associations with virtual scenes in the game, and their corresponding texts cannot be written based solely on pronunciation.
[0416] (4) Speech recognition decoder (the decoding network mentioned above)
[0417] The decoder is one of the core components of the speech recognition system. It combines the output of the acoustic model and the language model and uses a search algorithm (such as the Viterbi algorithm) to find the most likely sequence of words. The decoder's task is to convert the feature vector sequence into the final text output.
[0418] In an optional embodiment, a weighted finite-state transducer (WFST) is used to process the relationship between states and transitions. In the speech recognition process, states can represent different meanings based on different inputs, and transitions represent the transformation between states. For example, when processing multiple speech units, each speech unit can be regarded as a state, and the transformation between speech units (such as the transition from speech unit 1 to speech unit 2) is the transformation between states; in the analysis of candidate vocabulary sequences, each word in the candidate vocabulary sequence can be regarded as a state, and the transformation between words (such as the transition from word 1 to word 2) is the transformation between states, etc.
[0419] Schematically, WFST can assign a weight to each transition. The weight usually represents the cost or probability of the transition, such as the weight of the transition from speech unit 1 to speech unit 2; or the weight of the transition from vocabulary 1 to vocabulary 2, etc.
[0420] In some embodiments, WFST is used to represent various models and transformation relationships, such as acoustic models, language models, preset dictionaries, etc. In speech recognition systems, WFST is used in speech recognition to combine different models (such as acoustic models, language models, and preset dictionaries) into a unified decoding network that can efficiently search for the most likely word sequence.
[0421] Schematically, the acoustic model represents the probability of phonemes or other basic speech units as WFST; the language model represents the probability of candidate vocabulary sequences as WFST; the preset dictionary represents the mapping between vocabulary and its corresponding phoneme sequence (speech unit sequence) as WFST. The relationship between the acoustic model, language model and WFST is: In the speech recognition system, the acoustic model, language model and dictionary are usually represented as independent WFSTs, and then they are merged into a comprehensive WFST through specific combination and optimization algorithms. This comprehensive WFST is used for decoding and recognition.
[0422] Optionally, the acoustic model (H) maps multiple acoustic features corresponding to the natural language command in the form of speech to phonemes or other basic speech units to obtain multiple speech unit sequences. This mapping can be represented as a WFST, called H. The context dependency model (C) is usually used in the acoustic model to obtain a speech unit sequence that is more closely related to the context.
[0423] The preset dictionary (L) maps vocabulary to a sequence of speech units to obtain multiple candidate vocabulary sequences. This mapping can also be expressed as a WFST, called L.
[0424] The language model (G) represents the probability of a candidate word sequence as a WFST, called G.
[0425] By combining H, C, L and G, the HCLG corresponding to the speech recognition system is obtained, that is, a comprehensive WFST is generated, called HCLG. This comprehensive WFST represents the complete mapping from feature vectors to word sequences. During the decoding process, the ASR system uses the HCLG, a comprehensive WFST, to search for the most likely candidate word sequence as the command analysis result.
[0426] Schematically, a natural language command in speech form is converted into multiple acoustic features, and the acoustic model (H) and context dependency model (C) in HCLG are used to score the multiple acoustic features to generate phoneme probabilities; then the preset dictionary (L) in HCLG is used to map the phoneme probabilities to word sequences; finally, the command analysis result is obtained through analysis by the language model (G), such as finding the most likely word sequence as the command analysis result through a decoder through a search algorithm (such as the Viterbi algorithm).
[0427] In an optional embodiment, it is necessary to prepare voice data and text data related to the game level. These data will be used to train the acoustic model and language model. Voice data includes recorded and collected audio data in the form of voice that players may use in the game; text data includes text previously entered by players and text related to the game level entered by the large language model, such as level names, action instructions, etc.
[0428] Optionally, the Mel Frequency Cepstral Coefficients (MFCC) and fbank filter bank extract features from natural language commands in speech form to train the acoustic model. After using the extracted features to train the acoustic model in acoustic model training, the model probability at the level of Connectionist Temporal Classification (CTC) will be obtained. The language model (LM) is trained using the data described above. Commonly used tools include SRI Language Modeling Toolkit, Ken Language Model Toolkit (KenLM), etc. Build a pronunciation dictionary containing all possible words and their corresponding phoneme sequences. The pronunciation dictionary format contains one word and its phoneme representation per line. Then use the composition tools arpa2fst, fstcompose, fstreduced, etc. to construct the speech recognition decoding word graph and obtain them respectively.
[0429] Step 850: perform input correction based on the large language model.
[0430] In an optional embodiment, a large language model is used to generate sample data for training the ASR system, so that the ASR system is trained with the sample data before the ASR system is applied; or, the ASR system is periodically trained with the sample data during the application of the ASR system; or, the ASR system is trained with the sample data in real time during the application of the ASR system, etc.
[0431] Schematically, a large language model (LLM) refers to a natural language processing (NLP) model that has been trained with large-scale text data and has high complexity and high performance. Large language models use deep learning technology, especially the Transformer architecture, to understand and generate natural language commands. Large language models perform well in various NLP tasks, such as text generation, translation, question answering, summarization, etc. Large-scale data training: Large language models are usually trained using a large amount of text data, which may include billions or even trillions of words. Large-scale data enables the model to learn richer language features and patterns. Complex model architecture: These models usually have complex architectures, including multi-layer transformers, attention mechanisms, etc. to capture the nuances in the language. High performance: Due to the use of large amounts of data and complex model architectures, large language models usually have high understanding and generation capabilities and can perform well in various language tasks.
[0432] The large language model has prompts, which refer to the initial input text provided to the model when using the large language model for natural language processing tasks. The design and selection of prompts have an important impact on the output quality and accuracy of the model. By carefully designing prompts, the model can be guided to generate text that is more in line with expectations. In other words, prompts provide contextual information for the model, helping the model understand the user's intentions and generate relevant text. For example, given a prompt "write an article about climate change", the model will generate an article related to climate change. The output style and format can also be controlled by prompts. For example, prompts can contain specific instructions or format requirements to help control the output style and format of the model. For example, a prompt of "explain quantum mechanics in concise language" will guide the model to generate concise and easy-to-understand explanations.
[0433] In addition, the prompt can also include the content of previous conversations to help the model understand the context of the current conversation and generate a more coherent response. The design of prompts Effective prompts need to consider the following aspects: Clear instructions; prompts should contain clear instructions to tell the model what task needs to be completed. For example: The prompt "Please summarize the main points of the following article" is clearer than the prompt "summarize". Prompts can also provide enough contextual information to help the model understand the task. For example: In a dialogue system, prompts can include the content of previous conversations. In addition, if there are specific format requirements for the output, they can be clearly stated in the prompt. For example: The prompt is "Please list the answers to the following questions in list form."
[0434] Based on the player's in-game instructions, dialogues, descriptions, etc. Provide context: Give the model enough contextual information so that it can generate reasonable speech input. Use examples: Guide the model to generate similar outputs by providing some examples. Provide clear formatting: Make sure it is clearly stated in the prompt words.
[0435] In some embodiments, the process of generating sample data based on a large language model is described as follows.
[0436] For example, suppose there is an adventure game in which players can command non-player characters through natural language commands in the form of voice. Therefore, it is hoped that the large language model can generate some possible sample data to conduct more targeted training of the natural language analysis model.
[0437] Optionally, preset instruction templates can be spliced to generate text corpus. For example: suppose the template is a string containing placeholders for inserting specific player instruction parameters. Placeholders can be represented by {}. For example: "{player_name} goes to {direction}{location}{action}", which means that the player name (such as the name of a non-player character) goes to a certain direction, a certain place and takes a certain action; suppose there are the following player instruction parameters: player_name = "Teammate No. 1 goes"; direction = "front"; location = "in the forest"; action = "throw a grenade"; the splicing instructions get template = "Teammate No. 1 {player_name} is at {location}{action}", that is: Teammate No. 1 goes to the forest in front and throws a virtual grenade.
[0438] Alternatively, if the player is a player in an adventure game, he can use natural language commands to command a virtual character. For example, the command is: 1. Go north; 2. Attack the enemy in front; 3. Use healing potion; 4. Open the backpack; 5. Check the map. This command can command the virtual character.
[0439] Based on this example instruction, sample data of some possible speech forms can be generated based on the large language model as data for training the natural language analysis model.
[0440] Indicatively, if you want the generated voice input to be more specific, you can provide more contextual information. For example, the prompt words provided to the large language model are "You are a player in an adventure game, your character is a warrior, and you are currently in the forest. You can command your character by voice input. Here are some example commands: 1. Go north; 2. Attack the enemy in front; 3. Use healing potion; 4. Open the backpack; 5. Check the map; Current game status: Role: Warrior; Location: Forest; Mission: Find hidden treasure; Now, please generate some possible player voice inputs."
[0441] Indicatively, if you want to generate more diverse voice input, you can add more examples and different scenarios to the prompt words. For example: the prompt words provided to the large language model are "You are a player in an adventure game, your character is a warrior, and you are currently in the forest. You can command your character through voice input commands; here are some example commands: 1. Go north; 2. Attack the enemy in front; 3. Use healing potions; 4. Open the backpack; 5. Check the map; 6. "Pick up the gems on the ground", 7. "Talk to the villagers", 8. "Enter the cave"; Current game state: Warrior; Location: Forest; Mission: Find hidden treasures; Now, please generate some possible player voice inputs" and so on.
[0442] Optionally, based on the above prompt words, the large language model may generate the following text: 1. Go east; 2. Use fireball to attack the enemy; 3. Check the task log; 4. Drink a bottle of magic potion; 5. Search the nearby area; 6. "Trade with the merchant", 7. "Enter the castle", 8. "Equip a new sword", etc.
[0443] Through the above process, a large amount of text simulating the player's speech in the scene can be generated through the human speaking habits built into the large language model, and the text can be used as sample data; and / or, the text can be converted into speech to obtain speech input as sample data, so as to use richer sample data to train the natural language analysis model.
[0444] In an optional embodiment, the text generated by the large language model is used as sample data, and the language model therein is updated by the sample data before being applied in the ASR system.
[0445] Step 860, activating the language model of the current game level.
[0446] In an optional embodiment, the language model (G) in the HCLG includes multiple language models, and the multiple language models correspond to multiple virtual scenes respectively. Based on the first virtual scene in which the target virtual character is located, the first language model corresponding to the first virtual scene is determined from the multiple language models (that is, the first language sub-network corresponding to the first virtual scene is determined from the language network as mentioned above).
[0447] Optionally, multiple language models are models obtained after training with corresponding virtual scenes. In the process of training the language model using the text generated by the large language model as sample data, the texts corresponding to the multiple virtual scenes are obtained, and the language models corresponding to the virtual scenes are trained with the texts.
[0448] Step 870: The ASR system selects a language model when decoding.
[0449] Illustratively, when the ASR system performs decoding processing, a first language model corresponding to the first virtual scene is selected so that the content input by the sound model is analyzed through the first language model.
[0450] Step 880, speech recognition decoding.
[0451] Schematically, multiple speech units corresponding to natural language commands are obtained through the acoustic model in the ASR system, and the speech unit is the basic building block of vocabulary pronunciation; the language model in the ASR system analyzes the sequence matching relationship between the multiple speech units and the selected words in the preset dictionary, and the command analysis result in text form is obtained, which is the decoding result. The selected words include at least multiple first-scene hot words and common words.
[0452] In an optional embodiment, when the target virtual character controlled by the player moves between different buildings, the spatial query logic will provide complete location information to help the player quickly locate the position of the enemy virtual character. For example: the target virtual character is outside the motel and hears virtual gunshots coming from the second floor of the motel. The system will prompt "the virtual gunshot is on the second floor of the motel". Players move in the same building: When the player moves in the same building, the spatial query logic will simplify the feedback information and only provide information about the sub-level areas. For example, the target virtual character is on the first floor of the motel and the enemy virtual character is on the second floor. The system will prompt "the enemy is on the second floor". Players are in the same room: When the target virtual character and the enemy virtual character are in the same room, the spatial query logic will provide more accurate orientation information to help players react quickly. For example, the player and the enemy are both on the second floor of the motel, and the system will prompt "the enemy is in the right front position". Through hierarchical feedback information, it helps players to understand the location of the enemy virtual character more intuitively, and provide the most relevant location information according to the different locations of the players.
[0453] In an optional embodiment, in game speech recognition, loading different language models and scene hot words according to different virtual scenes can significantly improve the accuracy and response speed of speech recognition. The following are steps and methods to achieve this goal.
[0454] First, it is necessary to be able to identify the virtual scene in the current game. The loading of the language model decoding word graph will be achieved through the following methods: (1) Game state detection: by detecting the state variables of the game (such as player position, task progress, current activity, etc.), the first virtual scene where the current target virtual character is located is determined; (2) Event triggering: switching scenes according to specific events (such as entering a certain area, starting a certain task), so as to determine the first virtual scene where the target virtual character is located; (3) Player input: the player can actively inform the current first virtual scene through voice or other input methods.
[0455] Different language models are trained or selected for different virtual scenarios. For example, the language model in the combat scenario can focus more on words and phrases related to the virtual combat, while the language model in the exploration scenario can focus more on words related to navigation and interaction.
[0456] Among them, a set of scene hot words is generated according to the virtual elements (such as virtual items, virtual characters, etc.) contained in the line of sight. These scene hot words are the most frequently triggered or important words in the current player scene. For example: in a battle scene, hot words may include "enemy", "attack", "retreat", etc., while in a transaction scene, hot words may include "buy", "sell", "item", etc.
[0457] In an illustrative manner, the corresponding language model and scene hot words are dynamically loaded according to the recognized virtual scene.
[0458] Among them, when a scene change is detected, the language model corresponding to the virtual scene is switched, which can be achieved by calling the API of the speech recognition engine; in addition, when a scene change is detected, the currently used hot word list is updated so that the speech recognition engine can recognize these words more accurately.
[0459] In an optional embodiment, in the game, according to the virtual scene where the target virtual character is located, the decoding strategy of the speech recognition system can also be dynamically adjusted to improve the accuracy of speech recognition and user experience. The following are some methods and steps.
[0460] Indicatively, the player's current scene is first identified and classified through information such as game status, player location, and current task.
[0461] Optionally, different decoding strategies are defined according to different virtual scenarios. These strategies include: (1) vocabulary adjustment: adjusting the vocabulary of speech recognition according to the virtual scenario, that is, the selected vocabulary in the above-mentioned preset dictionary; (2) language model adjustment: using different language models to process different virtual scenarios; (3) confidence threshold adjustment: adjusting the confidence threshold of speech recognition according to the virtual scenario, such as manually raising the confidence threshold. Only when the predicted probability is higher than the confidence threshold, the candidate vocabulary sequence corresponding to the maximum preset probability is used as the instruction output result.
[0462] Among them, by dynamically adjusting the decoder during the game running process, the decoding strategy of the speech recognizer can be dynamically adjusted according to the virtual scene where the target virtual character is located, thereby improving the flexibility of decoding.
[0463] In an optional embodiment, after the ASR system is launched, the language model (or the sound model, etc.) can be continuously optimized and adjusted according to the actual usage of the players, so that it can be more adapted to the needs of different virtual scenes. Optimizing the language model includes the following operations: (1) Hot word adjustment: Based on player feedback and usage data, the text recognized by the ASR system is collected and automatically rewritten using the correction engine. The hot word list is dynamically adjusted through the rewritten player text to ensure that it covers the most commonly used and important vocabulary, so that the hot words of the first scene can be obtained more accurately in the future; (2) Model update: Retrain the language model and / or sound model and adjust the parameters of the model; (3) Testing and deployment: The updated model undergoes sufficient control tests to ensure its performance and stability. Once the test passes, the trained model can be deployed in the game; (4) Continuous iteration: Speech recognition is a continuous improvement process. Feedback is regularly collected and analyzed, and then the model is updated to adapt to player behavior and changes in the game. Test in the actual game environment, collect data and feedback, and continuously iterate and improve the speech recognition system to improve the accuracy of command analysis results.
[0464] In an optional embodiment, by adjusting the language model of speech recognition according to the virtual scene, it is helpful to better understand and parse the natural language commands in the form of speech, select an appropriate language model from the predefined language model library, and when the scene changes, the language model for analysis can be automatically adjusted. This process can be achieved through a preset language model management system. In addition, the system can also establish an effective feedback mechanism, players can provide feedback and correct recognition errors or add new scene hot words through the cloud-based large language model. These feedbacks will update and improve the natural language analysis model online, and adjust and update the hot words and language models in time. In addition, in order to ensure the reliability and stability of the system, the existing large language model can also be used for offline testing and evaluation. Combined with the system automatic testing and player feedback data in different scenarios, the hot words and language models are further optimized according to the test feedback data to meet the needs of different scenarios. For example: when a player enters the interior of a city room, "turn on the lights", "turn off the lights", "turn up the temperature", etc. may be the scene hot words of the virtual indoor scene.
[0465] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.
[0466] In an optional embodiment, taking a virtual soldier including a main virtual character and a non-player virtual character in a virtual environment as an example, the process of obtaining a command analysis result in text form based on a natural language command in voice form is described as follows.
[0467] Indicatively, Fig. 9 As shown, the virtual environment includes a main virtual character 910 manually controlled by a player, a virtual soldier 911 (called No. 1) and a virtual soldier 912 (called No. 2) controlled by voice; if the main virtual character 910 is used as the target virtual character.
[0468] After receiving the natural language command, it is determined that the first virtual scene in which the main virtual character 910 is located is a "virtual outdoor scene", and multiple first scene hot words corresponding to the first virtual scene are obtained, including "truck", "virtual grass", "virtual house", "enter", "open the truck", "defense", "desert", "attack", "oak tree", "stable", etc.
[0469] Afterwards, the natural language command is converted into a text-based command analysis result based on multiple hot words in the first scenario, and the command analysis result is obtained as "No. 1, you go to the front Defense The command analysis result can specifically command the virtual soldier 911 to move forward and take a defensive posture to prevent the enemy virtual character from attacking the main virtual character first; if there is no constraint of the hot words in the first scene, it is easy to recognize the natural language command as "No. 1, you go to the front VoiceThis erroneous recognition result will command No. 1 to shout forward or make other vocal actions, thus affecting the player's virtual combat plan.
[0470] Alternatively, based on multiple hot words in the first scenario, the natural language command is converted into a command analysis result in text form, and the command analysis result is obtained as "No. 2, you go desert The command analysis result can be used to specifically command the virtual soldier 912 to move to the desert to check whether there are virtual enemy characters or virtual treasure chests and other virtual elements; if there is no constraint of the hot words in the first scene, it is easy to recognize the natural language command as "No. 2, you go Mountains "Explore it", which causes the virtual soldier 912 to move to a wrong position, and the human-computer interaction efficiency is low.
[0471] Alternatively, based on multiple hot words in the first scenario, the natural language command is converted into a command analysis result in text form, and the command analysis result is obtained as "No. 1, No. 2, give me attack ", the command analysis result can be used to specifically command virtual soldiers 911 and 912 to carry out virtual attacks on nearby attackable virtual characters; if there is no constraint of the hot words in the first scene, it is easy to recognize the natural language command as "No. 1, No. 2, give me supply ", which causes virtual soldier 911 and virtual soldier 912 to make erroneous behaviors that do not meet the player's expectations, which can neither protect the main virtual character nor easily allow the enemy virtual character to win the virtual game.
[0472] Alternatively, based on multiple first-scene hot words, the natural language command is converted into a command analysis result in text form, and the command analysis result is obtained as "No. 1, go find a nearby oak tree". The command analysis result can specifically command the virtual soldier 911 to find an oak tree in the nearby area so that the game task can be completed or an oak tree that can be used to avoid attacks can be found. If there is no constraint on the first-scene hot words, the natural language command is easily recognized as "No. 1, go find a nearby oak tree". Project Book ", causing the virtual soldier 911 to search nearby for virtual elements that do not meet the player's expectations and fail to meet the player's virtual combat needs.
[0473] Alternatively, the natural language command is converted into a command analysis result in text form based on multiple first scenario hot words, and the command analysis result is obtained as "No. 1, No. 2, go over there" stable OK", the command analysis result can be used to specifically instruct virtual soldiers 911 and 912 to search for nearby stables and move to the stables in a targeted manner; without the constraints of the hot words in the first scene, it is easy to recognize the natural language command as "No. 1, No. 2, go over there , right awayOK", which causes virtual soldier 911 and virtual soldier 912 to mistakenly believe that the main controlling virtual character currently hopes to complete the virtual game by himself, and therefore cannot provide better game assistance to the main controlling virtual character.
[0474] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.
[0475] Indicatively, Fig.10 As shown, the process of command analysis based on the game terminal interface is explained. The target virtual character is the main control virtual character 1010, and the first virtual scene in which it is located is a virtual battle scene, which also includes a non-player character 1020 (soldier A), and the received natural language command is "xiaobingA geiwogongji" in voice form; based on the first virtual scene corresponding to multiple first scene hot words, the natural language command is analyzed through the constraints of the first scene hot words, and the command analysis result is obtained as "soldier A, give me attack", that is, "gongji" is determined to be "attack" instead of "supply", avoiding errors in the analysis of natural language commands, and obtaining more accurate command analysis results, so as to better command non-player characters.
[0476] To sum up, taking the first virtual scene in which the target virtual character is located as a condition for analyzing natural language commands can provide more targeted text information for the analysis process through the first scene hot words, reduce the analysis ambiguity caused by the poor quality of natural language commands, not only help to understand natural language commands more accurately, but also make the command analysis results more closely matched with the first virtual scene, improve the accuracy of the command analysis results, greatly avoid invalid or erroneous commands to non-player characters, provide players with a more efficient and comfortable operating environment, and improve human-computer interaction efficiency; a small number of first scene hot words can also reduce the amount of data processing during command analysis, thereby improving command analysis efficiency.
[0477] In the embodiment of the present application, the player's voice can be perceived in the game to be more adapted to the game scene, thereby bringing higher accuracy in player intention recognition; it can make it easier for players to interact and operate in the game. Through the player's voice instructions, the player can achieve a faster and more natural operation method, improve the fluency and fun of the game; it can also make the virtual characters and NPCs in the game more intelligent and realistic. Players can get a more realistic game experience and enhance the interactivity of the game by talking to the characters in the game. In addition, players can complete various operations more quickly in the game, such as switching virtual props, releasing skills, etc. Compared with traditional key operations, faster response and execution, more accurate perception of user intentions, improve the efficiency of game operations, and bring competitive advantages to game developers. At the same time, the above-mentioned new interaction process is better distinguished from other games, which helps to attract more players. In short, the above-mentioned technical solution can enhance the game experience, enhance the interactivity of the game, improve the efficiency of game operations, expand the game audience, and enhance the competitiveness of the game.
[0478] An exemplary embodiment of the present application provides a method for controlling a virtual character. The method includes:
[0479] Displaying at least one of a main control virtual character and an NPC in a virtual environment;
[0480] Exemplarily, the main virtual character is a virtual character that the user directly controls in the virtual environment. There are also one or more non-player characters (NPC) in the virtual environment; the NPC and the main virtual character belong to the same virtual camp and are teammates of the main virtual character. The NPC follows the main virtual character in the virtual environment. In addition, the NPC can also be a follower character, a pet character, etc. controlled by the main virtual character. In addition, the NPC can also be a neutral character, and only cooperates with the main virtual character when certain conditions are met.
[0481] Obtain natural language commands;
[0482] Exemplarily, the natural language command includes the natural semantics of commanding the NPC; the NPC is commanded in the virtual scene through the natural language command, and the natural language command can be directly input by the user or extracted from the voice information input by the user. The present application does not limit the method of obtaining the natural language command.
[0483] The natural language command includes at least one of a behavior intention and a scene entity; the behavior intention is used to indicate the type of virtual activity performed by the NPC, such as indicating what type of virtual activity is performed in the virtual environment; the scene entity includes a first scene object, such as indicating what scene object in the virtual environment the virtual activity is performed on. The behavior intention and scene entity included in the natural language command are partial words in the natural language command.
[0484] In one example, the natural language command controls the NPC in terms of what type of virtual activity to perform and for what scene object in the virtual environment to perform the virtual activity; it is a complex instruction to control the NPC.
[0485] Controlling NPCs to perform virtual activities in response to natural language commands;
[0486] Exemplarily, controlling the NPC to perform virtual activities is determined based on the environmental perception information of the main virtual character and / or NPC. In some examples, the main virtual character and / or NPC obtains environmental perception information within the perception range of vision, hearing, movement trajectory and other perception methods. Exemplarily, the virtual activity is close to the current perception of the virtual environment by the main virtual character and / or NPC, and the NPC is controlled with reference to the current perception of the virtual character and / or NPC of the virtual environment. Exemplarily, the NPC has autonomous behavior capabilities, and the user only directs the NPC. The user gives commands, and the NPC understands and executes the commands based on its own autonomous behavior capabilities.
[0487] To sum up, the method provided in this embodiment determines the virtual activities that need to be performed by facing complex instructions for controlling the NPC in terms of what type of virtual activities to perform and for what scene objects in the virtual environment to perform the virtual activities, with reference to the environmental perception information of the master virtual character and / or the NPC; and ensures that the NPC and the master virtual character perform virtual activities in collaboration with each other by using the environmental perception information of the master virtual character and / or the NPC as a reference.
[0488] Fig.11 A flowchart of a method for controlling a virtual character provided by an exemplary embodiment of the present application is shown. The method comprises:
[0489] Step 1101: Obtain a spatial data set;
[0490] The spatial dataset of the virtual environment includes visual labels of scene objects, and the visual labels are used to describe the visual characteristics of the scene objects in at least one dimension, such as describing the scene objects in dimensions such as material, transparency, color, and shape.
[0491] Step 1102: displaying at least one of a main control virtual character and an NPC in a virtual environment;
[0492] Exemplarily, the main control virtual character is a virtual character that the user directly controls in the virtual environment. There are also one or more NPCs in the virtual environment; the NPCs and the main control virtual character belong to the same virtual camp and are teammates of the main control virtual character.
[0493] Step 1103: Acquire a natural language command in voice form;
[0494] The natural language command includes natural semantics for commanding the NPC, and the command information includes behavioral intention and scene entity; the behavioral intention is used to indicate the type of virtual activity performed by the NPC, and the scene entity is used to indicate the target entity of the virtual activity;
[0495] Step 1104: converting the natural language command in voice form into the natural language command in text form;
[0496] Exemplarily, speech-to-text (Automatic Speech Recognition, ASR) processing is performed on the natural language command in the form of speech to determine the natural language command in the form of text. Speech-to-text processing generally includes calling components such as acoustic models and language models to recognize the pronunciation, vocabulary, grammatical structure and other information of the natural language command in the form of speech, and convert it into a natural language command in the form of text.
[0497] Step 1105: performing intent recognition on the natural language command to obtain a first classification label;
[0498] Exemplarily, the first intention indication corresponding to the first classification label includes a command intention for a non-player character on an intention dimension; for example, at least one of the following: indicating which of a plurality of non-player characters the non-player character commanded by the command text is, indicating whether the virtual activity performed by the non-player character is related to initiating a virtual attack, and indicating what kind of virtual activity the non-player character performs.
[0499] Step 1106: Perform entity recognition on the natural language command to obtain a target entity;
[0500] The target entity is determined from the virtual environment in combination with the environmental perception information, which includes information perceived by at least one of the main virtual character and the non-player character from the virtual environment. The target entity may be the entity currently seen by the player, or may be an entity that the player has seen in the past.
[0501] Step 1107: In response to the natural language command, the NPC is controlled to perform a virtual activity according to the first intention corresponding to the first classification label, or the NPC is controlled to perform a virtual activity associated with the target entity, or the NPC is controlled to perform a virtual activity associated with the target entity according to the first intention corresponding to the first classification label;
[0502] Exemplarily, the NPC is controlled to perform a virtual activity according to the first classification tag and / or the instruction of the target entity.
[0503] Step 1108: reporting the NPC's feedback information;
[0504] Exemplarily, the game application can generate corresponding feedback information in real time according to the environmental perception information of the non-player character and make a broadcast. For example, when the environmental perception information of the non-player character triggers the broadcast condition, the game application can generate corresponding feedback information in real time according to the environmental perception information that triggers the broadcast condition; or, when the non-player character or the main control virtual character triggers the broadcast condition, the game application can generate corresponding feedback information in real time according to the environmental perception information of the non-player character.
[0505] The preprocessing stage of step 1101 can be implemented as follows:
[0506] Sub-step 1: Obtaining attribute text of scene objects in the virtual scene;
[0507] Exemplarily, attribute text is used to introduce the inherent attributes of scene objects in a virtual scene; on the one hand, attribute text realizes the description of scene objects in a textual mode, and while describing scene objects, it provides semantic information for the prediction of the visual labels of scene objects. On the other hand, attribute text is the description of scene objects in a virtual scene. In the face of a large number of scene objects in a virtual scene and the existence of reused object models, it can more accurately describe the inherent attributes of scene objects in a virtual scene.
[0508] In an optional implementation, the attribute text includes at least one of a name of the scene object in the virtual scene and a size of the scene object in the virtual scene.
[0509] Sub-step 2: obtaining an appearance image of a scene object in a virtual scene;
[0510] Exemplarily, the appearance image is used to describe the style of the scene object; the appearance image carries the appearance style of the scene object such as color, texture, shape, etc., and the relative position relationship between each sub-part in the picture mode, and can fully describe the scene object from the picture mode (or visual mode). In an optional implementation, the appearance image of the scene object includes images obtained by observing the scene object from at least two perspectives.
[0511] Sub-step 3: Call the multimodal model to perform prediction on the attribute text and appearance image of the scene object to obtain the visual label of the scene object;
[0512] Exemplarily, the multimodal model has the ability to perform model prediction on text information and image information of different modalities. In this embodiment, the input parameters of the multimodal model are the attribute text and appearance image of the scene object; the multimodal model predicts the visual label of the scene object from the two modalities of text modality and image modality, and the visual label is used to describe the visual features of the scene object in at least one dimension.
[0513] Optionally, the multimodal model includes a visual question answering model; the question statement carries attribute text of the scene objects; on the one hand, the question statement is used to guide the visual question answering model to convert the appearance image into the visual label of the scene object; on the other hand, while guiding the conversion of the visual label of the scene object, the question statement provides supplementary information of the scene object in text form.
[0514] Get the expected information of scene objects;
[0515] Exemplarily, the expected information is used to indicate the description dimension of the scene object expected in the visual tag, and / or the expected format of the visual tag; in one example, the expected information is used to indicate that the description dimension of the scene object expected in the visual tag includes but is not limited to at least one of the type description, material, transparency, color, surface features, and shape of the scene object. In another example, the expected information is used to indicate that the expected format of the visual tag is at least one of the following: Comma Separated Values (CSV), JavaScript Object Notation (JSON), and eXtensible Markup Language (XML).
[0516] Construct a question statement of the appearance image based on the expected information and attribute text;
[0517] Exemplarily, the first subpart of the question sentence is supplementary introduction information of the scene object, carrying attribute text of the scene object; the second subpart of the question sentence is an answer guidance sentence for the visual question answering model, carrying expected information.
[0518] Optionally, the spatial data set includes matching labels of scene objects in the virtual scene; correspondingly: performing pseudo-spoken rewriting on the visual labels of the scene objects to obtain matching labels that conform to spoken expressions of natural language.
[0519] Exemplarily, as described above, visual labels are used to describe the visual features of scene objects in at least one dimension, and the appearance image of the scene objects presents rich visual features of the scene objects. In the spoken expression of natural language, however, the description of scene objects cannot cover the visual features of scene objects in all dimensions. The purpose of performing a pseudo-colloquial rewrite on the visual labels of scene objects is to obtain a matching label that is closer to the spoken expression; it can be understood that performing a pseudo-colloquial rewrite can be to delete part of the content in the visual label, or to change the visual label to a label with the same semantics but different textual expression.
[0520] In an optional implementation, the pseudo-colloquial rewriting is performed by calling a natural language model; the visual labels of the scene objects are input into the natural language model, and matching labels that conform to the spoken expressions of the natural language are predicted. The pseudo-colloquial rewriting of the visual labels is implemented based on calling the natural language model. Exemplarily, the natural language model carries prior knowledge of the spoken expressions of the natural language. For example, the natural language model is implemented as a large language model (Large Language Model, LLM).
[0521] Optionally, the spatial dataset also includes spatial information of scene objects, and accordingly, also includes:
[0522] The spatial position of the scene object in the virtual scene is obtained, and the spatial position of the scene object is determined as auxiliary information of the visual label of the scene object.
[0523] Exemplarily, the spatial position of the scene object in the virtual scene is used to indicate the deployment of the scene object in the virtual scene, and the spatial information is used to indicate the position, size and other information of the scene object in the virtual scene after the scene object is deployed in the virtual scene.
[0524] Further, obtaining at least one of the coordinate position, orientation information, bounding box information, and cover point information of the scene object in the virtual scene;
[0525] Among them, the coordinate position (Location) is used to indicate the position of the scene object in the virtual scene, such as the coordinate information of the center point or preset point of the scene object in the virtual scene; the orientation (Rotation) is used to indicate the direction facing the scene object in the virtual scene, such as the direction facing the front of the scene object in the virtual scene; the bounding box (BoundingBox) is used to indicate the size of the scene object in the virtual scene; the cover points (Cover points) are used to indicate the recommended virtual character position points when the virtual character approaches the scene object, so that the scene object can mask the virtual character.
[0526] The intention recognition phase of step 1105 can be implemented as follows:
[0527] Sub-step 4: calling at least one hierarchical prediction network to perform intent recognition on the natural language command, and obtaining a first classification label of the natural language command intent to command the NPC;
[0528] Exemplarily, the hierarchical prediction network has the ability to predict the classification label corresponding to the command text. Exemplarily, the hierarchical prediction network includes at least two layers of sub-networks, the upper network and the lower network are cascaded, and the lower network further performs the prediction of the classification label according to the prediction result output by the upper network. At least two layers of sub-networks are constructed based on the tree structure of multiple classification labels. The hierarchical prediction network constructs at least two layers of sub-networks corresponding to the hierarchical structure for the tree structure of the classification labels, and splits the prediction task of a wide variety of classification labels into prediction sub-tasks performed by at least two layers of sub-networks, so as to reduce the complexity of classification prediction of each layer of sub-network.
[0529] In an optional implementation, each hierarchical prediction network in at least one hierarchical prediction network is used to predict the command intention of the command text for a non-player character on an intention dimension; the intention dimension includes: at least one of the subject dimension, the semantic dimension, and the behavior dimension. Exemplarily, in at least one hierarchical prediction network, the i-th layer subnetwork in each hierarchical prediction network is used to predict the first-level behavior label of the command text on an intention dimension, and the i+1-th layer subnetwork in the hierarchical prediction network is used to predict the second-level behavior label corresponding to the command text in the first-level behavior label, i is a positive integer; the high-level behavior label (such as the first-level behavior label) predicted by the high-level subnetwork (such as the i-th layer subnetwork) in the hierarchical prediction network includes multiple sublabels, or subordinate lower-level labels; the corresponding low-level subnetwork (such as the i+1-th layer subnetwork) needs to be called to further perform classification prediction (such as predicting the second-level behavior label). The prediction task of a wide variety of classification labels is split into prediction subtasks performed by at least two layers of subnetworks to reduce the complexity of classification prediction of each layer of subnetwork.
[0530] Furthermore, the intent dimension includes a subject dimension, the hierarchical prediction network includes a hierarchical subject prediction network, and the subject prediction network has the ability to predict the subject type in a natural language command; the subject type is used to indicate the identity of the NPC commanded by the natural language command; exemplarily, the subject prediction network is used to predict which of a plurality of non-player characters the non-player character commanded by the command text is, and the number of non-player characters commanded by the command text can be one or more.
[0531] And / or, the intent dimension includes a semantic dimension, the hierarchical prediction network includes a hierarchical semantic prediction network, the semantic prediction network has the ability to predict the semantic type in a natural language command; the semantic type is used to indicate the NPC's control method for initiating a virtual attack; exemplarily, the subject prediction network is used to predict whether the virtual activity performed by a non-player character commanded by the command text is related to initiating a virtual attack.
[0532] And / or, the intention dimension includes a behavior dimension, the hierarchical prediction network includes a hierarchical behavior prediction network, the behavior prediction network has the ability to predict the behavior intention type in the natural language command; the behavior intention type is used to indicate the behavior mode of the NPC for performing the virtual activity. Exemplarily, the behavior prediction network is used to predict what virtual activity the non-player character performs.
[0533] Accordingly, in step 1107, in response to the natural language command, the NPC is controlled to perform a virtual activity according to the first intention corresponding to the first classification label;
[0534] Exemplarily, the first intention corresponding to the first classification tag indicates a command intention for a non-player character in an intention dimension, such as at least one of the following: indicating which non-player character is commanded by the command text among a plurality of non-player characters, indicating whether the virtual activity performed by the non-player character is related to launching a virtual attack, and indicating what virtual activity the non-player character performs. According to the instruction of the first classification tag, the NPC is controlled to perform the virtual activity.
[0535] The target entity identification process of step 1106 can be implemented as follows:
[0536] Sub-step 5: querying and obtaining the target entity from the entity set of the virtual environment according to the target entity information of the target entity indicated by the natural language command and the environment perception information;
[0537] The target entity is determined from the virtual environment in combination with environmental perception information, and the environmental perception information includes information perceived by the main virtual character and / or the non-player character from the virtual environment.
[0538] Since the target entity information in natural language commands is usually vague, for example, a natural language command may be "move to the red truck", where the target entity information is "red truck", and there may be many red trucks in the virtual environment. It is impossible to accurately determine the target entity in the virtual environment based solely on the target entity information in the natural language command.
[0539] Therefore, the embodiment of the present application provides a method for determining the target entity by combining the target entity information and the environmental perception information. For the previous example, the "red truck" expressed by the player in the natural language command should be a red truck that the player can see. Therefore, combined with the field of view of the main control virtual character, the red truck within the field of view of the main control system character can be screened out from the multiple red trucks in the virtual environment. The red truck is the target entity indicated by the player in the natural language command.
[0540] Since natural language commands are issued by players based on their perception of the virtual environment, in order to accurately identify the target entity indicated in the natural language command, the game application will combine the environmental perception information when the player issues the natural language command to identify the target entity in the natural language command. Based on the player's perception of the virtual environment, the entity that the player can perceive in the virtual environment and is closest to the target entity information is inferred as the target entity.
[0541] Exemplarily, the virtual environment includes multiple candidate entities that match the natural language command, and the target entity is an entity selected from the multiple candidate entities based on the environmental perception information of the main virtual character or the non-player character. For example, the game application first selects multiple candidate entities that match the target entity information in the natural language command from the entity set, and then selects the entity that can be perceived by the main virtual character or the non-player character from the multiple candidate entities as the target entity based on the environmental perception information.
[0542] For example, the environmental perception information of the main virtual character may include: the pictures or entities that the main virtual character can see when observing the virtual environment, the environmental sound effects that the main virtual character can hear, the source direction of the environmental sound effects, the type and volume of the environmental sound effects, the perception information obtained by the main virtual character by using perception skills (for example, visual information, auditory information, sound wave information, light reflection information, etc.), and the perception information obtained by the main virtual character by using sensing props (for example, sensor signals from sensors, positioning signals from positioning props, etc.).
[0543] The environmental perception information of the non-player character may include: the images or entities that the non-player character can see when observing the virtual environment, the environmental sounds that the non-player character can hear, the source direction of the environmental sounds, the type and volume of the environmental sounds, the perception information obtained by the non-player character by using perception skills (for example, visual information, auditory information, sound wave information, light reflection information, etc.), and the perception information obtained by the non-player character by using sensory props (for example, sensor signals of sensors, positioning signals of positioning props, etc.).
[0544] It should be noted that the environmental perception information may include the information perceived by the controlling virtual character and the non-player character in real time when receiving the natural language command, and may also include the information perceived by the controlling virtual character and the non-player character in history before receiving the natural language command. That is, the target entity may be the entity currently seen by the player, or it may be the entity that the player has seen in the past. Therefore, the game application needs to combine the real-time environmental perception information and the historical environmental perception information of the controlling virtual character and / or the non-player character to identify the target entity.
[0545] For example, when the natural language command is "Let's go back to the hotel we just passed by", the game application needs to query the hotels that the main virtual character has passed by based on the historical environmental perception information of the main virtual character.
[0546] In an optional embodiment, the client queries and obtains the target entity from the entity set of the virtual environment according to the target entity information and the environment perception information indicated by the natural language command. In another optional embodiment, the client reports the natural language command to the server, and the server queries and obtains the target entity from the entity set of the virtual environment according to the target entity information and the environment perception information indicated by the natural language command.
[0547] Optionally, the game application infers the target entity based on the target entity information, the environment perception information and the pre-processed virtual environment data, wherein the pre-processed virtual environment data includes a set of entities in the virtual environment.
[0548] The entity set includes entity information of each entity in the virtual environment. The entity information includes at least one of the following: name, type, location, feature, text label, embedding vector of text label, image, embedding vector of image.
[0549] The text label may be at least one word obtained after segmenting a feature (for example, a descriptive text of an entity appearance), and the embedding model is called to obtain an embedding vector corresponding to each text label.
[0550] The image may include an image obtained by observing a three-dimensional model of the entity from at least one direction, for example, the image may include three views of the entity. The embedding vector of the image is the embedding vector of the image feature label. The image feature label is obtained by calling the multimodal model to perform feature recognition based on the image and text label of the entity.
[0551] Exemplarily, the entity set includes a first type of entity and a second type of entity, the first type of entity includes at least one of an area and a building, and the second type of entity includes a physical object. The text labels of the first type of entity are obtained by manual labeling. For example, the text label of a room of a building is manually labeled as: an area, a building, a floor, and a room. The text labels of the second type of entity are automatically generated by calling a large language model.
[0552] The method for generating text labels for the second type of entity may include: obtaining at least one image of the entity, inputting the at least one image into a large language model, and obtaining a description text of the entity; segmenting the description text to obtain at least one text label of the entity, each text label including a word obtained after the segmentation process; performing a vectorization operation on the text label to obtain an embedding vector corresponding to the text label.
[0553] The embedding vector is the high-dimensional vector data of the text label. By pre-generating the embedding vector of the text label, there is no need to repeatedly generate vectors for the text label of each entity when matching the target entity from the entity set based on the target entity information, which can improve the matching efficiency of the target entity information and the entity information.
[0554] It should be noted that this method does not use the description text directly as the text label, but uses the word segmentation result of the description text as the text label. This is because the description text is usually of different lengths. For longer description texts, their embedding vectors are less effective in actual search and similarity comparison. For example, when searching for "a small blue truck with rusty body" using "truck", "a small blue truck with rusty body" is likely to be ranked after "a car" in the recall results. In fact, for the searched "truck", no matter how complex its additional description is, "truck" should be searched first. Therefore, when performing similarity search, the comparison should be one feature word at a time, not the description text.
[0555] In addition, in order to further extract the image features of the entity, the method provided in the embodiment of the present application can also further extract hidden visual features in the image of the entity based on the text label and image of the entity. Taking the first entity as an example, the method includes: obtaining at least one perspective image of the first entity; obtaining at least one text label of the first entity; calling the multimodal model to extract the image features of the first entity based on at least one perspective image and at least one text label of the first entity, and obtain the image embedding vector of the first entity.
[0556] Since text labels are manually annotated, or generated by a large language model based on human language characteristics; and text labels extracted based on human language habits may ignore some features of the entity. For example, when manually annotating a text label for an oil drum, only more eye-catching text labels such as "metal" and "rust" may be annotated, while detailed features such as "rust", "blue", "yellow", and "paint peeling in the lower right corner" are ignored. Therefore, the method provided in the embodiment of the present application also uses a multimodal model based on the text label and entity image of the entity, so that the multimodal model extracts more image features from it; performs image similarity matching with the target entity based on the image features to improve the recognition accuracy of the target entity.
[0557] Among them, the training method of the multimodal model can be: inputting sample entity images and sample labels into the pre-trained multimodal model, fine-tuning the pre-trained multimodal model according to the loss of the predicted labels and sample labels output by the pre-trained multimodal model, and obtaining a multimodal model that can output image feature labels based on the input text labels and images, and the image feature labels include more detailed description texts proposed from the image. For example, the text description of "a metal oil barrel" can only be associated with the two text labels "metal" and "oil barrel", while the Clip (multimodal) model can also identify the hidden visual information in the image from the image of the oil barrel, such as "rusty", "blue", and "yellow" image feature labels, which are features that do not appear in the text labels. Therefore, combining the Clip model for image feature search can further improve the accuracy of entity search.
[0558] In an optional embodiment, the game application may query the target entity using the following method.
[0559] (1) Parse the natural language command to obtain target entity information; the target entity information includes at least one of the following: entity type, entity name, entity location, and entity feature.
[0560] Optionally, the game application calls the large language model to parse the natural language command to obtain target entity information of the target entity, and calls the embedding model to vectorize the target entity information to obtain a target embedding vector of the target entity information.
[0561] For example, if the natural language command is "Come to the blue truck in front", the target entity information that can be parsed by the large language model includes: the direction information "front" and the entity name "blue truck".
[0562] For example, named entity recognition technology in the field of natural language processing can also be used to extract target entity information from natural language commands. Named entity recognition technology is used to identify entities with specific meanings in texts, such as names of people, places, locative words, adjectives, etc.
[0563] For example, if the natural language command is "Go to the first floor of the motel and find a paper box behind the red sofa", the named entity recognition technology can identify: the architectural noun "the first floor of the motel"; the object nouns "sofa" and "paper box"; the locative word "behind"; the adjective "red". After logical construction, a hierarchical scene query call form with search type, content, locative word, and constraint information can be formed. Among them, the search type is used to narrow the data retrieval scope based on similarity matching, the search content is a specific entity description, and the adjective will be merged into the description text for query together, and the locative word is used to form a constraint on the search scope. The floor constraint is determined by the player's location. In indoor scenes, the upper and lower ranges need to be narrowed to avoid finding invisible entities across floors.
[0564] (2) Calculate the similarity between the target entity information and each entity information in the entity set.
[0565] Exemplarily, the entity set includes entity information of at least one entity, and the entity information of each entity may include at least one of a text embedding vector of a text label and an embedding vector of an image feature. Then the target entity information may calculate similarity with the text embedding vector and the image embedding vector respectively, and the entity with higher similarity is determined as the target entity.
[0566] The following are methods for similarity matching with text embedding vectors and similarity matching with image embedding vectors.
[0567] 1) Similarity includes the text similarity between the target entity information and the text embedding vector.
[0568] Taking the first entity in the entity set as an example, the game application (client or server) segments the target entity information to obtain at least one target entity tag; converts the at least one target entity tag into at least one target embedding vector; obtains the entity information of the first entity, the entity information includes a text embedding vector, and the text embedding vector is an embedding vector obtained based on the text tag conversion of the first entity; calculates the text parent similarity between at least one target embedding vector and the text embedding vector respectively, and obtains at least one text parent similarity corresponding to at least one target embedding vector; and determines the sum of the at least one text parent similarity as the text similarity between the target entity information and the entity information of the first entity.
[0569] For example, the number of target entity tags is two, target entity tag 1 and target entity tag 2, and the number of entities in the entity set is also two, entity 1 and entity 2. Entity 1 corresponds to text embedding vector 1, and entity 2 corresponds to text embedding vector 2. Then similarity 1 between target entity tag 1 and text embedding vector 1 is calculated, and similarity 2 between target entity tag 2 and text embedding vector 1 is calculated, and the sum of similarity 1 and similarity 2 is determined as the text similarity between the target entity information and entity 1. Similarity 3 between target entity tag 1 and text embedding vector 2 is calculated, and similarity 4 between target entity tag 2 and text embedding vector 2 is calculated, and the sum of similarity 3 and similarity 4 is determined as the text similarity between the target entity information and entity 2.
[0570] Exemplarily, the entity information of the first entity includes at least one text embedding vector; the at least one target embedding vector includes a first target embedding vector; the game application calculates the text sub-similarity between the first target embedding vector and the at least one text embedding vector respectively to obtain at least one text sub-similarity; the highest value of the at least one text sub-similarity is determined as the text parent similarity corresponding to the first target embedding vector.
[0571] For example, the number of target entity tags is 1: target entity tag 1. The number of entities in the entity set is also 1: entity 1. Entity 1 corresponds to text embedding vector 1 and text embedding vector 3. Then the similarity 1 between target entity tag 1 and text embedding vector 1 is calculated, and the similarity 5 between target entity tag 1 and text embedding vector 3 is calculated, and the larger value of similarity 1 and similarity 5 is determined as the similarity between the target entity information (target entity tag 1) and entity 1.
[0572] For example, for the input target entity information "blue car", the target entity information is first segmented into target entity labels "blue" and "car", which means that the queried target entity has two features. Then each feature is calculated with each entity in the entity set for similarity, and each feature takes the maximum similarity. Finally, the sum of the similarity scores corresponding to all query features is taken, which represents the text similarity between the target entity information and the queried entity.
[0573] For example, the target entity information includes two target entity tags "blue" and "car", and the two target entity tags are converted into two target embedding vectors. Then the entity information in the entity set is obtained. For example, the entity set includes entity 1 and entity 2. The text tags of entity 1 include "metal", "old", "car", "truck", "blue", and "damaged", and the text tags of entity 2 include "metal", "scratches", "car", "damaged", "money transporter", and "black".
[0574] Calculate the text similarity between the target entity information and entity 1: first calculate the similarity between the target entity label "blue" and each text label of entity 1, and take the maximum value. For example, the similarity between the target entity label "blue" and the text label "blue" of entity 1 is 1 (1 is the maximum value); then calculate the similarity between the target entity label "car" and each text label of entity 1, and take the maximum value. For example, the similarity between the target entity label "car" and the text label "car" of entity 1 is 1. Then sum the two similarities of the target entity labels "car" and "blue" to obtain the final similarity between the target entity information and entity 1, which is 2.
[0575] Similarly, calculate the text similarity between the target entity information and entity 2: first calculate the similarity between the target entity label "blue" and each text label of entity 2, and take the maximum value. For example, the similarity between the target entity label "blue" and the text label "black" of entity 1 is 0.91; then calculate the similarity between the target entity label "car" and each text label of entity 1, and take the maximum value. For example, the similarity between the target entity label "car" and the text label "car" of entity 2 is 1 (1 is the maximum value). Then sum the two similarities of the target entity labels "car" and "blue" to obtain the final similarity between the target entity information and entity 2, which is 1.91.
[0576] It can be seen that the similarity between the target entity information and entity 1 is 2, which is higher than the similarity between the target entity information and entity 2, which is 1.91.
[0577] 2) Similarity includes the image similarity between the target entity information and the image embedding vector.
[0578] Taking the first entity in the entity set as an example, the game application (client or server) segments the target entity information to obtain at least one target entity tag; converts the at least one target entity tag into at least one target embedding vector; obtains the entity information of the first entity, the entity information includes an image embedding vector, and the image embedding vector is an embedding vector obtained based on the image extraction of the first entity; calculates the image parent similarity of at least one target embedding vector and the text embedding vector respectively, and obtains at least one image parent similarity corresponding to the at least one target embedding vector; and determines the sum of the at least one image parent similarity as the image similarity between the target entity information and the entity information of the first entity.
[0579] For example, the number of target entity tags is two, target entity tag 1 and target entity tag 2, and the number of entities in the entity set is also two, entity 1 and entity 2. Entity 1 corresponds to image embedding vector 1, and entity 2 corresponds to image embedding vector 2. Then similarity 1 between target entity tag 1 and image embedding vector 1 is calculated, and similarity 2 between target entity tag 2 and image embedding vector 1 is calculated, and the sum of similarity 1 and similarity 2 is determined as the image similarity between the target entity information and entity 1. Similarity 3 between target entity tag 1 and image embedding vector 2 is calculated, and similarity 4 between target entity tag 2 and image embedding vector 2 is calculated, and the sum of similarity 3 and similarity 4 is determined as the image similarity between the target entity information and entity 2.
[0580] Exemplarily, the entity information of the first entity includes at least one image embedding vector; at least one target embedding vector includes a first target embedding vector; the game application calculates the image sub-similarity between the first target embedding vector and the at least one image embedding vector respectively to obtain at least one image sub-similarity; and the highest value of the at least one image sub-similarity is determined as the image parent similarity corresponding to the first target embedding vector.
[0581] For example, the number of target entity tags is 1: target entity tag 1. The number of entities in the entity set is also 1: entity 1. Entity 1 corresponds to image embedding vector 1 and image embedding vector 3. Then the similarity 1 between target entity tag 1 and image embedding vector 1 is calculated, and the similarity 5 between target entity tag 1 and image embedding vector 3 is calculated, and the larger value of similarity 1 and similarity 5 is determined as the similarity between the target entity information (target entity tag 1) and entity 1.
[0582] In an optional embodiment, the target image embedding vector may be used for calculating the image similarity, that is, the target image embedding vector is used instead of the target embedding vector in the above method. The target image embedding vector may be obtained by: segmenting the target entity information to obtain at least one target entity tag; inputting at least one target entity tag into a multimodal model to obtain a target image embedding vector.
[0583] 3) Similarity includes text similarity and image similarity.
[0584] When the first entity and the target entity have corresponding text similarity and image similarity, the game application determines an average value of the text similarity and the image similarity as the similarity between the first entity and the target entity.
[0585] Alternatively, when the first entity and the target entity have corresponding text similarity and image similarity, the larger value of the text similarity and the image similarity is determined as the similarity between the first entity and the target entity.
[0586] Alternatively, when the first entity and the target entity have corresponding text similarity and image similarity, the sum of the text similarity and the image similarity is determined as the similarity between the first entity and the target entity.
[0587] (3) Determine the target entity from the entity set based on similarity and environmental perception information.
[0588] In an optional embodiment, the game application first performs a similarity search based on the target entity information to obtain at least one candidate entity with a high similarity, and sorts them according to the similarity; then, based on the position, location, orientation and other information of the main virtual character and / or the non-player character, the game application performs perceptual filtering and sorting, and finally screens out the target entity.
[0589] Exemplarily, the game application determines a target perception range based on target entity information and environmental perception information; and obtains the target entity by filtering entities perceived within the target perception range based on similarity.
[0590] Alternatively, the game application determines x entities with the highest similarity from the entity set according to the similarity to form a candidate entity list; determines the target perception range according to the target entity information and the environmental perception information, and determines the entity in the target perception range among the x entities as the target entity. x is a positive integer.
[0591] Exemplarily, the game application determines the entity with the highest similarity in the perception range as the target entity.
[0592] It should be noted that when there are multiple entities with equal similarity within the target perception range, the multiple entities can be sorted according to the distance between the entity and the controlling virtual character. For example, when there are at least two entities with the highest similarity in the perception range, the entity with the highest similarity in the perception range and the closest distance to the controlling virtual character is determined as the target entity.
[0593] In an optional embodiment, the game application determines the entity with the highest similarity within the perception range and above the threshold as the target entity. Exemplarily, the number of the perception range is at least one. For example, the perception range includes at least one of the following: the visual range of the main virtual character, the hearing range of the main virtual character, the perception range of the main virtual character's perception of virtual props, the visual range of the non-player character, the hearing range of the non-player character, and the perception range of the non-player character's perception of virtual props.
[0594] Exemplarily, the game application may sequentially traverse at least one of the above-mentioned perception ranges, or the game application may sequentially traverse at least one perception range determined according to the natural language command. For example, if the natural language command is "Can you see the truck in front? Move to the truck", the game application may first traverse the visual range of the main virtual character, and then traverse the visual range of the non-player character, and match the entity with the highest similarity and above the threshold.
[0595] Exemplarily, when there are at least two perception ranges, the traversal order of the at least two perception ranges may be preset, or the traversal order of the at least two perception ranges may be determined according to the intention indicated in the natural language command. For example, when the intention is to search for an item, the visual range is traversed first. When the intention is to find a combat scene, the auditory range is traversed first.
[0596] If there are multiple levels of nested instructions in the natural language command, the first target entity will be searched first, and then the next target entity will be searched based on the position of the previous target entity. In the case where the natural language command includes a second target entity determined based on the position of the first target entity, the first target entity is queried from the entity set of the virtual environment based on the first target entity information and environmental perception information of the first target entity indicated by the natural language command; then the second target entity is queried from the entity set of the virtual environment based on the second target entity information, the first target entity and environmental perception information of the second target entity.
[0597] In response to natural language commands, the NPC is controlled to perform a virtual activity associated with a target entity.
[0598] Exemplarily, a large language model is called to identify the behavioral intent of natural language commands, and a control instruction or a control instruction sequence is generated according to the behavioral intent. According to the control instruction or the control instruction sequence, a non-player character is controlled to complete an activity based on a target entity.
[0599] The broadcasting of the NPC feedback information in step 1108 may be implemented as follows:
[0600] Sub-step 6: reporting feedback information of the non-player character, wherein the feedback text corresponding to the feedback information is a non-fixed text generated based on the environmental perception information of the non-player character;
[0601] Exemplarily, the game application can generate corresponding feedback information in real time according to the environmental perception information of the non-player character and make a broadcast. For example, when the environmental perception information of the non-player character triggers the broadcast condition, the game application can generate corresponding feedback information in real time according to the environmental perception information that triggers the broadcast condition; or, when the non-player character or the main control virtual character triggers the broadcast condition, the game application can generate corresponding feedback information in real time according to the environmental perception information of the non-player character.
[0602] For example, the feedback text is obtained by inferring the large language model based on the static entity data of the three-dimensional virtual environment and the dynamic environment perception information of the non-player characters. The large language model will generate different feedback texts according to the different perception situations of the non-player characters.
[0603] For example, the feedback text can also be obtained by inferring the static entity data of the three-dimensional virtual environment and the dynamic environment perception information of the non-player character through the language model, and the feedback text can be in line with the personality characteristics of the non-player character. For example, when the non-player character is a strong uncle, the feedback text can adopt a heroic tone; when the non-player character is a reporter, the feedback text can adopt a news report tone.
[0604] It should be noted that due to the randomness of the results generated by the large language model, in the same scene, when the environmental perception information of the non-player character is the same, the generated feedback text may also be different.
[0605] The environmental perception information may include information perceived by the non-player character in real time, or may include information perceived by the non-player character historically.
[0606] Exemplarily, the feedback information is inferred by the game application based on the environmental perception information of the non-player character and the static entity data of the three-dimensional virtual environment. The feedback information can be a voice broadcast or a text broadcast. The static entity data includes data of relatively unchanged entities in the three-dimensional virtual environment, for example, model information of various buildings, terrains, and vehicles in the three-dimensional virtual environment.
[0607] Exemplarily, the feedback information may include at least one of the following: instant feedback, execution feedback, and dynamic feedback. Instant feedback and execution feedback are both feedback information generated for natural language commands, and dynamic feedback is feedback information spontaneously generated by non-player characters.
[0608] 1. Instant feedback is feedback information generated immediately when a natural language command is received. Instant feedback can include reply broadcast and instant broadcast. Reply broadcast is used to resume the inquiry in the natural language command; instant broadcast is used to feedback the reception of the natural language command.
[0609] For example, the client can command the non-player character to move in the three-dimensional virtual environment according to the intention of the natural language command. Then, when the client receives the natural language command, it can generate instant feedback corresponding to the natural language command, and the instant feedback is used to indicate that the natural language command has been received, or the instant feedback is used to instantly reply to the natural language command.
[0610] In an optional embodiment, the feedback information includes a reply report. The reply report includes a reply to the query natural language command generated in response to the query natural language command raised by the player. For example, when the query natural language command includes querying the location of the target entity, the corresponding reply report should include the query result of the location of the target entity.
[0611] When the behavioral intention of the received natural language command is an inquiry, the terminal broadcasts a reply message of the non-player character; the reply message includes reply content for the natural language command generated according to the environmental perception information of the non-player character on the three-dimensional virtual environment.
[0612] The natural language command is used to inquire about information from the non-player character. The natural language command may be referred to as an inquiry natural language command. The inquiry natural language command is a natural language command intended to be an inquiry. The inquiry natural language command contains at least one question.
[0613] Natural language commands are command instructions conveyed using human natural language. The terminal receives the natural language commands issued by the player and commands the non-player characters according to the intention corresponding to the natural language commands. For example, the terminal receives the voice audio of the player, converts the voice audio into text, and obtains the natural language command. Alternatively, the terminal receives the text of the natural language command input by the player.
[0614] Players can use spoken or written language to issue natural language commands. The game application will use the large language model to understand the natural language commands, extract the intentions expressed in the natural language commands, generate control instructions based on the intentions, and direct the activities of non-player characters.
[0615] For example, a natural language command may be "Come to me", and the game application may parse out that the intention of the natural language command is "control the non-player character to move to the location of the controlling virtual character". The game application then obtains the location of the controlling virtual character, generates a navigation route for the non-player character to move to the location of the controlling virtual character, and controls the non-player character to move to the location of the controlling virtual character along the navigation route.
[0616] For another example, a natural language command may be "go pick up a treasure chest", then the game application may parse out that the intention of the natural language command is "control the non-player character to move to the location of the treasure chest and pick up the treasure chest", then the game application may obtain the location of the treasure chest, control the non-player character to move to the location of the treasure chest, and perform the operation of picking up the treasure chest after arriving at the location of the treasure chest.
[0617] For example, when the natural language command includes an intelligence inquiry about the target entity, a first reply report is broadcast based on the non-player character's perception of the target entity; the first reply report includes the intelligence perception result of the non-player character about the target entity.
[0618] Alternatively, when the natural language command includes an inquiry about the location of the target entity, a second reply broadcast is broadcast based on the non-player character's perception of the target entity, and the second reply broadcast includes a description of the location of the target entity.
[0619] Exemplarily, a large language model is called to parse the received natural language command to obtain the intention of the natural language command. When the intention is of an inquiry type, the natural language command can be determined to be an inquiry natural language command. When the natural language command is an inquiry natural language command, the large language model can infer according to the intention of the natural language command, based on static entity data and environmental perception information, to obtain a reply text. That is, the large language model is called to parse the received natural language command, and the intention of the natural language command is output as an inquiry, as well as the reply text of the natural language command. The game application calls the text-to-speech service according to the processing logic of the inquiry intention, and converts the reply text into a reply broadcast, which is an audio broadcast, and sends the reply audio to the client for broadcast.
[0620] For example, when the large language model recognizes that the intention of the natural language command is to ask a question, since the user may ask hundreds of questions and it is impossible to store the answer to each question locally on the client, the game application will call the text-to-speech service to generate a reply broadcast in real time based on the reply text returned by the large language model, so that the game application can instantly generate a corresponding reply broadcast to respond to the player's question.
[0621] Exemplarily, a game application (client or server) calls a large language model to parse natural language commands, and infers a reply text based on static environmental data and environmental perception information of the three-dimensional virtual environment; and generates human voice audio based on the reply text to obtain a reply broadcast.
[0622] For example, the client reports the received natural language query command to the server, the server calls the large language model to parse the natural language query command, and infers the reply text based on the static environment data and environmental perception information of the three-dimensional virtual environment; generates human voice audio based on the reply text to obtain a reply broadcast; the server sends the reply broadcast to the client; and the client broadcasts the reply broadcast.
[0623] 2. Execution feedback is the feedback generated when a task is executed according to the intent indicated by the natural language command after receiving the natural language command. Execution feedback can also be called execution broadcast.
[0624] For example, when a natural language command is used to instruct a non-player character to perform a target task, the client can generate execution feedback during the non-player character's execution of the target task, and the execution feedback is used to indicate the execution status of the target task; or, the client can generate execution feedback after the non-player character completes the target task, and the execution feedback is used to indicate the execution result of the target task.
[0625] In an optional embodiment, the feedback information includes a feedback report (also referred to as execution feedback). The feedback report includes a report on the task execution status and task execution result generated in response to the task natural language command proposed by the player. For example, when the task natural language command includes moving to the target entity, the corresponding feedback report may include: moving to the target entity, or having reached the target entity.
[0626] When the behavioral intention of the received natural language command is to execute a task, the terminal broadcasts a feedback report; the feedback report includes an execution status after the natural language command is executed according to the environmental perception information of the non-player character on the three-dimensional virtual environment.
[0627] The natural language command is used to instruct the non-player character to perform a task. The natural language command may be referred to as a task natural language command. The task natural language command includes a description text of the task, for example, the description text of the task may include at least one of the following: the task name, the action that the non-player character needs to perform, the method used to perform the task, the task target, and the location of the task target.
[0628] For example, when the intention of the task natural language command includes searching for a target entity, a first feedback announcement is broadcast, and the first feedback announcement includes the search result of the non-player character for the target entity.
[0629] Alternatively, when the intention of the task natural language command includes using a virtual prop, a second feedback announcement is announced, and the second feedback announcement is used to indicate that the virtual prop is ready.
[0630] Alternatively, when the intention of the task natural language command includes the use of virtual props, a third feedback announcement is broadcast, and the third feedback announcement is used to indicate the result of the use of the virtual props.
[0631] Alternatively, when the intention of the task natural language command includes controlling the movement of the non-player character, a fourth feedback announcement is announced, and the fourth feedback announcement is used to indicate the movement result of the movement.
[0632] Alternatively, when the intention of the task natural language command includes performing an interaction with a target entity, a fifth feedback announcement is broadcast, and the fifth feedback announcement is used to indicate at least one of a search result of the non-player character for the target entity and an interaction result between the non-player character and the target entity.
[0633] Alternatively, when the intention of the task natural language command includes item interaction, a sixth feedback broadcast is broadcast, and the sixth feedback broadcast is used to indicate the item interaction result of the non-player character performing the item interaction.
[0634] Exemplarily, a large language model is called to parse the received natural language command to obtain the intention of the natural language command. When the intention is a task, the natural language command can be determined to be a task natural language command. When the natural language command is a task natural language command, the large language model can be inferred according to the intention of the natural language command, based on static entity data and environmental perception information, to obtain a task instruction sequence, and the task instruction sequence is used to control the non-player character to perform the task. When the game application receives the task instruction sequence, it controls the non-player character activity to perform the task according to the task instructions in the task instruction sequence. During the control of the non-player character according to the task instruction, the corresponding execution status can be obtained for feedback broadcast according to the feedback broadcast logic corresponding to different task instructions.
[0635] For example, since the instructions that can be executed by non-player characters in a three-dimensional virtual environment are finite and traversable, the client can locally store feedback broadcasts corresponding to each character's instructions. When the non-player character executes the corresponding instruction, the client can read the local feedback broadcast for voice broadcast.
[0636] Alternatively, when there is variable content in the feedback broadcast corresponding to a task instruction, for example, the target entity in the feedback broadcast is variable, or the position information in the feedback broadcast is variable; the client can also generate a broadcast voice with variable content based on the target entity or position information returned by the large language model, and splice it with the broadcast voice with immutable content stored locally to obtain the final feedback broadcast.
[0637] For example, when the task natural language command is "find a nearby treasure chest", calling the large language model to parse the task natural language command can get the intention of the task, the task goal is to find the target entity, and the target entity is a treasure chest; based on the static entity data and environmental perception information, reasoning can be obtained that there is a treasure chest in the left front of the main control virtual character. In the case that the target entity can be found, the feedback broadcast corresponding to the task goal is "find a at x", where x is the location of the target entity and a is the name of the target entity. The game application can call the online text-to-speech technology to generate the location voice according to the location information "left front" in the reasoning result; according to the name of the target entity "treasure chest", call the online text-to-speech technology to generate the name voice, and then splice the location voice and name voice into the feedback broadcast to get the final feedback broadcast.
[0638] Exemplarily, a game application (client or server) calls a large language model to parse natural language commands for a task, and infers task execution instructions based on static environmental data and environmental perception information of a three-dimensional virtual environment; controls a non-player character to perform the task according to the task execution instructions; generates feedback text according to the execution status of the non-player character performing the task; and generates human voice audio based on the feedback text to obtain a feedback broadcast.
[0639] For example, the client reports a task natural language command to the server; the server parses the task natural language command, and infers the task execution instructions based on the static environment data and environment perception information of the three-dimensional virtual environment; the non-player character is controlled to perform the task according to the task execution instructions; feedback text is generated according to the execution status of the non-player character performing the task; human voice audio is generated based on the feedback text to obtain a feedback broadcast; the server sends the feedback broadcast to the client; the client receives and broadcasts the feedback broadcast.
[0640] 3. Dynamic feedback is the feedback information that is spontaneously generated based on the environmental perception information of non-player characters. Dynamic feedback can also be called dynamic broadcast.
[0641] For example, the client can spontaneously generate dynamic feedback based on the perception of the three-dimensional virtual environment without receiving natural language commands, and the dynamic feedback is used to indicate abnormal situations found by the non-player character in the three-dimensional virtual environment. For example, abnormal situations may include: finding hostile virtual characters, finding changes in the status of friendly virtual characters, finding dangerous situations, finding traces of battle or looting, etc.
[0642] In an optional embodiment, the feedback information includes dynamic reporting (also referred to as "dynamic feedback"). The dynamic reporting includes reporting of abnormal situations perceived by non-player characters. For example, when a non-player character perceives being attacked, a report is made that the character is being attacked; when a non-player character perceives a dangerous situation ahead, a report is made that there is danger ahead.
[0643] The terminal broadcasts a dynamic report when the environmental perception information of the non-player character on the three-dimensional virtual environment meets the dynamic report conditions.
[0644] Dynamic reporting is a report generated by the game application based on the environmental perception information of non-player characters when it determines that there is an abnormal situation that needs to be reported. The dynamic report includes reminder information of the abnormal situation. For example, the abnormal situation may include at least one of the following: discovering a hostile virtual character, discovering a change in the status of a friendly virtual character, and discovering a new trace.
[0645] For example, in the case where the non-player character perceives a hostile virtual character, a first dynamic broadcast is broadcast; the first dynamic broadcast includes the position of the hostile virtual character perceived by the non-player character.
[0646] Alternatively, when the non-player character senses a dangerous situation, a second dynamic broadcast is broadcast; the second dynamic broadcast is used to prompt the dangerous situation.
[0647] Alternatively, when the non-player character senses a change in the state of the friendly virtual character, a third dynamic broadcast is broadcast; the third dynamic broadcast is used to prompt that the state of the friendly virtual character has changed.
[0648] Alternatively, when the non-player character finds new traces in the three-dimensional virtual environment, a fourth dynamic broadcast is broadcast; the fourth dynamic broadcast is used to prompt the new traces. The new traces may be combat traces and / or looting traces.
[0649] For example, since the number of dynamic announcements that can be triggered by non-player characters in a three-dimensional virtual environment is limited and traversable, the client can store dynamic announcement voices locally, and the client can read the dynamic announcement voices locally for announcement when the corresponding dynamic announcement voices are triggered. Similarly, some dynamic announcement voices can contain variable content, and the client can generate variable content in real time through text-to-speech technology, and splice it with the dynamic announcement template to obtain the final dynamic announcement.
[0650] In an optional embodiment, when generating feedback information, it may be necessary to query the location of the target entity in the three-dimensional virtual environment, or determine the target entity indicated in the natural language command from the three-dimensional virtual environment. In this case, an entity query method is required to query the target entity.
[0651] Before executing the target entity query, it is necessary to pre-build the static entity data of the 3D virtual environment. The static entity data construction can serve both the reasoning of the large language model and the real-time feedback text generation on the game application side after the reasoning results are returned.
[0652] In an optional embodiment, entities in the three-dimensional virtual environment are traversed, and entity information of each entity is exported from a game engine to obtain static entity data.
[0653] For example, for in-game scenes, editor tools have been developed to perform StaticMesh traversal of the entire scene, special Actor traversal (such as interactive doors, which do not belong to the StaticMesh category), vegetation traversal, and export the entity's position, orientation, and bounding box size as static entity data for subsequent labeling and reasoning by the large language model.
[0654] Among them, StaticMesh is a static geometry resource type in the Unreal Engine 4 (UE4) game engine, which is used to represent immutable 3D models, such as buildings, props, etc. Actor is a basic object class in the Unreal Engine 4 (UE4) game engine, representing an entity in the game world, such as a character, object, light source, etc. Actor can contain multiple components to achieve different functions.
[0655] Exemplarily, static entity data can also include spatial data, which is obtained by manual labeling. Spatial data is more complex than static entities because there are multiple floors in the space and multiple layers of nesting within the floors (such as the motel area includes the second floor, and the second floor area includes room 201). For this type of data, manual calibration is used to construct it. Add Volume to the scene in the editor, divide the area required for the entire map and label it accordingly. Volume is a special Actor class in the UnrealEngine 4 (UE4) game engine, which represents a three-dimensional area with specific functions, such as a trigger area, an audio area, etc.
[0656] After obtaining the static entity data, you can query entities or spaces based on the static entity data during the game.
[0657] For example, when a player issues a natural language command, the natural language command will be accompanied by the constructed static entity data and the real-time captured runtime data (including the environmental perception information of non-player characters and / or the main virtual character) to make an inference request to the large language model. In addition, when the NPC needs to provide dynamic feedback based on the runtime data, an entity query is also performed to generate feedback text based on the current state.
[0658] When a natural language command includes a target entity, the game application will query the target entity based on the position and orientation of the current controlling virtual character, the position and orientation of teammates or enemies, and based on the target entity information of the target entity described in the natural language command and the environmental perception information of the controlling virtual character and / or the non-player character, filter out a target entity that matches the target entity information from at least one candidate entity within the perception range of the controlling virtual character and / or the non-player character.
[0659] Optionally, the natural language command is used to instruct the non-player character to perform an activity related to the target entity. For example, the target entity may be a destination for the non-player character to move to, an object that the non-player character needs to observe, a target that the non-player character needs to attack, or an object that the non-player character needs to interact with.
[0660] The target entity is a virtual object that exists in a three-dimensional virtual environment. The target entity is the entity described in a natural language command. An entity may refer to a virtual object that is fixed in a three-dimensional virtual environment. For example, a target entity may be a virtual building, a virtual terrain, a virtual vehicle, a virtual plant, a virtual prop, a virtual item, etc. An entity may also refer to a virtual object that can interact with a virtual character in a three-dimensional virtual environment. For example, a target entity may be an interaction point in a three-dimensional virtual environment (e.g., a door, a window, a cabinet, a cellar, etc.), a virtual prop, a virtual character, a virtual light source, etc.
[0661] For example, a natural language command may be "Please help me open the door", then the target entity may be "door", and the non-player character needs to perform a door-related activity "open the door". Alternatively, a natural language command may be "Is the kitchen safe", then the target entity may be "kitchen", and the non-player character needs to perform a kitchen-related activity "check if there are enemy avatars or other dangerous situations in the kitchen".
[0662] Optionally, the natural language command includes target entity information of the target entity. The target entity information may be a description text of the target entity. For example, the target entity information may include at least one of the following contents: the name, type, location, and features of the target entity. The game application may find the target entity from among the many entities in the three-dimensional virtual environment based on the target entity information in the natural language.
[0663] The game application parses the natural language command, extracts the target entity information therein, determines the target entity from the entity set of the three-dimensional virtual environment based on the target entity information, and then controls the non-player character to perform activities related to the target entity according to the intention of the natural language command.
[0664] The target entity is determined from the three-dimensional virtual environment in combination with environmental perception information, and the environmental perception information includes information perceived by at least one of the main virtual character and the non-player character from the three-dimensional virtual environment.
[0665] Since the target entity information in natural language commands is usually vague, for example, a natural language command may be "move to the red truck", where the target entity information is "red truck", and there may be many red trucks in the three-dimensional virtual environment. It is impossible to accurately determine the target entity in the three-dimensional virtual environment based solely on the target entity information in the natural language command.
[0666] Therefore, the embodiment of the present application provides a method for determining the target entity by combining the target entity information and the environmental perception information. For the previous example, the "red truck" expressed by the player in the natural language command should be a red truck that the player can see. Therefore, combined with the field of view of the main control virtual character, the red truck within the field of view of the main control system character can be screened out from multiple red trucks in the three-dimensional virtual environment. The red truck is the target entity indicated by the player in the natural language command.
[0667] Since natural language commands are issued by players based on their perception of the 3D virtual environment, in order to accurately identify the target entity indicated in the natural language command, the game application will combine the environmental perception information when the player issues the natural language command to identify the target entity in the natural language command. Based on the player's perception of the 3D virtual environment, the entity that the player can perceive in the 3D virtual environment and that is closest to the target entity information is inferred as the target entity.
[0668] Exemplarily, the three-dimensional virtual environment includes multiple candidate entities that match the natural language command, and the target entity is an entity selected from the multiple candidate entities based on the environmental perception information of the main virtual character or the non-player character. For example, the game application first selects multiple candidate entities that match the target entity information in the natural language command from the entity set, and then selects the entity that can be perceived by the main virtual character or the non-player character from the multiple candidate entities as the target entity based on the environmental perception information.
[0669] For example, the perception range of the main virtual character is determined according to the position and orientation of the main virtual character. For example, the perception range is a sector area of a certain visual field size in front of the main virtual character; entities that are not within the sector range are filtered out. Then, the target entity information and the entity information of the candidate entities are sorted according to the similarity, and entities with higher similarity are selected as target entities. When there are two entities with the highest and equal similarity scores within the perception range, for example, the similarity scores of entity 2 and entity 3 are both 2 points, then entity 2, which is closer to the main virtual character, is selected as the target entity.
[0670] When it is necessary to generate feedback information based on the position description of the target entity, the game application can perform a spatial query to generate a position description based on the position relationship between the master virtual character and the target entity.
[0671] The main application scenario of spatial query is to obtain the location of the main virtual character, non-player characters, hostile virtual characters, gunshots, etc. For example, a report containing spatial information such as "enemies found on the second floor of the motel" can be made.
[0672] The spatial query logic defines the concepts of parent area and child area. The parent area refers to a large area in a 3D virtual environment that covers multiple buildings or multiple isolated spaces, such as a motel with front and back yards and multiple rooms and basements. The child area refers to an independent space in the parent area, which can be a closed space or a special function space, such as a motel guest room or kitchen.
[0673] In order to make the location description returned by the spatial query closer to human expression, when the main virtual character and the queried location are in different parent areas, the query result returns complete text information (for example, the main virtual character is outside the motel and the queried location is on the second floor of the motel, then "the enemy is on the second floor of the motel" is returned). When the main virtual character and the queried location are in the same parent area but different child areas, the query result only returns the child area information (for example, the main virtual character is on the first floor of the motel and the queried location is on the second floor, then "the enemy is on the second floor" is returned). When the main virtual character and the queried location are in the same parent area and the same child area, the query result only returns the direction relative to the main perspective player (for example, the main virtual character and the queried location are both on the second floor of the motel, then "the enemy is in the right front position" is returned).
[0674] Exemplarily, when the controlling virtual character and the target entity are located in different parent spaces, the position description includes the parent space and the child space where the target entity is located. When the controlling virtual character and the target entity are located in different child spaces of the same parent space, the position description includes the child space where the target entity is located. When the controlling virtual character and the target entity are located in the same child space of the same parent space, the position description includes the relative position of the target entity and the controlling virtual character; wherein a parent space includes at least one child space.
[0675] For the broadcast of the NPC feedback information in step 1108, the feedback information is implemented as spatial human voice audio processed by spatial sound effects, wherein the spatial sound effect processing process can be implemented as follows:
[0676] Sub-step 7: Obtaining human voice audio corresponding to the feedback text;
[0677] In the embodiment of the present application, the process of generating the feedback text is as shown in the above-mentioned sub-step 6, which will not be repeated here.
[0678] In some embodiments, the process of obtaining corresponding human voice audio based on feedback text can be implemented as follows: obtaining feedback text and character feature information corresponding to a non-player character; and generating human voice audio matching the character feature information according to the feedback text.
[0679] Illustratively, the character feature information is used to describe the attributes of the non-player character, that is, the character feature information is information used to describe the current state and characteristics of the non-player character. In some embodiments, the attributes of the non-player character include at least one of the basic attributes of the non-player character and the situational performance attributes. Among them, the basic attributes are attributes pre-configured for the non-player character, that is, the attributes of the non-player character that do not change due to the three-dimensional virtual environment or the current situation, such as the age information of the non-player character, the male / female information of the non-player character, the martial arts information of the non-player character, etc. The situational performance attributes are attributes determined in real time by the non-player character in the situation of the three-dimensional virtual environment, that is, attributes associated with the current situation, such as the emotional information and behavioral information of the non-player character in the current situation.
[0680] Optionally, the character feature information includes at least one of basic character information, character emotion information, and character behavior information. The basic character information is used to indicate the basic situation of the non-player character, and the basic character information is information about the basic attributes of the non-player character, such as the age information of the non-player character, the male / female information of the non-player character, and the martial arts information of the non-player character; the character emotion information is used to indicate the emotional state of the non-player character in the dialogue scenario corresponding to the feedback text, and the character emotion information is the situational performance attribute of the non-player character, such as the character emotion information indicates that the non-player character is in an excited state, an angry state, a sad state, etc.; the character behavior information is used to indicate the actions performed by the non-player character in the dialogue scenario, and the character behavior information is the situational performance attribute of the non-player character, such as being in a running state, an attacking state, an injured treatment state, etc.
[0681] In some embodiments, the generation of human voice audio is achieved through a pre-trained speech generation model, and the speech generation model is used to generate speech that matches the character feature information of the non-player character. In an illustrative example, the feedback text and the character feature information are input into the pre-trained speech generation model to obtain human voice audio as the human voice audio.
[0682] Optionally, the above-mentioned speech generation model can be implemented by a neural network model such as a convolutional neural network, a feedforward neural network, a residual network, a Transformer, a multimodal large language model (Multimodal Large Language Model, MLLM), etc., which is not specifically limited here.
[0683] In some embodiments, the speech generation model includes a text encoder, a role encoder and a decoder. The text encoder is used to perform text encoding on the input feedback text, that is, the feedback text is input into the text encoder to obtain a text encoding representation. The role encoder is used to perform feature encoding on the input role feature information, that is, the role feature information is input into the role encoder to obtain a role encoding representation.
[0684] Illustratively, after obtaining the text coding representation and the role coding representation, the text coding representation and the role coding representation are fused to obtain the coding representation input to the decoder, that is, the text coding representation and the role coding representa...
Claims
1. A method for voice conversion based on a virtual scene, characterized in that: The method comprises: Acquire a natural language command in voice form, the natural language command being used to instruct a non-player character, wherein a target virtual character is in a first virtual scene, and the target virtual character includes at least one of the non-player character and a master virtual character; Acquire a plurality of first scene hot words corresponding to the first virtual scene where the target virtual character is located, wherein the plurality of first scene hot words are scene-related words of the first virtual scene; A command analysis result is obtained by converting the natural language command into text based on the multiple first scene hot words.
2. The method according to claim 1, characterized in that The step of obtaining a plurality of first scene hot words corresponding to the first virtual scene where the target virtual character is located includes: Obtaining a first scene type corresponding to the first virtual scene in which the target virtual character is located; obtaining a hot word set corresponding to the first scene type, and using the words in the hot word set as the multiple first scene hot words, wherein the hot word set is a set of scene-related words collected based on the first scene type; or, Acquire the object perception range of the target virtual character in the first virtual scene, where the object perception range is the three-dimensional space range in which the target virtual character perceives the existence of other scene elements; acquire the multiple first scene hot words based on the environmental perception information within the object perception range; or, Obtain a first scene type corresponding to the first virtual scene; obtain a hot word set corresponding to the first scene type; obtain an object perception range of the target virtual character in the first virtual scene; and obtain at least two words from the hot word set as the multiple first scene hot words based on environmental perception information within the object perception range.
3. The method according to claim 2, characterized in that The object perception range includes at least one of the following: The visual perception range of the target virtual character; The auditory perception range of the target virtual character; The olfactory perception range of the target virtual character; The target virtual character's perception skill or the range of perception of the perception tool.
4. The method according to claim 3, characterized in that The obtaining of the object perception range of the target virtual character in the first virtual scene includes: When the object perception range includes a visual perception range, a preset visual cone is generated as the visual perception range with the position of the target virtual character as the starting point and the orientation of the target virtual character as the center line of the cone. The preset visual cone is a three-dimensional area that measures the visual perception range.
5. The method according to claim 2, characterized in that: The environmental perception information includes virtual elements; The acquiring of the plurality of first scene hot words based on the environmental perception information within the object perception range comprises: Determining the virtual elements in the first virtual scene that are within the perception range of the object; The element name of the virtual element is used as the first scene hot word; or the action name of the interactive action corresponding to the virtual element is used as the first scene hot word.
6. The method according to any one of claims 1 to 5, characterized in that: The first virtual scene is one of multiple virtual scenes in the virtual environment; After obtaining the natural language command in voice form, the method further includes: Based on the position of the target virtual character in the virtual environment, the first virtual scene where the target virtual character is located is determined from the multiple virtual scenes.
7. The method according to claim 6, characterized in that The method further comprises: In the case where the command analysis result is not generated, in response to the target virtual character moving from the first virtual scene to the second virtual scene, acquiring a plurality of second scene hot words corresponding to the second virtual scene; The command analysis result converts the natural language command into text form based on the multiple second scene hot words.
8. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Based on the first virtual game in which the target virtual character participates, the first virtual scene corresponding to the first virtual game is determined; wherein, a plurality of virtual games respectively correspond to a virtual scene, and the plurality of virtual games are simulated battle environments provided for the target virtual character.
9. The method according to claim 1, characterized in that: The step of obtaining a plurality of first scene hot words corresponding to the first virtual scene where the target virtual character is located includes: Acquire a plurality of scene hot words, where the plurality of scene hot words correspond to at least two virtual scenes; Based on the first virtual scene in which the target virtual character is located, the plurality of first scene hot words corresponding to the first virtual scene are acquired from the plurality of scene hot words.
10. The method according to claim 9, characterized in that The multiple scene hot words correspond to scene identifiers, and the scene identifiers are used to represent virtual scenes when collecting scene hot words; The acquiring, based on the first virtual scene in which the target virtual character is located, the plurality of first scene hot words corresponding to the first virtual scene from the plurality of scene hot words comprises: Based on the first virtual scene in which the target virtual character is located, a plurality of candidate scene hot words with a first scene identifier are obtained from the plurality of scene hot words, where the first scene identifier is a scene identifier corresponding to the first virtual scene; Based on the object perception range of the target virtual character in the first virtual scene, at least two candidate scene hot words are selected from the multiple candidate scene hot words as the first scene hot words.
11. The method according to any one of claims 1 to 5, characterized in that: The command analysis result of converting the natural language command into text form based on the multiple first scene hot words includes: Acquire a pre-trained natural language analysis model, wherein the pre-trained natural language analysis model includes an acoustic network, a language network, and a preset dictionary, wherein the preset dictionary includes the plurality of first scene hot words, other scene hot words, and common vocabulary; Acquiring a plurality of speech units corresponding to the natural language command through the acoustic network, wherein the speech unit is a basic unit of vocabulary pronunciation; The command analysis result in text form is obtained by analyzing the sequence matching relationship between the multiple voice units and the selected words in the preset dictionary through the language network, and the selected words at least include the multiple first scene hot words and the common words.
12. The method according to claim 11, characterized in that The selected vocabulary also includes the other scene hot words; Among them, the vocabulary in the preset dictionary includes analysis weights, the first analysis weights of the multiple first-scene hot words are higher than the second analysis weights of the other scene hot words, and the analysis weight refers to the degree of attention paid to the vocabulary when it participates in sequence matching.
13. The method according to claim 11, characterized in that The step of analyzing the sequence matching relationship between the plurality of speech units and the selected words in the preset dictionary through the language network to obtain the command analysis result in text form includes: Analyzing the matching relationship between the multiple speech units and the selected words in the preset dictionary to obtain multiple candidate word sequences, wherein the multiple candidate word sequences include at least one word from the multiple selected words, and the candidate word sequences are word sequences obtained based on the change relationship between the multiple speech units; The sequence semantics of the plurality of candidate vocabulary sequences are analyzed by the language network, and at least one candidate vocabulary sequence is obtained from the plurality of candidate vocabulary sequences as the command analysis result in text form.
14. The method according to claim 13, characterized in that The language network includes language sub-networks corresponding to at least two virtual scenes respectively; The step of analyzing the sequence semantics of the plurality of candidate vocabulary sequences by using the language network and obtaining at least one candidate vocabulary sequence from the plurality of candidate vocabulary sequences as the command analysis result in text form includes: Based on the first virtual scene in which the target virtual character is located, determining a first language sub-network corresponding to the first virtual scene from at least two language sub-networks; The sequence semantics of the plurality of candidate vocabulary sequences are analyzed by the first language sub-network, and at least one candidate vocabulary sequence is obtained from the plurality of candidate vocabulary sequences as the command analysis result in text form.
15. A speech conversion device based on a virtual scene, characterized in that: The device comprises: An acquisition module, configured to acquire a natural language command in a voice form, wherein the natural language command is used to instruct a non-player character, wherein a target virtual character is in a first virtual scene, and wherein the target virtual character includes at least one of the non-player character and a master virtual character; The acquisition module is further used to acquire a plurality of first scene hot words corresponding to the first virtual scene where the target virtual character is located, wherein the plurality of first scene hot words are scene-related words of the first virtual scene; An analysis module is used to convert the natural language command into a command analysis result in text form based on the multiple first scene hot words.
16. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the virtual scene-based voice conversion method as described in any one of claims 1 to 14.
17. A computer-readable storage medium, characterized in that: The storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the virtual scene-based voice conversion method as described in any one of claims 1 to 14.
18. A computer program product, characterized in that It comprises computer instructions, which, when executed by a processor, implement the virtual scene-based voice conversion method as claimed in any one of claims 1 to 14.
Citation Information
Patent Citations
Scene-based real-time voice recognition system and method
CN105448292A
Method and system for driving virtual character to act by real-time voice
CN111939558A
Speech recognition method, electronic equipment and storage device
CN112489651A
Speech recognition method and device, electronic equipment and storage medium
CN112562659A
Speech recognition method and device, computer readable storage medium and computer equipment
CN116935852A
Cited By
Conversation processing method and device, electronic equipment and storage medium
CN120673753A
Language control interaction method and system based on AR and VR
CN120913568A
