Virtual character control method and device, virtual character processing method and device, equipment and storage medium
By obtaining natural language commands and combining virtual roles and environment perception information, intelligently parsing and executing virtual activities, the problem of flexibility in natural language mode in virtual role control is solved, and the autonomy and collaborative execution ability of virtual characters in virtual environment is improved. .
Patent Information
- Application Number
- CN202411098396.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-22
AI Technical Summary
In the prior art, it is difficult to achieve flexible control of natural language methods for virtual characters in the virtual environment, and the fixed password types are single, so it is impossible to effectively respond to the virtual activity needs of complex scenarios.
By obtaining natural language commands, combining virtual roles and environment perception information, determining semantic understanding methods, executing virtual activities, and using multimodal models and prediction networks for semantic analysis and intent recognition, we can realize intelligent understanding of natural language commands of virtual characters and implement.
It improves human-computer interaction efficiency, allows users to flexibly control virtual roles in natural language, enhances the autonomy and collaborative execution capabilities of virtual roles, and reduces the complexity of user input.
Smart Images

Figure CN119003699A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of human-computer interaction technology, and in particular to a control method, processing method, device, equipment and storage medium for a virtual character. Background Art
[0002] A large number of scene objects are deployed in applications including virtual environments to provide a virtual space for virtual activities of virtual characters.
[0003] In the related art, natural semantics and the behavior of a virtual character are usually determined in a mapping manner. For example, controlling the virtual character to move forward is triggered by a fixed command “forward”.
[0004] However, the above fixed passwords are of a single type, and how to flexibly control virtual characters in a natural language manner is an urgent problem to be solved. Summary of the invention
[0005] The present application provides a control method, processing method, device, equipment and storage medium for a virtual character, and the technical solution is as follows:
[0006] According to one aspect of the present application, a method for controlling a virtual character is provided, the method comprising:
[0007] Displaying a host virtual character or NPC in a virtual environment;
[0008] Acquire command information, the command information including natural semantics for controlling the NPC, and the command information corresponds to at least two semantic understanding modes;
[0009] In response to the command information, controlling the NPC to perform a virtual activity according to a first semantic understanding method among the at least two semantic understanding methods;
[0010] The first semantic understanding method is determined from the at least two semantic understanding methods based on the environmental perception information of the master virtual character and / or the NPC.
[0011] According to one aspect of the present application, a method for controlling a virtual character is provided, the method comprising:
[0012] Displaying a host virtual character or NPC in a virtual environment;
[0013] Acquire command information, wherein the command information includes natural semantics for controlling the NPC, and the command information includes behavioral intention and scene entity; the behavioral intention is used to indicate the type of virtual activity performed by the NPC, and the scene entity is used to indicate the target entity of the virtual activity;
[0014] In response to the command information, controlling the NPC to perform a virtual activity;
[0015] The virtual activity is determined according to the environmental perception information of the main virtual character and / or the NPC.
[0016] According to one aspect of the present application, a method for processing command information is provided, the method comprising:
[0017] Acquire command information, the command information including natural semantics for controlling the NPC, and the command information corresponds to at least two semantic understanding modes;
[0018] Determining a first semantic understanding mode among the at least two semantic understanding modes of the command information based on the environment perception information of the master virtual character and / or the NPC;
[0019] Based on the command information, the NPC is controlled to perform a virtual activity according to a first semantic understanding method among the at least two semantic understanding methods.
[0020] According to one aspect of the present application, a method for processing command information is provided, the method comprising:
[0021] Acquire command information, wherein the command information includes natural semantics for controlling the NPC, and the command information includes behavioral intention and scene entity; the behavioral intention is used to indicate the type of virtual activity performed by the NPC, and the scene entity is used to indicate the target entity of the virtual activity;
[0022] Based on the command information, controlling the NPC to perform a virtual activity;
[0023] The virtual activity is determined according to the environmental perception information of the main virtual character and / or the NPC.
[0024] According to one aspect of the present application, a method for controlling a virtual character is provided, the method comprising:
[0025] Displaying at least one of a main control virtual character and an NPC in a virtual environment;
[0026] Acquire a natural language command, wherein the natural language command includes natural semantics for commanding the NPC, and the command information includes a behavior intention and a scene entity; the behavior intention is used to indicate the type of virtual activity performed by the NPC, and the scene entity is used to indicate a target entity for the virtual activity;
[0027] In response to the natural language command, controlling the NPC to perform the virtual activity;
[0028] The virtual activity is determined according to the environmental perception information of the main virtual character and / or the NPC.
[0029] In an optional implementation of the present application, the method further includes:
[0030] According to the target entity information of the target entity indicated by the natural language command and the environment perception information, query and obtain the target entity from the entity set of the virtual environment;
[0031] The step of controlling the NPC to perform the virtual activity in response to the natural language command comprises:
[0032] In response to the natural language command, controlling the NPC to perform the virtual activity associated with the target entity;
[0033] The environmental perception information includes information perceived by at least one of the main virtual character and the NPC from the virtual environment.
[0034] In an optional implementation of the present application, querying the target entity from an entity set of the virtual environment according to the target entity information of the target entity indicated by the natural language command and the environment perception information includes:
[0035] Parsing the natural language command to obtain the target entity information; the target entity information includes at least one of the following: entity type, entity name, entity location, entity feature;
[0036] Calculating the similarity between the target entity information and each entity information in the entity set;
[0037] The target entity is determined from the entity set according to the similarity and the environmental perception information.
[0038] In an optional implementation of the present application, determining the target entity from the entity set according to the similarity and the environment perception information includes:
[0039] Determine a target perception range according to the target entity information and the environmental perception information;
[0040] The target entity is obtained by screening entities perceived within the target perception range according to the similarity.
[0041] In an optional implementation of the present application, the similarity includes text similarity; the entity set includes entity information of the first entity;
[0042] The calculating the similarity between the target entity information and each entity information in the entity set includes:
[0043] Segmenting the target entity information to obtain at least one target entity tag;
[0044] converting the at least one target entity label into at least one target embedding vector;
[0045] Acquire entity information of the first entity, where the entity information includes a text embedding vector, where the text embedding vector is an embedding vector obtained by converting a text label of the first entity;
[0046] Calculating the text parent similarity between the at least one target embedding vector and the text embedding vector respectively, and obtaining at least one text parent similarity corresponding to the at least one target embedding vector respectively;
[0047] The sum of the at least one text parent similarity is determined as the text similarity between the target entity information and the entity information of the first entity.
[0048] In an optional implementation of the present application, the similarity includes image similarity; the entity set includes entity information of the first entity;
[0049] The calculating the similarity between the target entity information and each entity information in the entity set includes:
[0050] Segmenting the target entity information to obtain at least one target entity tag;
[0051] converting the at least one target entity label into at least one target embedding vector;
[0052] Acquire entity information of the first entity, the entity information including an image embedding vector, the image embedding vector being an embedding vector extracted based on the image of the first entity;
[0053] Calculating the image parent similarity of the at least one target embedding vector and the text embedding vector respectively, and obtaining at least one image parent similarity corresponding to the at least one target embedding vector respectively;
[0054] The sum of the at least one image parent similarity is determined as the image similarity between the target entity information and the entity information of the first entity.
[0055] In an optional implementation of the present application, the method further includes:
[0056] Calling at least one hierarchical prediction network to perform intent recognition on the natural language command to obtain a first classification label of the natural language command intended to command the NPC; the hierarchical prediction network includes at least two layers of sub-networks, and the at least two layers of sub-networks are constructed based on the tree structure of the multiple classification labels;
[0057] The step of controlling the NPC to perform the virtual activity in response to the natural language command comprises:
[0058] In response to the natural language command, the NPC is controlled to perform the virtual activity according to the first intention corresponding to the first classification label.
[0059] In an optional implementation of the present application, each hierarchical prediction network in the at least one hierarchical prediction network is used to predict the command intention of the natural language command for the NPC in an intention dimension;
[0060] The i-th layer sub-network in each hierarchical prediction network is used to predict the primary behavior label of the natural language command on the one intention dimension, and the i+1-th layer sub-network in the hierarchical prediction network is used to predict the secondary behavior label corresponding to the natural language command in the primary behavior label, where i is a positive integer;
[0061] The intention dimension includes at least one of a subject dimension, a semantic dimension, and a behavior dimension.
[0062] In an optional implementation of the present application, the intention dimension includes the subject dimension, the hierarchical prediction network includes a hierarchical subject prediction network, and the subject prediction network has the ability to predict the subject type in the natural language command; the subject type is used to indicate the identity of the NPC commanded by the natural language command;
[0063] And / or, the intention dimension includes the semantic dimension, the hierarchical prediction network includes a hierarchical semantic prediction network, the semantic prediction network has the ability to predict the semantic type in the natural language command; the semantic type is used to indicate the control method of the NPC for initiating a virtual attack;
[0064] And / or, the intention dimension includes the behavior dimension, the hierarchical prediction network includes a hierarchical behavior prediction network, and the behavior prediction network has the ability to predict the behavior intention type in natural language commands; the behavior intention type is used to indicate the behavior mode of the NPC in performing virtual activities.
[0065] In an optional implementation of the present application, the method further includes:
[0066] An information structure prediction network is called to perform classification prediction on the natural language command, and it is predicted that the natural language command corresponds to a first control label; the information structure prediction network has the ability to predict whether the natural language command carries natural semantics for commanding the NPC, and the first control label is used to indicate that the natural language command carries natural semantics for commanding the NPC.
[0067] In an optional implementation of the present application, the method further includes:
[0068] Acquire the attribute text of the scene object in the virtual scene, wherein the attribute text is used to introduce the inherent attributes of the scene object in the virtual scene;
[0069] Acquire an appearance image of the scene object in the virtual scene, wherein the appearance image is used to describe the style of the scene object;
[0070] A multimodal model is called to perform prediction on the attribute text and the appearance image of the scene object to obtain a visual label of the scene object, where the visual label is used to describe the visual features of the scene object in at least one dimension.
[0071] In an optional implementation of the present application, the multimodal model includes a visual question answering model, and calling the multimodal model to perform prediction on the attribute text and the appearance image of the scene object to obtain a visual label of the scene object includes:
[0072] Constructing a question sentence for the appearance image, wherein the question sentence carries the attribute text of the scene object;
[0073] The appearance image of the scene object and the question sentence are input into the visual question answering model to obtain an answer sentence, and the answer sentence is used as the visual label of the scene object.
[0074] In an optional implementation of the present application, constructing a question statement for the appearance image includes:
[0075] Acquiring expected information of the scene object, the expected information being used to indicate a description dimension of the scene object expected in the visual tag and / or an expected format of the visual tag;
[0076] constructing the question sentence of the appearance image according to the expected information and the attribute text;
[0077] Among them, the first sub-part of the question sentence is the supplementary introduction information of the scene object, carrying the attribute text of the scene object; the second sub-part of the question sentence is the answer guidance sentence for the visual question and answer model, carrying the expected information.
[0078] In an optional implementation of the present application, the attribute text of the scene object includes at least one of a name of the scene object in the virtual scene and a size of the scene object in the virtual scene;
[0079] And / or, the appearance image of the scene object includes images obtained by observing the scene object from at least two viewing angles.
[0080] In an optional implementation of the present application, the method further includes:
[0081] The visual label of the scene object is rewritten in a pseudo-spoken manner to obtain a matching label that conforms to a spoken expression of a natural language.
[0082] In an optional implementation of the present application, the environmental perception information further includes spatial information of the scene object, and the method further includes:
[0083] The spatial position of the scene object in the virtual scene is acquired, and the spatial position of the scene object is determined as auxiliary information of the visual label of the scene object.
[0084] In an optional implementation of the present application, the acquiring the spatial position of the scene object in the virtual scene includes:
[0085] Acquire at least one of the coordinate position, orientation information, bounding box information, and cover point information of the scene object in the virtual scene;
[0086] Among them, the coordinate position is used to indicate the position of the scene object in the virtual scene, the orientation information is used to indicate the direction facing the scene object in the virtual scene, the bounding box information is used to indicate the size of the scene object in the virtual scene, and the cover point information indicates the recommended virtual character standing position when the virtual character approaches the scene object.
[0087] In an optional implementation of the present application, the method further includes:
[0088] The feedback information of the NPC is broadcasted, and the feedback text corresponding to the feedback information is a non-fixed text generated based on the environmental perception information of the master virtual character and / or the NPC.
[0089] In an optional implementation of the present application, the feedback information includes a reply broadcast; the broadcasting of the feedback information of the NPC includes:
[0090] In the case where the behavioral intention of the natural language command is an inquiry, broadcasting the reply broadcast of the NPC; the reply broadcast includes reply content for the natural language command generated according to the environmental perception information of the NPC;
[0091] The natural language command is used to inquire about information from the non-player character.
[0092] In an optional implementation of the present application, the feedback information includes a feedback report; the report of the feedback information of the NPC includes:
[0093] In the case where the behavioral intention of the natural language command is to execute a task, broadcasting the feedback broadcast of the NPC; the feedback broadcast includes an execution status of commanding the NPC to execute the task according to the environmental perception information of the NPC;
[0094] The natural language command is used to instruct the NPC to perform a task.
[0095] In an optional implementation of the present application, the feedback information includes dynamic broadcasting; the broadcasting of the feedback information of the NPC includes:
[0096] In the case where the environmental perception information of the NPC meets the dynamic broadcast conditions, a dynamic broadcast is broadcast; the dynamic broadcast includes an abnormal situation identified according to the environmental perception information of the NPC.
[0097] In an optional implementation of the present application, the feedback information is spatial human voice audio of the non-player character;
[0098] The reporting of the NPC's feedback information includes:
[0099] Obtaining human voice audio corresponding to the feedback text;
[0100] Based on the relative position relationship between the non-player character and the master virtual character in the three-dimensional virtual environment, obtaining sound effect parameters corresponding to the non-player character;
[0101] The human voice audio is adjusted based on the sound effect parameters, and the spatial human voice audio corresponding to the non-player character is generated and broadcasted, wherein the spatial human voice audio is used to express the perceptual effect of the audio generated by the main control virtual character on the non-player character under the relative position relationship.
[0102] In an optional implementation of the present application, the acquiring of the sound effect parameters corresponding to the non-player character based on the relative position relationship between the non-player character and the master virtual character in the three-dimensional virtual environment includes:
[0103] Acquire first position information of the non-player character in the three-dimensional virtual environment; and acquire second position information of the master virtual character in the three-dimensional virtual environment;
[0104] Determining the relative position relationship between the non-player character and the master virtual character based on the first position information and the second position information;
[0105] Based on the relative position relationship, the sound effect parameters corresponding to the non-player character are generated.
[0106] In an optional implementation of the present application, the natural language command is a natural language command in voice form, and the method further includes:
[0107] Acquire a plurality of first scene hot words corresponding to a first virtual scene where a target virtual character is located, the target virtual character includes at least one of the NPC and the master virtual character, and the plurality of first scene hot words are scene-related words of the first virtual scene;
[0108] The natural language command in voice form is converted into text form based on the multiple first scene hot words.
[0109] In an optional implementation of the present application, the step of obtaining a plurality of first scene hot words corresponding to the first virtual scene where the target virtual character is located includes:
[0110] Acquire an object perception range of the target virtual character in the first virtual scene, the object perception range being a three-dimensional space range in which the target virtual character perceives the existence of other scene elements, the object perception range comprising at least one of the following: a visual perception range of the target virtual character; an auditory perception range of the target virtual character; an olfactory perception range of the target virtual character; or a range perceived by a perception skill or a perception tool possessed by the target virtual character;
[0111] The plurality of first scene hot words are acquired based on the environmental perception information within the perception range of the object.
[0112] In an optional implementation of the present application, converting the natural language command in voice form into text form based on the multiple first scene hot words includes:
[0113] In the case where the natural language command is analyzed by a natural language analysis model, a plurality of speech units corresponding to the natural language command are obtained by an acoustic network, wherein the speech unit is a basic constituent unit of vocabulary pronunciation, the natural language analysis model includes the acoustic network, a preset dictionary, and language sub-networks corresponding to at least two virtual scenes respectively, and the virtual environment includes the at least two virtual scenes;
[0114] Based on the first virtual scene in which the target virtual character is located, determining a first language sub-network corresponding to the first virtual scene from at least two language sub-networks;
[0115] The first language subnetwork analyzes the sequence matching relationship between the multiple voice units and the selected words in the preset dictionary, and converts the natural language command in the voice form into the text form, wherein the selected words at least include the multiple first scene hot words and the common words.
[0116] According to another aspect of the present application, a control device for a virtual character is provided, the device comprising:
[0117] A display module, used to display a main control virtual character or NPC in a virtual environment;
[0118] An acquisition module, used for acquiring command information, wherein the command information includes natural semantics for controlling the NPC, and the command information corresponds to at least two semantic understanding modes;
[0119] A control module, configured to control the NPC to perform a virtual activity in accordance with a first semantic understanding mode of the at least two semantic understanding modes in response to the command information;
[0120] The first semantic understanding method is determined from the at least two semantic understanding methods based on the environmental perception information of the master virtual character and / or the NPC.
[0121] In an optional design of the present application, the command information corresponds to at least two semantic understanding modes of scene entities, and the control module is further used for:
[0122] In response to the command information, controlling the NPC to perform the virtual activity on a first scene object located within the perception range of the master virtual character and / or the NPC;
[0123] There are multiple scene objects of the same type in the virtual environment, the multiple scene objects include the first scene object, and the perception range is a virtual area where the main control virtual character and / or the NPC obtains the environmental perception information.
[0124] In an optional design of the present application, the control module is also used for:
[0125] In response to the command information, the NPC is controlled to perform the virtual activity on the first scene object located in a field of view.
[0126] In an optional design of the present application, the field of view is the field of view observed by the main virtual character and / or the NPC; and / or, the field of view is an expanded field of view obtained using perception skills or perception tools.
[0127] In an optional design of the present application, in response to the command information, controlling the NPC to perform the virtual activity on a first scene object within the perception range of the master virtual character and / or the NPC includes:
[0128] In response to the command information, the NPC is controlled to perform the virtual activity on the first scene object located in a first virtual terrain, where the first virtual terrain is the virtual terrain to which the main control virtual character and / or the NPC is located.
[0129] In an optional design of the present application, the control module is also used for at least one of the following:
[0130] In the case where the first virtual terrain is an indoor terrain, in response to the command information, controlling the NPC to perform the virtual activity on the first scene object located in a first area, the first area being an indoor connected area including a first position where the master virtual character and / or the NPC is located;
[0131] In the case where the first virtual terrain is a street terrain adjacent to a building, in response to the command information, the NPC is controlled to perform the virtual activity on the first scene object located in a second area, the second area including buildings around a second position where the master virtual character and / or the NPC is located;
[0132] In the case where the first virtual terrain is an outdoor terrain away from buildings, in response to the command information, the NPC is controlled to perform the virtual activity on the first scene object located in a third area, and the third area is an area with the same virtual vegetation distribution as the third position where the main virtual character and / or the NPC is located.
[0133] In an optional design of the present application, the command information corresponds to at least two semantic understanding modes of word pronunciation, and the control module is further used for:
[0134] In response to the command information, converting the command information into command text according to the natural language pronunciation of scene objects in a perception range, wherein the perception range is a virtual area where the master virtual character and / or the NPC obtains the environmental perception information;
[0135] According to the command text, the NPC is controlled to execute the virtual activity according to the first semantic understanding method.
[0136] In an optional design of the present application, the control module is also used for:
[0137] When the similarity between the pronunciation of the first phoneme in the command information and the natural language pronunciation of the scene object exceeds a similarity threshold and a candidate text exists for the first phoneme, determining the text information of the first phoneme as the name text and / or introduction text of the scene object, and constructing the command text according to the text information;
[0138] The candidate text of the first phoneme does not include the name text and / or introduction text of the scene object.
[0139] In an optional design of the present application, the command information corresponds to at least two semantic understanding modes of the behavioral intention, and the control module is further used for:
[0140] In response to the command information, the NPC is controlled to perform the virtual activity according to a first behavioral intention; the first behavioral intention corresponds to a type of virtual activity permitted to be performed within a perception range, and the perception range is a virtual area in which the main virtual character and / or the NPC obtains the environmental perception information.
[0141] In an optional design of the present application, the control module is also used for:
[0142] In response to the command information, taking the first direction in the perception range as a reference direction and according to the directional words in the command information, controlling the NPC to perform the virtual activity based on a second direction;
[0143] The directional word is used to describe the angle between the second direction and the first direction, and / or to describe the second direction in a reference system constructed based on the first direction.
[0144] In an optional design of the present application, the control module is also used for:
[0145] Determining a command text in natural language according to the command information;
[0146] Calling a conditional random field to perform entity prediction on the command text to obtain an entity label carried by the command text;
[0147] Determining a first scene object in a perception range according to the entity label;
[0148] Calling a prediction network to perform intent recognition on the command text to obtain a classification label of the command text; the prediction network is used to predict a control method of the command text for the NPC in at least one behavior dimension;
[0149] According to the classification label, a control command of the NPC is determined, and based on the control command, the NPC is controlled to perform the virtual activity on the first scene object according to the first semantic understanding method.
[0150] In an optional design of the present application, the control module is also used for:
[0151] Calling a hierarchical prediction network to perform intent recognition on the command text to obtain the classification label of the command text;
[0152] The hierarchical prediction network includes at least two layers of sub-networks, and the at least two layers of sub-networks are constructed based on a tree structure of multiple classification labels.
[0153] In an optional design of the present application, the acquisition module is further used for:
[0154] Acquire visual labels of multiple candidate scene objects in the perception range;
[0155] The control module is also used for:
[0156] The similarities between the visual labels of the multiple candidate scene objects and the entity labels are respectively calculated to determine the first scene object with the highest similarity among the multiple candidate scene objects.
[0157] In an optional design of the present application, the acquisition module is further used for:
[0158] Acquire the spatial position of each scene object in the virtual environment;
[0159] The control module is also used for:
[0160] The plurality of candidate scene objects in the perception range are screened out from the respective scene objects in the virtual environment according to the spatial position.
[0161] In an optional design of the present application, the acquisition module is further used for:
[0162] Acquire at least one of the coordinate position, orientation information, bounding box information, and cover point information of each scene object in the virtual environment;
[0163] Among them, the coordinate position is used to indicate the position of the scene object in the virtual environment, the orientation information is used to indicate the direction facing the scene object in the virtual environment, the bounding box information is used to indicate the size of the scene object in the virtual environment, and the cover point information indicates the recommended virtual character standing position when the virtual character approaches the scene object.
[0164] In an optional design of the present application, the control module is also used for:
[0165] Acquire the attribute text of the scene object in the virtual environment, wherein the attribute text is used to introduce the inherent attributes of the scene object in the virtual environment;
[0166] Acquire an appearance image of the scene object in the virtual environment, wherein the appearance image is used to describe the style of the scene object;
[0167] Calling a multimodal model to perform prediction on the attribute text and the appearance image of the scene object to obtain a visual label of the scene object, where the visual label is used to describe the scene object in at least one dimension;
[0168] Building a spatial database of the virtual environment based on visual labels of various scene objects in the virtual environment;
[0169] In the spatial database, based on the perception range, visual labels of multiple candidate scene objects in the perception range are screened and obtained.
[0170] In an optional design of the present application, the device further includes:
[0171] A presentation module is used to present feedback information of the NPC to the command information, wherein the feedback information is used to indicate that the NPC has received the command information and / or that the NPC has executed the virtual activity indicated by the command information.
[0172] According to another aspect of the present application, a command information processing device is provided, the device comprising:
[0173] An acquisition module, used to acquire command information, wherein the command information includes natural semantics for controlling the NPC, and the command information corresponds to at least two semantic understanding modes;
[0174] A processing module, configured to determine a first semantic understanding mode among the at least two semantic understanding modes of the command information based on the environmental perception information of the master virtual character and / or the NPC;
[0175] A control module is used to control the NPC to perform virtual activities according to a first semantic understanding method among the at least two semantic understanding methods based on the command information.
[0176] In an optional design of the present application, the command information corresponds to at least two semantic understanding modes of scene entities;
[0177] The processing module is also used for:
[0178] Based on the environmental perception information, the command information is determined as executing the virtual activity on a first scene object located within the perception range of the master virtual character and / or the NPC;
[0179] Among them, there are multiple scene objects of the same type in the virtual environment where the master virtual character and / or the NPC is located, and the multiple scene objects include the first scene object. The perception range is the virtual area where the master virtual character and / or the NPC obtains the environmental perception information.
[0180] In an optional design of the present application, the processing module is further used for:
[0181] Based on the environmental perception information, determining the command information as executing the virtual activity on a first scene object located in a field of view;
[0182] Or, based on the environmental perception information, the command information is determined to execute the virtual activity for a first scene object located in a first virtual terrain; the first virtual terrain is the virtual terrain to which the main control virtual character and / or the NPC is located.
[0183] In an optional design of the present application, the command information corresponds to at least two semantic understanding modes of word pronunciation;
[0184] The processing module is also used for:
[0185] Based on the environmental perception information, determining the command information as a natural language pronunciation carrying scene objects in the perception range; and converting the command information into a command text according to the first semantic understanding method;
[0186] The perception range is a virtual area where the main virtual character and / or the NPC obtains the environmental perception information; the NPC performs the virtual activity based on the instruction of the command text.
[0187] In an optional design of the present application, the command information corresponds to at least two semantic understanding modes of behavioral intentions;
[0188] The processing module is also used for:
[0189] Based on the environmental perception information, the command information is determined to execute the virtual activity according to a first behavioral intention; the first behavioral intention corresponds to the type of virtual activity permitted to be executed within a perception range, and the perception range is a virtual area where the master virtual character and / or the NPC obtains the environmental perception information.
[0190] In an optional design of the present application, the processing module is further used for:
[0191] Determining a command text in natural language according to the command information;
[0192] Calling a conditional random field to perform entity prediction on the command text to obtain an entity label carried by the command text;
[0193] Determining a first scene object in a perception range according to the entity label;
[0194] Calling a prediction network to perform intent recognition on the command text to obtain a classification label of the command text; the prediction network is used to predict a control method of the command text for the NPC in at least one behavior dimension;
[0195] The controlling the NPC to perform a virtual activity according to a first semantic understanding mode of the at least two semantic understanding modes based on the command information includes:
[0196] According to the classification label corresponding to the command information and the first scene object, a control command of the NPC is determined, and a control command is sent, wherein the control command is used to control the NPC to perform the virtual activity for the first scene object according to the first semantic understanding method.
[0197] In an optional design of the present application, the processing module is further used for:
[0198] Calling a hierarchical prediction network to perform intent recognition on the command text to obtain the classification label of the command text;
[0199] The hierarchical prediction network includes at least two layers of sub-networks, and the at least two layers of sub-networks are constructed based on a tree structure of multiple classification labels.
[0200] In an optional design of the present application, the acquisition module is further used to acquire visual labels of multiple candidate scene objects in the perception range;
[0201] The processing module is further configured to respectively calculate the similarities between the visual labels of the plurality of candidate scene objects and the entity labels, and determine the first scene object having the highest similarity among the plurality of candidate scene objects.
[0202] In an optional design of the present application, the acquisition module is further used to acquire the spatial position of each scene object in the virtual environment;
[0203] The processing module is further used to screen out the multiple candidate scene objects in the perception range from the various scene objects in the virtual environment according to the spatial position.
[0204] In an optional design of the present application, the acquisition module is further used to:
[0205] Acquire at least one of the coordinate position, orientation information, bounding box information, and cover point information of each scene object in the virtual environment;
[0206] Among them, the coordinate position is used to indicate the position of the scene object in the virtual environment, the orientation information is used to indicate the direction facing the scene object in the virtual environment, the bounding box information is used to indicate the size of the scene object in the virtual environment, and the cover point information indicates the recommended virtual character standing position when the virtual character approaches the scene object.
[0207] In an optional design of the present application, the acquisition module is further used for:
[0208] Acquire the attribute text of the scene object in the virtual environment, wherein the attribute text is used to introduce the inherent attributes of the scene object in the virtual environment;
[0209] Acquire an appearance image of the scene object in the virtual environment, wherein the appearance image is used to describe the style of the scene object;
[0210] Calling a multimodal model to perform prediction on the attribute text and the appearance image of the scene object to obtain a visual label of the scene object, where the visual label is used to describe the scene object in at least one dimension;
[0211] Building a spatial database of the virtual environment based on visual labels of various scene objects in the virtual environment;
[0212] In the spatial database, based on the perception range, visual labels of multiple candidate scene objects in the perception range are screened and obtained.
[0213] According to another aspect of the present application, a control device for a virtual character is provided, the device comprising:
[0214] A display module, used to display at least one of a main control virtual character and an NPC in a virtual environment;
[0215] An acquisition module, used to acquire a natural language command, wherein the natural language command includes natural semantics for commanding the NPC, and the command information includes a behavior intention and a scene entity; the behavior intention is used to indicate the type of virtual activity performed by the NPC, and the scene entity is used to indicate the target entity of the virtual activity;
[0216] A control module, for controlling the NPC to perform the virtual activity in response to the natural language command;
[0217] The virtual activity is determined according to the environmental perception information of the main virtual character and / or the NPC.
[0218] According to another aspect of the present application, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to implement the control method of the virtual character as described above, and / or the method for processing command information.
[0219] According to another aspect of the present application, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, a code set or an instruction set is stored in the computer-readable storage medium, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the control method of the virtual character as described above, and / or the method for processing command information.
[0220] According to another aspect of the present application, a computer program product is provided, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor reads and executes the computer instructions from the computer-readable storage medium to implement the control method of the virtual character as described above, and / or the method for processing command information.
[0221] The beneficial effects of the technical solution provided by this application include at least:
[0222] By using the environmental perception information of the main virtual character and / or the NPC as a reference in at least two semantic understanding methods of the command information, it is determined that the virtual activity is executed in accordance with the first semantic understanding method; when there are at least two semantic understanding methods of the command information, on the one hand, there is no need for the user to add more basis for the command information, that is, the NPC can be controlled to execute the virtual activity according to the command information, thereby improving the efficiency of human-computer interaction; on the other hand, using the environmental perception information of the main virtual character and / or the NPC as a reference ensures that the NPC and the main virtual character collaborate in executing the virtual activity. BRIEF DESCRIPTION OF THE DRAWINGS
[0223] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0224] Figure 1 is a schematic diagram of controlling a non-player character in a virtual environment provided by an exemplary embodiment of the present application;
[0225] Figure 2 is a structural block diagram of a computer system provided by an exemplary embodiment of the present application;
[0226] Figure 3 is a flow chart of a method for controlling a virtual character provided by an exemplary embodiment of the present application;
[0227] Figure 4 is a flow chart of a method for controlling a virtual character provided by an exemplary embodiment of the present application;
[0228] Figure 5 is a flow chart of a method for controlling a virtual character provided by an exemplary embodiment of the present application;
[0229] Figure 6 is a flow chart of a method for controlling a virtual character provided by an exemplary embodiment of the present application;
[0230] Figure 7 is a flow chart of a method for controlling a virtual character provided by an exemplary embodiment of the present application;
[0231] Figure 8 is a schematic diagram of command information provided by an exemplary embodiment of the present application;
[0232] Fig. 9 is a schematic diagram of a visual label of a scene object provided by an exemplary embodiment of the present application;
[0233] Fig.10 is a flow chart of a method for controlling a virtual character provided by an exemplary embodiment of the present application;
[0234] Fig.11 is a flowchart of a method for processing command information provided by an exemplary embodiment of the present application;
[0235] Fig.12 is a schematic diagram of a method for processing command information provided by an exemplary embodiment of the present application;
[0236] Fig.13is a schematic diagram of a behavior tree of an NPC provided by an exemplary embodiment of the present application;
[0237] Fig.14 is a flow chart of a method for controlling a virtual character provided by an exemplary embodiment of the present application;
[0238] Fig.15 is a structural block diagram of a control device for a virtual character provided by an exemplary embodiment of the present application;
[0239] Fig.16 is a structural block diagram of a command information processing device provided by an exemplary embodiment of the present application;
[0240] Fig.17 It is a structural block diagram of a terminal provided by an exemplary embodiment of the present application.
[0241] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application. DETAILED DESCRIPTION
[0242] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.
[0243] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0244] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.
[0245] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the command information and other information involved in this application are all obtained with full authorization.
[0246] It should be understood that although the terms first, second, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first parameter may also be referred to as the second parameter, and similarly, the second parameter may also be referred to as the first parameter. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0247] The embodiments of the present application provide a solution for users to use natural language commands in the form of voice to command non-player characters (NPC) in a virtual environment. In this solution, in addition to the main virtual character controlled by the user, the user usually has one or more NPCs as teammates. The user can use a relatively casual and non-mechanical human conversation method to command the NPC to give the desired feedback to the scene objects in the virtual environment, thereby commanding the NPC to complete the task in collaboration with the main virtual character. Exemplarily, the NPC has autonomous behavior capabilities, and the user only commands the NPC. The user gives commands, and the NPC understands and executes the commands based on its own autonomous behavior capabilities. Exemplary reference Figure 1 , the user controls the main virtual character 10 to play the game, and the main virtual character 10 has an NPC teammate 20. There is a car 30 in the field of vision of the main virtual character 10. The user speaks a natural voice command in the form of speech: "No. 2, you go to ambush behind the car 30", and then the NPC teammate 20 will move to the back of the car 30 to ambush. It should be noted that Figure 1 There may be multiple cars in the scene shown, and the NPC teammate 20 will accurately understand that the car the user is talking about is the car 30 located within the user's field of vision. That is, the NPC teammate 20 has a relatively intelligent natural language understanding ability.
[0248] On the one hand, compared with the traditional technology of using mechanical fixed instructions to command NPC, the embodiment of the present application uses relatively random natural language commands to command NPC, which can provide users with more natural, more complex and more flexible language command capabilities.
[0249] On the other hand, the intelligence of the natural language understanding capability provided by the embodiment of the present application is reflected in that when understanding natural language commands, NPC teammates not only consider the literal information of natural language commands. NPC teammates will also combine the main virtual character and / or the NPC's own environmental perception information in the virtual environment to assist in understanding the semantics in natural language commands. That is, NPC not only considers the information of the natural language command mode, but also considers the perception information of other modes such as vision, hearing, radar, etc. to assist in the understanding and execution of natural language commands. Since natural language commands in the form of voice are spoken commands rather than written commands, there may be multiple candidate understanding methods or unclear places when considering the literal information of natural language commands alone. NPC teammates can determine a reasonable understanding method from multiple candidate understanding methods, or eliminate doubts about unclear places, by combining the main virtual character and / or the NPC's own environmental perception information in the virtual environment, thereby achieving a more intelligent natural language understanding capability. The above-mentioned environmental perception information includes, but is not limited to, at least one of the following:
[0250] Visual information perceived within the field of vision of the controlling virtual character;
[0251] Auditory information perceived within the hearing range of the controlling virtual character;
[0252] The information perceived by the master virtual character’s perception skills or perception tools;
[0253] Visual information perceived by the NPC within its field of vision;
[0254] Auditory information perceived within the NPC's hearing range;
[0255] Information perceived by the NPC's perception skills or perception tools.
[0256] Combined with reference Figure 1 The program includes at least one of the following five stages:
[0257] Phase 1: Preprocessing of spatial data;
[0258] The server will pre-process the scene objects in the virtual scene and construct the spatial data of the virtual scene.
[0259] The spatial data of the virtual scene includes the visual labels of the scene objects and the spatial positions of the scene objects. In some examples, the spatial data also includes the appearance images of the scene objects. The scene objects are any objects that appear in the virtual scene, such as Figure 1 Cars, walls, boxes, etc.
[0260] First, the pre-acquisition process of the visual labels of the scene objects is introduced. The attribute information of the scene objects in the virtual scene is obtained, such as at least one of the name and size of the scene objects; and the appearance image of the scene objects is obtained. The appearance image includes an image obtained by observing the scene objects from at least two perspectives. Observing the scene objects from multiple perspectives can ensure that the appearance image carries comprehensive appearance information of the scene objects.
[0261] The server calls the multimodal model to predict the attribute information and appearance image of the scene object. Based on the multimodal model's ability to process text dimension and image dimension information, it extracts hidden features from the attribute information of the text dimension and the appearance image of the image dimension, and decodes the hidden features to predict the visual label of the scene object. The visual label is used to describe the scene object in at least one dimension in a textual manner; for example, the scene object is described in dimensions such as material, transparency, color, and shape. The natural language model is called to optimize the visual label to obtain the visual label of the scene object; the natural language model has the ability to generate text, and rewrites the visual label input to the natural language model in a manner that conforms to the spoken expression of natural language, thereby optimizing the visual label and obtaining the optimized visual label of the scene object. The optimized visual label conforms to the spoken expression of natural language, and describes the scene object in at least one dimension in a manner close to the spoken expression.
[0262] For example, when describing a virtual bed in a virtual scene in spoken language, people usually pay attention to the color, material, and placement of the virtual bed, but usually neglect to pay attention to the craftsmanship used for the headboard and the side panels of the bed, such as carving, embossing, etc. The purpose of performing pseudo-spoken rewriting on the visual labels of scene objects is to obtain matching labels that are closer to spoken expressions; for example, using multiple words with the same semantics to enrich the labels of dimensions such as color, material, and placement that are paid attention to in spoken expressions, and deleting the labels of the surface feature dimensions of the headboard and the side panels of the bed that are neglected in spoken expressions.
[0263] Next, the spatial position of the scene object is introduced. The spatial position includes at least one of the following of the scene object: coordinate position (Location), such as the coordinate information of the center point or preset point of the scene object in the virtual scene; orientation (Rotation), such as the direction the front of the scene object faces in the virtual scene; bounding box (Bounding Box), used to indicate the size of the scene object in the virtual scene; cover points, used to indicate the recommended virtual character position points when the virtual character approaches the scene object, so that the scene object can cover the virtual character.
[0264] Next, the process of pre-acquiring the appearance image of the scene object is introduced. The appearance image is acquired in the process of predicting the visual label of the scene object, and the appearance image includes images obtained by observing the scene object from at least two perspectives.
[0265] Optionally, the spatial data of each scene object in the entire virtual environment may be pre-acquired and organized into a data set for use in a subsequent query process.
[0266] Phase 2: Speech recognition and intent recognition;
[0267] When the user controls the main virtual character, the terminal receives the natural language command input by the user in the form of voice; the server performs voice recognition and intent recognition on the natural language command to analyze the user's input instruction;
[0268] Speech recognition is to perform text conversion on the natural language instructions input by the user through voice, obtain the instruction text corresponding to the natural language instructions, and convert the speech information into text information. Call the encoding network to perform feature encoding on the instruction text to obtain the feature representation of the natural language instructions in the latent space; call the text segmentation network to perform segmentation on the feature representation to obtain multiple clauses in the instruction text, and then perform intent recognition on each of the multiple clauses. In the case that the natural language instructions input by the user through voice are long sentences, the segmentation of the instruction text corresponding to the natural language instructions can be achieved to meet the needs of analyzing the user's input instructions in the long sentence scenario.
[0269] The following introduces the intent recognition of each clause; the user's input instruction is obtained through intent recognition, and the user's input instruction includes an entity, a semantic type, a subject type, and an intent type in at least one clause.
[0270] On the one hand, the Conditional Random Field (CRF) method is used to identify entities in clauses, such as identifying entities in clauses as scene objects such as buildings, virtual items, and virtual vegetation in virtual scenes. The entities in the clauses are used to indicate the scene objects targeted by the virtual activities. For example, if the entity in the clause is a virtual building, and the clause carries the semantics of instructing the NPC to launch a virtual attack, then the virtual attack is executed against the virtual building, that is, a virtual attack is launched against the virtual building. On the other hand, the prediction network is called to predict the semantic type, subject type, and intention type in the clause respectively. Among them, the semantic type includes whether to instruct the NPC to launch a virtual attack; the subject type is used to indicate that the subject of the clause is one or more NPCs, or a master virtual character controlled by the user; the intention type is the expected virtual activity indicated by the clause, such as moving, launching a virtual attack, etc.
[0271] Phase 3: Query of spatial data;
[0272] For the entity in the input command, it is necessary to determine the spatial position of the entity in the virtual environment. In the case where there are multiple candidate entities in the virtual environment that match the entity in the input command, the target entity that matches the input command can be uniquely determined from the multiple candidate entities in combination with the environmental perception information of the main virtual character and / or NPC.
[0273] For example, the environment perception information is the field of view information of the main virtual character, the spatial data in the virtual environment is used as the query range, and the real-time position and orientation of the main virtual character is used as the query reference condition to query the spatial position of the target entity corresponding to the input instruction in the virtual environment, so as to subsequently control the NPC to move to the vicinity of the target entity, or to perform the behavior indicated by the input instruction on the target entity.
[0274] However, since the main virtual character and / or NPC may have multiple environmental perception information, multiple environmental perception information can be integrated to assist the determination process of the above-mentioned target entity, or multiple environmental perception information can be used according to priority to assist the determination process of the above-mentioned target entity, and there is no limitation on this.
[0275] Stage 4: NPC voice feedback;
[0276] NPC will give voice feedback to the user. The types of voice feedback include: at least one of instant feedback, command execution feedback, and dynamic feedback. Among them, instant feedback means that after the user issues a natural language command to the NPC, the NPC immediately gives voice feedback on the natural language command, such as "received", "start execution", "OK", etc.; execution command feedback means that after the NPC executes the control command indicated by the intention of the natural language command, the NPC gives voice feedback on the execution result of the control command, such as "successful execution", "failed to execute successfully", etc.; dynamic feedback means that when the user does not issue a natural language command to the NPC, the NPC gives voice feedback based on real-time environmental perception.
[0277] The above-mentioned voice feedback is obtained by converting text content into speech. The text content is inferred by a large language model combined with the above-mentioned spatial data and dynamic environmental perception information. Using a large language model to dynamically generate the text content of voice feedback can, on the one hand, break through the monotonous, mechanical, and easily repetitive voice feedback of fixed templated voice feedback; on the other hand, it does not need to save too many pre-made template voices in advance, which can avoid the problem of too large a total data volume on the client.
[0278] Stage 5: NPC behavior control;
[0279] Taking the user's input as input, and the spatial data of the virtual scene and the runtime information corresponding to the master virtual character as reference, the AI model is used to determine the target entity in the virtual space and the control instructions for the NPC;
[0280] Control instructions are instructions that can be used for NPC behavior tree execution; the subsequent actions that the control instruction indicates the NPC needs to perform can be a single action or a sequence of actions. In the case where the control instruction is a multi-action sequence instruction, the multi-action sequence instruction is used to instruct the NPC to perform a sequence of actions. The data structure storing the multi-action sequence instruction is in the form of a list, and the instructions to be executed in the sequence instruction are cached. When the NPC's behavior tree executes the current instruction, the cache is called back, and the instructions to be executed in the cache are continued to be sent for execution.
[0281] Hot word update function:
[0282] For phase 2, this solution also introduces a hot word system to improve the accuracy of speech recognition.
[0283] The hot word system is used to provide scene hot words related to the scenes involved in the natural language instructions during the speech recognition process, so that when a natural language instruction input by voice is received, the hot word library can be preferentially searched for words as the instruction analysis result based on the currently running virtual scene, so that the matching degree between the instruction analysis result and the virtual scene is higher.
[0284] The virtual scene includes at least one of a plurality of virtual scenes such as a virtual battle scene, a virtual transaction scene, a virtual office scene, and a virtual kitchen scene. Frequently used words in a plurality of virtual scenes, specific virtual elements existing in a virtual scene, etc. (such as virtual buildings and virtual characters that only exist in a specific scene type, etc.) are pre-analyzed as scene hot words, and a scene hot word library consisting of a plurality of scene hot words is obtained, and the plurality of scene hot words correspond to virtual scenes respectively. In addition, different scene language models are pre-trained for different virtual scenes, such as a scene language model in a battle scene that pays more attention to words related to battle, and a scene language model in a transaction scene that pays more attention to words related to transaction; the scene language model can perform a more targeted analysis of natural language commands received in a virtual scene.
[0285] For example: when a user controls a non-player character, a natural language command is received. Based on the game state, object position, and current task completion status, the first virtual scene in which the target virtual character (at least one of the main virtual character and the non-player character) is currently located is determined to be a virtual battle scene, and a battle scene language model corresponding to the virtual battle scene is obtained. In addition, based on the field of view of the target virtual character, the first scene hot words in the virtual battle scene are obtained, including: "truck", "virtual grass", "virtual house", "enter", "open truck", "defense", "desert", "attack", "oak tree", "stable", "firearms", etc. Under the constraints of the first scene hot words, the natural language command is decoded and processed through a pre-trained natural language analysis model (including an acoustic network, a language network, a preset dictionary, etc.), and the instruction analysis result is output to accurately control the non-player character. The instruction analysis result is illustrated as follows.
[0286] (1) The command analysis result is "No. 1, you go to the front Defense The command analysis result can be used to specifically control No. 1 to move forward and make a defensive posture, preventing the enemy virtual character from attacking the main control virtual character first; if there is no constraint of the hot words in the first scene, the natural language command is easily recognized as "No. 1, go to the front and make a sound", thereby affecting the player's virtual combat plan.
[0287] (2) The command analysis result is "No. 2, you go desert The command analysis result can be used to specifically control No. 2 to move to the desert to check whether there are virtual elements such as enemy virtual characters or virtual treasure chests. If there is no constraint of the hot words in the first scene, it is easy to recognize the natural language command as "No. 2, you go Mountains "Explore it", which causes No. 2 to move to the wrong position, and the human-computer interaction efficiency is low.
[0288] (3) The command analysis result is "No. 1, No. 2, give me attack ", the command analysis result can be used to specifically control No. 1 and No. 2 to carry out virtual attacks on nearby attackable virtual characters; if there is no constraint of the hot words in the first scene, it is easy to recognize the natural language command as "No. 1 No. 2, give me supply ", causing No. 1 and No. 2 to make incorrect behaviors that do not meet the players' expectations, which can neither protect the main virtual character nor easily allow the enemy virtual character to win the virtual game.
[0289] (4) The command analysis result is “No. 1, go find a nearby oak tree”. This command analysis result can be used to specifically control No. 1 to find an oak tree in the nearby area so that he can complete the game task or find an oak tree that can avoid attacks. If there is no constraint of the hot words in the first scene, it is easy to recognize the natural language command as “No. 1, go find a nearby oak tree”. Project Book ", causing No. 1 to search nearby for virtual elements that do not meet the player's expectations and fail to meet the player's virtual combat needs.
[0290] (5) The command analysis result is "No. 1, No. 2, go over there stable OK", the command analysis result can be used to specifically control No. 1 and No. 2 to search for nearby stables and move to the location of the stables; if there is no constraint of the hot words in the first scene, it is easy to recognize the natural language command as "No. 1, No. 2, go over there , right away OK", which makes No. 1 and No. 2 mistakenly believe that the main virtual character currently hopes to complete the virtual game by itself, and therefore cannot provide good game assistance to the main virtual character.
[0291] (6) The command analysis result is "No. 2, pick up the one in front firearms ", the command analysis result can be used to specifically control No. 2 to search for guns from the front and pick up the guns; if there is no constraint of the hot words in the first scene, it is easy to recognize the natural language command as "No. 2, pick up the gun in front Crab ", which makes it impossible for No. 2 to accurately find the virtual element that the main virtual character in front wants to pick up. It is possible that the player will be prompted with "Cannot find the crab", or the player may pick up the wrong virtual element "crab", interfering with the player's virtual combat process.
[0292] Ambient sound function:
[0293] In addition, the solution also provides a spatial audio enhancement solution for the ambient sound effects of the entire virtual scene.
[0294] The audio played by the terminal includes environmental audio and NPC audio. Environmental audio is audio generated based on the scene characteristics of the virtual scene where the main virtual object is currently located, so that the user can feel immersive; NPC audio is audio generated based on the characteristics of the NPC, so that the user can intuitively feel the character's emotions and physical state through the sound heard.
[0295] Environmental audio: Identify the scene elements in the virtual scene where the master virtual object is located, generate appropriate element sound effects in real time based on the scene elements / select appropriate element sound effects from the sound effect library, and synthesize the element sound effects to obtain environmental audio. For example, if the virtual scene is a forest at night, the scene elements include trees, owls, insects, etc., and the environmental audio includes the rustling of leaves, the calls of owls, the chirping of insects, etc.
[0296] NPC audio: Determine the NPCs included in the virtual scene, and generate corresponding character voices based on the NPC's character type, current emotion, and current behavior. For example, if the NPC is a middle-aged man running, the generated character voice is a deep male voice with a running panting sound effect.
[0297] When the terminal plays the ambient audio and / or NPC audio to the user, the terminal performs audio enhancement processing on the ambient audio and / or NPC audio to improve the realism of the audio.
[0298] Figure 2 The structure block diagram of a computer system provided by an exemplary embodiment of the present application is shown. The computer system 100 includes: a first terminal 110, a server 120, and a second terminal 130.
[0299] The first terminal 110 is installed and runs a client 111 that supports a virtual environment, and the client 111 can be a multiplayer online battle program. When the first terminal runs the client 111, the user interface of the client 111 is displayed on the screen of the first terminal 110. The client 111 can be any one of a battle royale shooting game, a virtual reality (VR) application, an augmented reality (AR) program, a three-dimensional map program, a virtual reality game, an augmented reality game, a casual game, a party gathering game, a first-person shooting game (First-Person Shooting Game, FPS), a third-person shooting game (Third-Person Shooting Game, TPS), a multiplayer online tactical competitive game (Multiplayer Online Battle Arena Games, MOBA), and a strategy game (Simulation Game, SLG). In this embodiment, the client 111 is an FPS game for example. The first terminal 110 is a terminal used by the first user 112. The first user 112 uses the first terminal 110 to control a first virtual object in a virtual environment to perform activities. The first virtual object may be referred to as a virtual object of the first user 112. The activities of the first virtual object include, but are not limited to, at least one of moving, jumping, teleporting, releasing skills, using props, adjusting body posture, crawling, walking, running, riding, flying, jumping, driving, picking up, shooting, attacking, and throwing. Schematically, the first virtual object is a first virtual object, such as a simulated human character or an animated human character.
[0300] The second terminal 130 is installed and runs a client 131 that supports a virtual environment, and the client 131 can be a multiplayer online battle program. When the second terminal 130 runs the client 131, the user interface of the client 131 is displayed on the screen of the second terminal 130. The client can be any one of a battle royale shooting game, a VR application, an AR program, a three-dimensional map program, a virtual reality game, an augmented reality game, an FPS, a TPS, a MOBA, and a SLG. In this embodiment, the client is an MOBA game as an example. The second terminal 130 is a terminal used by a second user 132. The second user 132 uses the second terminal 130 to control a second virtual object located in a virtual environment to perform activities. The second virtual object can be referred to as a virtual object of the second user 132. Schematically, the second virtual object is a second virtual object, such as a simulated character or an anime character.
[0301] Optionally, the first virtual object and the second virtual object are in the same virtual environment. Optionally, the first virtual object and the second virtual object may belong to the same camp, the same team, the same organization, have a friend relationship, or have temporary communication permissions. Optionally, the first virtual object and the second virtual object may belong to different camps, different teams, different organizations, or have a hostile relationship.
[0302] Optionally, the client installed on the first terminal 110 and the second terminal 130 is the same, or the client installed on the two terminals is the same type of client on different operating system platforms (Android or IOS). The first terminal 110 may refer to one of a plurality of terminals, and the second terminal 130 may refer to another of a plurality of terminals. This embodiment is only illustrated by taking the first terminal 110 and the second terminal 130 as examples. The device types of the first terminal 110 and the second terminal 130 are the same or different, and the device type includes at least one of: a smart phone, a tablet computer, an e-book reader, an MP3 player, an MP4 player, a laptop portable computer, and a desktop computer.
[0303] Figure 2 Only two terminals are shown, but in different embodiments, there are multiple other terminals 140 that can access the server 120. Optionally, there are one or more terminals 140 corresponding to the developer, and a development and editing platform for the client supporting the virtual environment is installed on the terminal 140. The developer can edit and update the client on the terminal 140, and transmit the updated client installation package to the server 120 through a wired or wireless network. The first terminal 110 and the second terminal 130 can download the client installation package from the server 120 to update the client.
[0304] The first terminal 110 , the second terminal 130 , and other terminals 140 are connected to the server 120 via a wireless network or a wired network.
[0305] The server 120 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. The server 120 is used to provide background services for the client supporting the three-dimensional virtual environment. Optionally, the server 120 undertakes the main computing work, and the terminal undertakes the secondary computing work; or, the server 120 undertakes the secondary computing work, and the terminal undertakes the main computing work; or, the server 120 and the terminal adopt a distributed computing architecture for collaborative computing.
[0306] In an illustrative example, the server 120 includes a processor 122, a user account database 123, a battle service module 124, and a user-oriented input / output interface (I / O interface) 125. The processor 122 is used to load instructions stored in the server 120 and process data in the user account database 123 and the battle service module 124; the user account database 123 is used to store data of user accounts used by the first terminal 110, the second terminal 130, and other terminals 140, such as the user account's avatar, the user account's nickname, the user account's combat power index, and the service area where the user account is located; the battle service module 124 is used to provide multiple battle rooms for users to fight, such as 1V1 battle, 3V3 battle, 5V5 battle, etc.; the user-oriented I / O interface 125 is used to establish communication and exchange data with the first terminal 110 and / or the second terminal 130 through a wireless network or a wired network.
[0307] The method provided in this application can be applied to at least one of the following scenarios, but is not limited to: virtual reality applications, three-dimensional map programs, casual games, party games, first-person shooting games (First-Person Shooting Game, FPS), third-person shooting games (Third-Person Shooting Game, TPS), multiplayer online tactical competitive games (Multiplayer Online Battle Arena Games, MOBA), multiplayer gunfight survival games, etc. The following embodiments are illustrated by the application in the game. The method provided in this application can be completely executed by the client, such as a stand-alone game; it can also be completely executed by the server, such as in a cloud game, the server performs all calculations, and the client is only responsible for displaying and collecting the user's operation behavior; the client and the server can also collaborate, with the client responsible for part and the server responsible for part.
[0308] Figure 3A flowchart of a method for controlling a virtual character provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device. The method includes:
[0309] Step 510: Displaying a main control virtual character or NPC in a virtual environment;
[0310] Exemplarily, the main control virtual character is a virtual character that the user directly controls in the virtual environment. There are also one or more non-player characters (NPCs) in the virtual environment; the NPCs and the main control virtual character belong to the same virtual camp and are teammates of the main control virtual character.
[0311] Step 520: Obtain command information;
[0312] Exemplarily, the command information includes natural semantics for controlling NPCs; commanding non-player characters in a virtual scene through command texts, the command texts may be directly input by the user, or may be extracted from voice information input by the user, and the present application does not limit the control method of the command texts. Exemplarily, the command information includes natural semantics for controlling NPCs, also known as natural language commands, or natural language commands in voice form.
[0313] For example, in addition to the main virtual character controlled by the user, the user also has one or more non-player characters (NPC) as teammates in the virtual environment. The user has control over the non-player characters and commands the non-player characters to make the desired feedback to the scene objects in the virtual environment through command text in natural language, thereby commanding the non-player characters to complete the task in collaboration with the main virtual character.
[0314] The command information corresponds to at least two semantic understanding methods; the at least two semantic understanding methods correspond to different NPC control methods in the virtual environment. Exemplarily, at least one of the scene entity, word pronunciation, and behavior intention indicated in the at least two semantic understanding methods is different.
[0315] For example, when there are at least two semantic understanding methods for the pronunciation of words in command information, it means that from the dimension of natural language, there are at least two semantic understanding methods for the command information itself. And when there are at least two semantic understanding methods for the scene entity / behavior intention of command information, it means that the command information is combined with the virtual environment, and there are at least two semantic understanding methods in the scenario of controlling NPC in the virtual environment.
[0316] Step 530: In response to the command information, controlling the NPC to perform a virtual activity according to a first semantic understanding method of at least two semantic understanding methods;
[0317] Exemplarily, the first semantic understanding method is one of at least two semantic understanding methods corresponding to the command information. The first semantic understanding method is determined in at least two semantic understanding methods based on the environmental perception information of the main virtual character and / or NPC. In some examples, the main virtual character and / or NPC obtains environmental perception information within the perception range of vision, hearing, movement trajectory and other perception methods. Based on the environmental perception information, the first semantic understanding method determined in at least two semantic understanding methods is close to the current perception of the virtual environment by the main virtual character and / or NPC, and the NPC is controlled with reference to the current perception of the virtual character and / or NPC to the virtual environment.
[0318] Exemplarily, since the first semantic understanding mode is determined in at least two semantic understanding modes based on the environmental perception information of the controlling virtual character and / or NPC; there are at least two situations in the virtual environment (at least one of the positions, virtual health values, virtual item holdings, virtual skill cooling time, virtual level, etc. of the controlling virtual character and / or NPC in the above two situations is different), facing the same command information, the first semantic understanding modes determined are different. That is to say, as the situation in the virtual environment changes, in response to the same command information, the NPC is controlled to perform different virtual activities.
[0319] To summarize, the method provided in this embodiment determines to execute virtual activities according to a first semantic understanding method by taking the environmental perception information of the controlling virtual character and / or the NPC as a reference in at least two semantic understanding methods of the command information; when there are at least two semantic understanding methods of the command information, on the one hand, there is no need for the user to add more basis for the command information, that is, the NPC can be controlled to execute virtual activities according to the command information, thereby improving the efficiency of human-computer interaction; on the other hand, taking the environmental perception information of the controlling virtual character and / or the NPC as a reference ensures that the NPC and the controlling virtual character collaboratively execute virtual activities.
[0320] Next, at least two semantic understanding methods corresponding to command information are introduced. In one implementation, command information corresponds to at least two semantic understanding methods of scene entities; in another implementation, command information corresponds to at least two semantic understanding methods of word pronunciation; in yet another implementation, command information corresponds to at least two semantic understanding methods of behavioral intentions.
[0321] The above three implementation methods will be introduced below in the form of separate embodiments.
[0322] There are at least two semantic understanding modes of scene entities corresponding to the command information.
[0323] Figure 4A flowchart of a method for controlling a virtual character provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device. Figure 3 In the illustrated embodiment, step 530 may be implemented as step 532:
[0324] Step 532: In response to the command information, control the NPC to perform a virtual activity on a first scene object located within the perception range of the master virtual character and / or the NPC;
[0325] Exemplarily, the perception range is a virtual area where the main virtual character and / or NPC obtains environmental perception information. The perception range can be a connected area in the virtual environment, or multiple independent areas that are not connected to each other. This application does not limit the shape of the perception range. In the perception range, the main virtual character and / or NPC can obtain information in the virtual environment through perception methods such as vision, hearing, and movement trajectory.
[0326] In this embodiment, there are multiple scene objects of the same type in the virtual environment, and at least two semantic understanding methods corresponding to the command information are used to indicate the execution of virtual activities for the multiple scene objects of the same type; that is, there are at least two semantic understanding methods corresponding to the scene entities. The multiple scene objects include the first scene object. Exemplarily, since the command text does not explicitly indicate which scene object among the multiple scene objects is to be executed for the virtual activity, the command text corresponds to at least two semantic understanding methods.
[0327] Taking multiple scene objects as virtual vehicles in a virtual environment as an example, there are multiple virtual vehicles in the virtual environment, distributed in various locations in the virtual environment. The command information is used to control the NPC to move to the location of the virtual vehicle. Since the command information does not clearly indicate the location of the virtual vehicle to be moved to, there are multiple semantic understanding methods, that is, moving to any location of a virtual vehicle satisfies the natural language semantics of the command information. In an implementation provided by the present application, among multiple semantic understanding methods, a first semantic understanding method is determined according to the perception range, and according to the first semantic understanding method, the NPC is controlled to move to the location of the virtual vehicle within the perception range, so that a semantic understanding method is determined in multiple semantic understanding methods with reference to the perception range, and the NPC is controlled to perform virtual activities for the first scene object; on the one hand, there is no need for the user to supplement more basis for the command information, which improves the efficiency of human-computer interaction; on the other hand, the NPC will not mistakenly perform virtual activities for scene objects outside the perception range of itself or the main virtual character, resulting in a departure from the collaboration with the main virtual character, or entering a range where no perception information is obtained (i.e., a strange range / dangerous range); it is ensured that the NPC and the main virtual character collaborate to perform virtual activities.
[0328] In an optional implementation, the NPC performs the virtual activity on the first scene object within the field of view; accordingly, this step can be implemented as follows:
[0329] In response to the command information, controlling the NPC to perform a virtual activity on a first scene object located in the field of view;
[0330] Exemplarily, the visual field is the observation range of the virtual environment by the controlling virtual character and / or NPC. Exemplarily, the visual field can be the visual perception range of the controlling virtual character and / or NPC itself, or the visual perception range of the perception skills or perception tools used.
[0331] The first semantic understanding mode is used to instruct the NPC to perform virtual activities for the first scene object located in the field of vision. It fully considers that visual perception is a perceptual way to obtain information in the virtual environment. It is a special design for controlling NPC in a natural semantic way. It fully considers that the user directly controls the main virtual character in the virtual environment. It fully considers that the desire to control the NPC to perform virtual activities is generated after the main virtual character observes the scene objects in the virtual environment. For example, when the main virtual character observes the virtual vehicle in the virtual environment, the desire to control the NPC to move is generated to use the virtual vehicle as a cover to cover the NPC and help the NPC avoid being discovered by the enemy virtual character. Only when the main virtual character observes the virtual vehicle in the virtual environment will the desire to use the virtual vehicle as a cover be generated. The first semantic understanding mode determined according to the field of vision is used to instruct the NPC to perform virtual activities for the first scene object located in the field of vision, and the scene object targeted by the virtual activity is limited to the field of vision. It can combine the command text of the natural semantic mode with the current visual observation of the virtual environment to flexibly control the virtual character.
[0332] Optionally, the field of view is the field of view observed by the controlling virtual character and / or NPC, and is the range of the observation screen obtained by a virtual camera model that follows the movement of the controlling virtual character and / or NPC; and / or, the field of view is an expanded field of view obtained by using perception skills or perception tools, and the user of the perception skills (such as skills to expand the field of view) or perception tools (such as virtual telescopes, virtual drones, etc.) includes the controlling virtual character and / or NPC.
[0333] In an optional implementation, the NPC performs the virtual activity on the first scene object in the first virtual terrain; accordingly, this step can be implemented as follows:
[0334] In response to the command information, controlling the NPC to perform a virtual activity on a first scene object located in the first virtual terrain;
[0335] Exemplarily, the first virtual terrain is the virtual terrain to which the main virtual character and / or NPC is located; exemplary, the first virtual terrain is a connected area including the main virtual character and / or NPC's location.
[0336] Next, the situations where the first virtual terrain is indoor terrain, block terrain, and outdoor terrain are introduced respectively:
[0337] In the case where the first virtual terrain is an indoor terrain, in response to the command information, controlling the NPC to perform a virtual activity on a first scene object located in the first area;
[0338] When the first virtual terrain is an indoor terrain, the first position where the master virtual character and / or NPC is located belongs to the indoor terrain; the first area is an indoor connected area including the first position where the master virtual character and / or NPC is located.
[0339] After the main virtual character and / or NPC enters the indoor area, the virtual activity is performed on the first scene object in the indoor area (i.e., the first area). It is fully considered that the indoor area in the virtual environment is an area rich in virtual resources / an area that provides cover for the virtual character; even if the indoor area is blocked by walls or has many indoor objects, the NPC can be controlled to perform virtual activities in the indoor area, and is not limited to the one virtual room where the main virtual character and / or NPC is located.
[0340] In the case where the first virtual terrain is a block terrain adjacent to the building, in response to the command information, the NPC is controlled to perform a virtual activity for the first scene object located in the second area;
[0341] When the first virtual terrain is a street terrain, the first position where the main virtual character and / or NPC is located is close to the building; the second area includes the buildings around the second position where the main virtual character and / or NPC is located.
[0342] When the main virtual character and / or NPC is located near a building, the virtual activities performed by the NPC can be performed against the building. It is fully considered that the indoor area in the virtual environment is an area rich in virtual resources / an area that provides cover for the virtual character; when the main virtual character and / or NPC is located near a building, the NPC can approach or even enter the building to obtain virtual resources or get cover from the building.
[0343] In a case where the first virtual terrain is an outdoor terrain far away from the building, in response to the command information, controlling the NPC to perform a virtual activity for the first scene object located in the third area;
[0344] When the first virtual terrain is an outdoor terrain, the first position where the main virtual character and / or NPC is located belongs to the outdoor terrain; the third area is an area with the same virtual vegetation distribution as the third position where the main virtual character and / or NPC is located.
[0345] In the case where the main virtual character and / or NPC is in outdoor terrain, the virtual activity performed by the NPC can be performed for the first scene object in the area with the same virtual vegetation distribution. The virtual terrain when the NPC performs the virtual activity has the same virtual vegetation distribution as the third position, which can ensure that the NPC is shielded by the virtual vegetation, and ensure that the shielding effect of the virtual vegetation obtained in the process of executing the virtual environment is the same as the shielding effect of the main virtual character and / or the NPC before performing the virtual activity.
[0346] In summary, the method provided in this embodiment determines to perform a virtual activity on a first scene object within the perception range by taking the perception range of the master virtual character and / or the NPC as a reference in at least two semantic understanding methods of the command information. In the case where there are at least two semantic understanding methods of the command information, on the one hand, there is no need for the user to add more basis for the command information, that is, the NPC can be controlled to perform virtual activities according to the command information, thereby improving the efficiency of human-computer interaction; on the other hand, the NPC will not mistakenly perform virtual activities on scene objects outside the perception range of itself or the master virtual character, thereby ensuring that the NPC and the master virtual character perform virtual activities in collaboration.
[0347] The command information corresponds to at least two semantic understanding modes of word pronunciation.
[0348] Figure 5 A flowchart of a method for controlling a virtual character provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device. Figure 3 In the illustrated embodiment, step 530 may be implemented as steps 534 and 535:
[0349] Step 534: In response to the command information, convert the command information into command text according to the natural language pronunciation of the scene objects in the perception range;
[0350] Exemplarily, the perception range is a virtual area where the main virtual character and / or NPC obtains environmental perception information; the perception range can be a connected area in the virtual environment, or multiple independent areas that are not connected to each other. This application does not limit the shape of the perception range. In the perception range, the main virtual character and / or NPC can obtain information in the virtual environment through perception methods such as vision, hearing, and movement trajectory.
[0351] In this embodiment, the command information is voice information. Due to the richness of natural language, the command information corresponds to at least two semantic understanding methods of word pronunciation. For example, in the Chinese phonetic expression, the pinyin pronunciation "qiang xie" can be the pronunciation of a virtual shooting prop "gun" or the pronunciation of a food "qiangxie" (a crab sashimi food). It is understandable that since there are such situations in natural language where the pronunciation is the same or similar but the meaning is different, the command information corresponds to at least two semantic understanding methods of word pronunciation.
[0352] In an implementation provided by the present application, among multiple semantic understanding methods, a first semantic understanding method is determined according to the perception range, and according to the first semantic understanding method, command information is converted into command text according to the natural language pronunciation of scene objects in the perception range. It is implemented that among multiple semantic understanding methods of word pronunciation, it is determined that the command text is converted according to the natural language pronunciation of scene objects in the perception range; accordingly, the command text includes the name text and / or introduction text of the scene object.
[0353] On the one hand, the natural language pronunciation of scene objects is used as a basis to avoid calling complex artificial neural networks to predict the semantics in voice information, reducing the computational complexity of converting command information into command text. On the other hand, full consideration is given to the fact that command information is often highly accompanied by information in the virtual environment when controlling NPCs, that is, voice control of NPCs requires the purpose of indicating virtual activities, such as picking up virtual guns, in which virtual guns are scene objects that appear in the virtual environment, and NPCs can be flexibly controlled with reference to scene objects within the perception range.
[0354] In an optional implementation, this step may be implemented as follows:
[0355] If the similarity between the pronunciation of the first phoneme in the command information and the natural language pronunciation of the scene object exceeds a similarity threshold and there is a candidate text for the first phoneme, determine the text information of the first phoneme as the name text and / or introduction text of the scene object, and construct the command text according to the text information;
[0356] Exemplarily, when the similarity between the pronunciation of the first phoneme in the command information and the natural language pronunciation of the scene object exceeds the similarity threshold, the pronunciation of the first phoneme in the command information is similar to the name text and / or introduction text of the scene object. The first phoneme has a candidate text, which is used to indicate that there is another candidate text with a pronunciation similar to the first phoneme. The candidate texts of the first phoneme do not include the name text and / or introduction text of the scene object.
[0357] In one example, the virtual environment is a shooting game in which the similarity between the first phoneme "qiangxie" in the command information and the natural language pronunciation of the scene object virtual firearm exceeds a similarity threshold, and there is a candidate text "qiangxie" for the first phoneme; the text information of the first phoneme is determined to be the name text "gun" of the scene object.
[0358] Step 535: According to the command text, control the NPC to perform the virtual activity according to the first semantic understanding method;
[0359] Exemplarily, the command text is converted according to a first semantic understanding method, and accordingly, the NPC is controlled to perform a virtual activity according to the first semantic understanding method.
[0360] To summarize, the method provided in this embodiment determines the command text of the command information according to the natural language pronunciation of the scene objects in the perception range in at least two semantic understanding methods of the command information; when there are at least two semantic understanding methods of the command information, on the one hand, there is no need for the user to add more basis for the command information, that is, the NPC can be controlled to perform virtual activities according to the command information, thereby improving the efficiency of human-computer interaction; on the other hand, full consideration is given to the fact that the command information is often highly accompanied by information in the virtual environment when controlling the NPC, thereby achieving flexible control of the NPC.
[0361] There are at least two semantic understanding methods for the command information corresponding to the behavioral intention.
[0362] Figure 6 A flowchart of a method for controlling a virtual character provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device. Figure 3 In the illustrated embodiment, step 530 may be implemented as step 536:
[0363] Step 536: In response to the command information, controlling the NPC to perform a virtual activity according to the first behavioral intention;
[0364] Exemplarily, the perception range is a virtual area where the main virtual character and / or NPC obtains environmental perception information. The perception range can be a connected area in the virtual environment, or multiple independent areas that are not connected to each other. This application does not limit the shape of the perception range. In the perception range, the main virtual character and / or NPC can obtain information in the virtual environment through perception methods such as vision, hearing, and movement trajectory.
[0365] In this embodiment, due to the richness of natural language, the command information corresponds to at least two semantic understanding methods of behavioral intentions. For example, the command information includes walking forward. Since the command information does not clearly indicate in which reference system to move forward, there are multiple ways to understand the behavioral intentions, which can be walking forward in the direction the NPC is facing, or walking forward with the marked location point as the destination. Exemplarily, the first behavioral intention determined in the semantics of multiple behavioral intentions according to the perception range corresponds to the type of virtual activity that is allowed to be executed in the perception range to ensure that the virtual activity can be performed in the virtual environment.
[0366] In an optional implementation, this step may be implemented as follows:
[0367] In response to the command information, the NPC is controlled to perform a virtual activity based on a second direction according to the directional words in the command information, taking the first direction in the perception range as a reference direction;
[0368] Exemplarily, the first direction includes but is not limited to the direction that the main virtual character or NPC is facing. The command information includes a direction word, which is used to describe the angle between the second direction and the first direction and / or describe the second direction in a reference system constructed based on the first direction.
[0369] Exemplarily, the first direction as the reference direction is to determine the first direction as the front; the reference system constructed based on the first direction is a plane coordinate system and / or a three-dimensional coordinate system constructed with the first direction as the positive semi-axis direction of x.
[0370] In summary, the method provided in this embodiment ensures that the virtual activity can be smoothly executed in the virtual environment by executing the virtual activity according to the type of virtual activity permitted to be executed in the perception range in at least two semantic understanding modes of the command information. According to the first direction in the perception range and the directional words included in the command information, it is determined that the virtual activity is executed based on the second direction; there is no need for the user to add more basis for the command information, that is, the NPC can be controlled to execute the virtual activity according to the command information, which improves the efficiency of human-computer interaction and avoids the problem that when the command information corresponds to at least two semantic understanding modes of behavioral intentions, the direction of executing the virtual activity cannot be determined according to the command information.
[0371] Figure 7 A flowchart of a method for controlling a virtual character provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device. Figure 3 In the illustrated embodiment, step 530 may be implemented as steps 542 to 550:
[0372] Step 542: Determine a command text in natural language form according to the command information;
[0373] Exemplarily, the command information is subjected to speech-to-text (Automatic Speech Recognition, ASR) processing to determine the acquired command text in natural language form. Speech-to-text processing generally includes calling components such as acoustic models and language models to recognize the pronunciation, vocabulary, grammatical structure and other information of the command information and convert it into a command text in text format.
[0374] Exemplarily, the command information is obtained in response to a trigger operation of a button-type control, and the command information is recorded in response to the trigger operation on the button-type control. Exemplarily, the command information is obtained when the user controls the main virtual character to perform a virtual activity in the virtual environment, and the microphone recording function is turned on during the process of the user controlling the main virtual character to perform a virtual activity in the virtual environment.
[0375] Step 544: calling the conditional random field to perform entity prediction on the command text to obtain the entity label carried by the command text;
[0376] Exemplarily, Conditional Random Fields (CRF) is a statistical model that handles complex dependencies between input and output by defining feature functions and corresponding parameter weights; it is capable of extracting corresponding entity labels from numerous expressions in natural semantics; the conditional random field can obtain the context information of the input command text sequence, calculate the conditional probability distribution, and obtain the entity label carried by the command text according to the conditional probability distribution.
[0377] Step 546: Determine a first scene object in the perception range according to the entity label;
[0378] Exemplarily, the perception range is a virtual area where the main virtual character and / or NPC obtains environmental perception information. The perception range can be a connected area in the virtual environment, or multiple independent areas that are not connected to each other. This application does not limit the shape of the perception range. In the perception range, the main virtual character and / or NPC can obtain information in the virtual environment through perception methods such as vision, hearing, and movement trajectory.
[0379] The first scene object is determined from the virtual environment in combination with environmental perception information, where the environmental perception information includes information perceived by at least one of the main virtual character and the non-player character within a perception range in the virtual environment.
[0380] The virtual environment includes multiple candidate entities that match the entity tags carried by the command text, and the first scene object is an entity selected from the multiple candidate entities based on the environmental perception information of the main virtual character or the non-player character. For example, the game application first selects multiple candidate entities that match the entity tags carried by the command text from the entity set, and then selects the entity that can be perceived by the main virtual character or the non-player character from the multiple candidate entities according to the environmental perception information as the first scene object. Exemplarily, the environmental perception information includes at least one of the following: visual information perceived within the field of view of the main virtual character; auditory information perceived within the hearing range of the main virtual character; information perceived by the perception skills or perception tools possessed by the main virtual character; visual information perceived within the field of view of the non-player character; auditory information perceived within the hearing range of the non-player character; information perceived by the perception skills or perception tools possessed by the non-player character.
[0381] Step 548: calling the prediction network to perform intent recognition on the command text to obtain a classification label of the command text;
[0382] The prediction network has the ability to predict the classification label corresponding to the command text. The prediction network is used to predict the control method of the command text for the NPC in at least one behavioral dimension;
[0383] Optionally, call the hierarchical prediction network to perform intent recognition on the command text to obtain a classification label for the command text;
[0384] Exemplarily, the hierarchical prediction network includes at least two layers of sub-networks, the upper network and the lower network are cascaded, and the lower network further performs the prediction of the classification label according to the prediction result output by the upper network. At least two layers of sub-networks are constructed based on the tree structure of multiple classification labels, and the hierarchical prediction network constructs at least two layers of sub-networks corresponding to the hierarchical structure for the tree structure of the classification labels, and splits the prediction task of a wide variety of classification labels into prediction sub-tasks performed by at least two layers of sub-networks, so as to reduce the complexity of classification prediction of each layer of sub-network.
[0385] Step 550: determining a control command of the NPC according to the classification label, and based on the control command, controlling the NPC to perform a virtual activity on the first scene object according to the first semantic understanding method;
[0386] Taking the classification label of the command text as input, the target entity in the virtual space and the control command for the NPC are determined; illustratively, each classification label in the command text has a corresponding control command, and the control command for the NPC is determined based on the correspondence between the classification label and the control command.
[0387] The subsequent action that the control command instructs the NPC to perform can be a single action or a sequence of actions. In the case where the control command is a sequence instruction of multiple actions, the sequence instruction of multiple actions is used to instruct the NPC to perform a sequence of actions. The data structure storing the sequence instruction of multiple actions is in the form of a list, and the instructions to be executed in the sequence instruction are cached. When the NPC's behavior tree executes the current instruction, the cache is called back, and the instructions to be executed in the cache are continued to be sent for execution.
[0388] To summarize, the method provided in this embodiment determines to execute virtual activities according to a first semantic understanding method by taking the environmental perception information of the controlling virtual character and / or the NPC as a reference in at least two semantic understanding methods of the command information; when there are at least two semantic understanding methods of the command information, on the one hand, there is no need for the user to add more basis for the command information, that is, the NPC can be controlled to execute virtual activities according to the command information, thereby improving the efficiency of human-computer interaction; on the other hand, taking the environmental perception information of the controlling virtual character and / or the NPC as a reference ensures that the NPC and the controlling virtual character collaboratively execute virtual activities.
[0389] In an optional implementation, before step 546, the following steps are further included:
[0390] Obtain the spatial position of each scene object in the virtual environment;
[0391] Exemplarily, the spatial position of the scene object in the virtual scene is used to indicate the deployment of the scene object in the virtual scene, and the spatial information is used to indicate the position, size and other information of the scene object in the virtual scene after the scene object is deployed in the virtual scene.
[0392] Optionally, a description of the spatial location follows.
[0393] Obtain at least one of the coordinate position, orientation information, bounding box information, and cover point information of each scene object in the virtual environment;
[0394] Among them, the coordinate position (Location) is used to indicate the position of the scene object in the virtual scene, such as the coordinate information of the center point or preset point of the scene object in the virtual scene; the orientation (Rotation) is used to indicate the direction facing the scene object in the virtual scene, such as the direction facing the front of the scene object in the virtual scene; the bounding box (BoundingBox) is used to indicate the size of the scene object in the virtual scene; the cover points are used to indicate the recommended virtual character position points when the virtual character approaches the scene object, so that the scene object can mask the virtual character.
[0395] ·Select multiple candidate scene objects in the perception range from various scene objects in the virtual environment according to the spatial position;
[0396] Exemplarily, the spatial position of the scene object in the virtual scene is used to indicate the deployment of the scene object in the virtual scene, and the spatial information is used to indicate the position, size and other information of the scene object in the virtual scene after the scene object is deployed in the virtual scene.
[0397] Exemplarily, the perception range is a virtual area where the main virtual character and / or NPC obtains environmental perception information. The perception range can be a connected area in the virtual environment, or multiple independent areas that are not connected to each other. This application does not limit the shape of the perception range. In the perception range, the main virtual character and / or NPC can obtain information in the virtual environment through perception methods such as vision, hearing, and movement trajectory.
[0398] According to the spatial positions, a plurality of scene objects located within the perception range in the virtual environment are determined, and the plurality of scene objects located within the perception range are determined as a plurality of candidate scene objects.
[0399] Obtain visual labels for multiple candidate scene objects in the perception range;
[0400] The visual tag is used to describe the scene object in at least one dimension. The description dimension of the scene object by the visual tag includes but is not limited to at least one of the type description, material, transparency, color, surface characteristics, and shape of the scene object.
[0401] Accordingly, step 546 can be implemented as follows:
[0402] Calculate the similarity between the visual labels of multiple candidate scene objects and the entity labels respectively, and determine the first scene object with the highest similarity among the multiple candidate scene objects.
[0403] Exemplarily, the entity tag is used to describe the scene object for which the command information indicates to perform the virtual activity. Exemplarily, some words in the command information correspond to the entity tag.
[0404] Based on the similarity between the entity tag and the visual tags of the plurality of candidate scene objects, the first scene object corresponding to the command information is screened out from the plurality of candidate scene objects in the perception range. The command information is used to command the non-player character in the virtual scene; the command information contains not only language to introduce which virtual scene object the virtual activity is directed to (for example, the command information indicates to move to the virtual building, i.e., the virtual activity of moving is performed on the virtual building), but also words indicating the identity of the non-player character performing the virtual activity.
[0405] Figure 8A schematic diagram of command information provided by an exemplary embodiment of the present application is shown. The multiple input words included in the command information 600 include "blue" and "car"; the virtual scene includes two scene objects; it can be understood that in different embodiments, the virtual scene may include more scene objects. The visual tags of scene object A 601 include metal, old, car, truck, blue, and damaged; the visual tags of scene object B 602 include metal, scratch, car, damaged, cash truck, and black. The similarity scores between the multiple input words included in the command information 600 and the visual tags of the scene objects are calculated respectively; the multiple similarity scores correspond to the multiple input words one by one; for the input word "blue" in the command information 600, the similarity score with the scene object A 601 is 1.0; for the input word "car" in the command information 600, the similarity score with the scene object A 601 is 1.0; the similarity between the scene object A 601 and the command information 600 is 1.0+1.0=2.0. Similarly, the similarity between scene object B 602 and command information 600 is calculated to be 1.0+0.91=1.91. The similarity between scene object A 601 and command information 600 is greater than the similarity between scene object B 602 and command information 600, and scene object A 601 is determined as the first scene object.
[0406] Next, the process of obtaining visual tags is introduced. Take a scene object in a virtual scene as an example:
[0407] Get the attribute text of scene objects in the virtual scene;
[0408] Exemplarily, attribute text is used to introduce the inherent attributes of scene objects in a virtual scene; on the one hand, attribute text realizes the description of scene objects in text mode, and while describing scene objects, it provides semantic information for the prediction of visual labels of scene objects. On the other hand, attribute text is a description of scene objects in a virtual scene. In the face of a large number of scene objects in a virtual scene and the existence of reused object models, it can more accurately describe the inherent attributes of scene objects in the virtual scene. For example, in a virtual environment, in order to avoid redesigning the object model of a scene object, the object model of a virtual table is subjected to at least one of the transformation operations such as scaling, stretching, and rotation, and the transformed object model is deployed in the virtual environment as a virtual bench; in the face of the above-mentioned model reuse, attribute text can describe the inherent characteristics of scene objects in the virtual scene.
[0409] In an optional implementation, the attribute text includes at least one of the name of the scene object in the virtual scene and the size of the scene object in the virtual scene; illustratively, the size of the scene object in the virtual scene is used to indicate the virtual space occupied by the scene object in the virtual scene, avoiding the impact on the description of the scene object when the size of the object model is changed when the object model is reused. The name in the virtual scene can reflect the purpose of the scene object in the virtual scene, avoiding the impact on the description of the scene object when two different scene objects are deployed according to the same object model when the object model is reused.
[0410] Obtain appearance images of scene objects in the virtual scene;
[0411] Exemplarily, appearance images are used to describe the style of scene objects. Based on the richness of natural language, players can describe scene objects in virtual scenes from multiple semantic perspectives. When constructing visual labels for scene objects, it is necessary to obtain comprehensive description information of the scene objects. Appearance images carry the color, texture, shape and other appearance styles of scene objects in the picture mode, as well as the relative positional relationships between various sub-parts, and can comprehensively describe scene objects from the picture mode (or visual mode).
[0412] In an optional implementation, the appearance image of the scene object includes an image obtained by observing the scene object from at least two viewing angles. Observing the scene object from at least two viewing angles in the appearance image can avoid the problem of occlusion of the spatial structure of the scene object and the inability to present all appearance styles of the scene object in a single viewing angle.
[0413] It should be noted that the appearance image can be obtained by the scene object in the virtual scene, such as capturing the appearance image of the scene object in the scene object; the appearance image can also be obtained outside the virtual scene, such as capturing the appearance image of the scene object in the development interface during the development and design process of the scene object.
[0414] Call the multimodal model to predict the attribute text and appearance image of the scene object to obtain the visual label of the scene object;
[0415] Exemplarily, the multimodal model has the ability to perform model prediction on text information and image information of different modalities. In this embodiment, the input parameters of the multimodal model are the attribute text and appearance image of the scene object; the multimodal model predicts the visual label of the scene object from the two modalities of text modality and image modality, and the visual label is used to describe the scene object in at least one dimension.
[0416] Exemplarily, the multimodal model includes an artificial neural network (ANN), and the network of the artificial neural network includes but is not limited to at least one of the following: Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Temporal Convolutional Networks (TCN), Long Short-Term Memory (LSTM), Multilayer Perceptron (MLP), Support Vector Machine (SVM). The present application does not limit the network structure of the multimodal network. Exemplarily, the multimodal model has the ability to extract the visual features of scene objects from the input information of text modality and picture modality to obtain the visual labels of the scene objects. Exemplarily, the multimodal model can be a classification model that assigns scene objects to preset type labels; it can also be a prediction model that predicts labels for scene objects that can describe the visual characteristics of scene objects.
[0417] Optionally, an anthropomorphic rewrite is performed on the visual label of the scene object to obtain a rewritten visual label that conforms to the spoken expression of natural language. The visual label is used to describe the scene object in at least one dimension, and the appearance image of the scene object presents rich visual features of the scene object. In the spoken expression of natural language, the description of the scene object cannot cover the visual features of the scene object in each dimension. For example, when describing a virtual bed in a virtual scene with spoken expression, people usually pay attention to the color, material, and placement of the virtual bed, but usually neglect to pay attention to the process used for the headboard and the side panels of the bed, such as carving, embossing, etc. The purpose of performing anthropomorphic rewriting on the visual label of the scene object is to obtain a matching label that is closer to the spoken expression; it can be understood that performing anthropomorphic rewriting can be to delete part of the content in the visual label, or to change the visual label to a label with the same semantics but different text expression. Exemplarily, performing anthropomorphic rewriting on the visual label is implemented based on calling a natural language model. Exemplarily, the natural language model carries prior knowledge of the spoken expression of natural language. For example, the natural language model is implemented as a large language model (LLM).
[0418] Furthermore, executing the call to the natural language model to perform anthropomorphic rewriting includes:
[0419] Obtain a first sample label pair;
[0420] Constructing a rewriting guidance sentence according to the first sample label pair and the visual label;
[0421] Input the rewriting guide sentence into the natural language model to predict the matching label of the spoken expression that conforms to the natural language;
[0422] In this example, the first sample label pair includes the first label before anthropomorphic rewriting and the second label obtained after anthropomorphic rewriting; taking the scene object as a virtual bed in a virtual scene as an example, the first label in the first sample label pair is: "Type description: wooden double bed; material: [wood]; transparency: opaque; color: [brown]; surface feature: [carving]; shape: [rectangular]". The second label is: "bed, brown, wooden, rectangular".
[0423] The rewriting guide sentence has the natural semantics of rewriting the visual label with reference to the first sample label pair; in one example, the visual label that needs to be rewritten in an anthropomorphic way is: "Type description: green plant pot; material: [plant]; transparency: opaque; color: [brown, green]; surface features: [smooth]; shape: [irregular shape]". Correspondingly, based on the first sample label pair and the visual label, the guide sentence obtained is: Example: rewrite "Type description: wooden double bed; material: [wood]; transparency: opaque; color: [brown]; surface features: [carving]; shape: [rectangular]" into "bed, brown, wooden, rectangular". According to the above example, "Type description: green plant pot; material: [plant]; transparency: opaque; color: [brown, green]; surface features: [smooth]; shape: [irregular shape]" is rewritten.
[0424] Fig. 9 A schematic diagram of a visual label of a scene object provided by an exemplary embodiment of the present application is shown. In one example, the question statement 402 is: "The following is a rendering of a three-dimensional model named A, with a size of B; please understand what the three-dimensional model is through the model name, image, and size information, and summarize the overall features, and finally output the visual label in Json format.". Among them, the first half is the first sub-part divided by the semicolon ";" in the question statement 402, which is the supplementary introduction information of the scene object. The supplementary introduction information of the scene object includes the attribute text of the scene object obtained from the data set 400, such as the name of the scene object in the virtual scene is A, and the size of the scene object in the virtual scene is B.
[0425] The second half is the second sub-part, which is the answer guide statement for the visual question answering model. Furthermore, the question statement includes a preset sentence template, and the attribute text of the scene object is filled into the preset sentence template to obtain the question statement. Among them, the filling position of the attribute text of the scene object is the position of the word "A" and the word "B" in the question sentence; wherein, the name of the scene object in the attribute text in the virtual scene is marked as the asset name (asset name), and the size of the scene object in the attribute text in the virtual scene is recorded as the asset size (asset size); fill it into the preset sentence template to obtain the question statement.
[0426] The appearance image 404 of the scene object includes images obtained by observing the scene object from nine perspectives. The question statement 402 and the appearance image 404 of the scene object are input into the visual question answering model 410 to obtain the visual label 415 of the scene object. Exemplarily, the visual label 415 of the scene object includes: "Type description: wooden double bed; material: [wood]; transparency: opaque; color: [brown]; surface feature: [carving]; shape: [rectangular]".
[0427] Further, the visual label 415 of the scene object is used to supplement the data set 400; after the visual label 415 is supplemented, the data set 400 includes the appearance image 404 of the scene object, the visual label 415, and the spatial position 420 of the scene object in the virtual scene, and the spatial position 420 includes the coordinate position, orientation information, bounding box information, and cover point information, which will be introduced in a separate embodiment below. In various embodiments of the present application, the data set 400 is also referred to as a spatial information library of the virtual scene.
[0428] Build a spatial database of the virtual environment based on the visual labels of each scene object in the virtual environment;
[0429] The spatial database includes visual labels of each scene object in the virtual environment.
[0430] Based on the perception range, the visual labels of multiple candidate scene objects in the perception range are screened in the spatial database.
[0431] Exemplarily, the spatial database includes the spatial position described above; according to the spatial position, multiple scene objects located within the perception range in the virtual environment are determined, and the multiple scene objects located within the perception range are determined as multiple candidate scene objects. And visual labels of the multiple candidate scene objects are obtained in the spatial database.
[0432] In summary, the method provided in this embodiment, by calling a multimodal model to perform predictions on attribute text and appearance images, realizes the prediction of visual labels of scene objects based on the description information of scene objects in text dimensions and image dimensions; in the process of predicting the visual labels of scene objects, the description information of scene objects in multiple dimensions is used, which can ensure that the characteristics of scene objects are fully described from the natural language semantics in the text dimension and the visual information in the image dimension, thereby ensuring the accuracy of visual labels and improving the efficiency of obtaining visual labels of scene objects.
[0433] Fig.10 A flowchart of a method for controlling a virtual character provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device. Figure 3 Based on the illustrated embodiment, step 560 is further included:
[0434] Step 560: presenting the NPC's feedback information on the command information;
[0435] Exemplarily, the feedback information is used to indicate that the NPC has received the command information and / or that the NPC has executed the virtual activity indicated by the command information.
[0436] Exemplarily, when the NPC starts to execute the virtual activity instructed by the command information, feedback information is presented to prompt the user that the command information has been issued, so as to prevent the user from repeatedly issuing the same command information without knowing it. Exemplarily, when the NPC executes the virtual activity indicated by the command information and / or completes the virtual activity, feedback information is presented to inform the user of the execution status of the command information.
[0437] For example, the feedback information is usually voice information, and the feedback information corresponds to the classification label obtained by the command information execution intention recognition. The feedback information has different feedback contents according to whether the virtual activity indicated by the command information is successfully completed.
[0438] Exemplarily, the feedback information may be stored in the local audio library of the terminal, and the feedback information may be obtained and played in the local audio library. The feedback information may also be generated by the text-to-speech (TTS) technology, which is at least one of the information such as the classification label obtained by the server according to the command information execution intent recognition and whether the virtual activity indicated by the command information is successfully completed, so as to obtain rich types of feedback information.
[0439] To sum up, the method provided in this embodiment determines to execute virtual activities according to the first semantic understanding method by taking the environmental perception information of the main virtual character and / or the NPC as a reference in at least two semantic understanding methods of command information; presenting the NPC's feedback information on the command information can improve the human-computer interaction efficiency of information transmission between the NPC and the main virtual character and increase the user's immersion.
[0440] An exemplary embodiment of the present application provides a method for controlling a virtual character. The method comprises:
[0441] Displaying a main control virtual character or NPC in a virtual environment;
[0442] Exemplarily, the main control virtual character is a virtual character that the user directly controls in the virtual environment. There are also one or more non-player characters (NPCs) in the virtual environment; the NPCs and the main control virtual character belong to the same virtual camp and are teammates of the main control virtual character.
[0443] Get command information;
[0444] Exemplarily, the command information includes natural semantics for controlling an NPC; commanding a non-player character in a virtual scene through command text, the command text may be directly input by the user or extracted from voice information input by the user, and the present application does not limit the control method of the command text.
[0445] The command information includes a behavior intention and a scene entity; the behavior intention is used to indicate the type of virtual activity performed by the NPC, such as indicating what type of virtual activity is performed in the virtual environment; the scene entity includes a first scene object, such as indicating what scene object in the virtual environment the virtual activity is performed on. The behavior intention and scene entity included in the command information are part of the words in the command information.
[0446] Exemplarily, the command information controls the NPC in two aspects: what type of virtual activity to perform and for what scene object in the virtual environment to perform the virtual activity; it is a complex instruction to control the NPC.
[0447] In response to the command information, control the NPC to perform virtual activities;
[0448] Exemplarily, controlling the NPC to perform a virtual activity is determined based on the environmental perception information of the main virtual character and / or the NPC. In some examples, the main virtual character and / or the NPC obtains environmental perception information within the perception range of vision, hearing, action trajectory, etc.
[0449] Exemplarily, the virtual activity is closely related to the current perception of the virtual environment by the main virtual character and / or NPC, and the NPC is controlled with reference to the current perception of the virtual character and / or NPC.
[0450] In summary, the method provided in this embodiment determines the virtual activities that need to be performed by facing complex instructions for controlling the NPC in terms of what type of virtual activities to perform and for what scene objects in the virtual environment to perform the virtual activities, with reference to the environmental perception information of the master virtual character and / or the NPC; and ensures that the NPC and the master virtual character perform virtual activities in collaboration with each other by using the environmental perception information of the master virtual character and / or the NPC as a reference.
[0451] In an optional implementation, the scene entity includes a first scene object; in response to the command information, controlling the NPC to perform a virtual activity may be implemented as follows:
[0452] In response to the command information, the NPC is controlled to perform a virtual activity on a first scene object located within the perception range of the master virtual character and / or the NPC;
[0453] There are multiple scene objects of the same type in the virtual environment, the multiple scene objects include a first scene object, and the perception range is a virtual area where the main control virtual character and / or NPC obtains environmental perception information.
[0454] Furthermore, in response to the command information, the NPC is controlled to perform a virtual activity on a first scene object located in the field of view;
[0455] Or, in response to the command information, the NPC is controlled to perform a virtual activity on a first scene object located in a first virtual terrain, where the first virtual terrain is a virtual terrain to which the main virtual character and / or the NPC is located.
[0456] In another optional implementation, the behavioral intention corresponds to the type of virtual activity that is permitted to be performed within the perception range; in response to the command information, controlling the NPC to perform the virtual activity can be implemented as follows:
[0457] In response to the command information, the NPC is controlled to perform a virtual activity according to the first behavioral intention; the perception range is a virtual area where the main virtual character and / or the NPC obtains environmental perception information.
[0458] In another optional implementation, the command information also includes the natural language pronunciation of the scene objects in the perception range; in response to the command information, controlling the NPC to perform the virtual activity can be implemented as follows:
[0459] In response to the command information, convert the command information into command text according to the natural language pronunciation of the scene objects in the perception range, the perception range being a virtual area where the main control virtual character and / or the NPC obtains environmental perception information;
[0460] According to the command text, control the NPC to perform virtual activities.
[0461] It should be noted that for the incomplete introduction to scene entities, behavioral intentions, and natural language pronunciations in the above text, please refer to the Figures 4 to 6 The corresponding embodiments will not be repeated here one by one.
[0462] In an optional implementation, in response to the command information, controlling the NPC to perform the virtual activity may be implemented as follows:
[0463] Determine a command text in a natural language form according to the command information;
[0464] Call the conditional random field to perform entity prediction on the command text to obtain the entity label carried by the command text;
[0465] Determine a first scene object in the perception range according to the entity label;
[0466] The prediction network is called to perform intent recognition on the command text to obtain a classification label of the command text; the prediction network is used to predict the control method of the command text for the NPC in at least one behavioral dimension;
[0467] According to the classification label, a control command of the NPC is determined, and based on the control command, the NPC is controlled to perform a virtual activity on the first scene object.
[0468] For an introduction to the above content, please refer to the above Figures 7 to 9 The corresponding embodiments will not be repeated here one by one.
[0469] Fig.11 A flowchart of a method for processing command information provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device. The method includes:
[0470] Step 710: Obtain command information;
[0471] Exemplarily, the command information includes natural semantics for controlling NPCs; commanding non-player characters in a virtual scene through command texts, the command texts may be directly input by the user, or may be extracted from voice information input by the user, and the present application does not limit the control method of the command texts. Exemplarily, the command information includes natural semantics for controlling NPCs, also known as natural language commands, or natural language commands in voice form.
[0472] Exemplarily, the main control virtual character is a virtual character that the user directly controls in the virtual environment. There are also one or more non-player characters (NPCs) in the virtual environment; the NPCs and the main control virtual character belong to the same virtual camp and are teammates of the main control virtual character.
[0473] In one example, the method is executed by a server, and the command information obtained by the server is reported by the terminal. The server processes the command information.
[0474] Step 720: Determine a first semantic understanding method from at least two semantic understanding methods of the command information based on the environment perception information of the controlling virtual character and / or the NPC;
[0475] The command information corresponds to at least two semantic understanding methods; the at least two semantic understanding methods correspond to different NPC control methods in the virtual environment. Exemplarily, at least one of the scene entity, word pronunciation, and behavior intention indicated in the at least two semantic understanding methods is different.
[0476] Exemplarily, the first semantic understanding method is one of at least two semantic understanding methods corresponding to the command information. The first semantic understanding method is determined in at least two semantic understanding methods based on the environmental perception information of the main virtual character and / or NPC. In some examples, the main virtual character and / or NPC obtains environmental perception information within the perception range of vision, hearing, movement trajectory and other perception methods. Based on the environmental perception information, the first semantic understanding method determined in at least two semantic understanding methods is close to the current perception of the virtual environment by the main virtual character and / or NPC, and the NPC is controlled with reference to the current perception of the virtual character and / or NPC to the virtual environment.
[0477] Step 730: Based on the command information, control the NPC to perform a virtual activity according to a first semantic understanding method of at least two semantic understanding methods;
[0478] Exemplarily, since the first semantic understanding mode is determined in at least two semantic understanding modes based on the environmental perception information of the controlling virtual character and / or NPC; there are at least two situations in the virtual environment (at least one of the positions, virtual health values, virtual item holdings, virtual skill cooling time, virtual level, etc. of the controlling virtual character and / or NPC in the above two situations is different), facing the same command information, the first semantic understanding modes determined are different. That is to say, as the situation in the virtual environment changes, in response to the same command information, the NPC is controlled to perform different virtual activities.
[0479] In one example, the server needs to send command information to the terminal, so that the terminal controls the NPC to perform virtual activities according to the first semantic understanding method of at least two semantic understanding methods according to the instruction of the command information.
[0480] To summarize, the method provided in this embodiment determines to execute virtual activities according to a first semantic understanding method by taking the environmental perception information of the controlling virtual character and / or the NPC as a reference in at least two semantic understanding methods of the command information; when there are at least two semantic understanding methods of the command information, on the one hand, there is no need for the user to add more basis for the command information, that is, the NPC can be controlled to execute virtual activities according to the command information, thereby improving the efficiency of human-computer interaction; on the other hand, taking the environmental perception information of the controlling virtual character and / or the NPC as a reference ensures that the NPC and the controlling virtual character collaboratively execute virtual activities.
[0481] In an optional implementation, and above Figures 4 to 6 Similar to the introduction in , in one implementation, the command information corresponds to at least two semantic understanding methods of scene entities; in another implementation, the command information corresponds to at least two semantic understanding methods of word pronunciations; in yet another implementation, the command information corresponds to at least two semantic understanding methods of behavioral intentions.
[0482] In one implementation, Fig.11 Step 720 in can be implemented as follows:
[0483] Based on the environmental perception information, the command information is determined as executing a virtual activity for a first scene object within the perception range of the master virtual character and / or the NPC; the command information corresponds to at least two semantic understanding modes of the scene entity;
[0484] Among them, there are multiple scene objects of the same type in the virtual environment where the main virtual character and / or NPC is located, and the multiple scene objects include the first scene object. The perception range is the virtual area where the main virtual character and / or NPC obtains environmental perception information.
[0485] Further optionally, based on the environmental perception information, determining the command information as executing a virtual activity for a first scene object located within the perception range of the master virtual character and / or the NPC may be implemented as follows:
[0486] Based on the environmental perception information, the command information is determined to execute a virtual activity for a first scene object located in a field of view;
[0487] Or, based on the environmental perception information, the command information is determined as executing a virtual activity for a first scene object located in a first virtual terrain; the first virtual terrain is the virtual terrain to which the main control virtual character and / or NPC is located.
[0488] In another implementation, Fig.11 Step 720 in can be implemented as follows:
[0489] Based on the environmental perception information, determining the command information as a natural language pronunciation carrying scene objects in the perception range; and converting the command information into a command text according to a first semantic understanding method; the command information corresponds to a semantic understanding method of at least two word pronunciations;
[0490] The perception range is a virtual area where the main virtual character and / or NPC obtains environmental perception information; the NPC performs virtual activities based on the instructions of the command text.
[0491] In yet another implementation, Fig.11 Step 720 in can be implemented as follows:
[0492] Based on the environmental perception information, the command information is determined to execute a virtual activity according to a first behavioral intention; the first behavioral intention corresponds to a type of virtual activity that is permitted to be executed in the perception range, and the perception range is a virtual area where the main virtual character and / or NPC obtains environmental perception information. The command information corresponds to at least two semantic understanding methods of behavioral intention;
[0493] It should be noted that for the introduction of semantic understanding methods of scene entities, word pronunciation, and behavior intention, please refer to the above Figures 4 to 6 The corresponding embodiments will not be repeated here one by one.
[0494] In an optional implementation, Fig.11 Step 720 in can be implemented as follows:
[0495] Determine the command text in natural language based on the command information;
[0496] Call the conditional random field to perform entity prediction on the command text to obtain the entity label carried by the command text;
[0497] Determine the first scene object in the perception range based on the entity label;
[0498] Calling the prediction network to perform intent recognition on the command text to obtain a classification label for the command text; the prediction network is used to predict the control method of the command text for the NPC in at least one behavioral dimension;
[0499] Accordingly, Fig.11Step 730 in can be implemented as follows:
[0500] Determine a control command of the NPC according to the classification label corresponding to the command information and the first scene object, and send the control command, where the control command is used to control the NPC to perform a virtual activity for the first scene object according to the first semantic understanding method.
[0501] In one example, the method is executed by a server, and the command information obtained by the server is reported by the terminal. The server processes the command information. In one example, the server needs to send the command information to the terminal, so that the terminal controls the NPC to perform virtual activities according to the first semantic understanding method of at least two semantic understanding methods according to the instructions of the command information.
[0502] Optionally, the call prediction network mentioned above performs intent recognition on the command text to obtain the classification label of the command text, which can be implemented as follows:
[0503] Call the hierarchical prediction network to perform intent recognition on the command text and obtain the classification label of the command text;
[0504] The hierarchical prediction network includes at least two layers of sub-networks, and at least two layers of sub-networks are constructed based on a tree structure of multiple classification labels.
[0505] Optionally, also include:
[0506] Obtain visual labels of multiple candidate scene objects in the perception range;
[0507] Determine the first scene object in the perception range according to the entity label, including:
[0508] The similarities between the visual labels of the multiple candidate scene objects and the entity labels are calculated respectively, and the first scene object with the highest similarity among the multiple candidate scene objects is determined.
[0509] Optionally, also include:
[0510] Obtaining the spatial position of each scene object in the virtual environment;
[0511] According to the spatial positions, multiple candidate scene objects in the perception range are screened out from various scene objects in the virtual environment.
[0512] Optionally, the acquisition of the spatial position of each scene object in the virtual environment in the above text can be implemented as follows:
[0513] Obtain at least one of the coordinate position, orientation information, bounding box information, and cover point information of each scene object in the virtual environment;
[0514] Among them, the coordinate position is used to indicate the position of the scene object in the virtual environment, the orientation information is used to indicate the direction facing the scene object in the virtual environment, the bounding box information is used to indicate the size of the scene object in the virtual environment, and the cover point information indicates the recommended virtual character standing position when the virtual character approaches the scene object.
[0515] Optionally, also include:
[0516] Acquire attribute text of scene objects in the virtual environment, where the attribute text is used to introduce inherent attributes of the scene objects in the virtual environment;
[0517] Acquire an appearance image of a scene object in a virtual environment, where the appearance image is used to describe the style of the scene object;
[0518] Calling the multimodal model to perform prediction on the attribute text and appearance image of the scene object to obtain a visual label of the scene object, where the visual label is used to describe the scene object in at least one dimension;
[0519] Based on the visual labels of each scene object in the virtual environment, a spatial database of the virtual environment is constructed;
[0520] Obtain visual labels for multiple candidate scene objects in the perception range, including:
[0521] Based on the perception range, visual labels of multiple candidate scene objects in the perception range are screened in the spatial database.
[0522] It should be noted that the command text, entity prediction, entity label, prediction network, intent recognition, control command and other information in this optional implementation are not fully introduced. Please refer to the above Figure 7 The corresponding embodiments, and Figure 7 The embodiments following the corresponding embodiments will not be repeated here one by one.
[0523] An exemplary embodiment of the present application provides a method for processing command information. The method includes:
[0524] Get command information;
[0525] Exemplarily, the main control virtual character is a virtual character that the user directly controls in the virtual environment. There are also one or more non-player characters (NPCs) in the virtual environment; the NPCs and the main control virtual character belong to the same virtual camp and are teammates of the main control virtual character.
[0526] Exemplarily, the command information includes natural semantics for controlling an NPC; commanding a non-player character in a virtual scene through command text, the command text may be directly input by the user or extracted from voice information input by the user, and the present application does not limit the control method of the command text.
[0527] The command information includes a behavior intention and a scene entity; the behavior intention is used to indicate the type of virtual activity performed by the NPC, such as indicating what type of virtual activity is performed in the virtual environment; the scene entity includes a first scene object, such as indicating what scene object in the virtual environment the virtual activity is performed on. The behavior intention and scene entity included in the command information are part of the words in the command information.
[0528] Exemplarily, the command information controls the NPC in two aspects: what type of virtual activity to perform and for what scene object in the virtual environment to perform the virtual activity; it is a complex instruction to control the NPC.
[0529] Based on the command information, control the NPC to perform virtual activities;
[0530] Exemplarily, controlling the NPC to perform a virtual activity is determined based on the environmental perception information of the main virtual character and / or the NPC. In some examples, the main virtual character and / or the NPC obtains environmental perception information within the perception range of vision, hearing, action trajectory, etc.
[0531] Exemplarily, the virtual activity is close to the current perception of the virtual environment by the main virtual character and / or NPC, and the NPC is controlled with reference to the current perception of the virtual character and / or NPC. In one example, the method is executed by a server, and the command information obtained by the server is reported by the terminal. The server processes the command information.
[0532] Optionally, the scene entity includes a first scene object; and / or the behavioral intention corresponds to a type of virtual activity permitted to be performed in the perception range; and / or the command information also includes a natural language pronunciation of the scene object in the perception range; For the incomplete introduction to the scene entity, behavioral intention, and natural language pronunciation in the above text, please refer to the above text. Figures 4 to 6 The corresponding embodiments will not be repeated here one by one.
[0533] Exemplarily, determining a command text in a natural language form according to the command information;
[0534] Call the conditional random field to perform entity prediction on the command text to obtain the entity label carried by the command text;
[0535] Determine a first scene object in the perception range according to the entity label;
[0536] The prediction network is called to perform intent recognition on the command text to obtain a classification label of the command text; the prediction network is used to predict the control method of the command text for the NPC in at least one behavioral dimension;
[0537] According to the classification label, a control command of the NPC is determined, and based on the control command, the NPC is controlled to perform a virtual activity on the first scene object.
[0538] For an introduction to the above content, please refer to the above Figures 7 to 9 The corresponding embodiments will not be repeated here one by one.
[0539] In summary, the method provided in this embodiment determines the virtual activities that need to be performed by facing complex instructions for controlling the NPC in terms of what type of virtual activities to perform and for what scene objects in the virtual environment to perform the virtual activities, with reference to the environmental perception information of the master virtual character and / or the NPC; and ensures that the NPC and the master virtual character perform virtual activities in collaboration with each other by using the environmental perception information of the master virtual character and / or the NPC as a reference.
[0540] Fig.12 A schematic diagram of a method for processing command information provided by an exemplary embodiment of the present application is shown. Command information 1002 is obtained, and command information 1002 is a control voice issued by a user to an NPC. Automatic Speech Recognition (ASR) service 1102 is requested to convert command information 1002 to obtain command text 1004 corresponding to command information 1002.
[0541] Auxiliary data 1006 is obtained, and the auxiliary data 1006 includes static data of the pre-built virtual environment and dynamic data in the virtual environment at the current moment.
[0542] Static data includes visual labels of scene objects, and visual labels are used to describe the scene objects in at least one dimension; for example, describing scene objects in dimensions such as material, transparency, color, and shape. Static data includes the spatial position of scene objects: the spatial position includes at least one of the following of scene objects: coordinate position (Location), such as the coordinate information of the center point or preset point of the scene object in the virtual scene; orientation (Rotation), such as the direction the front of the scene object faces in the virtual scene, and a bounding box (Bounding Box), which is used to indicate the size of the scene object in the virtual scene; cover points, which are used to indicate the recommended virtual character position points when the virtual character approaches the scene object, so that the scene object can mask the virtual character.
[0543] The dynamic data includes the basic information of the user's main virtual character and / or teammate character (NPC), and enemy character (such as coordinate position, direction, size of space occupied, etc.) obtained instantly when the user issues the command information 1002; the instructions and body postures currently being executed by the main virtual character and / or teammate character.
[0544] The model inference service 1104 is requested to perform inference prediction on the command text 1004 and the auxiliary data 1006, and obtain instruction data 1010 and feedback text 1012; wherein the instruction data 1010 is an instruction that can be used for the NPC's behavior tree execution, also known as a control instruction. The feedback text 1012 is a text presented in a voice manner to indicate that the NPC has received the command information and / or the NPC has executed the virtual activity indicated by the command information.
[0545] Exemplarily, the classification label of the command text 1004 is used as input to determine the target entity in the virtual space and the control command for the NPC; exemplary, each classification label in the command text 1004 has a corresponding control command, and the control command for the NPC is determined based on the correspondence between the classification label and the control command.
[0546] Exemplarily, the model inference service 1104 has the ability to predict the classification label and entity label of the command text 1004, such as calling the conditional random field to perform entity prediction on the command text to obtain the entity label carried by the command text; calling the prediction network to perform intent recognition on the command text to obtain the classification label of the command text; the entity label is used to determine the first scene object in the perception range, and the control command is used to control the NPC to perform the virtual activity for the first scene object according to the first semantic understanding method.
[0547] The instruction data 1010 and the feedback text 1012 are cached in the first component 1020 of the terminal, and the terminal collects and obtains the command information 1002. The first component 1020 of the terminal calls the second component 1120 on the server side based on the Unreal Engine Remote Procedure Call (RPC); the first component 1020 and the second component 1120 are both used to cache the instruction data 1010 and the feedback text 1012.
[0548] The subsequent action that the control command indicates the NPC needs to perform can be a single action or a sequence of actions. In the case where the control command is a sequence instruction of multiple actions, the sequence instruction of multiple actions is used to instruct the NPC to perform a sequence of actions. The data structure storing the sequence instruction of multiple actions is in the form of a list, and the instructions to be executed in the sequence instruction are cached. When the NPC's behavior tree executes the current instruction, the cache is called back, and the instructions to be executed in the cache are continued to be sent for execution. The second component 1120 sends the instruction data 1010 and the feedback text 1012 to the first voice feedback logic 1202, the activity execution logic 1204, the activity judgment logic 1206, and the second voice feedback logic 1208 in the order of the sequence instructions.
[0549] The first voice feedback logic 1202 is used to indicate the voice feedback presented when the NPC receives the command information; the activity execution logic 1204 is used to control the NPC to perform virtual activities according to the instruction data 1010; the activity judgment logic 1206 is used to determine whether the virtual activity performed by the NPC is ended; the second voice feedback logic 1208 is used to indicate the voice feedback presented when the NPC has executed the virtual activity indicated by the command information; illustratively, the feedback information in voice format obtained according to the feedback text 1012 can be converted by requesting the text-to-speech service 1132; it can also be existing audio obtained by querying the audio library 1134.
[0550] Exemplarily, the NPC has a behavior tree 1300, which controls the NPC's activities in the virtual environment with preset logic (such as automatically avoiding virtual attacks, etc.); the behavior tree 1300 is connected to the third voice feedback logic 1210, and the third voice feedback logic 1210 is used to provide voice feedback when performing virtual activities according to the preset logic, and broadcast the activities automatically performed by the NPC in the virtual environment.
[0551] It should be noted that the speech-to-text service 1102, the model reasoning service 1104, and the text-to-speech service 1132 in this embodiment can be implemented as a separate server, or can be implemented together with a background server that supports a virtual environment as a server cluster.
[0552] Next, the NPC control commands are introduced.
[0553] For example, Fig.12As described in the introduction, the model inference service is requested to perform inference prediction on the command text and auxiliary data to obtain the command data and feedback text; the request packet input to the model inference service includes at least one of the following fields: Content field: used to indicate the command text; CurDecision field: used to indicate that the NPC is currently executing the command; CurReceiver field: used to indicate the identity of the NPC who last commanded; CurTargetLocation field: used to indicate the target location of the last command of the NPC; MoveType field: used to indicate the current movement mode of the NPC (running, slow walking, normal walking, etc.); PostureType field: used to indicate the current posture of the NPC (standing, squatting, lying, etc.) ; Location field: used to indicate the position of the main virtual character and the NPC; Rotation field: used to indicate the direction of the main virtual character and the NPC; TraceInfo field: used to indicate the ray information of the main virtual character, such as the six directions of front, back, left, right, up and down, used to indicate whether there is any obstruction and whether it is indoors; EnemyLocation field: used to indicate the position of the current enemy character; EnemyRotation field: used to indicate the direction of the current enemy character; GunFireLocation field: used to indicate the position of the virtual attack sound effect (such as gunshot) in the current virtual environment; NormalSoundLocation field: used to indicate the position of the ambient sound in the current virtual environment.
[0554] The model reasoning service has the ability to predict the type of virtual activity that needs to be performed, the target entity for performing the virtual activity, the identity of the controlled NPC performing the virtual activity, and other information. The content inferred by the model reasoning service includes at least one of the following fields (also called the return packet content): Base field: used to indicate the reference system benchmark for the instruction to perform the virtual activity (for example, walk in front of object A, object A is the reference system benchmark); Fire field: used to indicate whether to allow a virtual attack; AbsLoc field: used to indicate the relative position of the virtual activity (such as front, back, left, and right); AbsLocDis field: used to indicate the relative distance of the virtual activity (such as 5 meters in front); Item field: used to indicate the identity of the item used (such as indicating a grenade, smoke bomb, or medical kit); Receiver field: used to indicate the recipient of the command information, that is, the identity of the NPC performing the virtual activity; VolumeId field: used to indicate the area scope for performing virtual activities; MoveType field: used to indicate the movement method for performing virtual activities; PostureType field: used to indicate the body posture for performing virtual activities; TargetLocation field: used to indicate the coordinates of the moving destination for performing virtual activities; TargetYaw field: used to indicate the expected direction for performing virtual activities; TargetType field: used to indicate the type of destination (whether it is an object, an evacuation point, or an indoor place); FeedbackText field: used to indicate the feedback text; Content_with_breakpoint field: used to indicate the clause segmentation result of the command text (number two goes to the front and number three comes back, which is divided into number two goes to the front and number three comes back); Decision field: used to indicate the type of virtual activity to be performed.
[0555] For example, when the value of the Decision field is -2, the type of virtual activity is to stay on the spot; when the value of the Decision field is -1, the type of virtual activity is to keep the historical commands; when the value of the Decision field is 0, the type of virtual activity is to go to the designated location and return after arriving at the location or discovering the enemy; when the value of the Decision field is 1, the type of virtual activity is to go to the designated location and will not be interrupted by discovering the enemy, and stay in place when arriving; when the value of the Decision field is 2, the type of virtual activity is to circle around the main virtual character and face the direction of possible enemy encounter; when the value of the Decision field is 3, the type of virtual activity is to stop any current behavior; when the value of the Decision field is 4, the virtual activity The type is to only adjust the NPC's body posture / movement speed; when the Decision field value is 5, the type of virtual activity is throwing / using items; when the Decision field value is 6, the type of virtual activity is location inquiry; when the Decision field value is 7, the type of virtual activity is intelligence inquiry; when the Decision field value is 8, the type of virtual activity is opening the door; when the Decision field value is 9, the type of virtual activity is closing the door; when the Decision field value is 10, the type of virtual activity is launching a virtual attack; when the Decision field value is 11, the type of virtual activity is avoiding virtual attacks; when the Decision field value is 12, the type of virtual activity is to evacuate away from the enemy character.
[0556] Fig.13 A schematic diagram of an NPC behavior tree provided by an exemplary embodiment of the present application is shown. The NPC behavior tree includes a general service 1500, which controls the NPC's activities such as observing and picking up items in a virtual environment. The subnodes of the general service 1500 include a preset service 1502, an interrupt service 1504, and a control service 1506.
[0557] The preset service 1502 is a service that is preset for the NPC to automatically execute in the virtual environment, such as the NPC automatically avoiding the thrown virtual grenade. The interrupt service 1504 is used to instruct the NPC to stop the currently executed virtual activity. For example, when the teammate character is attacked, the NPC stops the currently executed virtual activity and turns to assist the teammate character. The control service 1506 is a virtual activity executed in response to the user's command information. The control service includes the following types: standby activity 1510, inquiry activity 1520, movement activity 1530, protection activity 1540, reconnaissance activity 1550, door interaction activity 1560, and use of virtual props 1570; the above seven types of virtual activities have corresponding Decision fields, and the virtual activities on the corresponding behavior tree branches are executed according to the Decision field in the return packet content of the model reasoning service. It can be understood that some or all of the seven types of virtual activities also include subtypes, such as the standby activity 1510 includes two sub-activity types: change direction activity 1511 and wait for preset time activity 1512. The inquiry activity 1520 includes two sub-activity types: location inquiry 1521 and intelligence inquiry 1522. That is, the behavior tree of the NPC may include a more loaded tree structure, which is not limited in the present application.
[0558] An exemplary embodiment of the present application provides a method for controlling a virtual character. The method comprises:
[0559] Displaying at least one of a main control virtual character and an NPC in a virtual environment;
[0560] Exemplarily, the main virtual character is a virtual character that the user directly controls in the virtual environment. There are also one or more non-player characters (NPC) in the virtual environment; the NPC and the main virtual character belong to the same virtual camp and are teammates of the main virtual character. The NPC follows the main virtual character in the virtual environment. In addition, the NPC can also be a follower character, a pet character, etc. controlled by the main virtual character. In addition, the NPC can also be a neutral character, and only cooperates with the main virtual character when certain conditions are met.
[0561] Obtain natural language commands;
[0562] Exemplarily, the natural language command includes the natural semantics of commanding the NPC; the NPC is commanded in the virtual scene through the natural language command, and the natural language command can be directly input by the user or extracted from the voice information input by the user. The present application does not limit the method of obtaining the natural language command.
[0563] The natural language command includes at least one of a behavior intention and a scene entity; the behavior intention is used to indicate the type of virtual activity performed by the NPC, such as indicating what type of virtual activity is performed in the virtual environment; the scene entity includes a first scene object, such as indicating what scene object in the virtual environment the virtual activity is performed on. The behavior intention and scene entity included in the natural language command are partial words in the natural language command.
[0564] In one example, the natural language command controls the NPC in terms of what type of virtual activity to perform and for what scene object in the virtual environment to perform the virtual activity; it is a complex instruction to control the NPC.
[0565] Controlling NPCs to perform virtual activities in response to natural language commands;
[0566] Exemplarily, controlling the NPC to perform virtual activities is determined based on the environmental perception information of the main virtual character and / or NPC. In some examples, the main virtual character and / or NPC obtains environmental perception information within the perception range of vision, hearing, movement trajectory and other perception methods. Exemplarily, the virtual activity is close to the current perception of the virtual environment by the main virtual character and / or NPC, and the NPC is controlled with reference to the current perception of the virtual character and / or NPC of the virtual environment. Exemplarily, the NPC has autonomous behavior capabilities, and the user only directs the NPC. The user gives commands, and the NPC understands and executes the commands based on its own autonomous behavior capabilities.
[0567] To sum up, the method provided in this embodiment determines the virtual activities that need to be performed by facing complex instructions for controlling the NPC in terms of what type of virtual activities to perform and for what scene objects in the virtual environment to perform the virtual activities, with reference to the environmental perception information of the master virtual character and / or the NPC; and ensures that the NPC and the master virtual character perform virtual activities in collaboration with each other by using the environmental perception information of the master virtual character and / or the NPC as a reference.
[0568] Fig.14 A flowchart of a method for controlling a virtual character provided by an exemplary embodiment of the present application is shown. The method comprises:
[0569] Step 201: Acquire a spatial data set;
[0570] The spatial dataset of the virtual environment includes visual labels of scene objects, and the visual labels are used to describe the visual characteristics of the scene objects in at least one dimension, such as describing the scene objects in dimensions such as material, transparency, color, and shape.
[0571] Step 202: displaying at least one of a main control virtual character and an NPC in a virtual environment;
[0572] Exemplarily, the main control virtual character is a virtual character that the user directly controls in the virtual environment. There are also one or more NPCs in the virtual environment; the NPCs and the main control virtual character belong to the same virtual camp and are teammates of the main control virtual character.
[0573] Step 203: Acquire a natural language command in voice form;
[0574] The natural language command includes natural semantics for commanding the NPC, and the command information includes behavioral intention and scene entity; the behavioral intention is used to indicate the type of virtual activity performed by the NPC, and the scene entity is used to indicate the target entity of the virtual activity;
[0575] Step 204: converting the natural language command in voice form into the natural language command in text form;
[0576] Exemplarily, speech-to-text (Automatic Speech Recognition, ASR) processing is performed on the natural language command in the form of speech to determine the natural language command in the form of text. Speech-to-text processing generally includes calling components such as acoustic models and language models to recognize the pronunciation, vocabulary, grammatical structure and other information of the natural language command in the form of speech, and convert it into a natural language command in the form of text.
[0577] Step 205: performing intent recognition on the natural language command to obtain a first classification label;
[0578] Exemplarily, the first intention indication corresponding to the first classification label includes a command intention for a non-player character on an intention dimension; for example, at least one of the following: indicating which of a plurality of non-player characters the non-player character commanded by the command text is, indicating whether the virtual activity performed by the non-player character is related to initiating a virtual attack, and indicating what kind of virtual activity the non-player character performs.
[0579] Step 206: Perform entity recognition on the natural language command to obtain a target entity;
[0580] The target entity is determined from the virtual environment in combination with the environmental perception information, which includes information perceived by at least one of the main virtual character and the non-player character from the virtual environment. The target entity may be the entity currently seen by the player, or may be an entity that the player has seen in the past.
[0581] Step 207: In response to the natural language command, the NPC is controlled to perform a virtual activity according to the first intention corresponding to the first classification label, or the NPC is controlled to perform a virtual activity associated with the target entity, or the NPC is controlled to perform a virtual activity associated with the target entity according to the first intention corresponding to the first classification label;
[0582] Exemplarily, the NPC is controlled to perform a virtual activity according to the first classification tag and / or the instruction of the target entity.
[0583] Step 208: reporting the NPC's feedback information;
[0584] Exemplarily, the game application can generate corresponding feedback information in real time according to the environmental perception information of the non-player character and make a broadcast. For example, when the environmental perception information of the non-player character triggers the broadcast condition, the game application can generate corresponding feedback information in real time according to the environmental perception information that triggers the broadcast condition; or, when the non-player character or the main control virtual character triggers the broadcast condition, the game application can generate corresponding feedback information in real time according to the environmental perception information of the non-player character.
[0585] The preprocessing stage of step 201 can be implemented as follows:
[0586] Sub-step 1: Obtaining attribute text of scene objects in the virtual scene;
[0587] Exemplarily, attribute text is used to introduce the inherent attributes of scene objects in a virtual scene; on the one hand, attribute text realizes the description of scene objects in a textual mode, and while describing scene objects, it provides semantic information for the prediction of the visual labels of scene objects. On the other hand, attribute text is the description of scene objects in a virtual scene. In the face of a large number of scene objects in a virtual scene and the existence of reused object models, it can more accurately describe the inherent attributes of scene objects in a virtual scene.
[0588] In an optional implementation, the attribute text includes at least one of a name of the scene object in the virtual scene and a size of the scene object in the virtual scene.
[0589] Sub-step 2: obtaining an appearance image of a scene object in a virtual scene;
[0590] Exemplarily, the appearance image is used to describe the style of the scene object; the appearance image carries the appearance style of the scene object such as color, texture, shape, etc., and the relative position relationship between each sub-part in the picture mode, and can fully describe the scene object from the picture mode (or visual mode). In an optional implementation, the appearance image of the scene object includes images obtained by observing the scene object from at least two perspectives.
[0591] Sub-step 3: Call the multimodal model to perform prediction on the attribute text and appearance image of the scene object to obtain the visual label of the scene object;
[0592] Exemplarily, the multimodal model has the ability to perform model prediction on text information and image information of different modalities. In this embodiment, the input parameters of the multimodal model are the attribute text and appearance image of the scene object; the multimodal model predicts the visual label of the scene object from the two modalities of text modality and image modality, and the visual label is used to describe the visual features of the scene object in at least one dimension.
[0593] Optionally, the multimodal model includes a visual question answering model; the question statement carries attribute text of the scene objects; on the one hand, the question statement is used to guide the visual question answering model to convert the appearance image into the visual label of the scene object; on the other hand, while guiding the conversion of the visual label of the scene object, the question statement provides supplementary information of the scene object in text form.
[0594] Get the expected information of scene objects;
[0595] Exemplarily, the expected information is used to indicate the description dimension of the scene object expected in the visual tag, and / or the expected format of the visual tag; in one example, the expected information is used to indicate that the description dimension of the scene object expected in the visual tag includes but is not limited to at least one of the type description, material, transparency, color, surface features, and shape of the scene object. In another example, the expected information is used to indicate that the expected format of the visual tag is at least one of the following: Comma Separated Values (CSV), JavaScript Object Notation (JSON), and eXtensible Markup Language (XML).
[0596] Construct a question statement of the appearance image based on the expected information and attribute text;
[0597] Exemplarily, the first subpart of the question sentence is supplementary introduction information of the scene object, carrying attribute text of the scene object; the second subpart of the question sentence is an answer guidance sentence for the visual question answering model, carrying expected information.
[0598] Optionally, the spatial data set includes matching labels of scene objects in the virtual scene; correspondingly: performing pseudo-spoken rewriting on the visual labels of the scene objects to obtain matching labels that conform to spoken expressions of natural language.
[0599] Exemplarily, as described above, visual labels are used to describe the visual features of scene objects in at least one dimension, and the appearance image of the scene objects presents rich visual features of the scene objects. In the spoken expression of natural language, however, the description of scene objects cannot cover the visual features of scene objects in all dimensions. The purpose of performing a pseudo-colloquial rewrite on the visual labels of scene objects is to obtain a matching label that is closer to the spoken expression; it can be understood that performing a pseudo-colloquial rewrite can be to delete part of the content in the visual label, or to change the visual label to a label with the same semantics but different textual expression.
[0600] In an optional implementation, the pseudo-colloquial rewriting is performed by calling a natural language model; the visual labels of the scene objects are input into the natural language model, and matching labels that conform to the spoken expressions of the natural language are predicted. The pseudo-colloquial rewriting of the visual labels is implemented based on calling the natural language model. Exemplarily, the natural language model carries prior knowledge of the spoken expressions of the natural language. For example, the natural language model is implemented as a large language model (Large Language Model, LLM).
[0601] Optionally, the spatial dataset also includes spatial information of scene objects, and accordingly, also includes:
[0602] The spatial position of the scene object in the virtual scene is obtained, and the spatial position of the scene object is determined as auxiliary information of the visual label of the scene object.
[0603] Exemplarily, the spatial position of the scene object in the virtual scene is used to indicate the deployment of the scene object in the virtual scene, and the spatial information is used to indicate the position, size and other information of the scene object in the virtual scene after the scene object is deployed in the virtual scene.
[0604] Further, obtaining at least one of the coordinate position, orientation information, bounding box information, and cover point information of the scene object in the virtual scene;
[0605] Among them, the coordinate position (Location) is used to indicate the position of the scene object in the virtual scene, such as the coordinate information of the center point or preset point of the scene object in the virtual scene; the orientation (Rotation) is used to indicate the direction facing the scene object in the virtual scene, such as the direction facing the front of the scene object in the virtual scene; the bounding box (BoundingBox) is used to indicate the size of the scene object in the virtual scene; the cover points (Cover points) are used to indicate the recommended virtual character position points when the virtual character approaches the scene object, so that the scene object can mask the virtual character.
[0606] The intention recognition phase of step 205 can be implemented as follows:
[0607] Sub-step 4: calling at least one hierarchical prediction network to perform intent recognition on the natural language command, and obtaining a first classification label of the natural language command intent to command the NPC;
[0608] Exemplarily, the hierarchical prediction network has the ability to predict the classification label corresponding to the command text. Exemplarily, the hierarchical prediction network includes at least two layers of sub-networks, the upper network and the lower network are cascaded, and the lower network further performs the prediction of the classification label according to the prediction result output by the upper network. At least two layers of sub-networks are constructed based on the tree structure of multiple classification labels. The hierarchical prediction network constructs at least two layers of sub-networks corresponding to the hierarchical structure for the tree structure of the classification labels, and splits the prediction task of a wide variety of classification labels into prediction sub-tasks performed by at least two layers of sub-networks, so as to reduce the complexity of classification prediction of each layer of sub-network.
[0609] In an optional implementation, each hierarchical prediction network in at least one hierarchical prediction network is used to predict the command intention of the command text for a non-player character on an intention dimension; the intention dimension includes: at least one of the subject dimension, the semantic dimension, and the behavior dimension. Exemplarily, in at least one hierarchical prediction network, the i-th layer subnetwork in each hierarchical prediction network is used to predict the first-level behavior label of the command text on an intention dimension, and the i+1-th layer subnetwork in the hierarchical prediction network is used to predict the second-level behavior label corresponding to the command text in the first-level behavior label, i is a positive integer; the high-level behavior label (such as the first-level behavior label) predicted by the high-level subnetwork (such as the i-th layer subnetwork) in the hierarchical prediction network includes multiple sublabels, or subordinate lower-level labels; the corresponding low-level subnetwork (such as the i+1-th layer subnetwork) needs to be called to further perform classification prediction (such as predicting the second-level behavior label). The prediction task of a wide variety of classification labels is split into prediction subtasks performed by at least two layers of subnetworks to reduce the complexity of classification prediction of each layer of subnetwork.
[0610] Furthermore, the intent dimension includes a subject dimension, the hierarchical prediction network includes a hierarchical subject prediction network, and the subject prediction network has the ability to predict the subject type in a natural language command; the subject type is used to indicate the identity of the NPC commanded by the natural language command; exemplarily, the subject prediction network is used to predict which of a plurality of non-player characters the non-player character commanded by the command text is, and the number of non-player characters commanded by the command text can be one or more.
[0611] And / or, the intent dimension includes a semantic dimension, the hierarchical prediction network includes a hierarchical semantic prediction network, the semantic prediction network has the ability to predict the semantic type in a natural language command; the semantic type is used to indicate the NPC's control method for initiating a virtual attack; exemplarily, the subject prediction network is used to predict whether the virtual activity performed by a non-player character commanded by the command text is related to initiating a virtual attack.
[0612] And / or, the intention dimension includes a behavior dimension, the hierarchical prediction network includes a hierarchical behavior prediction network, the behavior prediction network has the ability to predict the behavior intention type in the natural language command; the behavior intention type is used to indicate the behavior mode of the NPC for performing the virtual activity. Exemplarily, the behavior prediction network is used to predict what virtual activity the non-player character performs.
[0613] Accordingly, in step 207, in response to the natural language command, the NPC is controlled to perform a virtual activity according to the first intention corresponding to the first classification label;
[0614] Exemplarily, the first intention corresponding to the first classification tag indicates a command intention for a non-player character in an intention dimension, such as at least one of the following: indicating which non-player character is commanded by the command text among a plurality of non-player characters, indicating whether the virtual activity performed by the non-player character is related to launching a virtual attack, and indicating what virtual activity the non-player character performs. According to the instruction of the first classification tag, the NPC is controlled to perform the virtual activity.
[0615] The target entity identification process of step 206 can be implemented as follows:
[0616] Sub-step 5: querying and obtaining the target entity from the entity set of the virtual environment according to the target entity information of the target entity indicated by the natural language command and the environment perception information;
[0617] The target entity is determined from the virtual environment in combination with environmental perception information, and the environmental perception information includes information perceived by the main virtual character and / or the non-player character from the virtual environment.
[0618] Since the target entity information in natural language commands is usually vague, for example, a natural language command may be "move to the red truck", where the target entity information is "red truck", and there may be many red trucks in the virtual environment. It is impossible to accurately determine the target entity in the virtual environment based solely on the target entity information in the natural language command.
[0619] Therefore, the embodiment of the present application provides a method for determining the target entity by combining the target entity information and the environmental perception information. For the previous example, the "red truck" expressed by the player in the natural language command should be a red truck that the player can see. Therefore, combined with the field of view of the main control virtual character, the red truck within the field of view of the main control system character can be screened out from the multiple red trucks in the virtual environment. The red truck is the target entity indicated by the player in the natural language command.
[0620] Since natural language commands are issued by players based on their perception of the virtual environment, in order to accurately identify the target entity indicated in the natural language command, the game application will combine the environmental perception information when the player issues the natural language command to identify the target entity in the natural language command. Based on the player's perception of the virtual environment, the entity that the player can perceive in the virtual environment and is closest to the target entity information is inferred as the target entity.
[0621] Exemplarily, the virtual environment includes multiple candidate entities that match the natural language command, and the target entity is an entity selected from the multiple candidate entities based on the environmental perception information of the main virtual character or the non-player character. For example, the game application first selects multiple candidate entities that match the target entity information in the natural language command from the entity set, and then selects the entity that can be perceived by the main virtual character or the non-player character from the multiple candidate entities as the target entity based on the environmental perception information.
[0622] For example, the environmental perception information of the main virtual character may include: the pictures or entities that the main virtual character can see when observing the virtual environment, the environmental sound effects that the main virtual character can hear, the source direction of the environmental sound effects, the type and volume of the environmental sound effects, the perception information obtained by the main virtual character by using perception skills (for example, visual information, auditory information, sound wave information, light reflection information, etc.), and the perception information obtained by the main virtual character by using sensing props (for example, sensor signals from sensors, positioning signals from positioning props, etc.).
[0623] The environmental perception information of the non-player character may include: the images or entities that the non-player character can see when observing the virtual environment, the environmental sounds that the non-player character can hear, the source direction of the environmental sounds, the type and volume of the environmental sounds, the perception information obtained by the non-player character by using perception skills (for example, visual information, auditory information, sound wave information, light reflection information, etc.), and the perception information obtained by the non-player character by using sensory props (for example, sensor signals of sensors, positioning signals of positioning props, etc.).
[0624] It should be noted that the environmental perception information may include the information perceived by the controlling virtual character and the non-player character in real time when receiving the natural language command, and may also include the information perceived by the controlling virtual character and the non-player character in history before receiving the natural language command. That is, the target entity may be the entity currently seen by the player, or it may be the entity that the player has seen in the past. Therefore, the game application needs to combine the real-time environmental perception information and the historical environmental perception information of the controlling virtual character and / or the non-player character to identify the target entity.
[0625] For example, when the natural language command is "Let's go back to the hotel we just passed by", the game application needs to query the hotels that the main virtual character has passed by based on the historical environmental perception information of the main virtual character.
[0626] In an optional embodiment, the client queries and obtains the target entity from the entity set of the virtual environment according to the target entity information and the environment perception information indicated by the natural language command. In another optional embodiment, the client reports the natural language command to the server, and the server queries and obtains the target entity from the entity set of the virtual environment according to the target entity information and the environment perception information indicated by the natural language command.
[0627] Optionally, the game application infers the target entity based on the target entity information, the environment perception information and the pre-processed virtual environment data, wherein the pre-processed virtual environment data includes a set of entities in the virtual environment.
[0628] The entity set includes entity information of each entity in the virtual environment. The entity information includes at least one of the following: name, type, location, feature, text label, embedding vector of text label, image, embedding vector of image.
[0629] The text label may be at least one word obtained after segmenting a feature (for example, a descriptive text of an entity appearance), and the embedding model is called to obtain an embedding vector corresponding to each text label.
[0630] The image may include an image obtained by observing a three-dimensional model of the entity from at least one direction, for example, the image may include three views of the entity. The embedding vector of the image is the embedding vector of the image feature label. The image feature label is obtained by calling the multimodal model to perform feature recognition based on the image and text label of the entity.
[0631] Exemplarily, the entity set includes a first type of entity and a second type of entity, the first type of entity includes at least one of an area and a building, and the second type of entity includes a physical object. The text labels of the first type of entity are obtained by manual labeling. For example, the text label of a room of a building is manually labeled as: an area, a building, a floor, and a room. The text labels of the second type of entity are automatically generated by calling a large language model.
[0632] The method for generating text labels for the second type of entity may include: obtaining at least one image of the entity, inputting the at least one image into a large language model, and obtaining a description text of the entity; segmenting the description text to obtain at least one text label of the entity, each text label including a word obtained after the segmentation process; performing a vectorization operation on the text label to obtain an embedding vector corresponding to the text label.
[0633] The embedding vector is the high-dimensional vector data of the text label. By pre-generating the embedding vector of the text label, there is no need to repeatedly generate vectors for the text label of each entity when matching the target entity from the entity set based on the target entity information, which can improve the matching efficiency of the target entity information and the entity information.
[0634] It should be noted that this method does not use the description text directly as the text label, but uses the word segmentation result of the description text as the text label. This is because the description text is usually of different lengths. For longer description texts, their embedding vectors are less effective in actual search and similarity comparison. For example, when searching for "a small blue truck with rusty body" using "truck", "a small blue truck with rusty body" is likely to be ranked after "a car" in the recall results. In fact, for the searched "truck", no matter how complex its additional description is, "truck" should be searched first. Therefore, when performing similarity search, the comparison should be one feature word at a time, not the description text.
[0635] In addition, in order to further extract the image features of the entity, the method provided in the embodiment of the present application can also further extract hidden visual features in the image of the entity based on the text label and image of the entity. Taking the first entity as an example, the method includes: obtaining at least one perspective image of the first entity; obtaining at least one text label of the first entity; calling the multimodal model to extract the image features of the first entity based on at least one perspective image and at least one text label of the first entity, and obtain the image embedding vector of the first entity.
[0636] Since text labels are manually annotated, or generated by a large language model based on human language characteristics; and text labels extracted based on human language habits may ignore some features of the entity. For example, when manually annotating a text label for an oil drum, only more eye-catching text labels such as "metal" and "rust" may be annotated, while detailed features such as "rust", "blue", "yellow", and "paint peeling in the lower right corner" are ignored. Therefore, the method provided in the embodiment of the present application also uses a multimodal model based on the text label and entity image of the entity, so that the multimodal model extracts more image features from it; performs image similarity matching with the target entity based on the image features to improve the recognition accuracy of the target entity.
[0637] Among them, the training method of the multimodal model can be: inputting sample entity images and sample labels into the pre-trained multimodal model, fine-tuning the pre-trained multimodal model according to the loss of the predicted labels and sample labels output by the pre-trained multimodal model, and obtaining a multimodal model that can output image feature labels based on the input text labels and images, and the image feature labels include more detailed description texts proposed from the image. For example, the text description of "a metal oil barrel" can only be associated with the two text labels "metal" and "oil barrel", while the Clip (multimodal) model can also identify the hidden visual information in the image from the image of the oil barrel, such as "rusty", "blue", and "yellow" image feature labels, which are features that do not appear in the text labels. Therefore, combining the Clip model for image feature search can further improve the accuracy of entity search.
[0638] In an optional embodiment, the game application may query the target entity using the following method.
[0639] (1) Parse the natural language command to obtain target entity information; the target entity information includes at least one of the following: entity type, entity name, entity location, and entity feature.
[0640] Optionally, the game application calls the large language model to parse the natural language command to obtain target entity information of the target entity, and calls the embedding model to vectorize the target entity information to obtain a target embedding vector of the target entity information.
[0641] For example, if the natural language command is "Come to the blue truck in front", the target entity information that can be parsed by the large language model includes: the direction information "front" and the entity name "blue truck".
[0642] For example, named entity recognition technology in the field of natural language processing can also be used to extract target entity information from natural language commands. Named entity recognition technology is used to identify entities with specific meanings in texts, such as names of people, places, locative words, adjectives, etc.
[0643] For example, if the natural language command is "Go to the first floor of the motel and find a paper box behind the red sofa", the named entity recognition technology can identify: the architectural noun "the first floor of the motel"; the object nouns "sofa" and "paper box"; the locative word "behind"; the adjective "red". After logical construction, a hierarchical scene query call form with search type, content, locative word, and constraint information can be formed. Among them, the search type is used to narrow the data retrieval scope based on similarity matching, the search content is a specific entity description, and the adjective will be merged into the description text for query together, and the locative word is used to form a constraint on the search scope. The floor constraint is determined by the player's location. In indoor scenes, the upper and lower ranges need to be narrowed to avoid finding invisible entities across floors.
[0644] (2) Calculate the similarity between the target entity information and each entity information in the entity set.
[0645] Exemplarily, the entity set includes entity information of at least one entity, and the entity information of each entity may include at least one of a text embedding vector of a text label and an embedding vector of an image feature. Then the target entity information may calculate similarity with the text embedding vector and the image embedding vector respectively, and the entity with higher similarity is determined as the target entity.
[0646] The following are methods for similarity matching with text embedding vectors and similarity matching with image embedding vectors.
[0647] 1) Similarity includes the text similarity between the target entity information and the text embedding vector.
[0648] Taking the first entity in the entity set as an example, the game application (client or server) segments the target entity information to obtain at least one target entity tag; converts the at least one target entity tag into at least one target embedding vector; obtains the entity information of the first entity, the entity information includes a text embedding vector, and the text embedding vector is an embedding vector obtained based on the text tag conversion of the first entity; calculates the text parent similarity between at least one target embedding vector and the text embedding vector respectively, and obtains at least one text parent similarity corresponding to at least one target embedding vector; and determines the sum of the at least one text parent similarity as the text similarity between the target entity information and the entity information of the first entity.
[0649] For example, the number of target entity tags is two, target entity tag 1 and target entity tag 2, and the number of entities in the entity set is also two, entity 1 and entity 2. Entity 1 corresponds to text embedding vector 1, and entity 2 corresponds to text embedding vector 2. Then similarity 1 between target entity tag 1 and text embedding vector 1 is calculated, and similarity 2 between target entity tag 2 and text embedding vector 1 is calculated, and the sum of similarity 1 and similarity 2 is determined as the text similarity between the target entity information and entity 1. Similarity 3 between target entity tag 1 and text embedding vector 2 is calculated, and similarity 4 between target entity tag 2 and text embedding vector 2 is calculated, and the sum of similarity 3 and similarity 4 is determined as the text similarity between the target entity information and entity 2.
[0650] Exemplarily, the entity information of the first entity includes at least one text embedding vector; the at least one target embedding vector includes a first target embedding vector; the game application calculates the text sub-similarity between the first target embedding vector and the at least one text embedding vector respectively to obtain at least one text sub-similarity; the highest value of the at least one text sub-similarity is determined as the text parent similarity corresponding to the first target embedding vector.
[0651] For example, the number of target entity tags is 1: target entity tag 1. The number of entities in the entity set is also 1: entity 1. Entity 1 corresponds to text embedding vector 1 and text embedding vector 3. Then the similarity 1 between target entity tag 1 and text embedding vector 1 is calculated, and the similarity 5 between target entity tag 1 and text embedding vector 3 is calculated, and the larger value of similarity 1 and similarity 5 is determined as the similarity between the target entity information (target entity tag 1) and entity 1.
[0652] For example, for the input target entity information "blue car", the target entity information is first segmented into target entity labels "blue" and "car", which means that the queried target entity has two features. Then each feature is calculated with each entity in the entity set for similarity, and each feature takes the maximum similarity. Finally, the sum of the similarity scores corresponding to all query features is taken, which represents the text similarity between the target entity information and the queried entity.
[0653] For example, the target entity information includes two target entity tags "blue" and "car", and the two target entity tags are converted into two target embedding vectors. Then the entity information in the entity set is obtained. For example, the entity set includes entity 1 and entity 2. The text tags of entity 1 include "metal", "old", "car", "truck", "blue", and "damaged", and the text tags of entity 2 include "metal", "scratches", "car", "damaged", "money transporter", and "black".
[0654] Calculate the text similarity between the target entity information and entity 1: first calculate the similarity between the target entity label "blue" and each text label of entity 1, and take the maximum value. For example, the similarity between the target entity label "blue" and the text label "blue" of entity 1 is 1 (1 is the maximum value); then calculate the similarity between the target entity label "car" and each text label of entity 1, and take the maximum value. For example, the similarity between the target entity label "car" and the text label "car" of entity 1 is 1. Then sum the two similarities of the target entity labels "car" and "blue" to obtain the final similarity between the target entity information and entity 1, which is 2.
[0655] Similarly, calculate the text similarity between the target entity information and entity 2: first calculate the similarity between the target entity label "blue" and each text label of entity 2, and take the maximum value. For example, the similarity between the target entity label "blue" and the text label "black" of entity 1 is 0.91; then calculate the similarity between the target entity label "car" and each text label of entity 1, and take the maximum value. For example, the similarity between the target entity label "car" and the text label "car" of entity 2 is 1 (1 is the maximum value). Then sum the two similarities of the target entity labels "car" and "blue" to obtain the final similarity between the target entity information and entity 2, which is 1.91.
[0656] It can be seen that the similarity between the target entity information and entity 1 is 2, which is higher than the similarity between the target entity information and entity 2, which is 1.91.
[0657] 2) Similarity includes the image similarity between the target entity information and the image embedding vector.
[0658] Taking the first entity in the entity set as an example, the game application (client or server) segments the target entity information to obtain at least one target entity tag; converts the at least one target entity tag into at least one target embedding vector; obtains the entity information of the first entity, the entity information includes an image embedding vector, and the image embedding vector is an embedding vector obtained based on the image extraction of the first entity; calculates the image parent similarity of at least one target embedding vector and the text embedding vector respectively, and obtains at least one image parent similarity corresponding to the at least one target embedding vector; and determines the sum of the at least one image parent similarity as the image similarity between the target entity information and the entity information of the first entity.
[0659] For example, the number of target entity tags is two, target entity tag 1 and target entity tag 2, and the number of entities in the entity set is also two, entity 1 and entity 2. Entity 1 corresponds to image embedding vector 1, and entity 2 corresponds to image embedding vector 2. Then similarity 1 between target entity tag 1 and image embedding vector 1 is calculated, and similarity 2 between target entity tag 2 and image embedding vector 1 is calculated, and the sum of similarity 1 and similarity 2 is determined as the image similarity between the target entity information and entity 1. Similarity 3 between target entity tag 1 and image embedding vector 2 is calculated, and similarity 4 between target entity tag 2 and image embedding vector 2 is calculated, and the sum of similarity 3 and similarity 4 is determined as the image similarity between the target entity information and entity 2.
[0660] Exemplarily, the entity information of the first entity includes at least one image embedding vector; at least one target embedding vector includes a first target embedding vector; the game application calculates the image sub-similarity between the first target embedding vector and the at least one image embedding vector respectively to obtain at least one image sub-similarity; and the highest value of the at least one image sub-similarity is determined as the image parent similarity corresponding to the first target embedding vector.
[0661] For example, the number of target entity tags is 1: target entity tag 1. The number of entities in the entity set is also 1: entity 1. Entity 1 corresponds to image embedding vector 1 and image embedding vector 3. Then the similarity 1 between target entity tag 1 and image embedding vector 1 is calculated, and the similarity 5 between target entity tag 1 and image embedding vector 3 is calculated, and the larger value of similarity 1 and similarity 5 is determined as the similarity between the target entity information (target entity tag 1) and entity 1.
[0662] In an optional embodiment, the target image embedding vector may be used for calculating the image similarity, that is, the target image embedding vector is used instead of the target embedding vector in the above method. The target image embedding vector may be obtained by: segmenting the target entity information to obtain at least one target entity tag; inputting at least one target entity tag into a multimodal model to obtain a target image embedding vector.
[0663] 3) Similarity includes text similarity and image similarity.
[0664] When the first entity and the target entity have corresponding text similarity and image similarity, the game application determines an average value of the text similarity and the image similarity as the similarity between the first entity and the target entity.
[0665] Alternatively, when the first entity and the target entity have corresponding text similarity and image similarity, the larger value of the text similarity and the image similarity is determined as the similarity between the first entity and the target entity.
[0666] Alternatively, when the first entity and the target entity have corresponding text similarity and image similarity, the sum of the text similarity and the image similarity is determined as the similarity between the first entity and the target entity.
[0667] (3) Determine the target entity from the entity set based on similarity and environmental perception information.
[0668] In an optional embodiment, the game application first performs a similarity search based on the target entity information to obtain at least one candidate entity with a high similarity, and sorts them according to the similarity; then, based on the position, location, orientation and other information of the main virtual character and / or the non-player character, the game application performs perceptual filtering and sorting, and finally screens out the target entity.
[0669] Exemplarily, the game application determines a target perception range based on target entity information and environmental perception information; and obtains the target entity by filtering entities perceived within the target perception range based on similarity.
[0670] Alternatively, the game application determines x entities with the highest similarity from the entity set according to the similarity to form a candidate entity list; determines the target perception range according to the target entity information and the environmental perception information, and determines the entity in the target perception range among the x entities as the target entity. x is a positive integer.
[0671] Exemplarily, the game application determines the entity with the highest similarity in the perception range as the target entity.
[0672] It should be noted that when there are multiple entities with equal similarity within the target perception range, the multiple entities can be sorted according to the distance between the entity and the controlling virtual character. For example, when there are at least two entities with the highest similarity in the perception range, the entity with the highest similarity in the perception range and the closest distance to the controlling virtual character is determined as the target entity.
[0673] In an optional embodiment, the game application determines the entity with the highest similarity within the perception range and above the threshold as the target entity. Exemplarily, the number of the perception range is at least one. For example, the perception range includes at least one of the following: the visual range of the main virtual character, the hearing range of the main virtual character, the perception range of the main virtual character's perception of virtual props, the visual range of the non-player character, the hearing range of the non-player character, and the perception range of the non-player character's perception of virtual props.
[0674] Exemplarily, the game application may sequentially traverse at least one of the above-mentioned perception ranges, or the game application may sequentially traverse at least one perception range determined according to the natural language command. For example, if the natural language command is "Can you see the truck in front? Move to the truck", the game application may first traverse the visual range of the main virtual character, and then traverse the visual range of the non-player character, and match the entity with the highest similarity and above the threshold.
[0675] Exemplarily, when there are at least two perception ranges, the traversal order of the at least two perception ranges may be preset, or the traversal order of the at least two perception ranges may be determined according to the intention indicated in the natural language command. For example, when the intention is to search for an item, the visual range is traversed first. When the intention is to find a combat scene, the auditory range is traversed first.
[0676] If there are multiple levels of nested instructions in the natural language command, the first target entity will be searched first, and then the next target entity will be searched based on the position of the previous target entity. In the case where the natural language command includes a second target entity determined based on the position of the first target entity, the first target entity is queried from the entity set of the virtual environment based on the first target entity information and environmental perception information of the first target entity indicated by the natural language command; then the second target entity is queried from the entity set of the virtual environment based on the second target entity information, the first target entity and environmental perception information of the second target entity.
[0677] In response to natural language commands, the NPC is controlled to perform a virtual activity associated with a target entity.
[0678] Exemplarily, a large language model is called to identify the behavioral intent of natural language commands, and a control instruction or a control instruction sequence is generated according to the behavioral intent. According to the control instruction or the control instruction sequence, a non-player character is controlled to complete an activity based on a target entity.
[0679] The broadcasting of the NPC feedback information in step 208 may be implemented as follows:
[0680] Sub-step 6: reporting feedback information of the non-player character, wherein the feedback text corresponding to the feedback information is a non-fixed text generated based on the environmental perception information of the non-player character;
[0681] Exemplarily, the game application can generate corresponding feedback information in real time according to the environmental perception information of the non-player character and make a broadcast. For example, when the environmental perception information of the non-player character triggers the broadcast condition, the game application can generate corresponding feedback information in real time according to the environmental perception information that triggers the broadcast condition; or, when the non-player character or the main control virtual character triggers the broadcast condition, the game application can generate corresponding feedback information in real time according to the environmental perception information of the non-player character.
[0682] For example, the feedback text is obtained by inferring the large language model based on the static entity data of the three-dimensional virtual environment and the dynamic environment perception information of the non-player characters. The large language model will generate different feedback texts according to the different perception situations of the non-player characters.
[0683] For example, the feedback text can also be obtained by inferring the static entity data of the three-dimensional virtual environment and the dynamic environment perception information of the non-player character through the language model, and the feedback text can be in line with the personality characteristics of the non-player character. For example, when the non-player character is a strong uncle, the feedback text can adopt a heroic tone; when the non-player character is a reporter, the feedback text can adopt a news report tone.
[0684] It should be noted that due to the randomness of the results generated by the large language model, in the same scene, when the environmental perception information of the non-player character is the same, the generated feedback text may also be different.
[0685] The environmental perception information may include information perceived by the non-player character in real time, or may include information perceived by the non-player character historically.
[0686] Exemplarily, the feedback information is inferred by the game application based on the environmental perception information of the non-player character and the static entity data of the three-dimensional virtual environment. The feedback information can be a voice broadcast or a text broadcast. The static entity data includes data of relatively unchanged entities in the three-dimensional virtual environment, for example, model information of various buildings, terrains, and vehicles in the three-dimensional virtual environment.
[0687] Exemplarily, the feedback information may include at least one of the following: instant feedback, execution feedback, and dynamic feedback. Instant feedback and execution feedback are both feedback information generated for natural language commands, and dynamic feedback is feedback information spontaneously generated by non-player characters.
[0688] 1. Instant feedback is feedback information generated immediately when a natural language command is received. Instant feedback can include reply broadcast and instant broadcast. Reply broadcast is used to resume the inquiry in the natural language command; instant broadcast is used to feedback the reception of the natural language command.
[0689] For example, the client can command the non-player character to move in the three-dimensional virtual environment according to the intention of the natural language command. Then, when the client receives the natural language command, it can generate instant feedback corresponding to the natural language command, and the instant feedback is used to indicate that the natural language command has been received, or the instant feedback is used to instantly reply to the natural language command.
[0690] In an optional embodiment, the feedback information includes a reply report. The reply report includes a reply to the query natural language command generated in response to the query natural language command raised by the player. For example, when the query natural language command includes querying the location of the target entity, the corresponding reply report should include the query result of the location of the target entity.
[0691] When the behavioral intention of the received natural language command is an inquiry, the terminal broadcasts a reply message of the non-player character; the reply message includes reply content for the natural language command generated according to the environmental perception information of the non-player character on the three-dimensional virtual environment.
[0692] The natural language command is used to inquire about information from the non-player character. The natural language command may be referred to as an inquiry natural language command. The inquiry natural language command is a natural language command intended to be an inquiry. The inquiry natural language command contains at least one question.
[0693] Natural language commands are command instructions conveyed using human natural language. The terminal receives the natural language commands issued by the player and commands the non-player characters according to the intention corresponding to the natural language commands. For example, the terminal receives the voice audio of the player, converts the voice audio into text, and obtains the natural language command. Alternatively, the terminal receives the text of the natural language command input by the player.
[0694] Players can use spoken or written language to issue natural language commands. The game application will use the large language model to understand the natural language commands, extract the intentions expressed in the natural language commands, generate control instructions based on the intentions, and direct the activities of non-player characters.
[0695] For example, a natural language command may be "Come to me", and the game application may parse out that the intention of the natural language command is "control the non-player character to move to the location of the controlling virtual character". The game application then obtains the location of the controlling virtual character, generates a navigation route for the non-player character to move to the location of the controlling virtual character, and controls the non-player character to move to the location of the controlling virtual character along the navigation route.
[0696] For another example, a natural language command may be "go pick up a treasure chest", then the game application may parse out that the intention of the natural language command is "control the non-player character to move to the location of the treasure chest and pick up the treasure chest", then the game application may obtain the location of the treasure chest, control the non-player character to move to the location of the treasure chest, and perform the operation of picking up the treasure chest after arriving at the location of the treasure chest.
[0697] For example, when the natural language command includes an intelligence inquiry about the target entity, a first reply report is broadcast based on the non-player character's perception of the target entity; the first reply report includes the intelligence perception result of the non-player character about the target entity.
[0698] Alternatively, when the natural language command includes an inquiry about the location of the target entity, a second reply broadcast is broadcast based on the non-player character's perception of the target entity, and the second reply broadcast includes a description of the location of the target entity.
[0699] Exemplarily, a large language model is called to parse the received natural language command to obtain the intention of the natural language command. When the intention is of an inquiry type, the natural language command can be determined to be an inquiry natural language command. When the natural language command is an inquiry natural language command, the large language model can infer according to the intention of the natural language command, based on static entity data and environmental perception information, to obtain a reply text. That is, the large language model is called to parse the received natural language command, and the intention of the natural language command is output as an inquiry, as well as the reply text of the natural language command. The game application calls the text-to-speech service according to the processing logic of the inquiry intention, and converts the reply text into a reply broadcast, which is an audio broadcast, and sends the reply audio to the client for broadcast.
[0700] For example, when the large language model recognizes that the intention of the natural language command is to ask a question, since the user may ask hundreds of questions and it is impossible to store the answer to each question locally on the client, the game application will call the text-to-speech service to generate a reply broadcast in real time based on the reply text returned by the large language model, so that the game application can instantly generate a corresponding reply broadcast to respond to the player's question.
[0701] Exemplarily, a game application (client or server) calls a large language model to parse natural language commands, and infers a reply text based on static environmental data and environmental perception information of the three-dimensional virtual environment; and generates human voice audio based on the reply text to obtain a reply broadcast.
[0702] For example, the client reports the received natural language query command to the server, the server calls the large language model to parse the natural language query command, and infers the reply text based on the static environment data and environmental perception information of the three-dimensional virtual environment; generates human voice audio based on the reply text to obtain a reply broadcast; the server sends the reply broadcast to the client; and the client broadcasts the reply broadcast.
[0703] 2. Execution feedback is the feedback generated when a task is executed according to the intent indicated by the natural language command after receiving the natural language command. Execution feedback can also be called execution broadcast.
[0704] For example, when a natural language command is used to instruct a non-player character to perform a target task, the client can generate execution feedback during the non-player character's execution of the target task, and the execution feedback is used to indicate the execution status of the target task; or, the client can generate execution feedback after the non-player character completes the target task, and the execution feedback is used to indicate the execution result of the target task.
[0705] In an optional embodiment, the feedback information includes a feedback report (also referred to as execution feedback). The feedback report includes a report on the task execution status and task execution result generated in response to the task natural language command proposed by the player. For example, when the task natural language command includes moving to the target entity, the corresponding feedback report may include: moving to the target entity, or having reached the target entity.
[0706] When the behavioral intention of the received natural language command is to execute a task, the terminal broadcasts a feedback report; the feedback report includes an execution status after the natural language command is executed according to the environmental perception information of the non-player character on the three-dimensional virtual environment.
[0707] The natural language command is used to instruct the non-player character to perform a task. The natural language command may be referred to as a task natural language command. The task natural language command includes a description text of the task, for example, the description text of the task may include at least one of the following: the task name, the action that the non-player character needs to perform, the method used to perform the task, the task target, and the location of the task target.
[0708] For example, when the intention of the task natural language command includes searching for a target entity, a first feedback announcement is broadcast, and the first feedback announcement includes the search result of the non-player character for the target entity.
[0709] Alternatively, when the intention of the task natural language command includes using a virtual prop, a second feedback announcement is announced, and the second feedback announcement is used to indicate that the virtual prop is ready.
[0710] Alternatively, when the intention of the task natural language command includes the use of virtual props, a third feedback announcement is broadcast, and the third feedback announcement is used to indicate the result of the use of the virtual props.
[0711] Alternatively, when the intention of the task natural language command includes controlling the movement of the non-player character, a fourth feedback announcement is announced, and the fourth feedback announcement is used to indicate the movement result of the movement.
[0712] Alternatively, when the intention of the task natural language command includes performing an interaction with a target entity, a fifth feedback announcement is broadcast, and the fifth feedback announcement is used to indicate at least one of a search result of the non-player character for the target entity and an interaction result between the non-player character and the target entity.
[0713] Alternatively, when the intention of the task natural language command includes item interaction, a sixth feedback broadcast is broadcast, and the sixth feedback broadcast is used to indicate the item interaction result of the non-player character performing the item interaction.
[0714] Exemplarily, a large language model is called to parse the received natural language command to obtain the intention of the natural language command. When the intention is a task, the natural language command can be determined to be a task natural language command. When the natural language command is a task natural language command, the large language model can be inferred according to the intention of the natural language command, based on static entity data and environmental perception information, to obtain a task instruction sequence, and the task instruction sequence is used to control the non-player character to perform the task. When the game application receives the task instruction sequence, it controls the non-player character activity to perform the task according to the task instructions in the task instruction sequence. During the control of the non-player character according to the task instruction, the corresponding execution status can be obtained for feedback broadcast according to the feedback broadcast logic corresponding to different task instructions.
[0715] For example, since the instructions that can be executed by non-player characters in a three-dimensional virtual environment are finite and traversable, the client can locally store feedback broadcasts corresponding to each character's instructions. When the non-player character executes the corresponding instruction, the client can read the local feedback broadcast for voice broadcast.
[0716] Alternatively, when there is variable content in the feedback broadcast corresponding to a task instruction, for example, the target entity in the feedback broadcast is variable, or the position information in the feedback broadcast is variable; the client can also generate a broadcast voice with variable content based on the target entity or position information returned by the large language model, and splice it with the broadcast voice with immutable content stored locally to obtain the final feedback broadcast.
[0717] For example, when the task natural language command is "find a nearby treasure chest", calling the large language model to parse the task natural language command can get the intention of the task, the task goal is to find the target entity, and the target entity is a treasure chest; based on the static entity data and environmental perception information, reasoning can be obtained that there is a treasure chest in the left front of the main control virtual character. In the case that the target entity can be found, the feedback broadcast corresponding to the task goal is "found aaa at xxx", where xxx is the location of the target entity and aaa is the name of the target entity. The game application can call the online text-to-speech technology to generate the location voice according to the location information "left front" in the reasoning result; according to the name of the target entity "treasure chest", call the online text-to-speech technology to generate the name voice, and then splice the location voice and name voice into the feedback broadcast to get the final feedback broadcast.
[0718] Exemplarily, a game application (client or server) calls a large language model to parse natural language commands for a task, and infers task execution instructions based on static environmental data and environmental perception information of a three-dimensional virtual environment; controls a non-player character to perform the task according to the task execution instructions; generates feedback text according to the execution status of the non-player character performing the task; and generates human voice audio based on the feedback text to obtain a feedback broadcast.
[0719] For example, the client reports a task natural language command to the server; the server parses the task natural language command, and infers the task execution instructions based on the static environment data and environment perception information of the three-dimensional virtual environment; the non-player character is controlled to perform the task according to the task execution instructions; feedback text is generated according to the execution status of the non-player character performing the task; human voice audio is generated based on the feedback text to obtain a feedback broadcast; the server sends the feedback broadcast to the client; the client receives and broadcasts the feedback broadcast.
[0720] 3. Dynamic feedback is the feedback information generated spontaneously based on the environmental perception information of non-player characters. Dynamic feedback can also be called dynamic broadcast.
[0721] For example, the client can spontaneously generate dynamic feedback based on the perception of the three-dimensional virtual environment without receiving natural language commands, and the dynamic feedback is used to indicate abnormal situations found by the non-player character in the three-dimensional virtual environment. For example, abnormal situations may include: finding hostile virtual characters, finding changes in the status of friendly virtual characters, finding dangerous situations, finding traces of battle or looting, etc.
[0722] In an optional embodiment, the feedback information includes dynamic reporting (also referred to as "dynamic feedback"). The dynamic reporting includes reporting of abnormal situations perceived by non-player characters. For example, when a non-player character perceives being attacked, a report is made that the character is being attacked; when a non-player character perceives a dangerous situation ahead, a report is made that there is danger ahead.
[0723] The terminal broadcasts a dynamic report when the environmental perception information of the non-player character on the three-dimensional virtual environment meets the dynamic report conditions.
[0724] Dynamic reporting is a report generated by the game application based on the environmental perception information of non-player characters when it determines that there is an abnormal situation that needs to be reported. The dynamic report includes reminder information of the abnormal situation. For example, the abnormal situation may include at least one of the following: discovering a hostile virtual character, discovering a change in the status of a friendly virtual character, and discovering a new trace.
[0725] For example, in the case where the non-player character perceives a hostile virtual character, a first dynamic broadcast is broadcast; the first dynamic broadcast includes the position of the hostile virtual character perceived by the non-player character.
[0726] Alternatively, when the non-player character senses a dangerous situation, a second dynamic broadcast is broadcast; the second dynamic broadcast is used to prompt the dangerous situation.
[0727] Alternatively, when the non-player character senses a change in the state of the friendly virtual character, a third dynamic broadcast is broadcast; the third dynamic broadcast is used to prompt that the state of the friendly virtual character has changed.
[0728] Alternatively, when the non-player character finds new traces in the three-dimensional virtual environment, a fourth dynamic broadcast is broadcast; the fourth dynamic broadcast is used to prompt the new traces. The new traces may be combat traces and / or looting traces.
[0729] For example, since the number of dynamic announcements that can be triggered by non-player characters in a three-dimensional virtual environment is limited and traversable, the client can store dynamic announcement voices locally, and the client can read the dynamic announcement voices locally for announcement when the corresponding dynamic announcement voices are triggered. Similarly, some dynamic announcement voices can contain variable content, and the client can generate variable content in real time through text-to-speech technology, and splice it with the dynamic announcement template to obtain the final dynamic announcement.
[0730] In an optional embodiment, when generating feedback information, it may be necessary to query the location of the target entity in the three-dimensional virtual environment, or determine the target entity indicated in the natural language command from the three-dimensional virtual environment. In this case, an entity query method is required to query the target entity.
[0731] Before executing the target entity query, it is necessary to pre-build the static entity data of the 3D virtual environment. The static entity data construction can serve both the reasoning of the large language model and the real-time feedback text generation on the game application side after the reasoning results are returned.
[0732] In an optional embodiment, entities in the three-dimensional virtual environment are traversed, and entity information of each entity is exported from a game engine to obtain static entity data.
[0733] For example, for in-game scenes, editor tools have been developed to perform StaticMesh traversal of the entire scene, special Actor traversal (such as interactive doors, which do not belong to the StaticMesh category), vegetation traversal, and export the entity's position, orientation, and bounding box size as static entity data for subsequent labeling and reasoning by the large language model.
[0734] Among them, StaticMesh is a static geometry resource type in the Unreal Engine 4 (UE4) game engine, which is used to represent immutable 3D models, such as buildings, props, etc. Actor is a basic object class in the Unreal Engine 4 (UE4) game engine, representing an entity in the game world, such as a character, object, light source, etc. Actor can contain multiple components to achieve different functions.
[0735] Exemplarily, static entity data can also include spatial data, which is obtained by manual labeling. Spatial data is more complex than static entities because there are multiple floors in the space and multiple layers of nesting within the floors (such as the motel area includes the second floor, and the second floor area includes room 201). For this type of data, manual calibration is used to construct it. Add Volume to the scene in the editor, divide the area required for the entire map and label it accordingly. Volume is a special Actor class in the UnrealEngine4 (UE4) game engine, which represents a three-dimensional area with specific functions, such as a trigger area, audio area, etc.
[0736] After obtaining the static entity data, you can query entities or spaces based on the static entity data during the game.
[0737] For example, when a player issues a natural language command, the natural language command will be accompanied by the constructed static entity data and the real-time captured runtime data (including the environmental perception information of non-player characters and / or the main virtual character) to make an inference request to the large language model. In addition, when the NPC needs to provide dynamic feedback based on the runtime data, an entity query is also performed to generate feedback text based on the current state.
[0738] When a natural language command includes a target entity, the game application will query the target entity based on the position and orientation of the current controlling virtual character, the position and orientation of teammates or enemies, and based on the target entity information of the target entity described in the natural language command and the environmental perception information of the controlling virtual character and / or the non-player character, filter out a target entity that matches the target entity information from at least one candidate entity within the perception range of the controlling virtual character and / or the non-player character.
[0739] Optionally, the natural language command is used to instruct the non-player character to perform an activity related to the target entity. For example, the target entity may be a destination for the non-player character to move to, an object that the non-player character needs to observe, a target that the non-player character needs to attack, or an object that the non-player character needs to interact with.
[0740] The target entity is a virtual object that exists in a three-dimensional virtual environment. The target entity is the entity described in a natural language command. An entity may refer to a virtual object that is fixed in a three-dimensional virtual environment. For example, a target entity may be a virtual building, a virtual terrain, a virtual vehicle, a virtual plant, a virtual prop, a virtual item, etc. An entity may also refer to a virtual object that can interact with a virtual character in a three-dimensional virtual environment. For example, a target entity may be an interaction point in a three-dimensional virtual environment (e.g., a door, a window, a cabinet, a cellar, etc.), a virtual prop, a virtual character, a virtual light source, etc.
[0741] For example, a natural language command may be "Please help me open the door", then the target entity may be "door", and the non-player character needs to perform a door-related activity "open the door". Alternatively, a natural language command may be "Is the kitchen safe", then the target entity may be "kitchen", and the non-player character needs to perform a kitchen-related activity "check if there are enemy avatars or other dangerous situations in the kitchen".
[0742] Optionally, the natural language command includes target entity information of the target entity. The target entity information may be a description text of the target entity. For example, the target entity information may include at least one of the following contents: the name, type, location, and features of the target entity. The game application may find the target entity from among the many entities in the three-dimensional virtual environment based on the target entity information in the natural language.
[0743] The game application parses the natural language command, extracts the target entity information therein, determines the target entity from the entity set of the three-dimensional virtual environment based on the target entity information, and then controls the non-player character to perform activities related to the target entity according to the intention of the natural language command.
[0744] The target entity is determined from the three-dimensional virtual environment in combination with environmental perception information, and the environmental perception information includes information perceived by at least one of the main virtual character and the non-player character from the three-dimensional virtual environment.
[0745] Since the target entity information in natural language commands is usually vague, for example, a natural language command may be "move to the red truck", where the target entity information is "red truck", and there may be many red trucks in the three-dimensional virtual environment. It is impossible to accurately determine the target entity in the three-dimensional virtual environment based solely on the target entity information in the natural language command.
[0746] Therefore, the embodiment of the present application provides a method for determining the target entity by combining the target entity information and the environmental perception information. For the previous example, the "red truck" expressed by the player in the natural language command should be a red truck that the player can see. Therefore, combined with the field of view of the main control virtual character, the red truck within the field of view of the main control system character can be screened out from multiple red trucks in the three-dimensional virtual environment. The red truck is the target entity indicated by the player in the natural language command.
[0747] Since natural language commands are issued by players based on their perception of the 3D virtual environment, in order to accurately identify the target entity indicated in the natural language command, the game application will combine the environmental perception information when the player issues the natural language command to identify the target entity in the natural language command. Based on the player's perception of the 3D virtual environment, the entity that the player can perceive in the 3D virtual environment and that is closest to the target entity information is inferred as the target entity.
[0748] Exemplarily, the three-dimensional virtual environment includes multiple candidate entities that match the natural language command, and the target entity is an entity selected from the multiple candidate entities based on the environmental perception information of the main virtual character or the non-player character. For example, the game application first selects multiple candidate entities that match the target entity information in the natural language command from the entity set, and then selects the entity that can be perceived by the main virtual character or the non-player character from the multiple candidate entities as the target entity based on the environmental perception information.
[0749] For example, the perception range of the main virtual character is determined according to the position and orientation of the main virtual character. For example, the perception range is a sector area of a certain visual field size in front of the main virtual character; entities that are not within the sector range are filtered out. Then, the target entity information and the entity information of the candidate entities are sorted according to the similarity, and entities with higher similarity are selected as target entities. When there are two entities with the highest and equal similarity scores within the perception range, for example, the similarity scores of entity 2 and entity 3 are both 2 points, then entity 2, which is closer to the main virtual character, is selected as the target entity.
[0750] When it is necessary to generate feedback information based on the position description of the target entity, the game application can perform a spatial query to generate a position description based on the position relationship between the master virtual character and the target entity.
[0751] The main application scenario of spatial query is to obtain the location of the main virtual character, non-player characters, hostile virtual characters, gunshots, etc. For example, a report containing spatial information such as "enemies found on the second floor of the motel" can be made.
[0752] The spatial query logic defines the concepts of parent area and child area. The parent area refers to a large area in a 3D virtual environment that covers multiple buildings or multiple isolated spaces, such as a motel with front and back yards and multiple rooms and basements. The child area refers to an independent space in the parent area, which can be a closed space or a special function space, such as a motel guest room or kitchen.
[0753] In order to make the location description returned by the spatial query closer to human expression, when the main virtual character and the queried location are in different parent areas, the query result returns complete text information (for example, the main virtual character is outside the motel and the queried location is on the second floor of the motel, then "the enemy is on the second floor of the motel" is returned). When the main virtual character and the queried location are in the same parent area but different child areas, the query result only returns the child area information (for example, the main virtual character is on the first floor of the motel and the queried location is on the second floor, then "the enemy is on the second floor" is returned). When the main virtual character and the queried location are in the same parent area and the same child area, the query result only returns the direction relative to the main perspective player (for example, the main virtual character and the queried location are both on the second floor of the motel, then "the enemy is in the right front position" is returned).
[0754] Exemplarily, when the controlling virtual character and the target entity are located in different parent spaces, the position description includes the parent space and the child space where the target entity is located. When the controlling virtual character and the target entity are located in different child spaces of the same parent space, the position description includes the child space where the target entity is located. When the controlling virtual character and the target entity are located in the same child space of the same parent space, the position description includes the relative position of the target entity and the controlling virtual character; wherein a parent space includes at least one child space.
[0755] For the broadcast of the NPC feedback information in step 208, the feedback information is implemented as spatial human voice audio processed by spatial sound effects, wherein the spatial sound effect processing process can be implemented as follows:
[0756] Sub-step 7: Obtaining human voice audio corresponding to the feedback text;
[0757] In the embodiment of the present application, the process of generating the feedback text is as shown in the above-mentioned sub-step 6, which will not be repeated here.
[0758] In some embodiments, the process of obtaining corresponding human voice audio based on feedback text can be implemented as follows: obtaining feedback text and character feature information corresponding to a non-player character; and generating human voice audio matching the character feature information according to the feedback text.
[0759] Illustratively, the character feature information is used to describe the attributes of the non-player character, that is, the character feature information is information used to describe the current state and characteristics of the non-player character. In some embodiments, the attributes of the non-player character include at least one of the basic attributes of the non-player character and the situational performance attributes. Among them, the basic attributes are attributes pre-configured for the non-player character, that is, the attributes of the non-player character that do not change due to the three-dimensional virtual environment or the current situation, such as the age information of the non-player character, the male / female information of the non-player character, the martial arts information of the non-player character, etc. The situational performance attributes are attributes determined in real time by the non-player character in the situation of the three-dimensional virtual environment, that is, attributes associated with the current situation, such as the emotional information and behavioral information of the non-player character in the current situation.
[0760] Optionally, the character feature information includes at least one of basic character information, character emotion information, and character behavior information. The basic character information is used to indicate the basic situation of the non-player character, and the basic character information is information about the basic attributes of the non-player character, such as the age information of the non-player character, the male / female information of the non-player character, and the martial arts information of the non-player character; the character emotion information is used to indicate the emotional state of the non-player character in the dialogue scenario corresponding to the feedback text, and the character emotion information is the situational performance attribute of the non-player character, such as the character emotion information indicates that the non-player character is in an excited state, an angry state, a sad state, etc.; the character behavior information is used to indicate the actions performed by the non-player character in the dialogue scenario, and the character behavior information is the situational performance attribute of the non-player character, such as being in a running state, an attacking state, an injured treatment state, etc.
[0761] In some embodiments, the generation of human voice audio is achieved through a pre-trained speech generation model, and the speech generation model is used to generate speech that matches the character feature information of the non-player character. In an illustrative example, the feedback text and the character feature information are input into the pre-trained speech generation model to obtain human voice audio as the human voice audio.
[0762] Optionally, the above-mentioned speech generation model can be implemented by a neural network model such as a convolutional neural network, a feedforward neural network, a residual network, a Transformer, a multimodal large language model (Multimodal Large Language Model, MLLM), etc., which is not specifically limited here.
[0763] In some embodiments, the speech generation model includes a text encoder, a role encoder and a decoder. The text encoder is used to perform text encoding on the input feedback text, that is, the feedback text is input into the text encoder to obtain a text encoding representation. The role encoder is used to perform feature encoding on the input role feature information, that is, the role feature information is input into the role encoder to obtain a role encoding representation.
[0764] Illustratively, after obtaining the text coding representation and the role coding representation, the text coding representation and the role coding representation are fused to obtain the coding representation input to the decoder, that is, the text coding representation and the role coding representation are fused to obtain a joint coding representation, and the joint coding representation is input into the decoder to generate human voice audio.
[0765] In some embodiments, the character encoder includes at least one of a first sub-encoder, a second sub-encoder, and a third sub-encoder. The first sub-encoder is used to perform feature encoding based on the basic attributes of the non-player character, that is, when the character feature information includes the character basic information, the character basic information is input into the first sub-encoder to obtain a first character encoding representation; the second sub-encoder is used to perform feature encoding based on the emotional state of the non-player character, that is, when the character feature information includes the character emotional information, the character emotional information is input into the second sub-encoder to obtain a second character encoding representation; the third sub-encoder is used to perform feature encoding based on the behavior state of the non-player character, that is, when the character feature information includes the character behavioral information, the character behavioral information is input into the third sub-encoder to obtain a third character encoding representation.
[0766] Sub-step 8: based on the relative position relationship between the non-player character and the main control virtual character in the three-dimensional virtual environment, obtaining sound effect parameters corresponding to the non-player character;
[0767] Illustratively, the relative position relationship between the non-player character and the controlling virtual character in the three-dimensional virtual environment is used to indicate the relative relationship between a first position corresponding to the non-player character and a second position corresponding to the controlling virtual character in the three-dimensional virtual environment.
[0768] Optionally, the above-mentioned relative orientation relationship includes at least one of position distance, position direction, and spatial surrounding situation between the non-player character and the main virtual character, wherein the position distance is used to indicate the distance relationship between the non-player character and the main virtual character, the position direction is used to indicate the angular relationship between the first position of the non-player character and the orientation direction of the main virtual character, and the spatial surrounding situation is used to indicate the surround situation of the main virtual character by the spatial elements forming the virtual space in the three-dimensional virtual environment, and the propagation effect of the space on the audio when the non-player character emits audio in the virtual space.
[0769] In some embodiments, the determination of the relative position relationship between the non-player character and the main virtual character can be implemented as follows: obtaining first position information of the non-player character in the three-dimensional virtual environment, obtaining second position information of the main virtual character in the three-dimensional virtual environment, and determining the relative position relationship between the non-player character and the main virtual character based on the first position information and the second position information.
[0770] In some embodiments, the first position information of the non-player character and the second position information of the controlling virtual character are position information determined based on the same preset coordinate system. Optionally, the preset coordinate system can be implemented as a world coordinate system corresponding to the three-dimensional virtual environment, or the preset coordinate system can be implemented as a coordinate system established with the controlling virtual character as the origin.
[0771] In some embodiments, when the sound effect parameters corresponding to the non-player characters are pre-generated and stored in a database, the sound effect parameters corresponding to the non-player characters are obtained from the database. In other embodiments, the sound effect parameters corresponding to the non-player characters may also be generated in real time, that is, the sound effect parameters corresponding to the non-player characters are generated based on the relative position relationship.
[0772] Optionally, the sound effect parameters corresponding to the non-player character may include at least one of the following types:
[0773] The first one is the distance type;
[0774] Illustratively, the distance-type sound effect parameter is used to indicate that the sound effect of the human voice audio is adjusted based on the distance between the non-player character and the main virtual character. In some embodiments, the distance-type sound effect parameter can adjust the audio volume of the human voice audio to simulate the effect of distance on the sound volume; and / or, the distance-type sound effect parameter can adjust the audio delay of the human voice audio to simulate the effect of distance on the sound propagation time.
[0775] The second type is the direction type;
[0776] Illustratively, the direction-type sound effect parameter is used to indicate the adjustment of the sound effect of the human voice audio based on the direction of the non-player character relative to the main virtual character. In some embodiments, the direction-type sound effect parameter can adjust the audio volume of the human voice audio to simulate whether the main virtual character is facing the scene audio element; and / or, the direction-type sound effect parameter can adjust the volume parameters of the human voice audio in different channels to simulate the direction of the audio emitted by the scene sound source element.
[0777] The third type is the spatial special effects type;
[0778] Illustratively, the sound effect parameters of the spatial special effect type are used to indicate that the virtual space formed based on the spatial elements adjusts the sound effects in the virtual space, that is, the sound effect parameters of the spatial special effect type are used to indicate the effect of the virtual space on the sound performance of the audio emitted by the non-player characters. In some embodiments, the sound effect parameters of the spatial special effect type can add echo sound effects and reverberation sound effects to simulate the effect of sound in the space.
[0779] Optionally, the sound effect parameters of the spatial special effect type include at least one of a reverberation time parameter, a pre-delay parameter, a wetness (Wet / DryMix) parameter, a space width parameter, and a distance effect parameter. Among them, the reverberation time parameter is used to indicate the speed at which the audio decays in the virtual space; the pre-delay parameter is used to indicate the time difference between the audio directly reaching the main virtual character and the audio reaching the main virtual character after the first reflection; the wetness parameter is used to indicate the ratio between the audio directly transmitted and the audio transmitted through reflection; the space width parameter is used to indicate the degree of diffusion of the audio on the horizontal plane in the virtual space; and the distance effect parameter is used to indicate the attenuation of the audio with the propagation distance during the propagation process.
[0780] The fourth type is custom type;
[0781] Schematically, the custom type of sound effect parameters are user-defined parameters for sound effect adjustment. Optionally, the user can customize the overall volume of the scene audio, whether to add background music, customize the volume of different types of audio, etc.
[0782] In some embodiments, the client provides a customization interface for sound effect parameters to the user, and the user configures the customized sound effect parameters through the customization interface.
[0783] In some embodiments, a parameter prediction model for personalized learning of the custom sound effect parameters is configured in the custom sound effect parameter server for the user. Schematically, the custom sound effect parameters of multiple candidate accounts are obtained, the multiple custom sound effect parameters are input into the parameter prediction model to be trained, the parameter prediction model to be trained is iteratively trained to obtain the parameter prediction model, and the sound effect parameters generated by the system (for example, the sound effect parameters of the above-mentioned distance type, direction type, and spatial special effect type) are optimized through the parameter prediction model.
[0784] Sub-step 9: adjusting the vocal audio based on the sound effect parameters, generating and broadcasting the spatial vocal audio corresponding to the non-player character;
[0785] Among them, the spatial human voice audio is used to express the perceptual effect of the audio produced by the main virtual character on the non-player character under the relative position relationship.
[0786] In an embodiment of the present application, the human voice audio is adjusted according to the sound effect parameters corresponding to the non-player character, thereby generating the spatial human voice audio corresponding to the non-player character. That is, the spatial human voice audio is the audio data corresponding to the non-player character obtained after the human voice audio is adjusted by the sound effect parameters. In some embodiments, the spatial human voice audio includes the audio after the sound effect parameters are adjusted, as well as at least one of the audio duration information, the audio playback start timestamp, the audio playback end timestamp, the audio playback condition information, and the element identification information corresponding to the audio.
[0787] Optionally, when the sound effect parameters indicate that the volume of the non-player character's vocal audio is to be adjusted, the volume of the non-player character's vocal audio on at least one channel is adjusted based on the sound effect parameters. Optionally, when the sound effect parameters indicate that the playback delay of the non-player character's vocal audio is to be adjusted, the audio playback start time corresponding to the non-player character's vocal audio is adjusted based on the sound effect parameters. Optionally, when the sound effect parameters indicate that the speed of the non-player character's vocal audio is to be adjusted, the playback speed of the non-player character's vocal audio is adjusted based on the sound effect parameters. Optionally, when the sound effect parameters indicate the pitch of the non-player character's vocal audio, the frequency in the spectrum of the non-player character's vocal audio is adjusted based on the sound effect parameters. When the sound effect parameters indicate the timbre of the non-player character's vocal audio, a filter corresponding to the non-player character is determined based on the sound effect parameters, and the timbre corresponding to the vocal audio is adjusted through the filter.
[0788] Illustratively, the client plays the spatial human voice audio to realize the broadcasting of non-player character feedback information.
[0789] The speech-to-text conversion stage of step 204 can be implemented as follows:
[0790] Sub-step 10: obtaining a plurality of first scene hot words corresponding to the first virtual scene where the target virtual character is located;
[0791] The target virtual character includes at least one of an NPC and a master virtual character.
[0792] In a illustrative manner, the main virtual character is the virtual character commanded by the player, and the NPC is the virtual character that assists the main virtual character in a virtual game. Usually, a virtual game includes a main virtual character and at least one NPC. There are certain differences between the main virtual character and the NPC.
[0793] Optionally, the main virtual character is the virtual character mainly controlled by the player, and the NPC is the virtual character chosen by the player to command, such as the main virtual character is the virtual character manually controlled by the player during the game, and the player's manual operations on the terminal interface are used to control the main virtual character; the NPC is the virtual character commanded by the player in voice form, and the player occasionally issues natural language commands in voice form to command the NPC.
[0794] In some embodiments, the first virtual scene is determined based on the target virtual character, and the target virtual character is at least one of an NPC and a master virtual character, which means that the process of determining the first virtual scene is implemented as at least one of the following.
[0795] (1) If the main virtual character is selected as the target virtual character, the virtual scene where the main virtual character is located is used as the first virtual scene;
[0796] (2) If an NPC is selected as the target virtual character, the virtual scene where the NPC is located is used as the first virtual scene; if there are multiple NPCs, the virtual scene including the most NPCs can be used as the first virtual scene. Alternatively, the NPC closest to the main virtual character can be selected as the target virtual character, and the virtual scene where the NPC is located can be used as the first virtual scene. Alternatively, the NPC with the largest character attribute value (such as at least one of the attribute values such as virtual health, virtual mana, and virtual defense) can be selected as the target virtual character, and the virtual scene where the NPC is located can be used as the first virtual scene.
[0797] (3) If the main virtual character and the NPC are selected as the target virtual characters, the virtual scene in which the main virtual character and the NPC are located can be used as the first virtual scene, etc.
[0798] In some embodiments, the target virtual character is determined based on the player's selection; or, the target virtual character is set by system default.
[0799] In some embodiments, a plurality of first scene hot words are determined based on the first virtual scene.
[0800] Among them, the plurality of first scene hot words are scene-related words of the first virtual scene, that is, there is an association relationship between the first scene hot words and the first virtual scene, and the first scene hot words are words that describe the first virtual scene.
[0801] Optionally, the first scene hot words include the scene status of the first virtual scene, such as: the first virtual scene is a virtual restaurant, and the first scene hot words include business status, rest status, closed status, etc.
[0802] Optionally, the first scene hot words include element names of virtual elements in the first virtual scene, such as: the first virtual scene is a virtual restaurant, and the first scene hot words include virtual tables, virtual chairs, virtual kitchen utensils, virtual plates, etc.
[0803] Optionally, the first scene hot words include interactive vocabulary for interacting with virtual elements in the first virtual scene, such as: the first virtual scene is a virtual battlefield, and the first scene hot words include attack, attack virtual character, defense, building barriers, etc.
[0804] In some embodiments, after receiving a natural language command, multiple first scene hot words are collected based on the first virtual scene; or, after receiving a natural language command, multiple first scene hot words corresponding to the first virtual scene are filtered from multiple scene hot words pre-acquired based on the first virtual scene, etc.
[0805] In an optional embodiment, the object perception range of the target virtual character in the first virtual scene is obtained.
[0806] Among them, the object perception range is the three-dimensional space range in which the target virtual character perceives the existence of other scene elements.
[0807] Optionally, when the target virtual character is the master virtual character, the object perception range of the master virtual character in the first virtual scene is obtained; when the target virtual character is an NPC, the object perception range of the NPC in the first virtual scene is obtained; or, when the target virtual character is the master virtual character and the NPC, the object perception range of the master virtual character in the first virtual scene is obtained and the object perception range of the NPC in the first virtual scene is obtained.
[0808] In some embodiments, the object perception range includes at least one of the following.
[0809] (1) The visual perception range of the target virtual character.
[0810] In schematic form, the visual perception range refers to the spatial range that an observer can visually perceive or understand, and is usually used to describe the size of the area that an organism or technical device can visually cover or perceive. The visual perception range of the target virtual character is the three-dimensional spatial range perceived by the target virtual character as an observer through observation.
[0811] Optionally, based on the position and visual observation ability of the target virtual character in the first virtual scene, the visual perception range of the target virtual character is acquired.
[0812] (2) The auditory perception range of the target virtual character.
[0813] In schematic form, the auditory perception range refers to the spatial range that a listener can perceive or understand, and is usually used to describe the range of sounds that an organism or technical device can cover or perceive. The auditory perception range of the target virtual character is the three-dimensional spatial range that the target virtual character can perceive through its auditory perception ability, with the target virtual character as the listener.
[0814] Optionally, based on the position and auditory perception ability of the target virtual character in the first virtual scene, the auditory perception range of the target virtual character is acquired.
[0815] (3) The olfactory perception range of the target virtual character.
[0816] In schematic form, the olfactory perception range refers to the spatial range of the odor perceived or recognized by the sense of smell. The olfactory perception range of the target virtual character is the three-dimensional spatial range that the target virtual character can perceive by the sense of smell.
[0817] Optionally, based on the position and olfactory perception ability of the target virtual character in the first virtual scene, the olfactory perception range of the target virtual character is acquired.
[0818] (4) The range of perception of the target virtual character’s perception skills or perception tools.
[0819] Illustratively, the perception skill is the ability of the target virtual character to perceive the first virtual scene, and the perception skill includes at least one of a perception acquisition form and a perception enhancement form.
[0820] Optionally, the perception acquisition form is used to indicate the process of acquiring a perception skill that is not possessed. After the perception skill is possessed through the perception acquisition form, the target virtual character can perform targeted scene perception of the first virtual scene through the perception skill. Optionally, the perception enhancement form is used to indicate the process of enhancing a certain perception ability by possessing the perception skill when the target virtual character possesses the perception ability.
[0821] Illustratively, the perception prop is a virtual prop used to perceive the first virtual scene, and the perception prop includes at least one of a perception acquisition form and a perception enhancement form.
[0822] Optionally, different perception skills or perception tools each correspond to a preset effective range, which represents a range that can be perceived when the perception skill or perception tool is applied, and the preset effective range represents the object perception range.
[0823] In some embodiments, the object perception range is a spherical range, an annular range, an irregular three-dimensional space range, etc. The shape of the object perception range is not limited here.
[0824] In an optional embodiment, a plurality of first scene hot words are acquired based on environmental perception information within a perception range of the object.
[0825] Schematically, the environmental perception information is used to represent the environmental information perceived and acquired by the target virtual character within the object perception range. Optionally, the environmental perception information includes virtual elements determined by emitting rays, and may also include regional structure information determined by analyzing the geometric structure and layout information of the virtual environment.
[0826] Illustratively, after determining the object perception range corresponding to the target virtual character, based on the range of the object perception range in the first virtual scene and the environmental perception information determined based on perception of the target virtual character within the object perception range, multiple first scene hot words representing the state of the target virtual character are obtained.
[0827] In an optional embodiment, the environmental perception information includes virtual elements; and the virtual elements in the first virtual scene that are within the perception range of the object are determined.
[0828] Schematically, the object perception range is a part of the three-dimensional space range in the first virtual scene; the first virtual scene includes a large number of virtual elements, and the virtual elements are elements that constitute the virtual scene, such as virtual elements including virtual ground, virtual buildings, virtual trees, virtual characters, virtual stones, etc.
[0829] In some embodiments, when determining the object perception range, the virtual elements within the object perception range are determined from the multiple virtual elements corresponding to the first virtual scene.
[0830] In some embodiments, the element name of the virtual element is used as the first scene hot word; or, the action name of the interactive action corresponding to the virtual element is used as the first scene hot word.
[0831] Sub-step 11: converting the natural language command in voice form into text form based on multiple first scenario hot words;
[0832] Indicatively, the first scene hot words are words that are highly associated with the first virtual scene. Therefore, if you want to command the NPC through natural language commands to better cooperate with the activities of the main virtual character, you can use the first scene hot words as constraints on the basis of the first virtual scene. In this way, in the process of converting the natural language commands into text form, the text content of the command analysis results is more consistent with the first virtual scene, avoiding command analysis results that are incompatible with the first virtual scene.
[0833] In an optional embodiment, when a natural language command is analyzed by a natural language analysis model, a plurality of speech units corresponding to the natural language command are acquired through an acoustic network.
[0834] Among them, phonetic units are the basic building blocks of vocabulary pronunciation.
[0835] In some embodiments, a command feature representation corresponding to a natural language command is extracted; and the command feature representation is analyzed through an acoustic network.
[0836] Schematically, taking the command feature representation as an example, the goal of the acoustic network is to map a continuous sequence of acoustic features to a sequence of speech units, such as phonemes or segments. It learns to predict possible sequences of speech units (phonemes or segments) given acoustic features.
[0837] Optionally, the acoustic network outputs one or more possible speech unit sequences, wherein the speech unit sequence includes multiple speech units. Different speech unit sequences may contain different speech units, or may just have different orders of speech units. The multiple speech unit sequences represent the arrangement of speech units that the language network considers to be most likely when understanding the input natural language command.
[0838] The natural language analysis model includes an acoustic network, a preset dictionary, and language sub-networks corresponding to at least two virtual scenes.
[0839] Schematically, the virtual environment includes at least two virtual scenes. For example, the virtual environment is a large virtual world, which includes a virtual kitchen 1, a virtual kitchen 2, a virtual office, etc. The virtual kitchen 1, the virtual kitchen 2, the virtual office, etc. can each be regarded as a virtual scene.
[0840] In an optional embodiment, based on a first virtual scene in which the target virtual character is located, a first lan...
Claims
1. A method for controlling a virtual character, characterized in that: The method comprises: Displaying a host virtual character or NPC in a virtual environment; Acquire command information, the command information including natural semantics for controlling the NPC, and the command information corresponds to at least two semantic understanding modes; In response to the command information, controlling the NPC to perform a virtual activity according to a first semantic understanding method among the at least two semantic understanding methods; The first semantic understanding method is determined from the at least two semantic understanding methods based on the environmental perception information of the master virtual character and / or the NPC.
2. The method according to claim 1, characterized in that: The command information corresponds to at least two semantic understanding modes of the scene entity, and in response to the command information, controlling the NPC to perform a virtual activity according to a first semantic understanding mode of the at least two semantic understanding modes includes: In response to the command information, controlling the NPC to perform the virtual activity on a first scene object located within the perception range of the master virtual character and / or the NPC; There are multiple scene objects of the same type in the virtual environment, the multiple scene objects include the first scene object, and the perception range is a virtual area where the main control virtual character and / or the NPC obtains the environmental perception information.
3. The method according to claim 2, characterized in that In response to the command information, controlling the NPC to perform the virtual activity on a first scene object located within the perception range of the master virtual character and / or the NPC includes: In response to the command information, the NPC is controlled to perform the virtual activity on the first scene object located in a field of view.
4. The method according to claim 3, characterized in that The field of view is the field of view observed by the controlling virtual character and / or the NPC; and / or, the field of view is an expanded field of view obtained by using a perception skill or a perception tool.
5. The method according to claim 2, characterized in that: In response to the command information, controlling the NPC to perform the virtual activity on a first scene object located within the perception range of the master virtual character and / or the NPC includes: In response to the command information, the NPC is controlled to perform the virtual activity on the first scene object located in a first virtual terrain, where the first virtual terrain is the virtual terrain to which the main control virtual character and / or the NPC is located.
6. The method according to claim 5, characterized in that In response to the command information, controlling the NPC to perform the virtual activity on the first scene object located in the first virtual terrain includes at least one of the following: In the case where the first virtual terrain is an indoor terrain, in response to the command information, controlling the NPC to perform the virtual activity on the first scene object located in a first area, the first area being an indoor connected area including a first position where the master virtual character and / or the NPC is located; In the case where the first virtual terrain is a street terrain adjacent to a building, in response to the command information, the NPC is controlled to perform the virtual activity on the first scene object located in a second area, the second area including buildings around a second position where the master virtual character and / or the NPC is located; In the case where the first virtual terrain is an outdoor terrain away from buildings, in response to the command information, the NPC is controlled to perform the virtual activity on the first scene object located in a third area, and the third area is an area with the same virtual vegetation distribution as the third position where the main virtual character and / or the NPC is located.
7. The method according to claim 1, characterized in that The command information corresponds to at least two semantic understanding modes of word pronunciation, and in response to the command information, controlling the NPC to perform a virtual activity according to a first semantic understanding mode of the at least two semantic understanding modes includes: In response to the command information, converting the command information into command text according to the natural language pronunciation of scene objects in a perception range, wherein the perception range is a virtual area where the master virtual character and / or the NPC obtains the environmental perception information; According to the command text, the NPC is controlled to execute the virtual activity according to the first semantic understanding method.
8. The method according to claim 7, characterized in that The step of converting the command information into a command text according to the natural language pronunciation of the scene object in the perception range in response to the command information includes: When the similarity between the pronunciation of the first phoneme in the command information and the natural language pronunciation of the scene object exceeds a similarity threshold and a candidate text exists for the first phoneme, determining the text information of the first phoneme as the name text and / or introduction text of the scene object, and constructing the command text according to the text information; The candidate text of the first phoneme does not include the name text and / or introduction text of the scene object.
9. The method according to claim 1, characterized in that: The command information corresponds to at least two semantic understanding modes of the behavioral intention, and in response to the command information, controlling the NPC to perform a virtual activity according to a first semantic understanding mode of the at least two semantic understanding modes includes: In response to the command information, the NPC is controlled to perform the virtual activity according to a first behavioral intention; the first behavioral intention corresponds to a type of virtual activity permitted to be performed within a perception range, and the perception range is a virtual area in which the main virtual character and / or the NPC obtains the environmental perception information.
10. The method according to claim 9, characterized in that In response to the command information, controlling the NPC to perform the virtual activity according to the first behavioral intention includes: In response to the command information, taking the first direction in the perception range as a reference direction and according to the directional words in the command information, controlling the NPC to perform the virtual activity based on a second direction; The directional word is used to describe the angle between the second direction and the first direction, and / or to describe the second direction in a reference system constructed based on the first direction.
11. The method according to claim 1, characterized in that: In response to the command information, controlling the NPC to perform a virtual activity according to a first semantic understanding method among the at least two semantic understanding methods includes: Determining a command text in natural language according to the command information; Calling a conditional random field to perform entity prediction on the command text to obtain an entity label carried by the command text; Determining a first scene object in a perception range according to the entity label; Calling a prediction network to perform intent recognition on the command text to obtain a classification label of the command text; the prediction network is used to predict a control method of the command text for the NPC in at least one behavior dimension; According to the classification label, a control command of the NPC is determined, and based on the control command, the NPC is controlled to perform the virtual activity on the first scene object according to the first semantic understanding method.
12. The method according to claim 11, characterized in that The call prediction network performs intent recognition on the command text to obtain a classification label of the command text, including: Calling a hierarchical prediction network to perform intent recognition on the command text to obtain the classification label of the command text; The hierarchical prediction network includes at least two layers of sub-networks, and the at least two layers of sub-networks are constructed based on a tree structure of multiple classification labels.
13. The method according to claim 11, characterized in that The method further comprises: Acquire visual labels of multiple candidate scene objects in the perception range; The determining the first scene object in the perception range according to the entity tag includes: The similarities between the visual labels of the multiple candidate scene objects and the entity labels are respectively calculated to determine the first scene object with the highest similarity among the multiple candidate scene objects.
14. The method according to claim 13, characterized in that The method further comprises: Acquire the spatial position of each scene object in the virtual environment; The plurality of candidate scene objects in the perception range are screened out from the respective scene objects in the virtual environment according to the spatial position.
15. The method according to claim 14, characterized in that The obtaining of the spatial position of each scene object in the virtual environment comprises: Acquire at least one of the coordinate position, orientation information, bounding box information, and cover point information of each scene object in the virtual environment; Among them, the coordinate position is used to indicate the position of the scene object in the virtual environment, the orientation information is used to indicate the direction facing the scene object in the virtual environment, the bounding box information is used to indicate the size of the scene object in the virtual environment, and the cover point information indicates the recommended virtual character standing position when the virtual character approaches the scene object.
16. The method according to claim 13, characterized in that The method further comprises: Acquire the attribute text of the scene object in the virtual environment, wherein the attribute text is used to introduce the inherent attributes of the scene object in the virtual environment; Acquire an appearance image of the scene object in the virtual environment, wherein the appearance image is used to describe the style of the scene object; Calling a multimodal model to perform prediction on the attribute text and the appearance image of the scene object to obtain a visual label of the scene object, where the visual label is used to describe the scene object in at least one dimension; Building a spatial database of the virtual environment based on visual labels of various scene objects in the virtual environment; The obtaining of visual labels of a plurality of candidate scene objects in the perception range includes: In the spatial database, based on the perception range, visual labels of multiple candidate scene objects in the perception range are screened and obtained.
17. The method according to any one of claims 1 to 16, characterized in that: The method further comprises: Present feedback information of the NPC to the command information, wherein the feedback information is used to indicate that the NPC has received the command information and / or that the NPC has executed the virtual activity indicated by the command information.
18. A method for controlling a virtual character, characterized in that: The method comprises: Displaying a host virtual character or NPC in a virtual environment; Acquire command information, wherein the command information includes natural semantics for controlling the NPC, and the command information includes behavioral intention and scene entity; the behavioral intention is used to indicate the type of virtual activity performed by the NPC, and the scene entity is used to indicate the target entity of the virtual activity; In response to the command information, controlling the NPC to perform a virtual activity; The virtual activity is determined according to the environmental perception information of the main virtual character and / or the NPC.
19. The method according to claim 18, characterized in that The scene entity includes a first scene object; and in response to the command information, controlling the NPC to perform a virtual activity includes: In response to the command information, controlling the NPC to perform the virtual activity on the first scene object located within the perception range of the master virtual character and / or the NPC; There are multiple scene objects of the same type in the virtual environment, the multiple scene objects include the first scene object, and the perception range is a virtual area where the main control virtual character and / or the NPC obtains the environmental perception information.
20. The method according to claim 19, characterized in that In response to the command information, controlling the NPC to perform the virtual activity on the first scene object located within the perception range of the master virtual character and / or the NPC includes: In response to the command information, controlling the NPC to perform the virtual activity on the first scene object located in a field of view; Or, in response to the command information, the NPC is controlled to perform the virtual activity on the first scene object located in a first virtual terrain, where the first virtual terrain is the virtual terrain to which the main control virtual character and / or the NPC is located.
21. The method according to claim 18, characterized in that The behavioral intention includes a first behavioral intention; In response to the command information, controlling the NPC to perform a virtual activity comprises: In response to the command information, the NPC is controlled to perform the virtual activity according to the first behavioral intention; the perception range is a virtual area where the main virtual character and / or the NPC obtains the environmental perception information.
22. The method according to claim 18, characterized in that The command information also includes the natural language pronunciation of scene objects in the perception range; and in response to the command information, controlling the NPC to perform a virtual activity includes: In response to the command information, converting the command information into command text according to the natural language pronunciation of scene objects in a perception range, wherein the perception range is a virtual area where the master virtual character and / or the NPC obtains the environmental perception information; According to the command text, the NPC is controlled to perform the virtual activity.
23. The method according to claim 18, characterized in that In response to the command information, controlling the NPC to perform a virtual activity comprises: Determining a command text in natural language according to the command information; Calling a conditional random field to perform entity prediction on the command text to obtain an entity label carried by the command text; Determining a first scene object in a perception range according to the entity label; Calling a prediction network to perform intent recognition on the command text to obtain a classification label of the command text; the prediction network is used to predict a control method of the command text for the NPC in at least one behavior dimension; A control command of the NPC is determined according to the classification label, and based on the control command, the NPC is controlled to perform the virtual activity on the first scene object.
24. A method for processing command information, characterized in that: The method comprises: Acquire command information, the command information including natural semantics for controlling the NPC, and the command information corresponds to at least two semantic understanding modes; Determining a first semantic understanding mode among the at least two semantic understanding modes of the command information based on the environment perception information of the master virtual character and / or the NPC; Based on the command information, the NPC is controlled to perform a virtual activity according to a first semantic understanding method among the at least two semantic understanding methods.
25. A method for processing command information, characterized in that: The method comprises: Acquire command information, wherein the command information includes natural semantics for controlling the NPC, and the command information includes behavioral intention and scene entity; the behavioral intention is used to indicate the type of virtual activity performed by the NPC, and the scene entity is used to indicate the target entity of the virtual activity; Based on the command information, controlling the NPC to perform a virtual activity; The virtual activity is determined according to the environmental perception information of the main virtual character and / or the NPC.
26. A control device for a virtual character, characterized in that: The device comprises: A display module, used for displaying a main control virtual character or NPC in a virtual environment; An acquisition module, used for acquiring command information, wherein the command information includes natural semantics for controlling the NPC, and the command information corresponds to at least two semantic understanding modes; A control module, configured to control the NPC to perform a virtual activity in accordance with a first semantic understanding mode of the at least two semantic understanding modes in response to the command information; The first semantic understanding method is determined from the at least two semantic understanding methods based on the environmental perception information of the master virtual character and / or the NPC.
27. A command information processing device, characterized in that: The device comprises: An acquisition module, used to acquire command information, wherein the command information includes natural semantics for controlling the NPC, and the command information corresponds to at least two semantic understanding modes; A processing module, configured to determine a first semantic understanding mode among the at least two semantic understanding modes of the command information based on the environmental perception information of the master virtual character and / or the NPC; A control module is used to control the NPC to perform virtual activities according to a first semantic understanding method among the at least two semantic understanding methods based on the command information.
28. A control device for a virtual character, characterized in that: The device comprises: A display module, used for displaying a main control virtual character or NPC in a virtual environment; An acquisition module, used to acquire command information, wherein the command information includes natural semantics for controlling the NPC, and the command information includes behavioral intention and scene entity; the behavioral intention is used to indicate the type of virtual activity performed by the NPC, and the scene entity is used to indicate the target entity of the virtual activity; A control module, for controlling the NPC to perform a virtual activity in response to the command information; The virtual activity is determined according to the environmental perception information of the main virtual character and / or the NPC.
29. A command information processing device, characterized in that: The device comprises: An acquisition module, used to acquire command information, wherein the command information includes natural semantics for controlling the NPC, and the command information includes behavioral intention and scene entity; the behavioral intention is used to indicate the type of virtual activity performed by the NPC, and the scene entity is used to indicate the target entity of the virtual activity; A control module, used for controlling the NPC to perform a virtual activity based on the command information; The virtual activity is determined according to the environmental perception information of the main virtual character and / or the NPC.
30. A computer device, characterized in that: The computer device includes: a processor and a memory, wherein the memory stores at least one program; the processor is used to execute the at least one program in the memory to implement the virtual character control method as described in any one of claims 1 to 23, and / or the command information processing method as described in claim 24 or 25.
31. A computer-readable storage medium, characterized in that: The readable storage medium stores executable instructions, which are loaded and executed by a processor to implement the virtual character control method as described in any one of claims 1 to 23, and / or the command information processing method as described in claim 24 or 25.
32. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. The processor reads and executes the computer instructions from the computer-readable storage medium to implement the virtual character control method as described in any one of claims 1 to 23, and / or the command information processing method as described in claim 24 or 25.
Citation Information
Cited By
Virtual character control method and apparatus, processing method and apparatus, device, and storage medium
WO2026031857A1
