Scene object processing method and device, equipment and storage medium
The method improves the efficiency and accuracy of visual label prediction for scene objects in virtual environments by using modality images and multimodal models, allowing for natural language command capabilities for NPCs.
Patent Information
- Application Number
- CN202411097944.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-08-09
AI Technical Summary
Existing methods for marking and searching scene objects in virtual scenes are inefficient due to the large number of objects, requiring manual tagging and inefficient search processes.
A method involving obtaining modality images and classification labels, using an image translation model and classification network to predict probabilities, and training the model based on structural consistency errors, along with a multimodal model to generate visual labels for scene objects, enabling efficient batch annotation.
Enhances the accuracy and efficiency of visual label prediction for scene objects by utilizing multimodal models to describe objects in multiple dimensions, ensuring comprehensive characterization and enabling natural language command capabilities for NPCs in virtual environments.
Smart Images

Figure CN119091340A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and storage medium for processing scene objects. Background Art
[0002] A large number of scene objects are deployed in an application program including a virtual scene to provide a virtual space for the virtual activities of virtual characters.
[0003] In the related art, it is usually necessary to record the visual tags of scene objects when drawing or writing scene object material files; or after completing the deployment of the virtual scene, mark the visual tags of the scene objects one by one, and then search for scene objects in the virtual scene based on the visual tags.
[0004] However, there are a large number of scene objects in the virtual scene, and the above marking method is inefficient. Summary of the invention
[0005] The present application provides a method, device, equipment and storage medium for processing scene objects, and the technical solution is as follows:
[0006] According to one aspect of the present application, a method for processing a scene object is provided, the method comprising:
[0007] Obtaining a first modality image, a second modality image, and a classification label of the second modality image;
[0008] Inputting the first modality image and the second modality image into the image translation model to obtain a first translation image, where the first translation image is an image generated when the image translation model predicts the first modality image based on the second modality image;
[0009] Inputting the first translated image into a classification network to obtain a first prediction probability, and inputting the second modality image into the classification network to obtain a second prediction probability, wherein the first prediction probability is a probability that the image content of the first translated image belongs to the target tissue, and the second prediction probability is a probability that the image content of the second modality image belongs to the target tissue;
[0010] Based on the first predicted probability, the second predicted probability and the classification label, calculating a structural consistency error, wherein the structural consistency error is used to represent the difference between the first predicted probability and the second predicted probability and the classification label respectively;
[0011] Based on the structural consistency error, the image translation model is trained by backward error propagation to obtain a trained image translation model.
[0012] According to another aspect of the present application, a scene object processing device is provided, the device comprising:
[0013] An acquisition module, used for acquiring attribute text of a scene object in a virtual scene, wherein the attribute text is used for introducing the inherent attributes of the scene object in the virtual scene;
[0014] The acquisition module is further used to acquire an appearance image of the scene object in the virtual scene, wherein the appearance image is used to describe the style of the scene object;
[0015] A processing module is used to call a multimodal model to perform prediction on the attribute text and the appearance image of the scene object to obtain a visual label of the scene object, wherein the visual label is used to describe the visual features of the scene object in at least one dimension.
[0016] In an optional design of the present application, the multimodal model includes a visual question answering model, and the processing module is further used to:
[0017] Constructing a question sentence for the appearance image, wherein the question sentence carries the attribute text of the scene object;
[0018] The appearance image of the scene object and the question sentence are input into the visual question answering model to obtain an answer sentence, and the answer sentence is used as the visual label of the scene object.
[0019] In an optional design of the present application, the question statement includes at least two sub-statements; and the processing module is further used for:
[0020] Inputting the appearance image of the scene object and a first sentence of the at least two sub-sentences into the visual question answering model to obtain a first answer sub-sentence;
[0021] Repeat the above steps until at least two answer sub-sentences corresponding to the at least two sub-sentences are obtained, wherein the at least two sub-sentences are used to inquire about the visual features of the appearance image from multiple dimensions;
[0022] Sentence aggregation is performed on the at least two answer sub-sentences to extract the visual label of the scene object.
[0023] In an optional design of the present application, the processing module is further used for:
[0024] Acquiring expected information of the scene object, the expected information being used to indicate a description dimension of the scene object expected in the visual tag and / or an expected format of the visual tag;
[0025] constructing the question sentence of the appearance image according to the expected information and the attribute text;
[0026] Among them, the first sub-part of the question sentence is the supplementary introduction information of the scene object, carrying the attribute text of the scene object; the second sub-part of the question sentence is the answer guidance sentence for the visual question and answer model, carrying the expected information.
[0027] In an optional design of the present application, the multimodal model includes a picture description model, and the processing module is further used to:
[0028] Inputting the appearance image of the scene object into the picture description model to predict a description text of the scene object;
[0029] Sentence aggregation is performed on the description text and the attribute text to extract the visual label of the scene object.
[0030] In an optional design of the present application, the attribute text of the scene object includes at least one of the name of the scene object in the virtual scene and the size of the scene object in the virtual scene;
[0031] And / or, the appearance image of the scene object includes images obtained by observing the scene object from at least two viewing angles.
[0032] In an optional design of the present application, the processing module is further used for:
[0033] The visual label of the scene object is rewritten in a pseudo-spoken manner to obtain a matching label that conforms to a spoken expression of a natural language.
[0034] In an optional design of the present application, the processing module is further used for:
[0035] The visual label of the scene object is input into a natural language model to predict the matching label that conforms to the spoken expression of the natural language, and the natural language model carries prior knowledge of the spoken expression of the natural language.
[0036] In an optional design of the present application, the processing module is further used for:
[0037] Obtaining a first sample label pair, the first sample label pair comprising a first label before being rewritten into a pseudo-colloquial language and a second label obtained after being rewritten into a pseudo-colloquial language;
[0038] Constructing a rewriting guide sentence according to the first sample label pair and the visual label, wherein the rewriting guide sentence has natural semantics of rewriting the visual label with reference to the first sample label pair;
[0039] The rewriting guide sentence is input into a natural language model to predict the matching label that conforms to the spoken expression of natural language.
[0040] In an optional design of the present application, the acquisition module is further used for:
[0041] The spatial position of the scene object in the virtual scene is acquired, and the spatial position of the scene object is determined as auxiliary information of the visual label of the scene object.
[0042] In an optional design of the present application, the processing module is further used for:
[0043] Acquire at least one of the coordinate position, orientation information, bounding box information, and cover point information of the scene object in the virtual scene;
[0044] Among them, the coordinate position is used to indicate the position of the scene object in the virtual scene, the orientation information is used to indicate the direction facing the scene object in the virtual scene, the bounding box information is used to indicate the size of the scene object in the virtual scene, and the cover point information indicates the recommended virtual character standing position when the virtual character approaches the scene object.
[0045] In an optional design of the present application, the processing module is further used to construct a spatial information library of the virtual scene according to the visual labels of the plurality of scene objects in the virtual scene;
[0046] The acquisition module is also used to acquire natural language commands;
[0047] The processing module is further configured to filter out a first scene object corresponding to the natural language command from a plurality of scene objects in the virtual scene based on the similarity between the natural language command and the visual label in the spatial information library;
[0048] The processing module is further used to execute a virtual activity for the first scene object based on the natural language command.
[0049] In an optional design of the present application, the natural language command includes a plurality of input words; the processing module is further configured to:
[0050] Determining, one by one, a similarity score between each word in the plurality of input words and the visual label of the first scene object;
[0051] Determining a sum of a plurality of similarity scores as the similarity between the first scene object and the natural language command, wherein the plurality of similarity scores correspond one-to-one to the plurality of input words;
[0052] When the similarity between the first scene object and the natural language command exceeds the similarity between other scene objects in the spatial information library and the natural language command, the first scene object is determined as the object corresponding to the natural language command.
[0053] In an optional design of the present application, the processing module is further used to construct a spatial information library of the virtual scene according to the appearance images of multiple scene objects in the virtual scene;
[0054] The acquisition module is also used to acquire natural language commands;
[0055] The processing module is further configured to call a similarity matching network to predict the similarity between the natural language command and the appearance images of the multiple scene objects in the spatial information library, and obtain a second scene object having the highest similarity with the natural language command among the multiple scene objects;
[0056] The processing module is further used to execute a virtual activity for the second scene object based on the natural language command.
[0057] According to another aspect of the present application, a computer device is provided, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method for processing scene objects as described above.
[0058] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the method for processing scene objects as described above.
[0059] According to another aspect of the present application, a computer program product is provided, which includes computer instructions stored in a computer-readable storage medium, and a processor reads and executes the computer instructions from the computer-readable storage medium to implement the method for processing scene objects as described above.
[0060] The beneficial effects of the technical solution provided by this application include at least:
[0061] By calling the multimodal model to perform predictions on attribute text and appearance images, the visual labels of scene objects are predicted based on the description information of scene objects in the text dimension and the image dimension. In the process of predicting the visual labels of scene objects, the description information of scene objects in multiple dimensions is used to ensure that the characteristics of scene objects are fully described from the natural language semantics in the text dimension and the visual information in the image dimension, thereby ensuring the accuracy of the visual labels. Calling the multimodal model to perform predictions on attribute text and appearance images can realize batch annotation of visual labels of scene objects, thereby improving the efficiency of obtaining visual labels of scene objects. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0063] Figure 1 is a schematic diagram of controlling a non-player character in a virtual environment provided by an exemplary embodiment of the present application;
[0064] Figure 2 is a structural block diagram of a computer system provided by an exemplary embodiment of the present application;
[0065] Figure 3 is a schematic diagram of a method for processing scene objects provided by an exemplary embodiment of the present application;
[0066] Figure 4 is a flow chart of a method for processing scene objects provided by an exemplary embodiment of the present application;
[0067] Figure 5 is a flow chart of a method for processing scene objects provided by an exemplary embodiment of the present application;
[0068] Figure 6 is a schematic diagram of a visual label of a scene object provided by an exemplary embodiment of the present application;
[0069] Figure 7 is a flow chart of a method for processing scene objects provided by an exemplary embodiment of the present application;
[0070] Figure 8 is a flow chart of a method for processing scene objects provided by an exemplary embodiment of the present application;
[0071] Fig. 9 is a schematic diagram of information of scene objects provided by an exemplary embodiment of the present application;
[0072] Fig.10 is a flow chart of a method for processing scene objects provided by an exemplary embodiment of the present application;
[0073] Fig.11 is a schematic diagram of a natural language command provided by an exemplary embodiment of the present application;
[0074] Fig.12 is a schematic diagram of an appearance image of a scene object provided by an exemplary embodiment of the present application;
[0075] Fig.13 is a schematic diagram of a spatial information library of a virtual scene provided by an exemplary embodiment of the present application;
[0076] Fig.14 is a flow chart of a method for controlling a virtual character provided by an exemplary embodiment of the present application;
[0077] Fig.15 is a structural block diagram of a scene object processing device provided by an exemplary embodiment of the present application;
[0078] Fig.16 It is a structural block diagram of a terminal provided by an exemplary embodiment of the present application.
[0079] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application. DETAILED DESCRIPTION
[0080] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.
[0081] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0082] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.
[0083] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the attribute text, appearance image and other information involved in this application are all obtained with full authorization.
[0084] It should be understood that although the terms first, second, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first parameter may also be referred to as the second parameter, and similarly, the second parameter may also be referred to as the first parameter. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0085] The embodiments of the present application provide a solution for users to use natural language commands in the form of voice to command non-player characters (NPC) in a virtual environment. In this solution, in addition to the main virtual character controlled by the user, the user usually has one or more NPCs as teammates. The user can use a relatively casual and non-mechanical human conversation method to command the NPC to give the desired feedback to the scene objects in the virtual environment, thereby commanding the NPC to complete the task in collaboration with the main virtual character. Exemplarily, the NPC has autonomous behavior capabilities, and the user only commands the NPC. The user gives commands, and the NPC understands and executes the commands based on its own autonomous behavior capabilities. Exemplary reference Figure 1 , the user controls the main virtual character 10 to play the game, and the main virtual character 10 has an NPC teammate 20. There is a car 30 in the field of vision of the main virtual character 10. The user speaks a natural voice command in the form of speech: "No. 2, you go to ambush behind the car 30", and then the NPC teammate 20 will move to the back of the car 30 to ambush. It should be noted that Figure 1 There may be multiple cars in the scene shown, and the NPC teammate 20 will accurately understand that the car the user is talking about is the car 30 located within the user's field of vision. That is, the NPC teammate 20 has a relatively intelligent natural language understanding ability.
[0086] On the one hand, compared with the traditional technology of using mechanical fixed instructions to command NPC, the embodiment of the present application uses relatively random natural language commands to command NPC, which can provide users with more natural, more complex and more flexible language command capabilities.
[0087] On the other hand, the intelligence of the natural language understanding capability provided by the embodiment of the present application is reflected in that when understanding natural language commands, NPC teammates not only consider the literal information of natural language commands. NPC teammates will also combine the main virtual character and / or the NPC's own environmental perception information in the virtual environment to assist in understanding the semantics in natural language commands. That is, NPC not only considers the information of the natural language command mode, but also considers the perception information of other modes such as vision, hearing, radar, etc. to assist in the understanding and execution of natural language commands. Since natural language commands in the form of voice are spoken commands rather than written commands, there may be multiple candidate understanding methods or unclear places when considering the literal information of natural language commands alone. NPC teammates can determine a reasonable understanding method from multiple candidate understanding methods, or eliminate doubts about unclear places, by combining the main virtual character and / or the NPC's own environmental perception information in the virtual environment, thereby achieving a more intelligent natural language understanding capability. The above-mentioned environmental perception information includes, but is not limited to, at least one of the following:
[0088] Visual information perceived within the field of vision of the controlling virtual character;
[0089] Auditory information perceived within the hearing range of the controlling virtual character;
[0090] The information perceived by the master virtual character’s perception skills or perception tools;
[0091] Visual information perceived within the NPC's field of view;
[0092] Auditory information perceived within the NPC's hearing range;
[0093] Information perceived by the NPC's perception skills or perception tools.
[0094] Combined with reference Figure 1 The program includes at least one of the following five stages:
[0095] Phase 1: Preprocessing of spatial data;
[0096] The server will pre-process the scene objects in the virtual scene and construct the spatial data of the virtual scene.
[0097] The spatial data of the virtual scene includes the visual labels of the scene objects and the spatial positions of the scene objects. In some examples, the spatial data also includes the appearance images of the scene objects. The scene objects are any objects that appear in the virtual scene, such as Figure 1 Cars, walls, boxes, etc.
[0098] First, the pre-acquisition process of the visual labels of the scene objects is introduced. The attribute information of the scene objects in the virtual scene is obtained, such as at least one of the name and size of the scene objects; and the appearance image of the scene objects is obtained. The appearance image includes an image obtained by observing the scene objects from at least two perspectives. Observing the scene objects from multiple perspectives can ensure that the appearance image carries comprehensive appearance information of the scene objects.
[0099] The server calls the multimodal model to predict the attribute information and appearance image of the scene object. Based on the multimodal model's ability to process text dimension and image dimension information, it extracts hidden features from the attribute information of the text dimension and the appearance image of the image dimension, and decodes the hidden features to predict the visual label of the scene object. The visual label is used to describe the scene object in at least one dimension in a textual manner; for example, the scene object is described in dimensions such as material, transparency, color, and shape. The natural language model is called to optimize the visual label to obtain the visual label of the scene object; the natural language model has the ability to generate text, and rewrites the visual label input to the natural language model in a manner that conforms to the spoken expression of natural language, thereby optimizing the visual label and obtaining the optimized visual label of the scene object. The optimized visual label conforms to the spoken expression of natural language, and describes the scene object in at least one dimension in a manner close to the spoken expression.
[0100] For example, when describing a virtual bed in a virtual scene in spoken language, people usually pay attention to the color, material, and placement of the virtual bed, but usually neglect to pay attention to the craftsmanship used for the headboard and the side panels of the bed, such as carving, embossing, etc. The purpose of performing pseudo-spoken rewriting on the visual labels of scene objects is to obtain matching labels that are closer to spoken expressions; for example, using multiple words with the same semantics to enrich the labels of dimensions such as color, material, and placement that are paid attention to in spoken expressions, and deleting the labels of the surface feature dimensions of the headboard and the side panels of the bed that are neglected in spoken expressions.
[0101] Next, the spatial position of the scene object is introduced. The spatial position includes at least one of the following of the scene object: coordinate position (Location), such as the coordinate information of the center point or preset point of the scene object in the virtual scene; orientation (Rotation), such as the direction the front of the scene object faces in the virtual scene; bounding box (Bounding Box), used to indicate the size of the scene object in the virtual scene; cover points, used to indicate the recommended virtual character position points when the virtual character approaches the scene object, so that the scene object can cover the virtual character.
[0102] Next, the process of pre-acquiring the appearance image of the scene object is introduced. The appearance image is acquired in the process of predicting the visual label of the scene object, and the appearance image includes images obtained by observing the scene object from at least two perspectives.
[0103] Optionally, the spatial data of each scene object in the entire virtual environment may be pre-acquired and organized into a data set for use in a subsequent query process.
[0104] Phase 2: Speech recognition and intent recognition;
[0105] When the user controls the main virtual character, the terminal receives the natural language command input by the user in the form of voice; the server performs voice recognition and intent recognition on the natural language command to analyze the user's input instruction;
[0106] Speech recognition is to perform text conversion on the natural language instructions input by the user through voice, obtain the instruction text corresponding to the natural language instructions, and convert the speech information into text information. Call the encoding network to perform feature encoding on the instruction text to obtain the feature representation of the natural language instructions in the latent space; call the text segmentation network to perform segmentation on the feature representation to obtain multiple clauses in the instruction text, and then perform intent recognition on each of the multiple clauses. In the case that the natural language instructions input by the user through voice are long sentences, the segmentation of the instruction text corresponding to the natural language instructions can be achieved to meet the needs of analyzing the user's input instructions in the long sentence scenario.
[0107] The following introduces the intent recognition of each clause; the user's input instruction is obtained through intent recognition, and the user's input instruction includes an entity, a semantic type, a subject type, and an intent type in at least one clause.
[0108] On the one hand, the Conditional Random Field (CRF) method is used to identify entities in clauses, such as identifying entities in clauses as scene objects such as buildings, virtual items, and virtual vegetation in virtual scenes. The entities in the clauses are used to indicate the scene objects targeted by the virtual activities. For example, if the entity in the clause is a virtual building, and the clause carries the semantics of instructing the NPC to launch a virtual attack, then the virtual attack is executed against the virtual building, that is, a virtual attack is launched against the virtual building. On the other hand, the prediction network is called to predict the semantic type, subject type, and intention type in the clause respectively. Among them, the semantic type includes whether to instruct the NPC to launch a virtual attack; the subject type is used to indicate that the subject of the clause is one or more NPCs, or a master virtual character controlled by the user; the intention type is the expected virtual activity indicated by the clause, such as moving, launching a virtual attack, etc.
[0109] Phase 3: Query of spatial data;
[0110] For the entity in the input command, it is necessary to determine the spatial position of the entity in the virtual environment. In the case where there are multiple candidate entities in the virtual environment that match the entity in the input command, the target entity that matches the input command can be uniquely determined from the multiple candidate entities in combination with the environmental perception information of the main virtual character and / or NPC.
[0111] For example, the environment perception information is the field of view information of the main virtual character, the spatial data in the virtual environment is used as the query range, and the real-time position and orientation of the main virtual character is used as the query reference condition to query the spatial position of the target entity corresponding to the input instruction in the virtual environment, so as to subsequently control the NPC to move to the vicinity of the target entity, or to perform the behavior indicated by the input instruction on the target entity.
[0112] However, since the main virtual character and / or NPC may have multiple environmental perception information, multiple environmental perception information can be integrated to assist the determination process of the above-mentioned target entity, or multiple environmental perception information can be used according to priority to assist the determination process of the above-mentioned target entity, and there is no limitation on this.
[0113] Stage 4: NPC voice feedback;
[0114] NPC will give voice feedback to the user. The types of voice feedback include: at least one of instant feedback, command execution feedback, and dynamic feedback. Among them, instant feedback means that after the user issues a natural language command to the NPC, the NPC immediately gives voice feedback on the natural language command, such as "received", "start execution", "OK", etc.; execution command feedback means that after the NPC executes the control command indicated by the intention of the natural language command, the NPC gives voice feedback on the execution result of the control command, such as "successful execution", "failed to execute successfully", etc.; dynamic feedback means that when the user does not issue a natural language command to the NPC, the NPC gives voice feedback based on real-time environmental perception.
[0115] The above-mentioned voice feedback is obtained by converting text content into speech. The text content is inferred by a large language model combined with the above-mentioned spatial data and dynamic environmental perception information. Using a large language model to dynamically generate the text content of voice feedback can, on the one hand, break through the monotonous, mechanical, and easily repetitive voice feedback of fixed templated voice feedback; on the other hand, it does not need to save too many pre-made template voices in advance, which can avoid the problem of too large a total data volume on the client.
[0116] Stage 5: NPC behavior control;
[0117] Taking the user's input as input, and the spatial data of the virtual scene and the runtime information corresponding to the master virtual character as reference, the AI model is used to determine the target entity in the virtual space and the control instructions for the NPC;
[0118] Control instructions are instructions that can be used for NPC behavior tree execution; the subsequent actions that the control instruction indicates the NPC needs to perform can be a single action or a sequence of actions. In the case where the control instruction is a multi-action sequence instruction, the multi-action sequence instruction is used to instruct the NPC to perform a sequence of actions. The data structure storing the multi-action sequence instruction is in the form of a list, and the instructions to be executed in the sequence instruction are cached. When the NPC's behavior tree executes the current instruction, the cache is called back, and the instructions to be executed in the cache are continued to be sent for execution.
[0119] Hot word update function:
[0120] For phase 2, this solution also introduces a hot word system to improve the accuracy of speech recognition.
[0121] The hot word system is used to provide scene hot words related to the scenes involved in the natural language instructions during the speech recognition process, so that when a natural language instruction input by voice is received, the hot word library can be preferentially searched for words as the instruction analysis result based on the currently running virtual scene, so that the matching degree between the instruction analysis result and the virtual scene is higher.
[0122] The virtual scene includes at least one of a plurality of virtual scenes such as a virtual battle scene, a virtual transaction scene, a virtual office scene, and a virtual kitchen scene. Frequently used words in a plurality of virtual scenes, specific virtual elements existing in a virtual scene, etc. (such as virtual buildings and virtual characters that only exist in a specific scene type, etc.) are pre-analyzed as scene hot words, and a scene hot word library consisting of a plurality of scene hot words is obtained, and the plurality of scene hot words correspond to virtual scenes respectively. In addition, different scene language models are pre-trained for different virtual scenes, such as a scene language model in a battle scene that pays more attention to words related to battle, and a scene language model in a transaction scene that pays more attention to words related to transaction; the scene language model can perform a more targeted analysis of natural language commands received in a virtual scene.
[0123] For example: when a user controls a non-player character, a natural language command is received. Based on the game state, object position, and current task completion status, the first virtual scene in which the target virtual character (at least one of the main virtual character and the non-player character) is currently located is determined to be a virtual battle scene, and a battle scene language model corresponding to the virtual battle scene is obtained. In addition, based on the field of view of the target virtual character, the first scene hot words in the virtual battle scene are obtained, including: "truck", "virtual grass", "virtual house", "enter", "open truck", "defense", "desert", "attack", "oak tree", "stable", "firearms", etc. Under the constraints of the first scene hot words, the natural language command is decoded and processed through a pre-trained natural language analysis model (including an acoustic network, a language network, a preset dictionary, etc.), and the instruction analysis result is output to accurately control the non-player character. The instruction analysis result is illustrated as follows.
[0124] (1) The command analysis result is "No. 1, you go to the front Defense The command analysis result can be used to specifically control No. 1 to move forward and make a defensive posture, preventing the enemy virtual character from attacking the main control virtual character first; if there is no constraint of the hot words in the first scene, the natural language command is easily recognized as "No. 1, go to the front and make a sound", thereby affecting the player's virtual combat plan.
[0125] (2) The command analysis result is "No. 2, you go desert The command analysis result can be used to specifically control No. 2 to move to the desert to check whether there are virtual elements such as enemy virtual characters or virtual treasure chests. If there is no constraint of the hot words in the first scene, it is easy to recognize the natural language command as "No. 2, you go Mountains "Explore it", which causes No. 2 to move to the wrong position, and the human-computer interaction efficiency is low.
[0126] (3) The command analysis result is "No. 1, No. 2, give me attack ", the command analysis result can be used to specifically control No. 1 and No. 2 to carry out virtual attacks on nearby attackable virtual characters; if there is no constraint of the hot words in the first scene, it is easy to recognize the natural language command as "No. 1 No. 2, give me supply ", causing No. 1 and No. 2 to make incorrect behaviors that do not meet the players' expectations, which can neither protect the main virtual character nor easily allow the enemy virtual character to win the virtual game.
[0127] (4) The command analysis result is “No. 1, go find a nearby oak tree”. This command analysis result can be used to specifically control No. 1 to find an oak tree in the nearby area so that he can complete the game task or find an oak tree that can avoid attacks. If there is no constraint of the hot words in the first scene, it is easy to recognize the natural language command as “No. 1, go find a nearby oak tree”. Project Book ", causing No. 1 to search nearby for virtual elements that do not meet the player's expectations and fail to meet the player's virtual combat needs.
[0128] (5) The command analysis result is "No. 1, No. 2, go over there stable OK", the command analysis result can be used to specifically control No. 1 and No. 2 to search for nearby stables and move to the location of the stables; if there is no constraint of the hot words in the first scene, it is easy to recognize the natural language command as "No. 1, No. 2, go over there , right away OK", which makes No. 1 and No. 2 mistakenly believe that the main virtual character currently hopes to complete the virtual game by itself, and therefore cannot provide good game assistance to the main virtual character.
[0129] (6) The command analysis result is "No. 2, pick up the one in front firearms ", the command analysis result can be used to specifically control No. 2 to search for guns from the front and pick up the guns; if there is no constraint of the hot words in the first scene, it is easy to recognize the natural language command as "No. 2, pick up the gun in front Crab ", which makes it impossible for No. 2 to accurately find the virtual element that the main virtual character in front wants to pick up. It is possible that the player will be prompted with "Cannot find the crab", or the player may pick up the wrong virtual element "crab", interfering with the player's virtual combat process.
[0130] Ambient sound function:
[0131] In addition, the solution also provides a spatial audio enhancement solution for the ambient sound effects of the entire virtual scene.
[0132] The audio played by the terminal includes environmental audio and NPC audio. Environmental audio is audio generated based on the scene characteristics of the virtual scene where the main virtual object is currently located, so that the user can feel immersive; NPC audio is audio generated based on the characteristics of the NPC, so that the user can intuitively feel the character's emotions and physical state through the sound heard.
[0133] Environmental audio: Identify the scene elements in the virtual scene where the master virtual object is located, generate appropriate element sound effects in real time based on the scene elements / select appropriate element sound effects from the sound effect library, and synthesize the element sound effects to obtain environmental audio. For example, if the virtual scene is a forest at night, the scene elements include trees, owls, insects, etc., and the environmental audio includes the rustling of leaves, the calls of owls, the chirping of insects, etc.
[0134] NPC audio: Determine the NPCs included in the virtual scene, and generate corresponding character voices based on the NPC's character type, current emotion, and current behavior. For example, if the NPC is a middle-aged man running, the generated character voice is a deep male voice with a running panting sound effect.
[0135] When the terminal plays the ambient audio and / or NPC audio to the user, the terminal performs audio enhancement processing on the ambient audio and / or NPC audio to improve the realism of the audio.
[0136] Figure 2 The structure block diagram of a computer system provided by an exemplary embodiment of the present application is shown. The computer system 100 includes: a first terminal 110, a server 120, and a second terminal 130.
[0137] The first terminal 110 is installed and runs a client 111 that supports a virtual environment, and the client 111 can be a multiplayer online battle program. When the first terminal runs the client 111, the user interface of the client 111 is displayed on the screen of the first terminal 110. The client 111 can be any one of a battle royale shooting game, a virtual reality (VR) application, an augmented reality (AR) program, a three-dimensional map program, a virtual reality game, an augmented reality game, a casual game, a party gathering game, a first-person shooting game (First-Person Shooting Game, FPS), a third-person shooting game (Third-Person Shooting Game, TPS), a multiplayer online tactical competitive game (Multiplayer Online Battle Arena Games, MOBA), and a strategy game (Simulation Game, SLG). In this embodiment, the client 111 is an FPS game for example. The first terminal 110 is a terminal used by the first user 112. The first user 112 uses the first terminal 110 to control a first virtual object in a virtual environment to perform activities. The first virtual object may be referred to as a virtual object of the first user 112. The activities of the first virtual object include, but are not limited to, at least one of moving, jumping, teleporting, releasing skills, using props, adjusting body posture, crawling, walking, running, riding, flying, jumping, driving, picking up, shooting, attacking, and throwing. Schematically, the first virtual object is a first virtual object, such as a simulated human character or an animated human character.
[0138] The second terminal 130 is installed and runs a client 131 that supports a virtual environment, and the client 131 can be a multiplayer online battle program. When the second terminal 130 runs the client 131, the user interface of the client 131 is displayed on the screen of the second terminal 130. The client can be any one of a battle royale shooting game, a VR application, an AR program, a three-dimensional map program, a virtual reality game, an augmented reality game, an FPS, a TPS, a MOBA, and a SLG. In this embodiment, the client is an MOBA game as an example. The second terminal 130 is a terminal used by a second user 132. The second user 132 uses the second terminal 130 to control a second virtual object located in a virtual environment to perform activities. The second virtual object can be referred to as a virtual object of the second user 132. Schematically, the second virtual object is a second virtual object, such as a simulated character or an anime character.
[0139] Optionally, the first virtual object and the second virtual object are in the same virtual environment. Optionally, the first virtual object and the second virtual object may belong to the same camp, the same team, the same organization, have a friend relationship, or have temporary communication permissions. Optionally, the first virtual object and the second virtual object may belong to different camps, different teams, different organizations, or have a hostile relationship.
[0140] Optionally, the client installed on the first terminal 110 and the second terminal 130 is the same, or the client installed on the two terminals is the same type of client on different operating system platforms (Android or IOS). The first terminal 110 may refer to one of a plurality of terminals, and the second terminal 130 may refer to another of a plurality of terminals. This embodiment is only illustrated by taking the first terminal 110 and the second terminal 130 as examples. The device types of the first terminal 110 and the second terminal 130 are the same or different, and the device type includes at least one of: a smart phone, a tablet computer, an e-book reader, an MP3 player, an MP4 player, a laptop portable computer, and a desktop computer.
[0141] Figure 2 Only two terminals are shown, but in different embodiments, there are multiple other terminals 140 that can access the server 120. Optionally, there are one or more terminals 140 corresponding to the developer, and a development and editing platform for the client supporting the virtual environment is installed on the terminal 140. The developer can edit and update the client on the terminal 140, and transmit the updated client installation package to the server 120 through a wired or wireless network. The first terminal 110 and the second terminal 130 can download the client installation package from the server 120 to update the client.
[0142] The first terminal 110 , the second terminal 130 , and other terminals 140 are connected to the server 120 via a wireless network or a wired network.
[0143] The server 120 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. The server 120 is used to provide background services for the client supporting the three-dimensional virtual environment. Optionally, the server 120 undertakes the main computing work, and the terminal undertakes the secondary computing work; or, the server 120 undertakes the secondary computing work, and the terminal undertakes the main computing work; or, the server 120 and the terminal adopt a distributed computing architecture for collaborative computing.
[0144] In an illustrative example, the server 120 includes a processor 122, a user account database 123, a battle service module 124, and a user-oriented input / output interface (I / O interface) 125. The processor 122 is used to load instructions stored in the server 120 and process data in the user account database 123 and the battle service module 124; the user account database 123 is used to store data of user accounts used by the first terminal 110, the second terminal 130, and other terminals 140, such as the user account's avatar, the user account's nickname, the user account's combat power index, and the service area where the user account is located; the battle service module 124 is used to provide multiple battle rooms for users to fight, such as 1V1 battle, 3V3 battle, 5V5 battle, etc.; the user-oriented I / O interface 125 is used to establish communication and exchange data with the first terminal 110 and / or the second terminal 130 through a wireless network or a wired network.
[0145] The method provided in this application can be applied to at least one of the following scenarios, but is not limited to: virtual reality applications, three-dimensional map programs, casual games, party games, first-person shooting games (First-Person Shooting Game, FPS), third-person shooting games (Third-Person Shooting Game, TPS), multiplayer online tactical competitive games (Multiplayer Online Battle Arena Games, MOBA), multiplayer gunfight survival games, etc. The following embodiments are illustrated by the application in the game. The method provided in this application can be completely executed by the client, such as a stand-alone game; it can also be completely executed by the server, such as in a cloud game, the server performs all calculations, and the client is only responsible for displaying and collecting the user's operation behavior; the client and the server can also collaborate, with the client responsible for part and the server responsible for part.
[0146] Figure 3A schematic diagram of a method for processing scene objects provided by an exemplary embodiment of the present application is shown.
[0147] Question sentence 302 is: "Below is a rendering of a 3D model named A, with a size of B; please understand what the 3D model is through the model name, image, and size information, and summarize the overall features, and finally output the visual label in Json format.". The first half is the first sub-part, which is the supplementary introduction information of the scene object, divided by the semicolon ";" in question sentence 302. The supplementary introduction information of the scene object includes the attribute text of the acquired scene object, such as the name of the scene object in the virtual scene is A, and the size of the scene object in the virtual scene is B. The second half is the second sub-part, which is the answer guidance sentence for the visual question answering model.
[0148] The appearance image 304 of the scene object includes images obtained by observing the scene object from nine perspectives. The question sentence 302 and the appearance image 304 of the scene object are input into the multimodal model 310 to obtain the visual label 315 of the scene object. Exemplarily, the visual label 315 of the scene object includes: "Type description: wooden double bed; material: [wood]; transparency: opaque; color: [brown]; surface feature: [carving]; shape: [rectangular]".
[0149] The visual label 315 of the scene object is input into the natural language model 320, and the matching label 325 that conforms to the spoken expression of the natural language is predicted; the natural language model 320 is implemented as a large language model (LLM). The natural language model 320 carries the prior knowledge of the spoken expression of the natural language, and performs a pseudo-spoken rewrite on the visual label 315 of the scene object based on the prior knowledge of the spoken expression of the natural language, so as to obtain a rewritten label (also called a matching label 325) that is closer to the spoken expression; the rewritten labels include: bed, brown, wooden, rectangular.
[0150] The spatial position 327 of the scene object in the virtual scene is obtained. The spatial position 327 includes the coordinate position, orientation information, bounding box information, and cover point information of the scene object in the virtual scene; it is used to indicate the position, size, and other information of the scene object in the virtual scene. The matching label 325 and the spatial position 327 are used together as the label of the scene object in the virtual scene to facilitate searching for the scene object in the virtual scene. Exemplary, Figure 3 The embodiment shown is Figure 1 Stage 1 pointed out in: A way to implement preprocessing of spatial data.
[0151] Figure 4A flowchart of a method for processing scene objects provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device. The method includes:
[0152] Step 510: Obtaining attribute text of scene objects in the virtual scene;
[0153] Exemplarily, attribute text is used to introduce the inherent attributes of scene objects in a virtual scene; on the one hand, attribute text realizes the description of scene objects in text mode, and while describing scene objects, it provides semantic information for the prediction of visual labels of scene objects. On the other hand, attribute text is a description of scene objects in a virtual scene. In the face of a large number of scene objects in a virtual scene and the existence of reused object models, it can more accurately describe the inherent attributes of scene objects in the virtual scene. For example, in a virtual environment, in order to avoid redesigning the object model of a scene object, the object model of a virtual table is subjected to at least one of the transformation operations such as scaling, stretching, and rotation, and the transformed object model is deployed in the virtual environment as a virtual bench; in the face of the above-mentioned model reuse, attribute text can describe the inherent characteristics of scene objects in the virtual scene.
[0154] In an optional implementation, the attribute text includes at least one of the name of the scene object in the virtual scene and the size of the scene object in the virtual scene; illustratively, the size of the scene object in the virtual scene is used to indicate the virtual space occupied by the scene object in the virtual scene, avoiding the impact on the description of the scene object when the size of the object model is changed when the object model is reused. The name in the virtual scene can reflect the purpose of the scene object in the virtual scene, avoiding the impact on the description of the scene object when two different scene objects are deployed according to the same object model when the object model is reused.
[0155] Step 520: Acquire an appearance image of a scene object in the virtual scene;
[0156] Exemplarily, appearance images are used to describe the style of scene objects. Based on the richness of natural language, players can describe scene objects in virtual scenes from multiple semantic perspectives. When constructing visual labels for scene objects, it is necessary to obtain comprehensive description information of the scene objects. Appearance images carry the color, texture, shape and other appearance styles of scene objects in the picture mode, as well as the relative positional relationships between various sub-parts, and can comprehensively describe scene objects from the picture mode (or visual mode).
[0157] In an optional implementation, the appearance image of the scene object includes an image obtained by observing the scene object from at least two viewing angles. Observing the scene object from at least two viewing angles in the appearance image can avoid the problem of occlusion of the spatial structure of the scene object and the inability to present all appearance styles of the scene object in a single viewing angle.
[0158] It should be noted that the appearance image can be obtained by the scene object in the virtual scene, such as capturing the appearance image of the scene object in the scene object; the appearance image can also be obtained outside the virtual scene, such as capturing the appearance image of the scene object in the development interface during the development and design process of the scene object.
[0159] Step 530: calling the multimodal model to perform prediction on the attribute text and appearance image of the scene object to obtain the visual label of the scene object;
[0160] Exemplarily, the multimodal model has the ability to perform model prediction on text information and image information of different modalities. In this embodiment, the input parameters of the multimodal model are the attribute text and appearance image of the scene object; the multimodal model predicts the visual label of the scene object from the two modalities of text modality and image modality, and the visual label is used to describe the visual features of the scene object in at least one dimension.
[0161] Exemplarily, the multimodal model includes an artificial neural network (ANN), and the network of the artificial neural network includes but is not limited to at least one of the following: Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Temporal Convolutional Networks (TCN), Long Short-Term Memory (LSTM), Multilayer Perceptron (MLP), Support Vector Machine (SVM). The present application does not limit the network structure of the multimodal network. Exemplarily, the multimodal model has the ability to extract the visual features of scene objects from the input information of text modality and picture modality to obtain the visual labels of the scene objects. Exemplarily, the multimodal model can be a classification model that assigns scene objects to preset type labels; it can also be a prediction model that predicts labels for scene objects that can describe the visual characteristics of scene objects.
[0162] In summary, the method provided in this embodiment predicts the visual labels of scene objects based on the description information of scene objects in the text dimension and the image dimension by calling a multimodal model to perform predictions on attribute text and appearance images; in the process of predicting the visual labels of scene objects, the description information of scene objects in multiple dimensions is used to ensure that the characteristics of scene objects are fully described from the natural language semantics in the text dimension and the visual information in the image dimension, thereby ensuring the accuracy of the visual labels; calling a multimodal model to perform predictions on attribute text and appearance images can realize batch annotation of visual labels of scene objects, thereby improving the efficiency of obtaining visual labels of scene objects.
[0163] Next, the multimodal model is further introduced;
[0164] Figure 5 A flowchart of a method for processing scene objects provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device. Figure 4 In the illustrated embodiment, step 530 may be implemented as steps 532 and 534:
[0165] Step 532: constructing a question statement for the appearance image;
[0166] Exemplarily, the question statement carries attribute text of the scene objects; on the one hand, the question statement is used to guide the visual question answering model to convert the appearance image into the visual labels of the scene objects; on the other hand, while guiding the conversion of the visual labels of the scene objects, the question statement provides supplementary information of the scene objects in text form.
[0167] In an optional implementation, step 532 in this embodiment may be implemented as the following sub-steps:
[0168] Get the expected information of scene objects;
[0169] Exemplarily, the expected information is used to indicate the description dimension of the scene object expected in the visual tag, and / or the expected format of the visual tag; in one example, the expected information is used to indicate that the description dimension of the scene object expected in the visual tag includes but is not limited to at least one of the type description, material, transparency, color, surface features, and shape of the scene object. In another example, the expected information is used to indicate that the expected format of the visual tag is at least one of the following: Comma Separated Values (CSV), JavaScript Object Notation (JSON), eXtensible Markup Language (XML), and Excel.
[0170] Constructing a query statement of the appearance image based on the expected information and attribute text;
[0171] Exemplarily, the first subpart of the question sentence is supplementary introduction information of the scene object, carrying attribute text of the scene object; the second subpart of the question sentence is an answer guidance sentence for the visual question answering model, carrying expected information.
[0172] Figure 6 A schematic diagram of a visual label of a scene object provided by an exemplary embodiment of the present application is shown. In one example, the question statement 402 is: "The following is a rendering of a three-dimensional model named A, with a size of B; please understand what the three-dimensional model is through the model name, image, and size information, and summarize the overall features, and finally output the visual label in Json format.". Among them, the first half is the first sub-part divided by the semicolon ";" in the question statement 402, which is the supplementary introduction information of the scene object. The supplementary introduction information of the scene object includes the attribute text of the scene object obtained from the data set 400, such as the name of the scene object in the virtual scene is A, and the size of the scene object in the virtual scene is B.
[0173] The second half is the second sub-part, which is the answer guide statement for the visual question answering model. Furthermore, the question statement includes a preset sentence template, and the attribute text of the scene object is filled into the preset sentence template to obtain the question statement. Among them, the filling position of the attribute text of the scene object is the position of the word "A" and the word "B" in the question sentence; wherein, the name of the scene object in the attribute text in the virtual scene is marked as the asset name (asset name), and the size of the scene object in the attribute text in the virtual scene is recorded as the asset size (asset size); fill it into the preset sentence template to obtain the question statement.
[0174] Step 534: input the appearance image of the scene object and the question sentence into the visual question answering model to obtain an answer sentence, and use the answer sentence as a visual label of the scene object;
[0175] Exemplarily, the appearance image of the scene object and the question sentence are input into the visual question answering model, and the question sentence prompts the guidance information of the visual label generated by the visual question answering model in text form, and at the same time provides supplementary information of the scene object to further describe the scene object. Figure 6, the appearance image 404 of the scene object includes images obtained by observing the scene object from nine perspectives. The question statement 402 and the appearance image 404 of the scene object are input into the visual question answering model 410 to obtain the visual label 415 of the scene object. Exemplarily, the visual label 415 of the scene object includes: "Type description: wooden double bed; material: [wood]; transparency: opaque; color: [brown]; surface feature: [carving]; shape: [rectangular]".
[0176] Further, the visual label 415 of the scene object is used to supplement the data set 400; after the visual label 415 is supplemented, the data set 400 includes the appearance image 404 of the scene object, the visual label 415, and the spatial position 420 of the scene object in the virtual scene, and the spatial position 420 includes the coordinate position, orientation information, bounding box information, and cover point information, which will be introduced in a separate embodiment below. In various embodiments of the present application, the data set 400 is also referred to as a spatial information library of the virtual scene.
[0177] In this embodiment, the visual question answering model (Visual Question Answering, VQA) can be implemented as at least one of the following: Large Language Model with Vision Ability (LLAVA), Mini Generative Pre-trained Transformer 4 (MiniGPT4), Cognitive Visual Language Model (CogVLM), Generative Pre-trained Transformer 4with Vision (GPT4V), and Gemini model (Gemini).
[0178] In an optional implementation, step 534 in this embodiment may be implemented as the following sub-steps:
[0179] Inputting the appearance image of the scene object and the first sentence of at least two sub-sentences into the visual question answering model to obtain a first answer sub-sentence;
[0180] Exemplarily, the visual question answering model takes the appearance image and the first sentence as input parameters, and the obtained first answer sub-sentence is the answer sentence under the guidance of the first sentence. Exemplarily, at least two sub-sentences are used to inquire about the visual features of the appearance image from multiple dimensions; the first answer sub-sentence is used to answer the visual features of the appearance image inquired in one dimension corresponding to the first sentence.
[0181] Repeat the above steps until at least two answer sub-sentences corresponding to at least two sub-sentences are obtained;
[0182] Each of the at least two sub-sentences and the appearance image is input into the visual question answering model to obtain a corresponding answer sub-sentence, that is, at least two answer sub-sentences corresponding to the at least two sub-sentences are obtained.
[0183] Perform sentence aggregation on at least two answer sub-sentences to extract visual labels of scene objects;
[0184] Exemplarily, sentence aggregation is performed on at least two answer sub-sentences, and part or all of the content is extracted from the at least two answer sub-sentences to obtain visual labels of scene objects. Exemplarily, sentence aggregation can be performed on each input answer sub-sentence based on an artificial neural network; or words can be intercepted at a preset position of the answer sub-sentence to achieve sentence aggregation.
[0185] In summary, the method provided in this embodiment predicts the attribute text and appearance image based on the description information of the scene objects in the text dimension and the image dimension by calling the visual question answering model to perform predictions on the attribute text and the appearance image; the question statement for the appearance image includes the attribute text, which provides supplementary information for the scene object while guiding the visual question answering model to generate an answer statement; in the process of predicting the visual label of the scene object, the description information of the scene object in multiple dimensions is used, which can ensure that the characteristics of the scene object are fully described from the natural language semantics in the text dimension and the visual information in the image dimension, thereby ensuring the accuracy of the visual label and improving the efficiency of obtaining the visual label of the scene object.
[0186] Figure 7 A flowchart of a method for processing scene objects provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device. Figure 4 In the illustrated embodiment, step 530 may be implemented as steps 536 and 538:
[0187] Step 536: input the appearance image of the scene object into the picture description model to predict the description text of the scene object;
[0188] Exemplarily, the image description model (Image Caption) has the ability to describe the appearance image of the input model in text form. By calling the image description model, the appearance image of the graphic modality is converted into a description text of the text modality. Exemplarily, the image description model includes but is not limited to at least one of the following: an image captioning model (Show and Tell), a display attention and narration model (Show, Attend and Tell, SAAT), a bootstrap latent integer picture model (Bootstrap Latent Integer Picture Models, BLIP), and a transformer-based image captioning model (Transformer-based Image Captioning Models, TICM).
[0189] Step 538: Perform sentence aggregation on the description text and the attribute text to extract visual labels of scene objects;
[0190] Exemplarily, part or all of the content in the description text and attribute text is determined as the visual label of the scene object obtained in the image area; exemplary, sentence aggregation is performed on the description text and attribute text, which can be based on an artificial neural network to perform sentence aggregation on the input description text and attribute text; or words can be intercepted at preset positions in the description text and attribute text to achieve sentence aggregation.
[0191] In summary, the method provided in this embodiment, by calling the image description model to perform prediction on the appearance image, realizes the description text of the scene objects in the text dimension and the image dimension, and converts it into the description text in the text dimension; further aggregates the description text and the attribute text predicted by the image description model, and obtains the description information from multiple dimensions of the description text from the image dimension and the attribute text from the text dimension; it can ensure that the characteristics of the scene objects are fully described from the natural language semantics in the text dimension and the visual information in the image dimension, thereby ensuring the accuracy of the visual labels and improving the efficiency of obtaining the visual labels of the scene objects.
[0192] In an optional implementation, a pseudo-colloquial rewriting is performed on the visual label, and the pseudo-colloquial rewriting is introduced as follows:
[0193] Figure 8 A flowchart of a method for processing scene objects provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device. Figure 4 Based on the embodiment shown, step 540 and step 545 are also included:
[0194] Step 540: Performing a pseudo-spoken rewrite on the visual labels of the scene objects to obtain matching labels that conform to spoken expressions of natural language;
[0195] Exemplarily, as described above, visual labels are used to describe the visual features of scene objects in at least one dimension, and the appearance images of scene objects present rich visual features of scene objects. In the spoken expression of natural language, the description of scene objects cannot cover the visual features of scene objects in all dimensions. For example, when describing a virtual bed in a virtual scene in spoken expression, people usually pay attention to the color, material, placement, etc. of the virtual bed, but usually neglect to pay attention to the craftsmanship used for the headboard and the side panels of the bed, such as carving, embossing, etc. The purpose of performing a pseudo-colloquial rewrite on the visual label of the scene object is to obtain a matching label that is closer to the spoken expression; it can be understood that performing a pseudo-colloquial rewrite can be to delete part of the content in the visual label, or to change the visual label to a label with the same semantics but different textual expression.
[0196] In an optional implementation, the pseudo-colloquial rewriting is performed by calling a natural language model; accordingly, step 540 in this embodiment can be implemented as follows:
[0197] Input the visual labels of scene objects into the natural language model to predict matching labels that match the spoken expressions of natural language;
[0198] In this implementation, the pseudo-colloquial rewriting of the visual label is implemented based on calling a natural language model. Exemplarily, the natural language model carries prior knowledge of spoken expressions of natural language. For example, the natural language model is implemented as a large language model (LLM).
[0199] Furthermore, the execution of calling the natural language model to perform the pseudo-spoken rewriting includes:
[0200] Obtain a first sample label pair;
[0201] Constructing a rewriting guidance sentence according to the first sample label pair and the visual label;
[0202] Input the rewriting guide sentence into the natural language model to predict the matching label of the spoken expression that conforms to the natural language;
[0203] In this example, the first sample label pair includes a first label before being rewritten in pseudo-colloquial language and a second label obtained by being rewritten in pseudo-colloquial language; taking the scene object as a virtual bed in a virtual scene as an example, the first label in the first sample label pair is: "Type description: wooden double bed; material: [wood]; transparency: opaque; color: [brown]; surface feature: [carving]; shape: [rectangular]". The second label is: "bed, brown, wooden, rectangular".
[0204] The rewriting guide sentence has the natural semantics of rewriting the visual label with reference to the first sample label pair; in one example, the visual label that needs to be rewritten in a pseudo-colloquial way is: "Type description: green plant pot; material: [plant]; transparency: opaque; color: [brown, green]; surface features: [smooth]; shape: [irregular shape]". Correspondingly, based on the first sample label pair and the visual label, the guide sentence obtained is: Example: rewrite "Type description: wooden double bed; material: [wood]; transparency: opaque; color: [brown]; surface features: [carving]; shape: [rectangular]" into "bed, brown, wooden, rectangular". According to the above example, "Type description: green plant pot; material: [plant]; transparency: opaque; color: [brown, green]; surface features: [smooth]; shape: [irregular shape]" is rewritten.
[0205] Step 545: obtaining the spatial position of the scene object in the virtual scene, and determining the spatial position of the scene object as auxiliary information of the visual label of the scene object;
[0206] Exemplarily, the spatial position of the scene object in the virtual scene is used to indicate the deployment of the scene object in the virtual scene, and the spatial information is used to indicate the position, size and other information of the scene object in the virtual scene after the scene object is deployed in the virtual scene.
[0207] In an optional implementation, step 545 in this embodiment may be implemented as follows:
[0208] Obtain at least one of the coordinate position, orientation information, bounding box information, and cover point information of the scene object in the virtual scene;
[0209] Among them, the coordinate position (Location) is used to indicate the position of the scene object in the virtual scene, such as the coordinate information of the center point or preset point of the scene object in the virtual scene; the orientation (Rotation) is used to indicate the direction facing the scene object in the virtual scene, such as the direction facing the front of the scene object in the virtual scene; the bounding box (BoundingBox) is used to indicate the size of the scene object in the virtual scene; the cover points are used to indicate the recommended virtual character position points when the virtual character approaches the scene object, so that the scene object can mask the virtual character.
[0210] Fig. 9 FIG. 1 shows a schematic diagram of scene object information provided by an exemplary embodiment of the present application. Various types of scene object information are stored in a table, such as Fig. 9As shown, the first column of the table records the asset names of the scene objects, indicating the names of the texture resources and skeleton model resources of the scene objects. The second column records the entity names of the scene objects, which are used to indicate the names of the scene objects in the virtual scene. The third column is the image storage path of the appearance image of the scene object. The fourth column is the spatial position of the coordinates, orientation, etc. of the appearance image, also known as tactical information. The fifth column is the asset scaling ratio of the scene object, which is used to indicate the size of the asset. The sixth column is the image name of the appearance image of the scene object. The seventh column is the appearance image of the scene object.
[0211] In summary, the method provided in this embodiment predicts attribute text and appearance images by calling a multimodal model, thereby realizing the prediction of visual labels of scene objects based on the description information of scene objects in the text dimension and the image dimension; in the process of predicting the visual labels of scene objects, the description information of scene objects in multiple dimensions is used, which can ensure that the characteristics of scene objects are fully described from the natural language semantics in the text dimension and the visual information in the image dimension, thereby ensuring the accuracy of visual labels. Further, the visual labels are rewritten in a pseudo-colloquial way to obtain matching labels that are closer to spoken expressions, thereby improving the accuracy of matching labels to describe scene objects in spoken expression scenes. Further, the spatial position of scene objects in the virtual scene is obtained, and the position and size of the scene in the virtual scene are described, which provides more basis for screening scene objects in the virtual scene.
[0212] Next, determining the first scene object in the virtual scene is introduced. Fig.10 A flowchart of a method for processing scene objects provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device. Figure 4 Based on the embodiment shown, steps 552 to 558 and steps 562 to 568 are also included:
[0213] Step 552: constructing a spatial information library of the virtual scene according to the visual labels of the multiple scene objects in the virtual scene;
[0214] For example, please refer to the above embodiments for the introduction of visual tags, which will not be repeated here. The spatial information library includes visual tags of multiple scene objects in the virtual scene. Figure 8 In the paper, a method for rewriting visual labels in a pseudo-colloquial way is introduced. Figure 8 When the present embodiment is combined with the present embodiment to form a new embodiment, the spatial information library is constructed based on the rewritten labels.
[0215] Step 554: Obtain natural language commands;
[0216] Exemplarily, natural language commands are commands for directing non-player characters in a virtual scene, and users can input natural language commands in text or voice. Natural language commands usually include multiple words to indicate virtual activities that need to be directed to non-player characters in the virtual scene.
[0217] Step 556: based on the similarity between the natural language command and the visual label in the spatial information library, screening out a first scene object corresponding to the natural language command from a plurality of scene objects in the virtual scene;
[0218] Exemplarily, based on the similarity between some or all words in the natural language command and the visual tags in the spatial information library, the first scene object corresponding to the natural language command is screened out from multiple scene objects in the virtual scene. As described above, the natural language command is used to command non-player characters in the virtual scene; the natural language command contains not only language introductions about which virtual scene object the virtual activity is directed to (for example, if the natural language command instructs to move to a virtual building, that is, the virtual activity of moving is performed on the virtual building), but also words indicating the identity of the non-player character performing the virtual activity.
[0219] In an optional implementation, the natural language command includes multiple input words; accordingly, step 556 can be implemented as follows:
[0220] Determining, one by one, a similarity score between each of the plurality of input words and the visual label of the first scene object;
[0221] refer to Fig.11 , the multiple input words included in the natural language command 600 include "blue" and "car"; the virtual scene includes two scene objects; it can be understood that in different embodiments, the virtual scene may include more scene objects. Fig.11 In the example, the visual tags of scene object A 601 include metal, old, car, truck, blue, and damaged; the visual tags of scene object B 602 include metal, scratch, car, damaged, cash transporter, and black. The similarity scores between the visual tags of the scene objects for the multiple input words included in the natural language command 600 are calculated respectively.
[0222] Determining the sum of the multiple similarity scores as the similarity between the first scene object and the natural language command;
[0223] Exemplarily, multiple similarity scores correspond to multiple input words one by one; each input word has a similarity score with the visual label of the first scene object, and the similarity score is the maximum value of the cosine similarity between the input word and each visual label of the first scene object. Exemplarily, the cosine similarity can be implemented as the cosine similarity between the input word and the visual label in the latent space of natural semantics. Fig.11 , multiple similarity scores correspond to multiple input words one by one; for the input word "blue" in the natural language command 600, the similarity score with the scene object A 601 is 1.0; for the input word "car" in the natural language command 600, the similarity score with the scene object A 601 is 1.0; the similarity between the scene object A 601 and the natural language command 600 is 1.0+1.0=2.0. Similarly, the similarity between the scene object B 602 and the natural language command 600 is calculated to be 1.0+0.91=1.91.
[0224] When the similarity between the first scene object and the natural language command exceeds the similarity between other scene objects in the spatial information library and the natural language command, determining the first scene object as the object corresponding to the natural language command;
[0225] Exemplarily, the first scene object is the scene object in the virtual scene that has the greatest similarity to the natural language command. Fig.11 In the example, the similarity between the scene object A 601 and the natural language command 600 is greater than the similarity between the scene object B 602 and the natural language command 600, and the scene object A 601 is determined as the first scene object.
[0226] Step 558: Execute a virtual activity for the first scene object based on the natural language command;
[0227] Exemplarily, the virtual activity indicated by the natural language command is performed on the first scene object, such as launching a virtual attack on the first scene object, moving to the location of the first scene object, opening or closing the first scene object, picking up the first scene object, etc.
[0228] Step 562: constructing a spatial information library of the virtual scene according to the appearance images of the plurality of scene objects in the virtual scene;
[0229] For example, for the introduction of the appearance image, please refer to the above embodiments, which will not be repeated here. The spatial information library includes appearance images of multiple scene objects in the virtual scene.
[0230] Fig.12A schematic diagram of an appearance image of a scene object provided by an exemplary embodiment of the present application is shown. First, an object traversal 452 is performed to obtain each scene object (Actor) 454 in the virtual scene, and the assets 456 referenced by each scene object are identified and extracted. The assets include but are not limited to three-dimensional static meshes and map resources. For the identified assets of the scene object, multiple perspectives 458 are traversed around the asset as the center; for example, nine perspectives are switched to observe the scene object from nine perspectives.
[0231] Exemplarily, by using a multi-view scene screenshot tool, the scene objects can be observed from nine perspectives, and the nine-square appearance image of the scene objects can be obtained, and the three-dimensional model structure of the scene objects can be converted into two-dimensional image information. Exemplarily, by traversing the assets referenced by each scene object in the virtual scene, the three-dimensional model structure of each asset is captured from multiple angles and finally stored as two-dimensional image information. The appearance image of the scene object can be easily obtained, and then the game asset dataset can be constructed.
[0232] The segmentation channels 460 are traversed, and information is captured in different channels 462. The segmentation channels are traversed first, and information is captured in each segmentation channel of the same perspective. Then, the perspectives are traversed to obtain information at different perspectives.
[0233] The introduction of multiple segmentation channels is as follows: On the one hand, the virtual camera model is called to shoot the asset. The selection of viewing angle is based on a preset set of angles to ensure comprehensive coverage of all directions of the asset. On the other hand, the position transformation information of the asset in the virtual scene is recorded, including the position and rotation of the asset, to ensure that the position and posture of the asset in the scene can be accurately restored. On the other hand, the scaling information of the asset is recorded so as to analyze and process the actual size and proportion of the scene objects. On the other hand, in order to describe the space occupancy of the scene objects in the virtual scene, the bounding box size of the scene objects is also calculated and recorded.
[0234] Step 564: Obtain natural language commands;
[0235] Exemplarily, natural language commands are commands for directing non-player characters in a virtual scene, and users can input natural language commands in text or voice. Natural language commands usually include multiple words to indicate virtual activities that need to be directed to non-player characters in the virtual scene.
[0236] Step 566: calling a similarity matching network to predict the similarity between the natural language command and the appearance images of the multiple scene objects in the spatial information library, and obtaining a second scene object with the highest similarity to the natural language command among the multiple scene objects;
[0237] Exemplarily, the similarity matching network has the ability to predict the similarity between image information and text information. As described above, the natural language command can be input in text or voice mode. When the natural language command is input in voice mode, the voice information is converted into text format information through automatic speech recognition (Automatic Speech Recognition, ASR).
[0238] Exemplarily, the similarity matching network can be implemented as a contrastive language-image pretraining model (CLIP). The network parameters of CLIP carry rich visual information. Based on the appearance images of multiple scene objects in the spatial information library and the natural language information in the natural language commands, the similarity relationship between the image information and the text information is predicted.
[0239] Step 568: Execute a virtual activity for the second scene object based on the natural language command;
[0240] Exemplarily, the virtual activity indicated by the natural language command is performed on the second scene object, such as launching a virtual attack on the second scene object, moving to the location of the second scene object, opening or closing the second scene object, picking up the second scene object, etc.
[0241] It should be noted that the second scene object in this step and the first scene object in step 558 may be the same or different, and there is no limitation on this.
[0242] It should be noted that steps 552 to 558 in this embodiment can be Figure 4 The steps in the embodiment are combined into a new embodiment and implemented separately; steps 562 to 568 in this embodiment can be combined with Figure 4 The various steps in the embodiment are combined into a new embodiment and implemented separately; this application does not limit this.
[0243] In an optional example, the spatial information library of the virtual scene is based on appearance images and visual labels of multiple scene objects in the virtual scene; it also includes identity tags of the scene objects to index the scene objects. Fig.13A schematic diagram of a spatial information library of a virtual scene provided by an exemplary embodiment of the present application is shown. Taking the virtual scene including five scene objects as an example, the identity of the first object 621 is SM_xxx01; the spatial information library includes appearance images obtained by observing the first object 621 from nine perspectives, and the visual labels of the first object 621 are: type description: a small building with a loft; material: concrete, wood, glass; transparency: opaque; color: brown, gray; surface features: old, damaged; shape: rectangle, triangle. Fig.13 The identification marks, appearance images, and visual labels of the second object 622 to the fifth object 625 are also shown.
[0244] First, execute the similarity based on the natural language command and the visual tags in the spatial information library shown in step 556, and screen out the first scene object corresponding to the natural language command from the multiple scene objects in the virtual scene; in step 556, if there are multiple scene objects with similar similarities, and they are the most similar scene objects in the virtual scene (there are multiple similar scene objects that cannot be distinguished); or if the similarity of the first scene object is lower than the preset confidence threshold (the confidence of the scene object determined according to the similarity of the visual tags is insufficient), execute the call of the similarity matching network prediction shown in step 566.
[0245] In summary, the method provided in this embodiment predicts the attribute text and appearance image by calling a multimodal model, thereby realizing the prediction of the visual label of the scene object based on the description information of the scene object in the text dimension and the image dimension; ensuring the accuracy of the visual label and improving the efficiency of obtaining the visual label of the scene object. Two similarity evaluation methods are provided, namely, the similarity between the natural language command and the visual label and the similarity between the natural language command and the appearance image. The first scene object is determined in the virtual scene, and the virtual activity to be performed on the first scene object as indicated by the natural language command can be determined.
[0246] An exemplary embodiment of the present application provides a method for controlling a virtual character. The method comprises:
[0247] Displaying at least one of a main control virtual character and an NPC in a virtual environment;
[0248] Exemplarily, the main virtual character is a virtual character that the user directly controls in the virtual environment. There are also one or more non-player characters (NPC) in the virtual environment; the NPC and the main virtual character belong to the same virtual camp and are teammates of the main virtual character. The NPC follows the main virtual character in the virtual environment. In addition, the NPC can also be a follower character, a pet character, etc. controlled by the main virtual character. In addition, the NPC can also be a neutral character, and only cooperates with the main virtual character when certain conditions are met.
[0249] Obtain natural language commands;
[0250] Exemplarily, the natural language command includes the natural semantics of commanding the NPC; the NPC is commanded in the virtual scene through the natural language command, and the natural language command can be directly input by the user or extracted from the voice information input by the user. The present application does not limit the method of obtaining the natural language command.
[0251] The natural language command includes at least one of a behavior intention and a scene entity; the behavior intention is used to indicate the type of virtual activity performed by the NPC, such as indicating what type of virtual activity is performed in the virtual environment; the scene entity includes a first scene object, such as indicating what scene object in the virtual environment the virtual activity is performed on. The behavior intention and scene entity included in the natural language command are partial words in the natural language command.
[0252] In one example, the natural language command controls the NPC in terms of what type of virtual activity to perform and for what scene object in the virtual environment to perform the virtual activity; it is a complex instruction to control the NPC.
[0253] Controlling NPCs to perform virtual activities in response to natural language commands;
[0254] Exemplarily, controlling the NPC to perform virtual activities is determined based on the environmental perception information of the main virtual character and / or NPC. In some examples, the main virtual character and / or NPC obtains environmental perception information within the perception range of vision, hearing, movement trajectory and other perception methods. Exemplarily, the virtual activity is close to the current perception of the virtual environment by the main virtual character and / or NPC, and the NPC is controlled with reference to the current perception of the virtual character and / or NPC of the virtual environment. Exemplarily, the NPC has autonomous behavior capabilities, and the user only directs the NPC. The user gives commands, and the NPC understands and executes the commands based on its own autonomous behavior capabilities.
[0255] To sum up, the method provided in this embodiment determines the virtual activities that need to be performed by facing complex instructions for controlling the NPC in terms of what type of virtual activities to perform and for what scene objects in the virtual environment to perform the virtual activities, with reference to the environmental perception information of the master virtual character and / or the NPC; and ensures that the NPC and the master virtual character perform virtual activities in collaboration with each other by using the environmental perception information of the master virtual character and / or the NPC as a reference.
[0256] Fig.14 A flowchart of a method for controlling a virtual character provided by an exemplary embodiment of the present application is shown. The method comprises:
[0257] Step 201: Acquire a spatial data set;
[0258] The spatial dataset of the virtual environment includes visual labels of scene objects, and the visual labels are used to describe the visual characteristics of the scene objects in at least one dimension, such as describing the scene objects in dimensions such as material, transparency, color, and shape.
[0259] Step 202: displaying at least one of a main control virtual character and an NPC in a virtual environment;
[0260] Exemplarily, the main control virtual character is a virtual character that the user directly controls in the virtual environment. There are also one or more NPCs in the virtual environment; the NPCs and the main control virtual character belong to the same virtual camp and are teammates of the main control virtual character.
[0261] Step 203: Acquire a natural language command in voice form;
[0262] The natural language command includes natural semantics for commanding the NPC, and the command information includes behavioral intention and scene entity; the behavioral intention is used to indicate the type of virtual activity performed by the NPC, and the scene entity is used to indicate the target entity of the virtual activity;
[0263] Step 204: converting the natural language command in voice form into the natural language command in text form;
[0264] Exemplarily, speech-to-text (Automatic Speech Recognition, ASR) processing is performed on the natural language command in the form of speech to determine the natural language command in the form of text. Speech-to-text processing generally includes calling components such as acoustic models and language models to recognize the pronunciation, vocabulary, grammatical structure and other information of the natural language command in the form of speech, and convert it into a natural language command in the form of text.
[0265] Step 205: performing intent recognition on the natural language command to obtain a first classification label;
[0266] Exemplarily, the first intention indication corresponding to the first classification label includes a command intention for a non-player character on an intention dimension; for example, at least one of the following: indicating which of a plurality of non-player characters the non-player character commanded by the command text is, indicating whether the virtual activity performed by the non-player character is related to initiating a virtual attack, and indicating what kind of virtual activity the non-player character performs.
[0267] Step 206: Perform entity recognition on the natural language command to obtain a target entity;
[0268] The target entity is determined from the virtual environment in combination with the environmental perception information, which includes information perceived by at least one of the main virtual character and the non-player character from the virtual environment. The target entity may be the entity currently seen by the player, or may be an entity that the player has seen in the past.
[0269] Step 207: In response to the natural language command, the NPC is controlled to perform a virtual activity according to the first intention corresponding to the first classification label, or the NPC is controlled to perform a virtual activity associated with the target entity, or the NPC is controlled to perform a virtual activity associated with the target entity according to the first intention corresponding to the first classification label;
[0270] Exemplarily, the NPC is controlled to perform a virtual activity according to the first classification tag and / or the instruction of the target entity.
[0271] Step 208: reporting the NPC's feedback information;
[0272] Exemplarily, the game application can generate corresponding feedback information in real time according to the environmental perception information of the non-player character and make a broadcast. For example, when the environmental perception information of the non-player character triggers the broadcast condition, the game application can generate corresponding feedback information in real time according to the environmental perception information that triggers the broadcast condition; or, when the non-player character or the main control virtual character triggers the broadcast condition, the game application can generate corresponding feedback information in real time according to the environmental perception information of the non-player character.
[0273] The preprocessing stage of step 201 can be implemented as follows:
[0274] Sub-step 1: Obtaining attribute text of scene objects in the virtual scene;
[0275] Exemplarily, attribute text is used to introduce the inherent attributes of scene objects in a virtual scene; on the one hand, attribute text realizes the description of scene objects in a textual mode, and while describing scene objects, it provides semantic information for the prediction of the visual labels of scene objects. On the other hand, attribute text is the description of scene objects in a virtual scene. In the face of a large number of scene objects in a virtual scene and the existence of reused object models, it can more accurately describe the inherent attributes of scene objects in a virtual scene.
[0276] In an optional implementation, the attribute text includes at least one of a name of the scene object in the virtual scene and a size of the scene object in the virtual scene.
[0277] Sub-step 2: obtaining an appearance image of a scene object in a virtual scene;
[0278] Exemplarily, the appearance image is used to describe the style of the scene object; the appearance image carries the appearance style of the scene object such as color, texture, shape, etc., and the relative position relationship between each sub-part in the picture mode, and can fully describe the scene object from the picture mode (or visual mode). In an optional implementation, the appearance image of the scene object includes images obtained by observing the scene object from at least two perspectives.
[0279] Sub-step 3: Call the multimodal model to perform prediction on the attribute text and appearance image of the scene object to obtain the visual label of the scene object;
[0280] Exemplarily, the multimodal model has the ability to perform model prediction on text information and image information of different modalities. In this embodiment, the input parameters of the multimodal model are the attribute text and appearance image of the scene object; the multimodal model predicts the visual label of the scene object from the two modalities of text modality and image modality, and the visual label is used to describe the visual features of the scene object in at least one dimension.
[0281] Optionally, the multimodal model includes a visual question answering model; the question statement carries attribute text of the scene objects; on the one hand, the question statement is used to guide the visual question answering model to convert the appearance image into the visual label of the scene object; on the other hand, while guiding the conversion of the visual label of the scene object, the question statement provides supplementary information of the scene object in text form.
[0282] Get the expected information of scene objects;
[0283] Exemplarily, the expected information is used to indicate the description dimension of the scene object expected in the visual tag, and / or the expected format of the visual tag; in one example, the expected information is used to indicate that the description dimension of the scene object expected in the visual tag includes but is not limited to at least one of the type description, material, transparency, color, surface features, and shape of the scene object. In another example, the expected information is used to indicate that the expected format of the visual tag is at least one of the following: Comma Separated Values (CSV), JavaScript Object Notation (JSON), and eXtensible Markup Language (XML).
[0284] Construct a question statement of the appearance image based on the expected information and attribute text;
[0285] Exemplarily, the first subpart of the question sentence is supplementary introduction information of the scene object, carrying attribute text of the scene object; the second subpart of the question sentence is an answer guidance sentence for the visual question answering model, carrying expected information.
[0286] Optionally, the spatial data set includes matching labels of scene objects in the virtual scene; correspondingly: performing pseudo-spoken rewriting on the visual labels of the scene objects to obtain matching labels that conform to spoken expressions of natural language.
[0287] Exemplarily, as described above, visual labels are used to describe the visual features of scene objects in at least one dimension, and the appearance image of the scene objects presents rich visual features of the scene objects. In the spoken expression of natural language, however, the description of scene objects cannot cover the visual features of scene objects in all dimensions. The purpose of performing a pseudo-colloquial rewrite on the visual labels of scene objects is to obtain a matching label that is closer to the spoken expression; it can be understood that performing a pseudo-colloquial rewrite can be to delete part of the content in the visual label, or to change the visual label to a label with the same semantics but different textual expression.
[0288] In an optional implementation, the pseudo-colloquial rewriting is performed by calling a natural language model; the visual labels of the scene objects are input into the natural language model, and matching labels that conform to the spoken expressions of the natural language are predicted. The pseudo-colloquial rewriting of the visual labels is implemented based on calling the natural language model. Exemplarily, the natural language model carries prior knowledge of the spoken expressions of the natural language. For example, the natural language model is implemented as a large language model (Large Language Model, LLM).
[0289] Optionally, the spatial dataset also includes spatial information of scene objects, and accordingly, also includes:
[0290] The spatial position of the scene object in the virtual scene is obtained, and the spatial position of the scene object is determined as auxiliary information of the visual label of the scene object.
[0291] Exemplarily, the spatial position of the scene object in the virtual scene is used to indicate the deployment of the scene object in the virtual scene, and the spatial information is used to indicate the position, size and other information of the scene object in the virtual scene after the scene object is deployed in the virtual scene.
[0292] Further, obtaining at least one of the coordinate position, orientation information, bounding box information, and cover point information of the scene object in the virtual scene;
[0293] Among them, the coordinate position (Location) is used to indicate the position of the scene object in the virtual scene, such as the coordinate information of the center point or preset point of the scene object in the virtual scene; the orientation (Rotation) is used to indicate the direction facing the scene object in the virtual scene, such as the direction facing the front of the scene object in the virtual scene; the bounding box (BoundingBox) is used to indicate the size of the scene object in the virtual scene; the cover points (Cover points) are used to indicate the recommended virtual character position points when the virtual character approaches the scene object, so that the scene object can mask the virtual character.
[0294] The intention recognition phase of step 205 can be implemented as follows:
[0295] Sub-step 4: calling at least one hierarchical prediction network to perform intent recognition on the natural language command, and obtaining a first classification label of the natural language command intent to command the NPC;
[0296] Exemplarily, the hierarchical prediction network has the ability to predict the classification label corresponding to the command text. Exemplarily, the hierarchical prediction network includes at least two layers of sub-networks, the upper network and the lower network are cascaded, and the lower network further performs the prediction of the classification label according to the prediction result output by the upper network. At least two layers of sub-networks are constructed based on the tree structure of multiple classification labels. The hierarchical prediction network constructs at least two layers of sub-networks corresponding to the hierarchical structure for the tree structure of the classification labels, and splits the prediction task of a wide variety of classification labels into prediction sub-tasks performed by at least two layers of sub-networks, so as to reduce the complexity of classification prediction of each layer of sub-network.
[0297] In an optional implementation, each hierarchical prediction network in at least one hierarchical prediction network is used to predict the command intention of the command text for a non-player character on an intention dimension; the intention dimension includes: at least one of the subject dimension, the semantic dimension, and the behavior dimension. Exemplarily, in at least one hierarchical prediction network, the i-th layer subnetwork in each hierarchical prediction network is used to predict the first-level behavior label of the command text on an intention dimension, and the i+1-th layer subnetwork in the hierarchical prediction network is used to predict the second-level behavior label corresponding to the command text in the first-level behavior label, i is a positive integer; the high-level behavior label (such as the first-level behavior label) predicted by the high-level subnetwork (such as the i-th layer subnetwork) in the hierarchical prediction network includes multiple sublabels, or subordinate lower-level labels; the corresponding low-level subnetwork (such as the i+1-th layer subnetwork) needs to be called to further perform classification prediction (such as predicting the second-level behavior label). The prediction task of a wide variety of classification labels is split into prediction subtasks performed by at least two layers of subnetworks to reduce the complexity of classification prediction of each layer of subnetwork.
[0298] Furthermore, the intent dimension includes a subject dimension, the hierarchical prediction network includes a hierarchical subject prediction network, and the subject prediction network has the ability to predict the subject type in a natural language command; the subject type is used to indicate the identity of the NPC commanded by the natural language command; exemplarily, the subject prediction network is used to predict which of a plurality of non-player characters the non-player character commanded by the command text is, and the number of non-player characters commanded by the command text can be one or more.
[0299] And / or, the intent dimension includes a semantic dimension, the hierarchical prediction network includes a hierarchical semantic prediction network, the semantic prediction network has the ability to predict the semantic type in a natural language command; the semantic type is used to indicate the NPC's control method for initiating a virtual attack; exemplarily, the subject prediction network is used to predict whether the virtual activity performed by a non-player character commanded by the command text is related to initiating a virtual attack.
[0300] And / or, the intention dimension includes a behavior dimension, the hierarchical prediction network includes a hierarchical behavior prediction network, the behavior prediction network has the ability to predict the behavior intention type in the natural language command; the behavior intention type is used to indicate the behavior mode of the NPC for performing the virtual activity. Exemplarily, the behavior prediction network is used to predict what virtual activity the non-player character performs.
[0301] Accordingly, in step 207, in response to the natural language command, the NPC is controlled to perform a virtual activity according to the first intention corresponding to the first classification label;
[0302] Exemplarily, the first intention corresponding to the first classification tag indicates a command intention for a non-player character in an intention dimension, such as at least one of the following: indicating which non-player character is commanded by the command text among a plurality of non-player characters, indicating whether the virtual activity performed by the non-player character is related to launching a virtual attack, and indicating what virtual activity the non-player character performs. According to the instruction of the first classification tag, the NPC is controlled to perform the virtual activity.
[0303] The target entity identification process of step 206 can be implemented as follows:
[0304] Sub-step 5: querying and obtaining the target entity from the entity set of the virtual environment according to the target entity information of the target entity indicated by the natural language command and the environment perception information;
[0305] The target entity is determined from the virtual environment in combination with environmental perception information, and the environmental perception information includes information perceived by the main virtual character and / or the non-player character from the virtual environment.
[0306] Since the target entity information in natural language commands is usually vague, for example, a natural language command may be "move to the red truck", where the target entity information is "red truck", and there may be many red trucks in the virtual environment. It is impossible to accurately determine the target entity in the virtual environment based solely on the target entity information in the natural language command.
[0307] Therefore, the embodiment of the present application provides a method for determining the target entity by combining the target entity information and the environmental perception information. For the previous example, the "red truck" expressed by the player in the natural language command should be a red truck that the player can see. Therefore, combined with the field of view of the main control virtual character, the red truck within the field of view of the main control system character can be screened out from the multiple red trucks in the virtual environment. The red truck is the target entity indicated by the player in the natural language command.
[0308] Since natural language commands are issued by players based on their perception of the virtual environment, in order to accurately identify the target entity indicated in the natural language command, the game application will combine the environmental perception information when the player issues the natural language command to identify the target entity in the natural language command. Based on the player's perception of the virtual environment, the entity that the player can perceive in the virtual environment and is closest to the target entity information is inferred as the target entity.
[0309] Exemplarily, the virtual environment includes multiple candidate entities that match the natural language command, and the target entity is an entity selected from the multiple candidate entities based on the environmental perception information of the main virtual character or the non-player character. For example, the game application first selects multiple candidate entities that match the target entity information in the natural language command from the entity set, and then selects the entity that can be perceived by the main virtual character or the non-player character from the multiple candidate entities as the target entity based on the environmental perception information.
[0310] For example, the environmental perception information of the main virtual character may include: the pictures or entities that the main virtual character can see when observing the virtual environment, the environmental sound effects that the main virtual character can hear, the source direction of the environmental sound effects, the type and volume of the environmental sound effects, the perception information obtained by the main virtual character by using perception skills (for example, visual information, auditory information, sound wave information, light reflection information, etc.), and the perception information obtained by the main virtual character by using sensing props (for example, sensor signals from sensors, positioning signals from positioning props, etc.).
[0311] The environmental perception information of the non-player character may include: the images or entities that the non-player character can see when observing the virtual environment, the environmental sounds that the non-player character can hear, the source direction of the environmental sounds, the type and volume of the environmental sounds, the perception information obtained by the non-player character by using perception skills (for example, visual information, auditory information, sound wave information, light reflection information, etc.), and the perception information obtained by the non-player character by using sensory props (for example, sensor signals of sensors, positioning signals of positioning props, etc.).
[0312] It should be noted that the environmental perception information may include the information perceived by the controlling virtual character and the non-player character in real time when receiving the natural language command, and may also include the information perceived by the controlling virtual character and the non-player character in history before receiving the natural language command. That is, the target entity may be the entity currently seen by the player, or it may be the entity that the player has seen in the past. Therefore, the game application needs to combine the real-time environmental perception information and the historical environmental perception information of the controlling virtual character and / or the non-player character to identify the target entity.
[0313] For example, when the natural language command is "Let's go back to the hotel we just passed by", the game application needs to query the hotels that the main virtual character has passed by based on the historical environmental perception information of the main virtual character.
[0314] In an optional embodiment, the client queries and obtains the target entity from the entity set of the virtual environment according to the target entity information and the environment perception information indicated by the natural language command. In another optional embodiment, the client reports the natural language command to the server, and the server queries and obtains the target entity from the entity set of the virtual environment according to the target entity information and the environment perception information indicated by the natural language command.
[0315] Optionally, the game application infers the target entity based on the target entity information, the environment perception information and the pre-processed virtual environment data, wherein the pre-processed virtual environment data includes a set of entities in the virtual environment.
[0316] The entity set includes entity information of each entity in the virtual environment. The entity information includes at least one of the following: name, type, location, feature, text label, embedding vector of text label, image, embedding vector of image.
[0317] The text label may be at least one word obtained after segmenting a feature (for example, a descriptive text of an entity appearance), and the embedding model is called to obtain an embedding vector corresponding to each text label.
[0318] The image may include an image obtained by observing a three-dimensional model of the entity from at least one direction, for example, the image may include three views of the entity. The embedding vector of the image is the embedding vector of the image feature label. The image feature label is obtained by calling the multimodal model to perform feature recognition based on the image and text label of the entity.
[0319] Exemplarily, the entity set includes a first type of entity and a second type of entity, the first type of entity includes at least one of an area and a building, and the second type of entity includes a physical object. The text labels of the first type of entity are obtained by manual labeling. For example, the text label of a room of a building is manually labeled as: an area, a building, a floor, and a room. The text labels of the second type of entity are automatically generated by calling a large language model.
[0320] The method for generating text labels for the second type of entity may include: obtaining at least one image of the entity, inputting the at least one image into a large language model, and obtaining a description text of the entity; segmenting the description text to obtain at least one text label of the entity, each text label including a word obtained after the segmentation process; performing a vectorization operation on the text label to obtain an embedding vector corresponding to the text label.
[0321] The embedding vector is the high-dimensional vector data of the text label. By pre-generating the embedding vector of the text label, there is no need to repeatedly generate vectors for the text label of each entity when matching the target entity from the entity set based on the target entity information, which can improve the matching efficiency of the target entity information and the entity information.
[0322] It should be noted that this method does not use the description text directly as the text label, but uses the word segmentation result of the description text as the text label. This is because the description text is usually of different lengths. For longer description texts, their embedding vectors are less effective in actual search and similarity comparison. For example, when searching for "a small blue truck with rusty body" using "truck", "a small blue truck with rusty body" is likely to be ranked after "a car" in the recall results. In fact, for the searched "truck", no matter how complex its additional description is, "truck" should be searched first. Therefore, when performing similarity search, the comparison should be one feature word at a time, not the description text.
[0323] In addition, in order to further extract the image features of the entity, the method provided in the embodiment of the present application can also further extract hidden visual features in the image of the entity based on the text label and image of the entity. Taking the first entity as an example, the method includes: obtaining at least one perspective image of the first entity; obtaining at least one text label of the first entity; calling the multimodal model to extract the image features of the first entity based on at least one perspective image and at least one text label of the first entity, and obtain the image embedding vector of the first entity.
[0324] Since text labels are manually annotated, or generated by a large language model based on human language characteristics; and text labels extracted based on human language habits may ignore some features of the entity. For example, when manually annotating a text label for an oil drum, only more eye-catching text labels such as "metal" and "rust" may be annotated, while detailed features such as "rust", "blue", "yellow", and "paint peeling in the lower right corner" are ignored. Therefore, the method provided in the embodiment of the present application also uses a multimodal model based on the text label and entity image of the entity, so that the multimodal model extracts more image features from it; performs image similarity matching with the target entity based on the image features to improve the recognition accuracy of the target entity.
[0325] Among them, the training method of the multimodal model can be: inputting sample entity images and sample labels into the pre-trained multimodal model, fine-tuning the pre-trained multimodal model according to the loss of the predicted labels and sample labels output by the pre-trained multimodal model, and obtaining a multimodal model that can output image feature labels based on the input text labels and images, and the image feature labels include more detailed description texts proposed from the image. For example, the text description of "a metal oil barrel" can only be associated with the two text labels "metal" and "oil barrel", while the Clip (multimodal) model can also identify the hidden visual information in the image from the image of the oil barrel, such as "rusty", "blue", and "yellow" image feature labels, which are features that do not appear in the text labels. Therefore, combining the Clip model for image feature search can further improve the accuracy of entity search.
[0326] In an optional embodiment, the game application may query the target entity using the following method.
[0327] (1) Parse the natural language command to obtain target entity information; the target entity information includes at least one of the following: entity type, entity name, entity location, and entity feature.
[0328] Optionally, the game application calls the large language model to parse the natural language command to obtain target entity information of the target entity, and calls the embedding model to vectorize the target entity information to obtain a target embedding vector of the target entity information.
[0329] For example, if the natural language command is "Come to the blue truck in front", the target entity information that can be parsed by the large language model includes: the direction information "front" and the entity name "blue truck".
[0330] For example, named entity recognition technology in the field of natural language processing can also be used to extract target entity information from natural language commands. Named entity recognition technology is used to identify entities with specific meanings in texts, such as names of people, places, locative words, adjectives, etc.
[0331] For example, if the natural language command is "Go to the first floor of the motel and find a paper box behind the red sofa", the named entity recognition technology can identify: the architectural noun "the first floor of the motel"; the object nouns "sofa" and "paper box"; the locative word "behind"; the adjective "red". After logical construction, a hierarchical scene query call form with search type, content, locative word, and constraint information can be formed. Among them, the search type is used to narrow the data retrieval scope based on similarity matching, the search content is a specific entity description, and the adjective will be merged into the description text for query together, and the locative word is used to form a constraint on the search scope. The floor constraint is determined by the player's location. In indoor scenes, the upper and lower ranges need to be narrowed to avoid finding invisible entities across floors.
[0332] (2) Calculate the similarity between the target entity information and each entity information in the entity set.
[0333] Exemplarily, the entity set includes entity information of at least one entity, and the entity information of each entity may include at least one of a text embedding vector of a text label and an embedding vector of an image feature. Then the target entity information may calculate similarity with the text embedding vector and the image embedding vector respectively, and the entity with higher similarity is determined as the target entity.
[0334] The following are methods for similarity matching with text embedding vectors and similarity matching with image embedding vectors.
[0335] 1) Similarity includes the text similarity between the target entity information and the text embedding vector.
[0336] Taking the first entity in the entity set as an example, the game application (client or server) segments the target entity information to obtain at least one target entity tag; converts the at least one target entity tag into at least one target embedding vector; obtains the entity information of the first entity, the entity information includes a text embedding vector, and the text embedding vector is an embedding vector obtained based on the text tag conversion of the first entity; calculates the text parent similarity between at least one target embedding vector and the text embedding vector respectively, and obtains at least one text parent similarity corresponding to at least one target embedding vector; and determines the sum of the at least one text parent similarity as the text similarity between the target entity information and the entity information of the first entity.
[0337] For example, the number of target entity tags is two, target entity tag 1 and target entity tag 2, and the number of entities in the entity set is also two, entity 1 and entity 2. Entity 1 corresponds to text embedding vector 1, and entity 2 corresponds to text embedding vector 2. Then similarity 1 between target entity tag 1 and text embedding vector 1 is calculated, and similarity 2 between target entity tag 2 and text embedding vector 1 is calculated, and the sum of similarity 1 and similarity 2 is determined as the text similarity between the target entity information and entity 1. Similarity 3 between target entity tag 1 and text embedding vector 2 is calculated, and similarity 4 between target entity tag 2 and text embedding vector 2 is calculated, and the sum of similarity 3 and similarity 4 is determined as the text similarity between the target entity information and entity 2.
[0338] Exemplarily, the entity information of the first entity includes at least one text embedding vector; the at least one target embedding vector includes a first target embedding vector; the game application calculates the text sub-similarity between the first target embedding vector and the at least one text embedding vector respectively to obtain at least one text sub-similarity; the highest value of the at least one text sub-similarity is determined as the text parent similarity corresponding to the first target embedding vector.
[0339] For example, the number of target entity tags is 1: target entity tag 1. The number of entities in the entity set is also 1: entity 1. Entity 1 corresponds to text embedding vector 1 and text embedding vector 3. Then the similarity 1 between target entity tag 1 and text embedding vector 1 is calculated, and the similarity 5 between target entity tag 1 and text embedding vector 3 is calculated, and the larger value of similarity 1 and similarity 5 is determined as the similarity between the target entity information (target entity tag 1) and entity 1.
[0340] For example, for the input target entity information "blue car", the target entity information is first segmented into target entity labels "blue" and "car", which means that the queried target entity has two features. Then each feature is calculated with each entity in the entity set for similarity, and each feature takes the maximum similarity. Finally, the sum of the similarity scores corresponding to all query features is taken, which represents the text similarity between the target entity information and the queried entity.
[0341] For example, the target entity information includes two target entity tags "blue" and "car", and the two target entity tags are converted into two target embedding vectors. Then the entity information in the entity set is obtained. For example, the entity set includes entity 1 and entity 2. The text tags of entity 1 include "metal", "old", "car", "truck", "blue", and "damaged", and the text tags of entity 2 include "metal", "scratches", "car", "damaged", "money transporter", and "black".
[0342] Calculate the text similarity between the target entity information and entity 1: first calculate the similarity between the target entity label "blue" and each text label of entity 1, and take the maximum value. For example, the similarity between the target entity label "blue" and the text label "blue" of entity 1 is 1 (1 is the maximum value); then calculate the similarity between the target entity label "car" and each text label of entity 1, and take the maximum value. For example, the similarity between the target entity label "car" and the text label "car" of entity 1 is 1. Then sum the two similarities of the target entity labels "car" and "blue" to obtain the final similarity between the target entity information and entity 1, which is 2.
[0343] Similarly, calculate the text similarity between the target entity information and entity 2: first calculate the similarity between the target entity label "blue" and each text label of entity 2, and take the maximum value. For example, the similarity between the target entity label "blue" and the text label "black" of entity 1 is 0.91; then calculate the similarity between the target entity label "car" and each text label of entity 1, and take the maximum value. For example, the similarity between the target entity label "car" and the text label "car" of entity 2 is 1 (1 is the maximum value). Then sum the two similarities of the target entity labels "car" and "blue" to obtain the final similarity between the target entity information and entity 2, which is 1.91.
[0344] It can be seen that the similarity between the target entity information and entity 1 is 2, which is higher than the similarity between the target entity information and entity 2, which is 1.91.
[0345] 2) Similarity includes the image similarity between the target entity information and the image embedding vector.
[0346] Taking the first entity in the entity set as an example, the game application (client or server) segments the target entity information to obtain at least one target entity tag; converts the at least one target entity tag into at least one target embedding vector; obtains the entity information of the first entity, the entity information includes an image embedding vector, and the image embedding vector is an embedding vector obtained based on the image extraction of the first entity; calculates the image parent similarity of at least one target embedding vector and the text embedding vector respectively, and obtains at least one image parent similarity corresponding to the at least one target embedding vector; and determines the sum of the at least one image parent similarity as the image similarity between the target entity information and the entity information of the first entity.
[0347] For example, the number of target entity tags is two, target entity tag 1 and target entity tag 2, and the number of entities in the entity set is also two, entity 1 and entity 2. Entity 1 corresponds to image embedding vector 1, and entity 2 corresponds to image embedding vector 2. Then similarity 1 between target entity tag 1 and image embedding vector 1 is calculated, and similarity 2 between target entity tag 2 and image embedding vector 1 is calculated, and the sum of similarity 1 and similarity 2 is determined as the image similarity between the target entity information and entity 1. Similarity 3 between target entity tag 1 and image embedding vector 2 is calculated, and similarity 4 between target entity tag 2 and image embedding vector 2 is calculated, and the sum of similarity 3 and similarity 4 is determined as the image similarity between the target entity information and entity 2.
[0348] Exemplarily, the entity information of the first entity includes at least one image embedding vector; at least one target embedding vector includes a first target embedding vector; the game application calculates the image sub-similarity between the first target embedding vector and the at least one image embedding vector respectively to obtain at least one image sub-similarity; and the highest value of the at least one image sub-similarity is determined as the image parent similarity corresponding to the first target embedding vector.
[0349] For example, the number of target entity tags is 1: target entity tag 1. The number of entities in the entity set is also 1: entity 1. Entity 1 corresponds to image embedding vector 1 and image embedding vector 3. Then the similarity 1 between target entity tag 1 and image embedding vector 1 is calculated, and the similarity 5 between target entity tag 1 and image embedding vector 3 is calculated, and the larger value of similarity 1 and similarity 5 is determined as the similarity between the target entity information (target entity tag 1) and entity 1.
[0350] In an optional embodiment, the target image embedding vector may be used for calculating the image similarity, that is, the target image embedding vector is used instead of the target embedding vector in the above method. The target image embedding vector may be obtained by: segmenting the target entity information to obtain at least one target entity tag; inputting at least one target entity tag into a multimodal model to obtain a target image embedding vector.
[0351] 3) Similarity includes text similarity and image similarity.
[0352] When the first entity and the target entity have corresponding text similarity and image similarity, the game application determines an average value of the text similarity and the image similarity as the similarity between the first entity and the target entity.
[0353] Alternatively, when the first entity and the target entity have corresponding text similarity and image similarity, the larger value of the text similarity and the image similarity is determined as the similarity between the first entity and the target entity.
[0354] Alternatively, when the first entity and the target entity have corresponding text similarity and image similarity, the sum of the text similarity and the image similarity is determined as the similarity between the first entity and the target entity.
[0355] (3) Determine the target entity from the entity set based on similarity and environmental perception information.
[0356] In an optional embodiment, the game application first performs a similarity search based on the target entity information to obtain at least one candidate entity with a high similarity, and sorts them according to the similarity; then, based on the position, location, orientation and other information of the main virtual character and / or the non-player character, the game application performs perceptual filtering and sorting, and finally screens out the target entity.
[0357] Exemplarily, the game application determines a target perception range based on target entity information and environmental perception information; and obtains the target entity by filtering entities perceived within the target perception range based on similarity.
[0358] Alternatively, the game application determines x entities with the highest similarity from the entity set according to the similarity to form a candidate entity list; determines the target perception range according to the target entity information and the environmental perception information, and determines the entity in the target perception range among the x entities as the target entity. x is a positive integer.
[0359] Exemplarily, the game application determines the entity with the highest similarity in the perception range as the target entity.
[0360] It should be noted that when there are multiple entities with equal similarity within the target perception range, the multiple entities can be sorted according to the distance between the entity and the controlling virtual character. For example, when there are at least two entities with the highest similarity in the perception range, the entity with the highest similarity in the perception range and the closest distance to the controlling virtual character is determined as the target entity.
[0361] In an optional embodiment, the game application determines the entity with the highest similarity within the perception range and above the threshold as the target entity. Exemplarily, the number of the perception range is at least one. For example, the perception range includes at least one of the following: the visual range of the main virtual character, the hearing range of the main virtual character, the perception range of the main virtual character's perception of virtual props, the visual range of the non-player character, the hearing range of the non-player character, and the perception range of the non-player character's perception of virtual props.
[0362] Exemplarily, the game application may sequentially traverse at least one of the above-mentioned perception ranges, or the game application may sequentially traverse at least one perception range determined according to the natural language command. For example, if the natural language command is "Can you see the truck in front? Move to the truck", the game application may first traverse the visual range of the main virtual character, and then traverse the visual range of the non-player character, and match the entity with the highest similarity and above the threshold.
[0363] Exemplarily, when there are at least two perception ranges, the traversal order of the at least two perception ranges may be preset, or the traversal order of the at least two perception ranges may be determined according to the intention indicated in the natural language command. For example, when the intention is to search for an item, the visual range is traversed first. When the intention is to find a combat scene, the auditory range is traversed first.
[0364] If there are multiple levels of nested instructions in the natural language command, the first target entity will be searched first, and then the next target entity will be searched based on the position of the previous target entity. In the case where the natural language command includes a second target entity determined based on the position of the first target entity, the first target entity is queried from the entity set of the virtual environment based on the first target entity information and environmental perception information of the first target entity indicated by the natural language command; then the second target entity is queried from the entity set of the virtual environment based on the second target entity information, the first target entity and environmental perception information of the second target entity.
[0365] In response to natural language commands, the NPC is controlled to perform a virtual activity associated with a target entity.
[0366] Exemplarily, a large language model is called to identify the behavioral intent of natural language commands, and a control instruction or a control instruction sequence is generated according to the behavioral intent. According to the control instruction or the control instruction sequence, a non-player character is controlled to complete an activity based on a target entity.
[0367] The broadcasting of the NPC feedback information in step 208 may be implemented as follows:
[0368] Sub-step 6: reporting feedback information of the non-player character, wherein the feedback text corresponding to the feedback information is a non-fixed text generated based on the environmental perception information of the non-player character;
[0369] Exemplarily, the game application can generate corresponding feedback information in real time according to the environmental perception information of the non-player character and make a broadcast. For example, when the environmental perception information of the non-player character triggers the broadcast condition, the game application can generate corresponding feedback information in real time according to the environmental perception information that triggers the broadcast condition; or, when the non-player character or the main control virtual character triggers the broadcast condition, the game application can generate corresponding feedback information in real time according to the environmental perception information of the non-player character.
[0370] For example, the feedback text is obtained by inferring the large language model based on the static entity data of the three-dimensional virtual environment and the dynamic environment perception information of the non-player character. The large language model will generate different feedback texts according to the different perception situations of the non-player character.
[0371] For example, the feedback text can also be obtained by inferring the static entity data of the three-dimensional virtual environment and the dynamic environment perception information of the non-player character through the language model, and the feedback text can be in line with the personality characteristics of the non-player character. For example, when the non-player character is a strong uncle, the feedback text can adopt a heroic tone; when the non-player character is a reporter, the feedback text can adopt a news report tone.
[0372] It should be noted that due to the randomness of the results generated by the large language model, in the same scene, when the environmental perception information of the non-player character is the same, the generated feedback text may also be different.
[0373] The environmental perception information may include information perceived by the non-player character in real time, or may include information perceived by the non-player character historically.
[0374] Exemplarily, the feedback information is inferred by the game application based on the environmental perception information of the non-player character and the static entity data of the three-dimensional virtual environment. The feedback information can be a voice broadcast or a text broadcast. The static entity data includes data of relatively unchanged entities in the three-dimensional virtual environment, for example, model information of various buildings, terrains, and vehicles in the three-dimensional virtual environment.
[0375] Exemplarily, the feedback information may include at least one of the following: instant feedback, execution feedback, and dynamic feedback. Instant feedback and execution feedback are both feedback information generated for natural language commands, and dynamic feedback is feedback information spontaneously generated by non-player characters.
[0376] 1. Instant feedback is feedback information generated immediately when a natural language command is received. Instant feedback can include reply broadcast and instant broadcast. Reply broadcast is used to resume the inquiry in the natural language command; instant broadcast is used to feedback the reception of the natural language command.
[0377] For example, the client can command the non-player character to move in the three-dimensional virtual environment according to the intention of the natural language command. Then, when the client receives the natural language command, it can generate instant feedback corresponding to the natural language command, and the instant feedback is used to indicate that the natural language command has been received, or the instant feedback is used to instantly reply to the natural language command.
[0378] In an optional embodiment, the feedback information includes a reply report. The reply report includes a reply to the query natural language command generated in response to the query natural language command raised by the player. For example, when the query natural language command includes querying the location of the target entity, the corresponding reply report should include the query result of the location of the target entity.
[0379] When the behavioral intention of the received natural language command is an inquiry, the terminal broadcasts a reply message of the non-player character; the reply message includes reply content for the natural language command generated according to the environmental perception information of the non-player character on the three-dimensional virtual environment.
[0380] The natural language command is used to inquire about information from the non-player character. The natural language command may be referred to as an inquiry natural language command. The inquiry natural language command is a natural language command intended to be an inquiry. The inquiry natural language command contains at least one question.
[0381] Natural language commands are command instructions conveyed using human natural language. The terminal receives the natural language commands issued by the player and commands the non-player characters according to the intention corresponding to the natural language commands. For example, the terminal receives the voice audio of the player, converts the voice audio into text, and obtains the natural language command. Alternatively, the terminal receives the text of the natural language command input by the player.
[0382] Players can use spoken or written language to issue natural language commands. The game application will use the large language model to understand the natural language commands, extract the intentions expressed in the natural language commands, generate control instructions based on the intentions, and direct the activities of non-player characters.
[0383] For example, a natural language command may be "Come to me", and the game application may parse out that the intention of the natural language command is "control the non-player character to move to the location of the controlling virtual character". The game application then obtains the location of the controlling virtual character, generates a navigation route for the non-player character to move to the location of the controlling virtual character, and controls the non-player character to move to the location of the controlling virtual character along the navigation route.
[0384] For another example, a natural language command may be "go pick up a treasure chest", then the game application may parse out that the intention of the natural language command is "control the non-player character to move to the location of the treasure chest and pick up the treasure chest", then the game application may obtain the location of the treasure chest, control the non-player character to move to the location of the treasure chest, and perform the operation of picking up the treasure chest after arriving at the location of the treasure chest.
[0385] For example, when the natural language command includes an intelligence inquiry about the target entity, a first reply report is broadcast based on the non-player character's perception of the target entity; the first reply report includes the intelligence perception result of the non-player character about the target entity.
[0386] Alternatively, when the natural language command includes an inquiry about the location of the target entity, a second reply broadcast is broadcast based on the non-player character's perception of the target entity, and the second reply broadcast includes a description of the location of the target entity.
[0387] Exemplarily, a large language model is called to parse the received natural language command to obtain the intention of the natural language command. When the intention is of an inquiry type, the natural language command can be determined to be an inquiry natural language command. When the natural language command is an inquiry natural language command, the large language model can infer according to the intention of the natural language command, based on static entity data and environmental perception information, to obtain a reply text. That is, the large language model is called to parse the received natural language command, and the intention of the natural language command is output as an inquiry, as well as the reply text of the natural language command. The game application calls the text-to-speech service according to the processing logic of the inquiry intention, and converts the reply text into a reply broadcast, which is an audio broadcast, and sends the reply audio to the client for broadcast.
[0388] For example, when the large language model recognizes that the intention of the natural language command is to ask a question, since the user may ask hundreds of questions and it is impossible to store the answer to each question locally on the client, the game application will call the text-to-speech service to generate a reply broadcast in real time based on the reply text returned by the large language model, so that the game application can instantly generate a corresponding reply broadcast to respond to the player's question.
[0389] Exemplarily, a game application (client or server) calls a large language model to parse natural language commands, and infers a reply text based on static environmental data and environmental perception information of the three-dimensional virtual environment; and generates human voice audio based on the reply text to obtain a reply broadcast.
[0390] For example, the client reports the received natural language query command to the server, the server calls the large language model to parse the natural language query command, and infers the reply text based on the static environment data and environmental perception information of the three-dimensional virtual environment; generates human voice audio based on the reply text to obtain a reply broadcast; the server sends the reply broadcast to the client; and the client broadcasts the reply broadcast.
[0391] 2. Execution feedback is the feedback generated when a task is executed according to the intent indicated by the natural language command after receiving the natural language command. Execution feedback can also be called execution broadcast.
[0392] For example, when a natural language command is used to instruct a non-player character to perform a target task, the client can generate execution feedback during the non-player character's execution of the target task, and the execution feedback is used to indicate the execution status of the target task; or, the client can generate execution feedback after the non-player character completes the target task, and the execution feedback is used to indicate the execution result of the target task.
[0393] In an optional embodiment, the feedback information includes a feedback report (also referred to as execution feedback). The feedback report includes a report on the task execution status and task execution result generated in response to the task natural language command proposed by the player. For example, when the task natural language command includes moving to the target entity, the corresponding feedback report may include: moving to the target entity, or having reached the target entity.
[0394] When the behavioral intention of the received natural language command is to execute a task, the terminal broadcasts a feedback report; the feedback report includes an execution status after the natural language command is executed according to the environmental perception information of the non-player character on the three-dimensional virtual environment.
[0395] The natural language command is used to instruct the non-player character to perform a task. The natural language command may be referred to as a task natural language command. The task natural language command includes a description text of the task, for example, the description text of the task may include at least one of the following: the task name, the action that the non-player character needs to perform, the method used to perform the task, the task target, and the location of the task target.
[0396] For example, when the intention of the task natural language command includes searching for a target entity, a first feedback announcement is broadcast, and the first feedback announcement includes the search result of the non-player character for the target entity.
[0397] Alternatively, when the intention of the task natural language command includes using a virtual prop, a second feedback announcement is announced, and the second feedback announcement is used to indicate that the virtual prop is ready.
[0398] Alternatively, when the intention of the task natural language command includes the use of virtual props, a third feedback announcement is broadcast, and the third feedback announcement is used to indicate the result of the use of the virtual props.
[0399] Alternatively, when the intention of the task natural language command includes controlling the movement of the non-player character, a fourth feedback announcement is announced, and the fourth feedback announcement is used to indicate the movement result of the movement.
[0400] Alternatively, when the intention of the task natural language command includes performing an interaction with a target entity, a fifth feedback announcement is broadcast, and the fifth feedback announcement is used to indicate at least one of a search result of the non-player character for the target entity and an interaction result between the non-player character and the target entity.
[0401] Alternatively, when the intention of the task natural language command includes item interaction, a sixth feedback broadcast is broadcast, and the sixth feedback broadcast is used to indicate the item interaction result of the non-player character performing the item interaction.
[0402] Exemplarily, a large language model is called to parse the received natural language command to obtain the intention of the natural language command. When the intention is a task, the natural language command can be determined to be a task natural language command. When the natural language command is a task natural language command, the large language model can be inferred according to the intention of the natural language command, based on static entity data and environmental perception information, to obtain a task instruction sequence, and the task instruction sequence is used to control the non-player character to perform the task. When the game application receives the task instruction sequence, it controls the non-player character activity to perform the task according to the task instructions in the task instruction sequence. During the control of the non-player character according to the task instruction, the corresponding execution status can be obtained for feedback broadcast according to the feedback broadcast logic corresponding to different task instructions.
[0403] For example, since the instructions that can be executed by non-player characters in a three-dimensional virtual environment are finite and traversable, the client can locally store feedback broadcasts corresponding to each character's instructions. When the non-player character executes the corresponding instruction, the client can read the local feedback broadcast for voice broadcast.
[0404] Alternatively, when there is variable content in the feedback broadcast corresponding to a task instruction, for example, the target entity in the feedback broadcast is variable, or the position information in the feedback broadcast is variable; the client can also generate a broadcast voice with variable content based on the target entity or position information returned by the large language model, and splice it with the broadcast voice with immutable content stored locally to obtain the final feedback broadcast.
[0405] For example, when the task natural language command is "find a nearby treasure chest", calling the large language model to parse the task natural language command can get the intention of the task, the task goal is to find the target entity, and the target entity is a treasure chest; based on the static entity data and environmental perception information, reasoning can be obtained that there is a treasure chest in the left front of the main control virtual character. In the case that the target entity can be found, the feedback broadcast corresponding to the task goal is "found aaa at xxx", where xxx is the location of the target entity and aaa is the name of the target entity. The game application can call the online text-to-speech technology to generate the location voice according to the location information "left front" in the reasoning result; according to the name of the target entity "treasure chest", call the online text-to-speech technology to generate the name voice, and then splice the location voice and name voice into the feedback broadcast to get the final feedback broadcast.
[0406] Exemplarily, a game application (client or server) calls a large language model to parse natural language commands for a task, and infers task execution instructions based on static environmental data and environmental perception information of a three-dimensional virtual environment; controls a non-player character to perform the task according to the task execution instructions; generates feedback text according to the execution status of the non-player character performing the task; and generates human voice audio based on the feedback text to obtain a feedback broadcast.
[0407] For example, the client reports a task natural language command to the server; the server parses the task natural language command, and infers the task execution instructions based on the static environment data and environment perception information of the three-dimensional virtual environment; the non-player character is controlled to perform the task according to the task execution instructions; feedback text is generated according to the execution status of the non-player character performing the task; human voice audio is generated based on the feedback text to obtain a feedback broadcast; the server sends the feedback broadcast to the client; the client receives and broadcasts the feedback broadcast.
[0408] 3. Dynamic feedback is the feedback information that is spontaneously generated based on the environmental perception information of non-player characters. Dynamic feedback can also be called dynamic broadcast.
[0409] For example, the client can spontaneously generate dynamic feedback based on the perception of the three-dimensional virtual environment without receiving natural language commands, and the dynamic feedback is used to indicate abnormal situations found by the non-player character in the three-dimensional virtual environment. For example, abnormal situations may include: finding hostile virtual characters, finding changes in the status of friendly virtual characters, finding dangerous situations, finding traces of battle or looting, etc.
[0410] In an optional embodiment, the feedback information includes dynamic reporting (also referred to as "dynamic feedback"). The dynamic reporting includes reporting of abnormal situations perceived by non-player characters. For example, when a non-player character perceives being attacked, a report is made that the character is being attacked; when a non-player character perceives a dangerous situation ahead, a report is made that there is danger ahead.
[0411] The terminal broadcasts a dynamic report when the environmental perception information of the non-player character on the three-dimensional virtual environment meets the dynamic report conditions.
[0412] Dynamic reporting is a report generated by the game application based on the environmental perception information of non-player characters when it determines that there is an abnormal situation that needs to be reported. The dynamic report includes reminder information of the abnormal situation. For example, the abnormal situation may include at least one of the following: discovering a hostile virtual character, discovering a change in the status of a friendly virtual character, and discovering a new trace.
[0413] For example, in the case where the non-player character perceives a hostile virtual character, a first dynamic broadcast is broadcast; the first dynamic broadcast includes the position of the hostile virtual character perceived by the non-player character.
[0414] Alternatively, when the non-player character senses a dangerous situation, a second dynamic broadcast is broadcast; the second dynamic broadcast is used to prompt the dangerous situation.
[0415] Alternatively, when the non-player character senses a change in the state of the friendly virtual character, a third dynamic broadcast is broadcast; the third dynamic broadcast is used to prompt that the state of the friendly virtual character has changed.
[0416] Alternatively, when the non-player character finds new traces in the three-dimensional virtual environment, a fourth dynamic broadcast is broadcast; the fourth dynamic broadcast is used to prompt the new traces. The new traces may be combat traces and / or looting traces.
[0417] For example, since the number of dynamic announcements that can be triggered by non-player characters in a three-dimensional virtual environment is limited and traversable, the client can store dynamic announcement voices locally, and the client can read the dynamic announcement voices locally for announcement when the corresponding dynamic announcement voices are triggered. Similarly, some dynamic announcement voices can contain variable content, and the client can generate variable content in real time through text-to-speech technology, and splice it with the dynamic announcement template to obtain the final dynamic announcement.
[0418] In an optional embodiment, when generating feedback information, it may be necessary to query the location of the target entity in the three-dimensional virtual environment, or determine the target entity indicated in the natural language command from the three-dimensional virtual environment. In this case, an entity query method is required to query the target entity.
[0419] Before executing the target entity query, it is necessary to pre-build the static entity data of the 3D virtual environment. The static entity data construction can serve both the reasoning of the large language model and the real-time feedback text generation on the game application side after the reasoning results are returned.
[0420] In an optional embodiment, entities in the three-dimensional virtual environment are traversed, and entity information of each entity is exported from a game engine to obtain static entity data.
[0421] For example, for in-game scenes, editor tools have been developed to perform StaticMesh traversal of the entire scene, special Actor traversal (such as interactive doors, which do not belong to the StaticMesh category), vegetation traversal, and export the entity's position, orientation, and bounding box size as static entity data for subsequent labeling and reasoning by the large language model.
[0422] Among them, StaticMesh is a static geometry resource type in the Unreal Engine 4 (UE4) game engine, which is used to represent immutable 3D models, such as buildings, props, etc. Actor is a basic object class in the Unreal Engine 4 (UE4) game engine, representing an entity in the game world, such as a character, object, light source, etc. Actor can contain multiple components to achieve different functions.
[0423] Exemplarily, static entity data can also include spatial data, which is obtained by manual labeling. Spatial data is more complex than static entities because there are multiple floors in the space and multiple layers of nesting within the floors (such as the motel area includes the second floor, and the second floor area includes room 201). For this type of data, manual calibration is used to construct it. Add Volume to the scene in the editor, divide the area required for the entire map and label it accordingly. Volume is a special Actor class in the UnrealEngine4 (UE4) game engine, which represents a three-dimensional area with specific functions, such as a trigger area, audio area, etc.
[0424] After obtaining the static entity data, you can query entities or spaces based on the static entity data during the game.
[0425] For example, when a player issues a natural language command, the natural language command will be accompanied by the constructed static entity data and the real-time captured runtime data (including the environmental perception information of non-player characters and / or the main virtual character) to make an inference request to the large language model. In addition, when the NPC needs to provide dynamic feedback based on the runtime data, an entity query is also performed to generate feedback text based on the current state.
[0426] When a natural language command includes a target entity, the game application will query the target entity based on the position and orientation of the current controlling virtual character, the position and orientation of teammates or enemies, and based on the target entity information of the target entity described in the natural language command and the environmental perception information of the controlling virtual character and / or the non-player character, filter out a target entity that matches the target entity information from at least one candidate entity within the perception range of the controlling virtual character and / or the non-player character.
[0427] Optionally, the natural language command is used to instruct the non-player character to perform an activity related to the target entity. For example, the target entity may be a destination for the non-player character to move to, an object that the non-player character needs to observe, a target that the non-player character needs to attack, or an object that the non-player character needs to interact with.
[0428] The target entity is a virtual object that exists in a three-dimensional virtual environment. The target entity is the entity described in a natural language command. An entity may refer to a virtual object that is fixed in a three-dimensional virtual environment. For example, a target entity may be a virtual building, a virtual terrain, a virtual vehicle, a virtual plant, a virtual prop, a virtual item, etc. An entity may also refer to a virtual object that can interact with a virtual character in a three-dimensional virtual environment. For example, a target entity may be an interaction point in a three-dimensional virtual environment (e.g., a door, a window, a cabinet, a cellar, etc.), a virtual prop, a virtual character, a virtual light source, etc.
[0429] For example, a natural language command may be "Please help me open the door", then the target entity may be "door", and the non-player character needs to perform a door-related activity "open the door". Alternatively, a natural language command may be "Is the kitchen safe", then the target entity may be "kitchen", and the non-player character needs to perform a kitchen-related activity "check if there are enemy avatars or other dangerous situations in the kitchen".
[0430] Optionally, the natural language command includes target entity information of the target entity. The target entity information may be a description text of the target entity. For example, the target entity information may include at least one of the following contents: the name, type, location, and features of the target entity. The game application may find the target entity from among the many entities in the three-dimensional virtual environment based on the target entity information in the natural language.
[0431] The game application parses the natural language command, extracts the target entity information therein, determines the target entity from the entity set of the three-dimensional virtual environment based on the target entity information, and then controls the non-player character to perform activities related to the target entity according to the intention of the natural language command.
[0432] The target entity is determined from the three-dimensional virtual environment in combination with environmental perception information, and the environmental perception information includes information perceived by at least one of the main virtual character and the non-player character from the three-dimensional virtual environment.
[0433] Since the target entity information in natural language commands is usually vague, for example, a natural language command may be "move to the red truck", where the target entity information is "red truck", and there may be many red trucks in the three-dimensional virtual environment. It is impossible to accurately determine the target entity in the three-dimensional virtual environment based solely on the target entity information in the natural language command.
[0434] Therefore, the embodiment of the present application provides a method for determining the target entity by combining the target entity information and the environmental perception information. For the previous example, the "red truck" expressed by the player in the natural language command should be a red truck that the player can see. Therefore, combined with the field of view of the main control virtual character, the red truck within the field of view of the main control system character can be screened out from multiple red trucks in the three-dimensional virtual environment. The red truck is the target entity indicated by the player in the natural language command.
[0435] Since natural language commands are issued by players based on their perception of the 3D virtual environment, in order to accurately identify the target entity indicated in the natural language command, the game application will combine the environmental perception information when the player issues the natural language command to identify the target entity in the natural language command. Based on the player's perception of the 3D virtual environment, the entity that the player can perceive in the 3D virtual environment and that is closest to the target entity information is inferred as the target entity.
[0436] Exemplarily, the three-dimensional virtual environment includes multiple candidate entities that match the natural language command, and the target entity is an entity selected from the multiple candidate entities based on the environmental perception information of the main virtual character or the non-player character. For example, the game application first selects multiple candidate entities that match the target entity information in the natural language command from the entity set, and then selects the entity that can be perceived by the main virtual character or the non-player character from the multiple candidate entities as the target entity based on the environmental perception information.
[0437] For example, the perception range of the main virtual character is determined according to the position and orientation of the main virtual character. For example, the perception range is a sector area of a certain visual field size in front of the main virtual character; entities that are not within the sector range are filtered out. Then, the target entity information and the entity information of the candidate entities are sorted according to the similarity, and entities with higher similarity are selected as target entities. When there are two entities with the highest and equal similarity scores within the perception range, for example, the similarity scores of entity 2 and entity 3 are both 2 points, then entity 2, which is closer to the main virtual character, is selected as the target entity.
[0438] When it is necessary to generate feedback information based on the position description of the target entity, the game application can perform a spatial query to generate a position description based on the position relationship between the master virtual character and the target entity.
[0439] The main application scenario of spatial query is to obtain the location of the main virtual character, non-player characters, hostile virtual characters, gunshots, etc. For example, a report containing spatial information such as "enemies found on the second floor of the motel" can be made.
[0440] The spatial query logic defines the concepts of parent area and child area. The parent area refers to a large area in a 3D virtual environment that covers multiple buildings or multiple isolated spaces, such as a motel with front and back yards and multiple rooms and basements. The child area refers to an independent space in the parent area, which can be a closed space or a special function space, such as a motel guest room or kitchen.
[0441] In order to make the location description returned by the spatial query closer to human expression, when the main virtual character and the queried location are in different parent areas, the query result returns complete text information (for example, the main virtual character is outside the motel and the queried location is on the second floor of the motel, then "the enemy is on the second floor of the motel" is returned). When the main virtual character and the queried location are in the same parent area but different child areas, the query result only returns the child area information (for example, the main virtual character is on the first floor of the motel and the queried location is on the second floor, then "the enemy is on the second floor" is returned). When the main virtual character and the queried location are in the same parent area and the same child area, the query result only returns the direction relative to the main perspective player (for example, the main virtual character and the queried location are both on the second floor of the motel, then "the enemy is in the right front position" is returned).
[0442] Exemplarily, when the controlling virtual character and the target entity are located in different parent spaces, the position description includes the parent space and the child space where the target entity is located. When the controlling virtual character and the target entity are located in different child spaces of the same parent space, the position description includes the child space where the target entity is located. When the controlling virtual character and the target entity are located in the same child space of the same parent space, the position description includes the relative position of the target entity and the controlling virtual character; wherein a parent space includes at least one child space.
[0443] For the broadcast of the NPC feedback information in step 208, the feedback information is implemented as spatial human voice audio processed by spatial sound effects, wherein the spatial sound effect processing process can be implemented as follows:
[0444] Sub-step 7: Obtaining human voice audio corresponding to the feedback text;
[0445] In the embodiment of the present application, the process of generating the feedback text is as shown in the above-mentioned sub-step 6, which will not be repeated here.
[0446] In some embodiments, the process of obtaining corresponding human voice audio based on feedback text can be implemented as follows: obtaining feedback text and character feature information corresponding to a non-player character; and generating human voice audio matching the character feature information according to the feedback text.
[0447] Illustratively, the character feature information is used to describe the attributes of the non-player character, that is, the character feature information is information used to describe the current state and characteristics of the non-player character. In some embodiments, the attributes of the non-player character include at least one of the basic attributes of the non-player character and the situational performance attributes. Among them, the basic attributes are attributes pre-configured for the non-player character, that is, the attributes of the non-player character that do not change due to the three-dimensional virtual environment or the current situation, such as the age information of the non-player character, the male / female information of the non-player character, the martial arts information of the non-player character, etc. The situational performance attributes are attributes determined in real time by the non-player character in the situation of the three-dimensional virtual environment, that is, attributes associated with the current situation, such as the emotional information and behavioral information of the non-player character in the current situation.
[0448] Optionally, the character feature information includes at least one of basic character information, character emotion information, and character behavior information. The basic character information is used to indicate the basic situation of the non-player character, and the basic character information is information about the basic attributes of the non-player character, such as the age information of the non-player character, the male / female information of the non-player character, and the martial arts information of the non-player character; the character emotion information is used to indicate the emotional state of the non-player character in the dialogue scenario corresponding to the feedback text, and the character emotion information is the situational performance attribute of the non-player character, such as the character emotion information indicates that the non-player character is in an excited state, an angry state, a sad state, etc.; the character behavior information is used to indicate the actions performed by the non-player character in the dialogue scenario, and the character behavior information is the situational performance attribute of the non-player character, such as being in a running state, an attacking state, an injured treatment state, etc.
[0449] In some embodiments, the generation of human voice audio is achieved through a pre-trained speech generation model, and the speech generation model is used to generate speech that matches the character feature information of the non-player character. In an illustrative example, the feedback text and the character feature information are input into the pre-trained speech generation model to obtain human voice audio as the human voice audio.
[0450] Optionally, the above-mentioned speech generation model can be implemented by a neural network model such as a convolutional neural network, a feedforward neural network, a residual network, a Transformer, a multimodal large language model (Multimodal Large Language Model, MLLM), etc., which is not specifically limited here.
[0451] In some embodiments, the speech generation model includes a text encoder, a role encoder and a decoder. The text encoder is used to perform text encoding on the input feedback text, that is, the feedback text is input into the text encoder to obtain a text encoding representation. The role encoder is used to perform feature encoding on the input role feature information, that is, the role feature information is input into the role encoder to obtain a role encoding representation.
[0452] Illustratively, after obtaining the text coding representation and the role coding representation, the text coding representation and the role coding representation are fused to obtain the coding representation input to the decoder, that is, the text coding representation and the role coding representation are fused to obtain a joint coding representation, and the joint coding representation is input into the decoder to generate human voice audio.
[0453] In some embodiments, the character encoder includes at least one of a first sub-encoder, a second sub-encoder, and a third sub-encoder. The first sub-encoder is used to perform feature encoding based on the basic attributes of the non-player character, that is, when the character feature information includes the character basic information, the character basic information is input into the first sub-encoder to obtain a first character encoding representation; the second sub-encoder is used to perform feature encoding based on the emotional state of the non-player character, that is, when the character feature information includes the character emotional information, the character emotional information is input into the second sub-encoder to obtain a second character encoding representation; the third sub-encoder is used to perform feature encoding based on the behavior state of the non-player character, that is, when the character feature information includes the character behavioral information, the character behavioral information is input into the third sub-encoder to obtain a third character encoding representation.
[0454] Sub-step 8: based on the relative position relationship between the non-player character and the main control virtual character in the three-dimensional virtual environment, obtaining sound effect parameters corresponding to the non-player character;
[0455] Illustratively, the relative position relationship between the non-player character and the controlling virtual character in the three-dimensional virtual environment is used to indicate the relative relationship between a first position corresponding to the non-player character and a second position corresponding to the controlling virtual character in the three-dimensional virtual environment.
[0456] Optionally, the above-mentioned relative orientation relationship includes at least one of position distance, position direction, and spatial surrounding situation between the non-player character and the main virtual character, wherein the position distance is used to indicate the distance relationship between the non-player character and the main virtual character, the position direction is used to indicate the angular relationship between the first position of the non-player character and the orientation direction of the main virtual character, and the spatial surrounding situation is used to indicate the surround situation of the main virtual character by the spatial elements forming the virtual space in the three-dimensional virtual environment, and the propagation effect of the space on the audio when the non-player character emits audio in the virtual space.
[0457] In some embodiments, the determination of the relative position relationship between the non-player character and the main virtual character can be implemented as follows: obtaining first position information of the non-player character in the three-dimensional virtual environment, obtaining second position information of the main virtual character in the three-dimensional virtual environment, and determining the relative position relationship between the non-player character and the main virtual character based on the first position information and the second position information.
[0458] In some embodiments, the first position information of the non-player character and the second position information of the controlling virtual character are position information determined based on the same preset coordinate system. Optionally, the preset coordinate system can be implemented as a world coordinate system corresponding to the three-dimensional virtual environment, or the preset coordinate system can be implemented as a coordinate system established with the controlling virtual character as the origin.
[0459] In some embodiments, when the sound effect parameters corresponding to the non-player characters are pre-generated and stored in a database, the sound effect parameters corresponding to the non-player characters are obtained from the database. In other embodiments, the sound effect parameters corresponding to the non-player characters may also be generated in real time, that is, the sound effect parameters corresponding to the non-player characters are generated based on the relative position relationship.
[0460] Optionally, the sound effect parameters corresponding to the non-player character may include at least one of the following types:
[0461] The first type is distance type
[0462] Illustratively, the distance-type sound effect parameter is used to indicate that the sound effect of the human voice audio is adjusted based on the distance between the non-player character and the main virtual character. In some embodiments, the distance-type sound effect parameter can adjust the audio volume of the human voice audio to simulate the effect of distance on the sound volume; and / or, the distance-type sound effect parameter can adjust the audio delay of the human voice audio to simulate the effect of distance on the sound propagation time.
[0463] Second, direction type
[0464] Illustratively, the direction-type sound effect parameter is used to indicate the adjustment of the sound effect of the human voice audio based on the direction of the non-player character relative to the main virtual character. In some embodiments, the direction-type sound effect parameter can adjust the audio volume of the human voice audio to simulate whether the main virtual character is facing the scene audio element; and / or, the direction-type sound effect parameter can adjust the volume parameters of the human voice audio in different channels to simulate the direction of the audio emitted by the scene sound source element.
[0465] The third type is space effects
[0466] Illustratively, the sound effect parameters of the spatial special effect type are used to indicate that the virtual space formed based on the spatial elements adjusts the sound effects in the virtual space, that is, the sound effect parameters of the spatial special effect type are used to indicate the effect of the virtual space on the sound performance of the audio emitted by the non-player characters. In some embodiments, the sound effect parameters of the spatial special effect type can add echo sound effects and reverberation sound effects to simulate the effect of sound in the space.
[0467] Optionally, the sound effect parameters of the spatial special effect type include at least one of a reverberation time parameter, a pre-delay parameter, a wetness (Wet / DryMix) parameter, a space width parameter, and a distance effect parameter. Among them, the reverberation time parameter is used to indicate the speed at which the audio decays in the virtual space; the pre-delay parameter is used to indicate the time difference between the audio directly reaching the main virtual character and the audio reaching the main virtual character after the first reflection; the wetness parameter is used to indicate the ratio between the audio directly transmitted and the audio transmitted through reflection; the space width parameter is used to indicate the degree of diffusion of the audio on the horizontal plane in the virtual space; and the distance effect parameter is used to indicate the attenuation of the audio with the propagation distance during the propagation process.
[0468] The fourth type is custom type
[0469] Schematically, the custom type of sound effect parameters are user-defined parameters for sound effect adjustment. Optionally, the user can customize the overall volume of the scene audio, whether to add background music, customize the volume of different types of audio, etc.
[0470] In some embodiments, the client provides a customization interface for sound effect parameters to the user, and the user configures the customized sound effect parameters through the customization interface.
[0471] In some embodiments, a parameter prediction model for personalized learning of the custom sound effect parameters is configured in the custom sound effect parameter server for the user. Schematically, the custom sound effect parameters of multiple candidate accounts are obtained, the multiple custom sound effect parameters are input into the parameter prediction model to be trained, the parameter prediction model to be trained is iteratively trained to obtain the parameter prediction model, and the sound effect parameters generated by the system (for example, the sound effect parameters of the above-mentioned distance type, direction type, and spatial special effect type) are optimized through the parameter prediction model.
[0472] Sub-step 9: adjusting the vocal audio based on the sound effect parameters, generating and broadcasting the spatial vocal audio corresponding to the non-player character;
[0473] Among them, the spatial human voice audio is used to express the perceptual effect of the audio produced by the main virtual character on the non-player character under the relative position relationship.
[0474] In an embodiment of the present application, the human voice audio is adjusted according to the sound effect parameters corresponding to the non-player character, thereby generating the spatial human voice audio corresponding to the non-player character. That is, the spatial human voice audio is the audio data corresponding to the non-player character obtained after the human voice audio is adjusted by the sound effect parameters. In some embodiments, the spatial human voice audio includes the audio after the sound effect parameters are adjusted, as well as at least one of the audio duration information, the audio playback start timestamp, the audio playback end timestamp, the audio playback condition information, and the element identification information corresponding to the audio.
[0475] Optionally, when the sound effect parameters indicate that the volume of the non-player character's vocal audio is to be adjusted, the volume of the non-player character's vocal audio on at least one channel is adjusted based on the sound effect parameters. Optionally, when the sound effect parameters indicate that the playback delay of the non-player character's vocal audio is to be adjusted, the audio playback start time corresponding to the non-player character's vocal audio is adjusted based on the sound effect parameters. Optionally, when the sound effect parameters indicate that the speed of the non-player character's vocal audio is to be adjusted, the playback speed of the non-player character's vocal audio is adjusted based on the sound effect parameters. Optionally, when the sound effect parameters indicate the pitch of the non-player character's vocal audio, the frequency in the spectrum of the non-player character's vocal audio is adjusted based on the sound effect parameters. When the sound effect parameters indicate the timbre of the non-player character's vocal audio, a filter corresponding to the non-player character is determined based on the sound effect parameters, and the timbre corresponding to the vocal audio is adjusted through the filter.
[0476] Illustratively, the client plays the spatial human voice audio to realize the broadcasting of non-player character feedback information.
[0477] The speech-to-text conversion stage of step 204 can be implemented as follows:
[0478] Sub-step 10: obtaining a plurality of first scene hot words corresponding to the first virtual scene where the target virtual character is located;
[0479] The target virtual character includes at least one of an NPC and a master virtual character.
[0480] In a illustrative manner, the main virtual character is the virtual character commanded by the player, and the NPC is the virtual character that assists the main virtual character in a virtual game. Usually, a virtual game includes a main virtual character and at least one NPC. There are certain differences between the main virtual character and the NPC.
[0481] Optionally, the main virtual character is the virtual character mainly controlled by the player, and the NPC is the virtual character chosen by the player to command, such as the main virtual character is the virtual character manually controlled by the player during the game, and the player's manual operations on the terminal interface are used to control the main virtual character; the NPC is the virtual character commanded by the player in voice form, and the player occasionally issues natural language commands in voice form to command the NPC.
[0482] In some embodiments, the first virtual scene is determined based on the target virtual character, and the target virtual character is at least one of an NPC and a master virtual character, which means that the process of determining the first virtual scene is implemented as at least one of the following.
[0483] (1) If the main virtual character is selected as the target virtual character, the virtual scene where the main virtual character is located is used as the first virtual scene;
[0484] (2) If an NPC is selected as the target virtual character, the virtual scene where the NPC is located is used as the first virtual scene; if there are multiple NPCs, the virtual scene including the most NPCs can be used as the first virtual scene. Alternatively, the NPC closest to the main virtual character can be selected as the target virtual character, and the virtual scene where the NPC is located can be used as the first virtual scene. Alternatively, the NPC with the largest character attribute value (such as at least one of the attribute values such as virtual health, virtual mana, and virtual defense) can be selected as the target virtual character, and the virtual scene where the NPC is located can be used as the first virtual scene.
[0485] (3) If the main virtual character and the NPC are selected as the target virtual characters, the virtual scene in which the main virtual character and the NPC are located can be used as the first virtual scene, etc.
[0486] In some embodiments, the target virtual character is determined based on the player's selection; or, the target virtual character is set by system default.
[0487] In some embodiments, a plurality of first scene hot words are determined based on the first virtual scene.
[0488] Among them, the plurality of first scene hot words are scene-related words of the first virtual scene, that is, there is an association relationship between the first scene hot words and the first virtual scene, and the first scene hot words are words that describe the first virtual scene.
[0489] Optionally, the first scene hot words include the scene status of the first virtual scene, such as: the first virtual scene is a virtual restaurant, and the first scene hot words include business status, rest status, closed status, etc.
[0490] Optionally, the first scene hot words include element names of virtual elements in the first virtual scene, such as: the first virtual scene is a virtual restaurant, and the first scene hot words include virtual tables, virtual chairs, virtual kitchen utensils, virtual plates, etc.
[0491] Optionally, the first scene hot words include interactive vocabulary for interacting with virtual elements in the first virtual scene, such as: the first virtual scene is a virtual battlefield, and the first scene hot words include attack, attack virtual character, defense, building barriers, etc.
[0492] In some embodiments, after receiving a natural language command, multiple first scene hot words are collected based on the first virtual scene; or, after receiving a natural language command, multiple first scene hot words corresponding to the first virtual scene are filtered from multiple scene hot words pre-acquired based on the first virtual scene, etc.
[0493] In an optional embodiment, the object perception range of the target virtual character in the first virtual scene is obtained.
[0494] Among them, the object perception range is the three-dimensional space range in which the target virtual character perceives the existence of other scene elements.
[0495] Optionally, when the target virtual character is the master virtual character, the object perception range of the master virtual character in the first virtual scene is obtained; when the target virtual character is an NPC, the object perception range of the NPC in the first virtual scene is obtained; or, when the target virtual character is the master virtual character and the NPC, the object perception range of the master virtual character in the first virtual scene is obtained and the object perception range of the NPC in the first virtual scene is obtained.
[0496] In some embodiments, the object perception range includes at least one of the following.
[0497] (1) The visual perception range of the target virtual character.
[0498] In schematic form, the visual perception range refers to the spatial range that an observer can visually perceive or understand, and is usually used to describe the size of the area that an organism or technical device can visually cover or perceive. The visual perception range of the target virtual character is the three-dimensional spatial range perceived by the target virtual character as an observer through observation.
[0499] Optionally, based on the position and visual observation ability of the target virtual character in the first virtual scene, the visual perception range of the target virtual character is acquired.
[0500] (2) The auditory perception range of the target virtual character.
[0501] In schematic form, the auditory perception range refers to the spatial range that a listener can perceive or understand, and is usually used to describe the range of sounds that an organism or technical device can cover or perceive. The auditory perception range of the target virtual character is the three-dimensional spatial range that the target virtual character can perceive through its auditory perception ability, with the target virtual character as the listener.
[0502] Optionally, based on the position and auditory perception ability of the target virtual character in the first virtual scene, the auditory perception range of the target virtual character is acquired.
[0503] (3) The olfactory perception range of the target virtual character.
[0504] In schematic form, the olfactory perception range refers to the spatial range of the odor perceived or recognized by the sense of smell. The olfactory perception range of the target virtual character is the three-dimensional spatial range that the target virtual character can perceive by the sense of smell.
[0505] Optionally, based on the position and olfactory perception ability of the target virtual character in the first virtual scene, the olfactory perception range of the target virtual character is acquired.
[0506] (4) The range of perception of the target virtual character’s perception skills or perception tools.
[0507] Illustratively, the perception skill is the ability of the target virtual character to perceive the first virtual scene, and the perception skill includes at least one of a perception acquisition form and a perception enhancement form.
[0508] Optionally, the perception acquisition form is used to indicate the process of acquiring a perception skill that is not possessed. After the perception skill is possessed through the perception acquisition form, the target virtual character can perform targeted scene perception of the first virtual scene through the perception skill. Optionally, the perception enhancement form is used to indicate the process of enhancing a certain perception ability by possessing the perception skill when the target virtual character possesses the perception ability.
[0509] Illustratively, the perception prop is a virtual prop used to perceive the first virtual scene, and the perception prop includes at least one of a perception acquisition form and a perception enhancement form.
[0510] Optionally, different perception skills or perception tools each correspond to a preset effective range, which represents a range that can be perceived when the perception skill or perception tool is applied, and the preset effective range represents the object perception range.
[0511] In some embodiments, the object perception range is a spherical range, an annular range, an irregular three-dimensional space range, etc. The shape of the object perception range is not limited here.
[0512] In an optional embodiment, a plurality of first scene hot words are acquired based on environmental perception information within a perception range of the object.
[0513] Schematically, the environmental perception information is used to represent the environmental information perceived and acquired by the target virtual character within the object perception range. Optionally, the environmental perception information includes virtual elements determined by emitting rays, and may also include regional structure information determined by analyzing the geometric structure and layout information of the virtual environment.
[0514] Illustratively, after determining the object perception range corresponding to the target virtual character, multiple first scene hot words representing the state of the target virtual character are obtained based on the range of the object perception range in the first virtual scene and the environmental perception information determined based on perception of the target virtual character within the object perception range.
[0515] In an optional embodiment, the environmental perception information includes virtual elements; and the virtual elements in the first virtual scene that are within the perception range of the object are determined.
[0516] Schematically, the object perception range is a part of the three-dimensional space range in the first virtual scene; the first virtual scene includes a large number of virtual elements, and the virtual elements are elements that constitute the virtual scene, such as virtual elements including virtual ground, virtual buildings, virtual trees, virtual characters, virtual stones, etc.
[0517] In some embodiments, when determining the object perception range, the virtual elements within the object perception range are determined from the multiple virtual elements corresponding to the first virtual scene.
[0518] In some embodiments, the element name of the virtual element is used as the first scene hot word; or, the action name of the interactive action corresponding to the virtual element is used as the first scene hot word.
[0519] Sub-step 11: converting the natural language command in voice form into text form based on multiple first scenario hot words;
[0520] Indicatively, the first scene hot words are words that are highly associated with the first virtual scene. Therefore, if you want to command the NPC through natural language commands to better cooperate with the activities of the main virtual character, you can use the first scene hot words as constraints on the basis of the first virtual scene. In this way, in the process of converting the natural language commands into text form, the text content of the command analysis results is more consistent with the first virtual scene, avoiding command analysis results that are incompatible with the first virtual scene.
[0521] In an optional embodiment, when a natural language command is analyzed by a natural language analysis model, a plurality of speech units corresponding to the natural language command are acquired through an acoustic network.
[0522] Among them, phonetic units are the basic building blocks of vocabulary pronunciation.
[0523] In some embodiments, a command feature representation corresponding to a natural language command is extracted; and the command feature representation is analyzed through an acoustic network.
[0524] Schematically, taking the command feature representation as an example, the goal of the acoustic network is to map a continuous sequence of acoustic features to a sequence of speech units, such as phonemes or segments. It learns to predict possible sequences of speech units (phonemes or segments) given acoustic features.
[0525] Optionally, the acoustic network outputs one or more possible speech unit sequences, wherein the speech unit sequence includes multiple speech units. Different speech unit sequences may contain different speech units, or may just have different orders of speech units. The multiple speech unit sequences represent the arrangement of speech units that the language network considers to be most likely when understanding the input natural language command.
[0526] The natural language analysis model includes an acoustic network, a preset dictionary, and language sub-networks corresponding to at least two virtual scenes.
[0527] Schematically, the virtual environment includes at least two virtual scenes. For example, the virtual environment is a large virtual world, which includes a virtual kitchen 1, a virtual kitchen 2, a virtual office, etc. The virtual kitchen 1, the virtual kitchen 2, the virtual office, etc. can each be regarded as a virtual scene.
[0528] In an optional embodiment, based on a first virtual scene in which the target virtual character is located, a first language sub-network corresponding to the first virtual scene is determined from at least two language sub-networks.
[0529] Schematically, multiple virtual scenes correspond to a language subnetwork respectively, and different language subnetworks are used to analyze the corresponding virtual scenes in a targeted manner. For example, virtual scene A corresponds to language subnetwork 1, and virtual scene B corresponds to language subnetwork 2; when analysis is required based on virtual scene A, language subnetwork 1 is used for language analysis; when analysis is required based on virtual scene B, language subnetwork 2 is used for language analysis, and so on.
[0530] Illustratively, the first virtual scene is a virtual scene among at least two virtual scenes. In addition to determining the first scene hot words based on the first virtual scene in which the target virtual character is located, a first language subnetwork corresponding to the first virtual scene is determined from at least two language subnetworks based on the first virtual scene. The first language subnetwork is a subnetwork layer that performs semantic analysis on natural language commands based on the first virtual scene.
[0531] In an optional embodiment, the first language sub-network analyzes the sequence matching relationship between multiple speech units and selected words in a preset dictionary to convert the natural language command in speech form into text form.
[0532] Optionally, different language sub-networks are language sub-networks pre-trained based on corresponding virtual scenes, wherein the first language sub-network is a language sub-network trained based on the first virtual scene.
[0533] In some embodiments, matching relationships between multiple speech units and selected words in a preset dictionary are analyzed to obtain multiple candidate word sequences.
[0534] The selected words include at least a plurality of first-scenario hot words and common words.
[0535] Schematically, the process of analyzing the matching relationship represents the process of analyzing the matching between the selected vocabulary and the speech unit. Schematically, each selected vocabulary corresponds to at least one speech unit, and the conversion probability of the speech unit to the selected vocabulary is determined by analyzing the multiple speech units and the selected vocabulary. The conversion probability represents the probability of matching between the selected vocabulary and the speech unit. The larger the conversion probability, the closer the speech unit is to the selected vocabulary, that is, the greater the possibility that the speech unit may express the selected vocabulary, and the stronger the matching relationship between the selected vocabulary and the speech unit.
[0536] Optionally, a plurality of speech units form at least one speech unit sequence, and based on a matching analysis process between the speech units and the selected vocabulary, at least one candidate vocabulary sequence corresponding to the at least one speech unit sequence is determined, and a plurality of candidate vocabulary sequences are obtained.
[0537] In some embodiments, the sequence semantics of multiple candidate vocabulary sequences are analyzed by the first language sub-network, and at least one candidate vocabulary sequence is obtained from the multiple candidate vocabulary sequences as a text form for natural language command conversion.
[0538] Schematically, sequence semantics is used to represent the semantic information expressed by the candidate vocabulary sequence, which can not only reflect the semantic changes of at least one word in the candidate vocabulary sequence, but also reflect the sentence semantics of the entire candidate vocabulary sequence.
[0539] Schematically, the natural language command in text form is called the command analysis result; multiple candidate vocabulary sequences are evaluated separately by the first language sub-network, so as to take at least one candidate vocabulary sequence that is more in line with the current semantic situation as the command analysis result, that is, the natural language command in speech form is converted into text form.
[0540] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.
[0541] Those skilled in the art can understand that the above embodiments can be implemented independently, or the above embodiments can be freely combined to form new embodiments to implement the scene object processing method of the present application.
[0542] Fig.15 The following is a block diagram of a scene object processing device provided by an exemplary embodiment of the present application. The device includes:
[0543] An acquisition module 810 is used to acquire an attribute text of a scene object in a virtual scene, wherein the attribute text is used to introduce an inherent attribute of the scene object in the virtual scene;
[0544] The acquisition module 810 is further used to acquire an appearance image of the scene object in the virtual scene, wherein the appearance image is used to describe the style of the scene object;
[0545] The processing module 820 is used to call the multimodal model to perform prediction on the attribute text and the appearance image of the scene object to obtain a visual label of the scene object, where the visual label is used to describe the visual features of the scene object in at least one dimension.
[0546] In an optional implementation of this embodiment, the multimodal model includes a visual question answering model, and the processing module 820 is further used to:
[0547] Constructing a question sentence for the appearance image, wherein the question sentence carries the attribute text of the scene object;
[0548] The appearance image of the scene object and the question sentence are input into the visual question answering model to obtain an answer sentence, and the answer sentence is used as the visual label of the scene object.
[0549] In an optional implementation of this embodiment, the question statement includes at least two sub-statements; the processing module 820 is further used for:
[0550] Inputting the appearance image of the scene object and a first sentence of the at least two sub-sentences into the visual question answering model to obtain a first answer sub-sentence;
[0551] Repeat the above steps until at least two answer sub-sentences corresponding to the at least two sub-sentences are obtained, wherein the at least two sub-sentences are used to inquire about the visual features of the appearance image from multiple dimensions;
[0552] Sentence aggregation is performed on the at least two answer sub-sentences to extract the visual label of the scene object.
[0553] In an optional implementation of this embodiment, the processing module 820 is further configured to:
[0554] Acquiring expected information of the scene object, the expected information being used to indicate a description dimension of the scene object expected in the visual tag and / or an expected format of the visual tag;
[0555] constructing the question sentence of the appearance image according to the expected information and the attribute text;
[0556] Among them, the first sub-part of the question sentence is the supplementary introduction information of the scene object, carrying the attribute text of the scene object; the second sub-part of the question sentence is the answer guidance sentence for the visual question and answer model, carrying the expected information.
[0557] In an optional implementation of this embodiment, the multimodal model includes a picture description model, and the processing module 820 is further used to:
[0558] Inputting the appearance image of the scene object into the picture description model to predict a description text of the scene object;
[0559] Sentence aggregation is performed on the description text and the attribute text to extract the visual label of the scene object.
[0560] In an optional implementation of this embodiment, the attribute text of the scene object includes at least one of a name of the scene object in the virtual scene and a size of the scene object in the virtual scene;
[0561] And / or, the appearance image of the scene object includes images obtained by observing the scene object from at least two viewing angles.
[0562] In an optional implementation of this embodiment, the processing module 820 is further configured to:
[0563] The visual label of the scene object is rewritten in a pseudo-spoken manner to obtain a matching label that conforms to a spoken expression of a natural language.
[0564] In an optional implementation of this embodiment, the processing module 820 is further configured to:
[0565] The visual label of the scene object is input into a natural language model to predict the matching label that conforms to the spoken expression of the natural language, and the natural language model carries prior knowledge of the spoken expression of the natural language.
[0566] In an optional implementation of this embodiment, the processing module 820 is further configured to:
[0567] Obtaining a first sample label pair, the first sample label pair comprising a first label before being rewritten into a pseudo-colloquial language and a second label obtained after being rewritten into a pseudo-colloquial language;
[0568] Constructing a rewriting guide sentence according to the first sample label pair and the visual label, wherein the rewriting guide sentence has natural semantics of rewriting the visual label with reference to the first sample label pair;
[0569] The rewriting guide sentence is input into a natural language model to predict the matching label that conforms to the spoken expression of natural language.
[0570] In an optional implementation of this embodiment, the acquisition module 810 is further configured to:
[0571] The spatial position of the scene object in the virtual scene is acquired, and the spatial position of the scene object is determined as auxiliary information of the visual label of the scene object.
[0572] In an optional implementation of this embodiment, the processing module 820 is further configured to:
[0573] Acquire at least one of the coordinate position, orientation information, bounding box information, and cover point information of the scene object in the virtual scene;
[0574] Among them, the coordinate position is used to indicate the position of the scene object in the virtual scene, the orientation information is used to indicate the direction facing the scene object in the virtual scene, the bounding box information is used to indicate the size of the scene object in the virtual scene, and the cover point information indicates the recommended virtual character standing position when the virtual character approaches the scene object.
[0575] In an optional implementation of this embodiment, the processing module 820 is further configured to construct a spatial information library of the virtual scene according to visual labels of a plurality of scene objects in the virtual scene;
[0576] The acquisition module 810 is also used to acquire natural language commands;
[0577] The processing module 820 is further configured to filter out a first scene object corresponding to the natural language command from among a plurality of scene objects in the virtual scene based on the similarity between the natural language command and the visual tag in the spatial information library;
[0578] The processing module 820 is further configured to execute a virtual activity for the first scene object based on the natural language command.
[0579] In an optional implementation of this embodiment, the natural language command includes multiple input words; the processing module 820 is further used to:
[0580] Determining, one by one, a similarity score between each word in the plurality of input words and the visual label of the first scene object;
[0581] Determining a sum of a plurality of similarity scores as the similarity between the first scene object and the natural language command, wherein the plurality of similarity scores correspond one-to-one to the plurality of input words;
[0582] When the similarity between the first scene object and the natural language command exceeds the similarity between other scene objects in the spatial information library and the natural language command, the first scene object is determined as the object corresponding to the natural language command.
[0583] In an optional implementation of this embodiment, the processing module 820 is further used to construct a spatial information library of the virtual scene according to the appearance images of multiple scene objects in the virtual scene;
[0584] The acquisition module 810 is also used to acquire natural language commands;
[0585] The processing module 820 is further configured to call a similarity matching network to predict the similarity between the natural language command and the appearance images of the multiple scene objects in the spatial information library, and obtain a second scene object with the highest similarity to the natural language command among the multiple scene objects;
[0586] The processing module 820 is further configured to execute a virtual activity for the second scene object based on the natural language command.
[0587] One point that needs to be explained is that the device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example to implement its functions. In actual applications, the above-mentioned functions can be assigned to different functional modules according to actual needs, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above.
[0588] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method; the technical effects achieved by each module performing operations are the same as those in the embodiment of the method, and will not be elaborated here.
[0589] An embodiment of the present application also provides a computer device, which includes: a processor and a memory, wherein a computer program is stored in the memory; the processor is used to execute the computer program in the memory to implement the scene object processing method provided by the above-mentioned method embodiments.
[0590] Fig.16The structure block diagram of a terminal provided by an exemplary embodiment of the present application is shown. The terminal 1900 may be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer or a desktop computer. The terminal 1900 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.
[0591] Typically, the terminal 1900 includes: a processor 1901 and a memory 1902. The processor 1901 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1901 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1901 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1901 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1901 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.
[0592] The memory 1902 may include one or more computer-readable storage media, which may be non-transitory. The memory 1902 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1902 is used to store at least one instruction, which is used to be executed by the processor 1901 to implement the method for processing scene objects provided in the method embodiment of the present application.
[0593] In some embodiments, the terminal 1900 may also optionally include: a peripheral device interface 1903 and at least one peripheral device. The processor 1901, the memory 1902 and the peripheral device interface 1903 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1903 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1904, a touch display screen 1905, a camera assembly 1906, an audio circuit 1907 and a power supply 1908.
[0594] The peripheral device interface 1903 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1901 and the memory 1902. In some embodiments, the processor 1901, the memory 1902, and the peripheral device interface 1903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1901, the memory 1902, and the peripheral device interface 1903 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.
[0595] The radio frequency circuit 1904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1904 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 1904 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 1904 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1904 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.
[0596] The touch display screen 1905 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. The touch display screen 1905 also has the energy to collect touch signals on the surface or above the surface of the touch display screen 1905. The touch signal may be input as a control signal to the processor 1901 for processing. At this time, the touch display screen 1905 may also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the touch display screen 1905 may be one, and the front panel of the terminal 1900 is set; in other embodiments, the touch display screen 1905 may be at least two, which are respectively set on different surfaces of the terminal 1900 or are folded; in still other embodiments, the touch display screen 1905 may be a flexible display screen, which is set on a curved surface or a folded surface of the terminal 1900. Even, the touch display screen 1905 may also be set to a non-rectangular irregular figure, that is, a special-shaped screen. The touch display screen 1905 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).
[0597] The camera assembly 1906 is used to capture images or videos. Optionally, the camera assembly 1906 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting with VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1906 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.
[0598] The audio circuit 1907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1901 for processing, or input them into the radio frequency circuit 1904 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 1900. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1901 or the radio frequency circuit 1904 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1907 may also include a headphone jack.
[0599] The power supply 1908 is used to power various components in the terminal 1900. The power supply 1908 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1908 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery charged through a wired line, and a wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.
[0600] In some embodiments, the terminal 1900 further includes one or more sensors 1909 . The one or more sensors 1909 include, but are not limited to: an acceleration sensor 1910 , a gyroscope sensor 1911 , a pressure sensor 1912 , an optical sensor 1913 , and a proximity sensor 1914 .
[0601] The acceleration sensor 1910 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 1900. For example, the acceleration sensor 1910 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 1901 can control the touch display screen 1905 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 1910. The acceleration sensor 1910 can also be used to collect game or user motion data. The gyroscope sensor 1911 can detect the body direction and rotation angle of the terminal 1900, and the gyroscope sensor 1911 can cooperate with the acceleration sensor 1910 to collect the user's 3D action on the terminal 1900. The processor 1901 can implement the following functions based on the data collected by the gyroscope sensor 1911: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.
[0602] The pressure sensor 1912 can be set on the side frame of the terminal 1900 and / or the lower layer of the touch display screen 1905. When the pressure sensor 1912 is set on the side frame of the terminal 1900, it can detect the user's holding signal of the terminal 1900, and the processor 1901 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 1912. When the pressure sensor 1912 is set on the lower layer of the touch display screen 1905, the processor 1901 controls the operability controls on the UI interface according to the user's pressure operation on the touch display screen 1905. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.
[0603] The optical sensor 1913 is used to collect the ambient light intensity. In one embodiment, the processor 1901 can control the display brightness of the touch display screen 1905 according to the ambient light intensity collected by the optical sensor 1913. Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 1905 is increased; when the ambient light intensity is low, the display brightness of the touch display screen 1905 is reduced. In another embodiment, the processor 1901 can also dynamically adjust the shooting parameters of the camera assembly 1906 according to the ambient light intensity collected by the optical sensor 1913.
[0604] The proximity sensor 1914, also called a distance sensor, is usually arranged on the front panel of the terminal 1900. The proximity sensor 1914 is used to collect the distance between the user and the front of the terminal 1900. In one embodiment, when the proximity sensor 1914 detects that the distance between the user and the front of the terminal 1900 is gradually decreasing, the processor 1901 controls the touch display screen 1905 to switch from the screen-on state to the screen-off state; when the proximity sensor 1914 detects that the distance between the user and the front of the terminal 1900 is gradually increasing, the processor 1901 controls the touch display screen 1905 to switch from the screen-off state to the screen-on state.
[0605] Those skilled in the art will appreciate that the above structure does not limit the terminal 1900 and may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.
[0606] In an exemplary embodiment, a chip is also provided, which includes a programmable logic circuit and / or program instructions. When the chip runs on a computer device, it is used to implement the scene object processing method described in the above aspects.
[0607] In an exemplary embodiment, a computer program product is also provided, the computer program product includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor reads and executes the computer instructions from the computer-readable storage medium to implement the scene object processing method provided by each of the above method embodiments.
[0608] In an exemplary embodiment, a computer-readable storage medium is further provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the scene object processing method provided by the above-mentioned method embodiments.
[0609] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0610] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.
[0611] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.
Claims
1. A method for processing scene objects, It is characterized in that The method comprises: Acquire attribute text of a scene object in a virtual scene, wherein the attribute text is used to introduce inherent attributes of the scene object in the virtual scene; Acquire an appearance image of the scene object in the virtual scene, wherein the appearance image is used to describe the style of the scene object; A multimodal model is called to perform prediction on the attribute text and the appearance image of the scene object to obtain a visual label of the scene object, where the visual label is used to describe the visual features of the scene object in at least one dimension.
2. The method according to claim 1, It is characterized in that The multimodal model includes a visual question answering model, and calling the multimodal model to perform prediction on the attribute text and the appearance image of the scene object to obtain a visual label of the scene object includes: Constructing a question sentence for the appearance image, wherein the question sentence carries the attribute text of the scene object; The appearance image of the scene object and the question sentence are input into the visual question answering model to obtain an answer sentence, and the answer sentence is used as the visual label of the scene object.
3. The method according to claim 2, It is characterized in that The question statement includes at least two sub-statements; the inputting the appearance image of the scene object and the question statement into the visual question answering model to obtain an answer statement, and using the answer statement as the visual label of the scene object includes: Inputting the appearance image of the scene object and a first sentence of the at least two sub-sentences into the visual question answering model to obtain a first answer sub-sentence; Repeat the above steps until at least two answer sub-sentences corresponding to the at least two sub-sentences are obtained, wherein the at least two sub-sentences are used to inquire about the visual features of the appearance image from multiple dimensions; Sentence aggregation is performed on the at least two answer sub-sentences to extract the visual label of the scene object.
4. The method according to claim 2, It is characterized in that The step of constructing a question statement for the appearance image includes: Acquiring expected information of the scene object, the expected information being used to indicate a description dimension of the scene object expected in the visual tag and / or an expected format of the visual tag; constructing the question sentence of the appearance image according to the expected information and the attribute text; Among them, the first sub-part of the question sentence is the supplementary introduction information of the scene object, carrying the attribute text of the scene object; the second sub-part of the question sentence is the answer guidance sentence for the visual question and answer model, carrying the expected information.
5. The method according to claim 1, It is characterized in that The multimodal model includes a picture description model, and calling the multimodal model to perform prediction on the attribute text and the appearance image of the scene object to obtain a visual label of the scene object includes: Inputting the appearance image of the scene object into the picture description model to predict a description text of the scene object; Sentence aggregation is performed on the description text and the attribute text to extract the visual label of the scene object.
6. The method according to any one of claims 1 to 5, It is characterized in that The attribute text of the scene object includes at least one of the name of the scene object in the virtual scene and the size of the scene object in the virtual scene; And / or, the appearance image of the scene object includes images obtained by observing the scene object from at least two viewing angles.
7. The method according to any one of claims 1 to 5, It is characterized in that The method further comprises: The visual label of the scene object is rewritten in a pseudo-spoken manner to obtain a matching label that conforms to a spoken expression of a natural language.
8. The method according to claim 7, It is characterized in that The performing a pseudo-spoken rewriting on the visual label of the scene object to obtain a matching label that conforms to a spoken expression of a natural language includes: The visual label of the scene object is input into a natural language model to predict the matching label that conforms to the spoken expression of the natural language, and the natural language model carries prior knowledge of the spoken expression of the natural language.
9. The method according to claim 8, It is characterized in that The step of inputting the visual label of the scene object into a natural language model to predict the matching label that conforms to the spoken expression of the natural language includes: Obtaining a first sample label pair, the first sample label pair comprising a first label before being rewritten into a pseudo-colloquial language and a second label obtained after being rewritten into a pseudo-colloquial language; Constructing a rewriting guide sentence according to the first sample label pair and the visual label, wherein the rewriting guide sentence has natural semantics of rewriting the visual label with reference to the first sample label pair; The rewriting guide sentence is input into a natural language model to predict the matching label that conforms to the spoken expression of natural language.
10. The method according to any one of claims 1 to 5, It is characterized in that The method further comprises: The spatial position of the scene object in the virtual scene is acquired, and the spatial position of the scene object is determined as auxiliary information of the visual label of the scene object.
11. The method according to claim 10, It is characterized in that The obtaining of the spatial position of the scene object in the virtual scene comprises: Acquire at least one of the coordinate position, orientation information, bounding box information, and cover point information of the scene object in the virtual scene; Among them, the coordinate position is used to indicate the position of the scene object in the virtual scene, the orientation information is used to indicate the direction facing the scene object in the virtual scene, the bounding box information is used to indicate the size of the scene object in the virtual scene, and the cover point information indicates the recommended virtual character standing position when the virtual character approaches the scene object.
12. The method according to any one of claims 1 to 5, It is characterized in that The method further comprises: Constructing a spatial information library of the virtual scene according to visual labels of a plurality of scene objects in the virtual scene; Get natural language commands; Based on the similarity between the natural language command and the visual label in the spatial information library, screening out a first scene object corresponding to the natural language command from a plurality of scene objects in the virtual scene; Based on the natural language command, a virtual activity is performed on the first scene object.
13. The method according to claim 12, It is characterized in that The natural language command includes a plurality of input words; and based on the similarity between the natural language command and the visual label in the spatial information library, selecting a first scene object corresponding to the natural language command from a plurality of scene objects in the virtual scene includes: Determining, one by one, a similarity score between each word in the plurality of input words and the visual label of the first scene object; Determining a sum of a plurality of similarity scores as the similarity between the first scene object and the natural language command, wherein the plurality of similarity scores correspond one-to-one to the plurality of input words; When the similarity between the first scene object and the natural language command exceeds the similarity between other scene objects in the spatial information library and the natural language command, the first scene object is determined as the object corresponding to the natural language command.
14. The method according to any one of claims 1 to 5, It is characterized in that The method further comprises: Constructing a spatial information library of the virtual scene according to appearance images of a plurality of scene objects in the virtual scene; Get natural language commands; Calling a similarity matching network to predict the similarity between the natural language command and the appearance images of the multiple scene objects in the spatial information library, and obtaining a second scene object with the highest similarity to the natural language command among the multiple scene objects; Based on the natural language command, a virtual activity is performed on the second scene object.
15. A device for processing scene objects, It is characterized in that The device comprises: An acquisition module, used for acquiring attribute text of a scene object in a virtual scene, wherein the attribute text is used for introducing the inherent attributes of the scene object in the virtual scene; The acquisition module is further used to acquire an appearance image of the scene object in the virtual scene, wherein the appearance image is used to describe the style of the scene object; A processing module is used to call a multimodal model to perform prediction on the attribute text and the appearance image of the scene object to obtain a visual label of the scene object, wherein the visual label is used to describe the visual features of the scene object in at least one dimension.
16. A computer device, It is characterized in that The computer device includes: a processor and a memory, wherein the memory stores at least one program; the processor is used to execute the at least one program in the memory to implement the scene object processing method as described in any one of claims 1 to 14.
17. A computer-readable storage medium, It is characterized in that The readable storage medium stores executable instructions, and the executable instructions are loaded and executed by a processor to implement the scene object processing method as described in any one of claims 1 to 14.
18. A computer program product, It is characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. The processor reads and executes the computer instructions from the computer-readable storage medium to implement the scene object processing method as described in any one of claims 1 to 14.
Citation Information
Patent Citations
Text-based text visual question-answering system and method
CN112507727A
Static label identification model training method, static label identification method and device
CN116051921A
Display device and visual question and answer method
CN116301337A
Open type question and answer and multi-mode large model training method and device and related equipment
CN117235232A
Fish and vegetable symbiosis scene image visual question and answer method and device and electronic equipment
CN117668169A
Cited By
Real-time object interaction method and system based on expanded reality
CN120276606A
