A control method of virtual human driven by behavior tree
The control method for virtual humans using a behavior tree with CBT executor and thread pool management addresses scalability and stability issues, enabling efficient parallel execution and reducing user complexity in large-scale scenarios.
Patent Information
- Application Number
- CN202310940537.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-28
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2043-07-28
AI Technical Summary
Existing behavior tree algorithms face challenges in terms of scalability, deadlock, and deadlock, and lack of effective multi-threaded scheduling and resource management, which affect the scalability and stability of the system, leading to unstable system performance, especially in large-scale scenes.
A control method for virtual humans driven by a behavior tree, utilizing CBT executor and thread pool management to enable parallel execution, node state synchronization, and data sharing, and Mutex, and Semaphore nodes to handle concurrent operations, ensuring efficient execution and stability.
The method enhances scalability and stability by enabling efficient parallel execution of behavior trees, reducing execution time and memory usage, and ensuring reliable control of multiple virtual humans and events, while reducing user learning and operation complexity.
Smart Images

Figure CN117032957B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a control method for a behavior tree-driven virtual human. Background Art
[0002] The existing behavior tree algorithm can ensure that the behavior logic between the virtual human and the user is correct and smooth and natural, and can realize the functions of task allocation, action planning and decision-making of the virtual character, thereby ensuring that the behavior of the virtual human meets the user's expectations. However, the existing behavior tree algorithm still has the following technical problems:
[0003] 1. The performance may be insufficient when processing complex scenes. The behavior tree algorithm executes slowly and is difficult to meet the needs of large-scale scenes. When processing large-scale scenes, problems such as long execution time and high memory usage are prone to occur, resulting in unstable system performance.
[0004] 2. During the execution of the algorithm, there are situations where multiple nodes modify the status or access shared data at the same time, which is prone to problems such as contention and deadlock.
[0005] 3. The support for parallel processing in a multi-threaded environment is low, and there is a lack of effective multi-threaded scheduling and resource management mechanisms, which affects the scalability and stability of the system.
[0006] 4. In terms of behavior control, traditional methods usually use fixed rules or scripts to program the behavior of virtual humans, resulting in a lack of flexibility and scalability in behavior. Previous research has attempted to solve the above problems. For example, in terms of behavior control, some studies have adopted technologies such as state machines and finite state machines, but these methods require manual writing and adjustment of a large number of rules, which is very troublesome. Summary of the invention
[0007] In view of the above technical problems, the present technical solution provides a control method for a virtual human driven by a behavior tree. By constructing behavior nodes and logic nodes, a hierarchical behavior tree structure is formed to describe the behavior of the virtual human. The behavior tree can be used to control multiple people and multiple events. The CBT executor and thread pool management technology are introduced to solve the parallel execution of the behavior tree and the deadlock or blocking problems that occur during the processing. The CBT executor is designed based on the parallelization principle of the behavior tree algorithm and can efficiently execute the behavior tree in a multi-threaded environment. The executor can decompose the behavior tree into multiple subtrees for parallel execution, and can also handle issues such as node state synchronization and data sharing, so as to quickly complete the execution of the entire behavior tree. The above problems can be effectively solved.
[0008] The present invention is achieved through the following technical solutions:
[0009] A control method for a virtual human driven by a behavior tree, which obtains the user's voice through voice drive, converts the user's voice describing the behavior process into an event list with time sequence information, and builds a behavior tree by matching the virtual character action instructions, with the purpose of reproducing the scene described by the user so that the user can see the described virtual human scene in the whole process; the specific steps include:
[0010] Step 1: Construct nodes, including virtual human behavior nodes and logic nodes;
[0011] Step 2: Get the user's input voice from the driver;
[0012] Step 3: Recognize the acquired speech;
[0013] Step 4: Process text information and build an event list;
[0014] The text is divided into three elements: subject, object, and behavior; the subject refers to the performer of the action, and the object can be a location or an object; four word libraries Z, X, W, and S are established to store the subject title, behavior, location information, and time;
[0015] Step 5: Construct an event set based on the results obtained in the event list; the formula for the event set is:
[0016] Ai=N(z,x,w,s)
[0017] Where i∈[1,n], n represents the total number of events; Parallel, Mutex, and Semaphore node types are introduced to support concurrent operations. Events are added to the logic nodes of the behavior tree according to the order in which the events occur; if s in the event cannot be ignored and s=1, it means that the current action of the virtual person occurs at the same time as the action of another virtual person, then the Parallel node is used;
[0018] Step 6: Build the behavior tree executor CBTEngine; the specific operation method is:
[0019] Initialize CBTEngine and create a thread pool. CBTEngine creates a certain number of threads in advance and creates a runtime instance for each behavior tree. Scan the task queues of all behavior trees. If there are tasks in the queues, assign the tasks to idle threads for execution. During the execution, CBTEngine will check other behavior tree tasks that may be running concurrently. If necessary, Mutex and Semaphore nodes will be used to ensure the correct execution of mutual exclusion and synchronization operations. At the same time, CBTEngine also has an exception recovery mechanism that can handle errors in the execution of behavior tree tasks and notify related operations in a timely manner.
[0020] Step 7: Complete the control of the virtual human according to the behavior tree actuator.
[0021] Furthermore, the behavior nodes described in step 1 include behaviors related to turning, grabbing, running, walking, and lifting; the virtual human logic node controls whether the behavior node is activated, and the behavior nodes and logic nodes of the virtual human are hierarchically formed into a tree structure.
[0022] Furthermore, the specific operation method of step 3 is:
[0023] Step 3.1: Perform signal processing and feature extraction on the acquired speech;
[0024] Step 3.2: Select an acoustic model and train it;
[0025] Step 3.3: After the acoustic model gives the pronunciation sequence, the language model finds the string sequence with the highest probability from the candidate text sequences; finds the corresponding Chinese characters or words according to the dictionary;
[0026] Step 3.4: Use the decoder to find the word sequence; in the search space composed of knowledge sources such as acoustic models, dictionaries, and language models, use a certain search algorithm to find the word sequence with the highest probability, and finally output the text.
[0027] Furthermore, the signal processing and feature extraction described in step 3.1 are operated as follows: the Whisper model is used for processing, the speech is obtained for preprocessing, the input audio is divided into 30-second blocks, converted into a Mel spectrogram, feature extraction is performed, and then passed to the encoder.
[0028] Furthermore, the specific operation method of step 3.2 is as follows: the output of the encoder is normalized using a normalization layer, the decoder is trained to predict the corresponding text, and mixed with special tags, and the tag guide model is converted into text.
[0029] Furthermore, the specific operation method of step 4 is:
[0030] Step 4.1: Traverse the entire sentence, search for the subject title in the sentence, and match it with the title vocabulary of subject Z, and mark the matched result as z;
[0031] Step 4.2: Search for behavior. Use the same method as step 4.1 to search for the behavior in the sentence and match it with the behavior vocabulary X. Mark the matched result as x.
[0032] Step 4.3: Search the object location information. When retrieving non-subject titles, the location information is dynamic information and needs to be updated each time it is called. If it is static location information, match it with the location information word library W and mark the searched location information attribute as w.
[0033] Step 4.4: Search the time and determine whether the sentence contains words in the time vocabulary; if there is a matching result, match it with the information in the time vocabulary S and mark the matching result as s; otherwise, do not process it.
[0034] Beneficial Effects
[0035] The bearing fault diagnosis method based on the improved deep residual shrinkage network proposed in the present invention has the following beneficial effects compared with the prior art:
[0036] (1) This technical solution uses speech recognition technology to obtain the user's voice through voice drive, and converts the user's voice describing the processing behavior process into an event list with time sequence information. The user can naturally describe the desired interaction scenario without manually writing complex control instructions, thereby reducing the user's learning cost and operation difficulty.
[0037] (2) This technical solution constructs different types of nodes such as sequential nodes, selection nodes, parallel nodes, etc., and constructs behavioral nodes and logic nodes to form a hierarchical behavior tree structure to describe the behavior of the virtual human and define the behavior rules and logic of the virtual human; these nodes can be organized according to the priority and dependency of the tasks, thereby realizing complex behavior strategies.
[0038] (3) This technical solution introduces parallel computing to achieve efficient execution of behavior trees, solving the problem of slow execution speed of behavior tree algorithms; through behavior trees, multiple people and multiple events are controlled, and CBT executors are introduced to achieve parallel execution of behavior trees, and node state synchronization is processed; CBT executors are designed based on the parallelization principle of behavior tree algorithms, and can efficiently execute behavior trees in a multi-threaded environment. The executor can decompose the behavior tree into multiple subtrees for parallel execution, and can also handle issues such as node state synchronization and data sharing, so as to quickly complete the execution of the entire behavior tree, thereby ensuring reliability and stability. Thread pool management technology is also introduced to effectively manage concurrent threads, thereby ensuring reliability and stability, and solving the problem of competition and deadlock when multiple nodes modify the state or access shared data at the same time.
[0039] (4) This technical solution builds a behavior tree by matching the action instructions of the virtual character, with the purpose of reproducing the scene described by the user, so that the user can see the described virtual human scene throughout the process, so that the user can enhance the experience and immersion. The voice driver is used to obtain the user's voice, and then the voice recognition algorithm, behavior tree algorithm and other technologies are used to reproduce the virtual scene.
[0040] (5) This technical solution generates a behavior tree by combining the constructed nodes and event lists to form a complete behavior tree. The behavior tree will guide the virtual human to perform corresponding behaviors in the interactive environment and dynamically adjust and update according to external events and conditions. It can also be expanded and improved by using different technologies and algorithms to improve its performance and effect. For example, by introducing a state management mechanism, the behavior tree executor can track and manage the state of the virtual human to better control and adjust the behavior. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 It is the overall flow chart of the present invention.
[0042] Figure 2 This is the overall flow chart of the Whisper-based speech recognition method in step 3.
[0043] Figure 3 This is the construction diagram of the behavior tree in step 5.
[0044] Figure 4 This is the execution architecture diagram of the behavior tree in step 6.
[0045] Figure 5 This is a diagram of the behavior tree execution in step 6. DETAILED DESCRIPTION
[0046] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Under the premise of not departing from the design concept of the present invention, various modifications and improvements made by ordinary persons in the art to the technical solutions of the present invention should all fall within the protection scope of the present invention.
[0047] Embodiment 1:
[0048] like Figure 1 As shown, a control method of a virtual human driven by a behavior tree is provided. The user's voice is obtained through voice drive, and the voice describing the processing behavior process is converted into an event list with time sequence information. A behavior tree is constructed by matching the action instructions of the virtual human. The purpose is to reproduce the scene described by the user so that the user can see the described virtual human scene in the whole process. The method includes the following steps:
[0049] Step 1: Construct nodes; including virtual human behavior nodes and logic nodes: behavior nodes include turning, grabbing, running, walking, lifting, etc., and virtual human logic nodes control whether the behavior nodes are activated. The virtual human behavior nodes and logic nodes are layered to form a tree structure.
[0050] Step 2: Get the audio from the microphone driver.
[0051] Step 3: Recognize the acquired voice; the specific operation method is as follows:
[0052] Step 3.1: Perform signal processing and feature extraction on the acquired speech;
[0053] Using the Whisper model, speech is obtained for preprocessing. The input audio is divided into 30-second chunks, converted into Mel spectrograms, feature extraction is performed, and then passed to the encoder.
[0054] Step 3.2: Select an acoustic model and train it;
[0055] The output of the encoder is normalized using a normalization layer, and the decoder is trained to predict the corresponding text and mixed with special tokens that guide the model to convert to text.
[0056] Step 3.3: After the acoustic model gives the pronunciation sequence, the language model finds the string sequence with the highest probability from the candidate text sequences; and finds the corresponding Chinese characters or words according to the dictionary.
[0057] Step 3.4: Use the decoder to find the word sequence; in the search space composed of knowledge sources such as acoustic models, dictionaries, and language models, use a certain search algorithm to find the word sequence with the highest probability, and finally output the text.
[0058] Step 4: Based on the fact that text information can be used as interactive information between virtual people, the text is divided into subject, object, and behavior. The subject is the executor of the behavior, and the object can be a location or an object. Four lexicons are established and recorded as Z, X, W, and S. Z is the subject's name lexicon, X is the behavior lexicon, W is the location information lexicon, and S is the time lexicon that represents the virtual person's event.
[0059] Step 4.1: Traverse the entire sentence, search for the subject title in the sentence, and match it with the title vocabulary of the Z subject, and mark the matched result as z.
[0060] Step 4.2: Search for behavior. Use the same method as step 4.1 to search for the behavior in the sentence and match it with the behavior vocabulary X. The matched result is marked as x.
[0061] Step 4.3: Search object location information. When retrieving non-subject titles, the location information is dynamic information and needs to be updated each time it is called. If it is static location information, match it with the location information word library W and mark the searched location information attribute as w.
[0062] Step 4.4: Search the time and determine whether the sentence contains words in the time vocabulary. If there is a matching result, match it with the information in the time vocabulary S and mark the matching result as s; otherwise, do not process it.
[0063] Step 4.5: Traverse the entire sentence, search for the subject, behavior, location information and time, match them with the vocabulary, and mark the matching words as z, x, w, s respectively, otherwise do not process them.
[0064] Step 5: Generate a behavior tree from the event sequence, introduce new node types such as Parallel, Mutex, and Semaphore to support concurrent operations.
[0065] Step 5.1: Each sentence corresponds to an event. Based on the results obtained in step 4, construct the event Ai = N (z, x, w, s) to form an event set A i (where i∈[1,n], n represents the total number of events), and sets the node type.
[0066] Step 5.1.1: Build a Parallel node: First, define the input parameters of the Parallel node, such as a list of child nodes or a selection strategy. Inside the Parallel node, create a counter to track the completion status of the child nodes. Each child node needs to report to the counter after execution is completed and check the status of other child nodes. Then, determine whether all child nodes have been completed according to the set strategy. If so, return success; otherwise, continue waiting.
[0067] Step 5.1.2: Construct a Mutex node: First, define the input parameters of the Mutex node, such as the name or identifier of the resource to be protected. Inside the Mutex node, introduce a mutual exclusion lock mechanism to ensure that only one virtual person can obtain the lock. When a virtual person tries to obtain the lock, if the lock is already occupied, the virtual person needs to wait until the lock is released. Only the virtual person who successfully obtains the lock can perform actions related to the resource.
[0068] Step 5.1.3: Build a Semaphore node: First, define the input parameters of the Semaphore node, such as the initial counter value and the maximum allowed concurrency. Inside the Semaphore node, a counter is maintained to track the number of virtual people who have currently acquired the lock. When a virtual person tries to acquire the semaphore, if the number of virtual people who have currently acquired the lock reaches the maximum concurrency, the virtual person needs to wait until another virtual person releases the semaphore. After acquiring the semaphore, the virtual person can perform actions related to the resource and release the semaphore after completion.
[0069] Step 5.2: Add the events to the logic nodes of the behavior tree in the order in which they occur. If the s in the event cannot be ignored and s=1, it means that the current action of the virtual person occurs at the same time as the action of another virtual person, then it is a Parallel node.
[0070] Step 6: Build the Behavior Tree Executor (CBTEngine), including the initialization module and thread pool management method to achieve concurrent execution of Behavior Tree tasks.
[0071] Step 6.1: Initialize it and create a thread pool. CBTEngine pre-creates a certain number of threads at startup, creates a thread pool, and creates a runtime instance for each behavior tree.
[0072] Step 6.2: Scan the task queues of all behavior trees. If there are tasks in the queue, take out the tasks for execution and return the results to the behavior tree.
[0073] Step 6.2.1: In the task queue in step 6.2, if the nodes in the behavior tree need to be executed in parallel, the task is submitted to the thread pool, and the thread pool will automatically select an idle thread to process the task.
[0074] Step 6.2.2: In the task queue in step 6.2, if the nodes in the behavior tree need to be synchronized or coordinated, message communication is used to achieve this.
[0075] Step 6.2.3: In the task queue in step 6.2, if there is a running error, the exception information is returned to the behavior tree and an attempt is made to restore the state of the node.
[0076] Step 7: Control the virtual person;
[0077] Step 7.1: Complete the control of the virtual human according to the behavior tree actuator.
[0078] Step 7.2: Execute the virtual human behavior nodes defined in the behavior tree to drive the virtual human to perform corresponding actions in the scene.
[0079] Step 7.3: Implement process control and interactive response of virtual human behavior by scheduling and executing the logic nodes in the behavior tree.
[0080] This embodiment uses voice driver to obtain the user's voice, converts the user's voice describing the process of processing behavior into an event list with time sequence information, and builds a behavior tree by matching the virtual character action instructions, with the purpose of reproducing the scene described by the user, so that the user can see the described virtual human scene throughout the process, so that the user can enhance the experience and immersion. The key to this method is to use voice driver to obtain user voice, and then use voice recognition algorithm, behavior tree algorithm and other technologies to reproduce the virtual scene.
[0081] Using speech recognition technology and behavior tree algorithm, the processing behavior process described by the user is automatically converted into a list of events arranged in chronological order, and a behavior tree is constructed by matching the action instructions of the virtual character to reproduce the scene described by the user. Speech recognition technology is the thinking tool of the entire system, which converts the user's speech description into a processable sequence of events; the behavior tree algorithm is the key factor to ensure that the behavior logic between the virtual human and the user is correct and smooth and natural. It allows users to describe the desired interactive scene through natural speech without having to manually write complex control instructions, thereby reducing the user's learning cost and operating difficulty. This technology has broad application prospects in the fields of smart education, human-computer interaction, etc.
[0082] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent transformation or modification made according to the essence of the present invention should be included in the protection scope of the present invention.
Claims
1. A control method for virtual human driven by behavior tree, Features: The user's voice is acquired through voice drive, and the voice describing the processing behavior process is converted into an event list with time sequence information. The behavior tree is constructed by matching the action instructions of the virtual character, so as to reproduce the scene described by the user and allow the user to see the described virtual human scene throughout the process. The specific steps include: Step 1: Construct nodes, including virtual human behavior nodes and logic nodes; Step 2: Get the user's input voice from the driver; Step 3: Recognize the acquired speech; Step 4: Process text information and build an event list; The text is divided into three elements: subject, object, and behavior; the subject refers to the performer of the action, and the object is the location or object; four word libraries Z, X, W, and S are established to store the subject's title, behavior, location information, and time; among them, Z is the subject's title word library, X is the behavior word library, W is the location information word library, and S is the time word library representing virtual human events; Step 5: Construct an event set based on the results obtained in the event list; the formula for the event set is: ; in, , n represents the total number of events; is an event set, z is the matching result tag obtained by matching with the Z subject’s title vocabulary, is the matching result tag obtained by matching with the behavior vocabulary X, is the matching result tag obtained by matching with the location information word library W, and s is the matching result tag obtained by matching with the information in the time word library S; Introduce Parallel, Mutex, and Semaphore node types to support concurrent operations. Add events to the logic nodes of the behavior tree according to the order in which the events occur. If the s in the event cannot be ignored and s=1, it means that the current action of the virtual person occurs at the same time as the action of another virtual person, then use the Parallel node. The specific operation method is as follows: Step 5.1: Each sentence corresponds to an event. According to the results obtained in step 4, construct the event , forming an event set A i , and set the node type; Step 5.1.1: Build the Parallel node: First, define the input parameters of the Parallel node, including the child node list or selection strategy; inside the Parallel node, create a counter to track the completion status of the child nodes; each child node needs to report to the counter after execution is completed and check the status of other child nodes; then determine whether all child nodes have been completed according to the set strategy, if so, return success; if not, continue to wait; Step 5.1.2: Construct a Mutex node: First, define the input parameters of the Mutex node, including the name or identifier of the resource to be protected; introduce a mutual exclusion lock mechanism inside the Mutex node to ensure that only one virtual person can obtain the lock; when a virtual person tries to obtain the lock, if the lock is occupied, the virtual person needs to wait until the lock is released; only the virtual person who successfully obtains the lock can perform actions related to the resource; Step 5.1.3: Build a Semaphore node: First, define the input parameters of the Semaphore node, including the initial counter value and the maximum allowed concurrency. Inside the Semaphore node, maintain a counter to track the number of virtual people who have currently acquired the lock. When a virtual person tries to acquire the semaphore, if the number of virtual people who have currently acquired the lock reaches the maximum concurrency, the virtual person needs to wait until another virtual person releases the semaphore. After acquiring the semaphore, the virtual person can perform actions related to the resource and release the semaphore after completion. Step 5.2: Add the events to the logic nodes of the behavior tree in the order in which they occur. If the s in the event cannot be ignored and s=1, it means that the current action of the virtual person occurs at the same time as the action of another virtual person, then it is a Parallel node. Step 6: Build the behavior tree executor CBTEngine; the specific operation method is: Initialize CBTEngine and create a thread pool. CBTEngine creates a certain number of threads in advance and creates a runtime instance for each behavior tree. Scan the task queues of all behavior trees. If there are tasks in the queues, assign the tasks to idle threads for execution. During the execution, CBTEngine will check other behavior tree tasks that may be running concurrently. If necessary, Mutex and Semaphore nodes will be used to ensure the correct execution of mutual exclusion and synchronization operations. At the same time, CBTEngine also has an exception recovery mechanism that can handle errors in the execution of behavior tree tasks and notify related operations in a timely manner. Step 7: Complete the control of the virtual human according to the behavior tree actuator.
2. According to the control method of a behavior tree driven virtual human according to claim 1, Features: The behavior nodes described in step 1 include behaviors related to turning, grabbing, running, walking, and lifting; the virtual human logic node controls whether the behavior node is activated, and the behavior nodes and logic nodes of the virtual human are hierarchically formed into a tree structure.
3. According to the control method of a behavior tree driven virtual human as described in claim 1, Features: The specific operation method of step 3 is: Step 3.1: Perform signal processing and feature extraction on the acquired speech; Step 3.2: Select an acoustic model and train it; Step 3.3: After the acoustic model gives the pronunciation sequence, the language model finds the string sequence with the highest probability from the candidate text sequences; finds the corresponding Chinese characters or words according to the dictionary; Step 3.4: Use the decoder to find the word sequence; in the search space composed of the acoustic model, dictionary and language model knowledge source, use a certain search algorithm to find the word sequence with the highest probability, and finally output the text.
4. The control method of a behavior tree driven virtual human according to claim 3, Features: The signal processing and feature extraction described in step 3.1 are operated as follows: using the Whisper model for processing, the speech is obtained for preprocessing, the input audio is divided into 30-second blocks, converted into a Mel spectrogram, feature extraction is performed, and then passed to the encoder.
5. The control method of a behavior tree driven virtual human according to claim 4, Features: The specific operation method of step 3.2 is as follows: the output of the encoder is normalized using a normalization layer, the decoder is trained to predict the corresponding text, and mixed with special tags, and the tag guide model is converted into text.
6. The control method of a behavior tree driven virtual human according to claim 1, Features: The specific operation method of step 4 is: Step 4.1: Traverse the entire sentence, search for the subject title in the sentence, and match it with the title vocabulary of the Z subject, and mark the matched result as z; Step 4.2: Search for behavior. Use the same method as step 4.1 to search for the behavior in the sentence and match it with the behavior vocabulary X. Mark the matched result as x. Step 4.3: Search the object location information. When retrieving non-subject titles, the location information is dynamic information and needs to be updated each time it is called. If it is static location information, match it with the location information word library W and mark the searched location information attribute as w. Step 4.4: Search the time to determine whether the sentence contains words in the time vocabulary; if there is a matching result, match it with the information in the time vocabulary S and mark the matching result as s; Otherwise no processing is performed.
