An intelligent virtual human assisted maintenance method and system based on SAC algorithm and FSM

Abstract

The application discloses an intelligent virtual human assisted maintenance method and system based on SAC algorithm and FSM, and belongs to the virtual human simulation field. The application combines deep reinforcement learning and finite state machine to design the autonomous maintenance behavior of the virtual human. The simple actions such as walking, running, standing and turning of the virtual human are controlled by using the state machine, and the complex actions such as maintenance are trained by using the deep reinforcement learning. Compared with the traditional virtual human assisted maintenance design, the application avoids excessive human intervention, reduces the system running memory consumption, effectively improves the maintenance efficiency, makes the virtual human have the fidelity, and effectively avoids the shortcomings of the state machine and the behavior tree method. The virtual human visual perception model based on the multi-Cube trigger is used, and the problems of the scene unable to be deeply optimized and the missed objects caused by the ray detection are avoided. The application makes the virtual human more intelligent, helps to enhance the reality and credibility of the virtual human, and improves the creativity, immersion and real-time interactivity of the virtual maintenance operation.

Description

Technical Field

[0001] This invention belongs to the field of virtual human simulation, and more specifically, relates to an intelligent virtual human-assisted maintenance method and system based on the SAC (Soft Actor Critic) algorithm and FSM (Finite-State Machine). Background Technology

[0002] Virtual humans represent the geometric and behavioral characteristics of humans in a computer-generated space and can be widely applied in fields such as human-computer interaction, human-computer efficiency, and gaming. With the rapid development of virtual reality technology, digital prototypes are gradually replacing physical models and full-scale prototypes used for engineering analysis during the development process. Virtual humans are also being introduced to fully consider factors such as maintainability and human-computer efficiency during the design phase. Maintenance simulation methods based on virtual humans replace physical prototypes with virtual prototypes and actual maintenance personnel with virtual humans. These virtual humans are endowed with a certain degree of autonomy, capable of perceiving the environment, planning a reasonable and optimized path, following it, and making decisions and executing maintenance actions upon reaching the destination. This achieves a visualized simulation of human maintenance of equipment.

[0003] Currently, virtual human behavior control and decision-making technologies are based on algorithms such as behavior trees, motion capture, finite state machines, or deep reinforcement learning (CN108803615B). However, behavior tree methods start from the root in each frame, potentially visiting all nodes and consuming more CPU; furthermore, even simple operations require node manipulation, making them extremely complex. Motion capture methods offer good accuracy, but have high system costs, cumbersome production processes, and significant post-processing workload. Finite state machine methods are not easily extensible, as new logic requires new states. In the field of virtual human-assisted maintenance, reinforcement learning algorithms face challenges such as high interaction with the environment and high sample complexity, leading to long training times for the agent and increased difficulty in algorithm convergence.

[0004] In virtual maintenance simulation, accessibility and operational space analysis primarily rely on collision detection between the virtual human and products, and between tools and products, to restrict the virtual human's movement and check whether the restricted virtual human has completed the maintenance work. Currently, in the field of virtual maintenance simulation, Unity's range detection and collision detection mostly use raycasting technology. This detection method consumes a large amount of computational resources and has the potential for missed detections. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide an intelligent virtual human-assisted maintenance method and system based on the SAC algorithm and FSM, which aims to solve the problems of low efficiency and high computational resource consumption in traditional virtual maintenance.

[0006] To achieve the above objectives, in a first aspect, the present invention provides an intelligent virtual human-assisted maintenance method based on the SAC algorithm and FSM, the method comprising:

[0007] S1. Obtain the 3D model of the virtual human, the 3D model of the object to be repaired, the 3D model of the repair tools, and the 3D model of the repair scene. Import each 3D model into the graphical development environment so that the virtual human, the object to be repaired, and the repair tools are in the repair scene.

[0008] S2. Set the virtual human's position in the maintenance scene as the starting point and the maintenance operation area as the ending point, perform path planning, and realize the virtual human's standing, walking, running, and turning actions based on FSM. The maintenance operation area is the object to be maintained and the designated space around it.

[0009] S3. Following the planned path, the virtual human is moved to the maintenance operation area by walking and running. During the movement, collision detection is performed based on the visual perception model. If the detected object is an obstacle, the virtual human avoids the obstacle by turning around. After the virtual human moves to the maintenance operation area, the visual perception model is used to detect whether there is a maintenance object in the field of vision. If not, the collision detection function of the field of vision perception model is called. The virtual human first adjusts its posture by turning around until the maintenance object can be perceived in the field of vision. Then, the angle detection function is called to continue to perform the turning action to fine-tune the angle between the front of the virtual human and the maintenance object. If the object exists, the angle detection function is called directly to perform the turning action to fine-tune the angle between the front of the virtual human and the maintenance object until the angle between the front of the virtual human and the maintenance object does not exceed the preset angle, and then proceed to S4.

[0010] S4. Based on the visual perception model, optimize the intelligent decision-making model using the SAC algorithm to determine the maintenance behavior of the virtual human.

[0011] Preferably, the implementation of the virtual human's standing, walking, running, and turning movements based on FSM is as follows:

[0012] The virtual human finite state machine can be represented as a quintuple M = (A, S, T, W, q0), where,

[0013] A represents a finite set of states, which is the set of all different state sequences of a virtual human, including four states: standing, walking, running, and turning around.

[0014] S represents the triggering input events for the virtual human's state transition, including perceiving obstacles, being on the planned path, arriving at the designated maintenance area, the angle between the maintenance object and the virtual human's front, and the time spent walking;

[0015] W represents the state transition function, which causes the state machine to transition from its current state to another state;

[0016] q0 represents the initial state of the virtual human state machine, where the initial state is station;

[0017] T represents a non-empty set of state transitions, defined as a quintuple T = (SState, TState, Constraint, Input-event, Action). SState represents the current state of the virtual human, TState represents the state the virtual human will become, Constraint represents the constraints of the triggering condition, and Input-event represents the condition that triggers the state transition. When this condition is met, the virtual human state machine will undergo a state transition, including perceiving an obstacle, avoiding the obstacle and returning to the planned path, the walking time reaching the preset time, the virtual human reaching the end of the planned path, and the angle between the virtual human's front and the repair object being less than or equal to the preset angle. Action represents the action executed after the state transition.

[0018] Preferably, when the virtual human begins to perform a task, the state machine is switched to walking, and every two seconds the state is switched to running. If the visual perception model detects an obstacle while walking or running, the state is switched to turning around. After avoiding the obstacle and returning to the planned route, the state is switched back to walking or running. When the virtual human reaches the designated maintenance area, the state machine is switched to standing, and the angle between the virtual human's front and the maintenance object is determined. If it is greater than a preset angle, the state is switched to turning around. If it is less than or equal to the preset angle, the state is switched back to standing. If it is less than or equal to the preset angle, the state machine remains unchanged.

[0019] Preferably, the visual perception model includes three Cube triggers, wherein the middle Cube trigger is arranged directly in front of the virtual person, and the two Cube triggers on the sides are symmetrically arranged in front of the left and right sides of the virtual person, so as to form a 60° fan-shaped field of view within the trigger range of the Cube triggers; the radius of the Cube trigger is 2 meters.

[0020] Preferably, step S4 includes:

[0021] (1) Use the ML-Agents component of Unity3D to build a deep reinforcement learning environment, initialize the Critic network parameters ω1 and ω2, the Actor network parameters θ and the experience pool D; modify the algorithm configuration file to set the maximum number of steps Max-Step to 10 million and the network layer num_layer to 8 layers.

[0022] (2) Add a base agent script, Agents, to the virtual human. The virtual human collects the three-dimensional spatial position vector P based on the observation information of the visual perception model. r The angle θ between the virtual human's front and the object being repaired r, Repair tool three-dimensional spatial position vector T r and the three-dimensional spatial position vector C of the repaired parts r And define the state space as The virtual human's hand repair actions are defined as a table, and the specific actions provided to the virtual human are indexes of this table, defining an action space. Where AC0, AC1, and AC2 represent hand grasping, hand releasing, and hand movement operations, respectively;

[0023] (3) Obtain observation information for the current step based on the virtual human visual perception model. and operation target T g The virtual human selects the appropriate action to perform. The reward R is obtained through the reward and punishment function. At this point, the state of the observed information is transformed. Will T g , R and Combine into a tuple And store it in experience pool D;

[0024] (4) The virtual human completes the operation in the manner of step (3) until the action on the object is completed. A batch of data of size N is randomly sampled from the experience pool D to update the Critic network parameters ω1, ω2 and the Actor network parameters θ, and then the target network is updated.

[0025] (5) Repeat steps (3) and (4) until the iteration reaches the maximum number of steps Max-Step, then terminate the training. After training is complete, copy the model into Unity.

[0026] Preferably, the reward function R is defined as follows:

[0027]

[0028] Where, r t 'r' represents the reward value immediately gained when the moved position is closer to the target than the previous position; 'r' is a positive value. 'r' represents the reward value gained every 0.5 seconds when the virtual human is in the action state; 'r' is a negative value. a This represents the reward value immediately obtained when the virtual character is greater than the safe distance from the nearest obstacle; the reward value is defined as positive. noa This represents the reward value immediately obtained when the virtual character's distance from the nearest obstacle is less than or equal to the safe distance; the reward value is conventionally negative. o This represents the reward value immediately obtained when the captured object becomes the target of the operation; the reward value is defined as positive. nooThis represents the reward value immediately obtained when the captured object is not the object to be operated on; the reward value is conventionally negative. p This represents the reward value immediately obtained when the object is moved to a specified area or into the user's hand; the reward value is defined as positive. nop This represents the reward value immediately obtained when the object being operated on is not moved to the designated area or into the user's hands, or when the operation is not completed within the specified time; the reward value is negative. f(t) d represents the position of the virtual human's hand relative to the object being manipulated during the current sampling period. f(t-1) The previous sampling period defines the position of the virtual human's hand from the object being manipulated; time represents the timing period when the virtual human begins its operation; d Smin This represents the safe distance between the virtual human's hand and the non-operational model; T represents the object the virtual human is grasping. target This indicates that the object being grabbed by the virtual human is the object being manipulated; Grab Collider The collision box representing the virtual human's hand, Target Collider NoTarget represents the collision box of the object being manipulated. Collider The collision box represents the non-operated object.

[0029] To achieve the above objectives, in a second aspect, the present invention provides an intelligent virtual human-assisted maintenance system based on the SAC algorithm and FSM, comprising: a processor and a memory; the memory for storing computer execution instructions; and the processor for executing the computer execution instructions, such that the method described in the first aspect is executed.

[0030] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:

[0031] This invention proposes an intelligent virtual human-assisted maintenance method and system based on the SAC algorithm and FSM. It combines finite state machines and deep reinforcement learning (SAC). Simple movements such as walking, running, standing, and turning of the virtual human are controlled using finite state machines, while complex movements such as maintenance are trained using deep reinforcement learning. This effectively improves the efficiency of virtual maintenance and maintenance operation training, significantly reducing the time and economic costs of existing maintenance training, and provides valuable insights for the development of future virtual maintenance systems. Furthermore, this invention employs a multi-Cube trigger virtual human visual perception model, avoiding the problems of insufficient scene optimization and missed object detection inherent in ray detection. This invention makes the virtual human more intelligent, enhancing its realism and credibility, and improving the conceptualization, immersion, and real-time interactivity of virtual maintenance operations. Attached Figure Description

[0032] Figure 1 The present invention provides a flowchart of an intelligent virtual human-assisted maintenance method based on SAC algorithm and FSM.

[0033] Figure 2 This is a schematic diagram illustrating the specific steps of this embodiment. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0035] Figure 1 This invention provides a flowchart of an intelligent virtual human-assisted maintenance method based on the SAC algorithm and FSM. (See flowchart for example.) Figure 1 As shown, the method includes:

[0036] Step S1. Obtain the 3D model of the virtual human, the 3D model of the object to be repaired, the 3D model of the repair tools, and the 3D model of the repair scene. Import each 3D model into a graphical representation. Development Environment This allows virtual humans, objects to be repaired, and repair tools to be placed within the repair scenario.

[0037] The outline of the virtual human model was built using 3ds Max polygon modeling software, and the virtual human skeleton was created. Skinning was then applied. Based on the design drawings, photos, and videos of the repair object, repair tools, and repair scene, the appearance and specific dimensions of the modeling object were obtained, along with the training environment required for modeling. The model obtained was then imported into Unity.

[0038] Step S2. Set the virtual human's position in the maintenance scene as the starting point and the maintenance operation area as the ending point, perform path planning, and realize the virtual human's standing, walking, running, and turning actions based on FSM. The maintenance operation area is the object to be maintained and the designated space around it.

[0039] Preferably, the implementation of the virtual human's standing, walking, running, and turning movements based on FSM is as follows:

[0040] The virtual human finite state machine can be represented as a quintuple M = (A, S, T, W, q0), where,

[0041] A represents a finite set of states, which is the set of all different state sequences of a virtual human, including four states: standing, walking, running, and turning around.

[0042] S represents the triggering input events for the virtual human's state transition, including perceiving obstacles, being on the planned path, arriving at the designated maintenance area, the angle between the maintenance object and the virtual human's front, and the time spent walking;

[0043] W represents the state transition function, which causes the state machine to transition from its current state to another state;

[0044] q0 represents the initial state of the virtual human state machine, where the initial state is station;

[0045] T represents a non-empty set of state transitions, defined as a quintuple T = (SState, TState, Constraint, Input-event, Action). SState represents the current state of the virtual human, TState represents the state the virtual human will become, Constraint represents the constraints of the triggering condition, and Input-event represents the condition that triggers the state transition. When this condition is met, the virtual human state machine will undergo a state transition, including perceiving an obstacle, avoiding the obstacle and returning to the planned path, the walking time reaching the preset time, the virtual human reaching the end of the planned path, and the angle between the virtual human's front and the repair object being less than or equal to the preset angle. Action represents the action executed after the state transition.

[0046] Preferably, when the virtual human begins to perform a task, the state machine is switched to walking, and every two seconds the state is switched to running. If the visual perception model detects an obstacle while walking or running, the state is switched to turning around. After avoiding the obstacle and returning to the planned route, the state is switched back to walking or running. When the virtual human reaches the designated maintenance area, the state machine is switched to standing, and the angle between the virtual human's front and the maintenance object is determined. If it is greater than a preset angle, the state is switched to turning around. If it is less than or equal to the preset angle, the state is switched back to standing. If it is less than or equal to the preset angle, the state machine remains unchanged.

[0047] Step S3. Following the planned path, the virtual human is moved to the maintenance operation area by walking and running. During the movement, collision detection is performed based on the visual perception model. If the detected object is an obstacle, the virtual human avoids the obstacle by turning around. After the virtual human moves to the maintenance operation area, the visual perception model detects whether there is a maintenance object within its field of vision. If not, the collision detection function of the field of vision model is called. The virtual human first adjusts its posture by turning around until it can perceive the maintenance object within its field of vision. Then, the angle detection function is called to continue turning around to fine-tune the angle between the virtual human's front and the maintenance object. If the object exists, the angle detection function is called directly to perform turning around to fine-tune the angle between the virtual human's front and the maintenance object until the angle between the virtual human's front and the maintenance object does not exceed the preset angle, and then proceed to S4.

[0048] Preferably, the visual perception model includes three Cube triggers, wherein the middle Cube trigger is arranged directly in front of the virtual person, and the two Cube triggers on the sides are symmetrically arranged in front of the left and right sides of the virtual person, so as to form a 60° fan-shaped field of view within the trigger range of the Cube triggers; the radius of the Cube trigger is 2 meters.

[0049] Step S4. Based on the visual perception model, optimize the intelligent decision-making model using the SAC algorithm to determine the virtual human's maintenance behavior.

[0050] Preferably, step S4 includes:

[0051] (1) Use the ML-Agents component of Unity3D to build a deep reinforcement learning environment, initialize the Critic network parameters ω1 and ω2, the Actor network parameters θ and the experience pool D; modify the algorithm configuration file to set the maximum number of steps Max-Step to 10 million and the network layer num_layer to 8 layers.

[0052] (2) Add a base agent script, Agents, to the virtual human. The virtual human collects the three-dimensional spatial position vector P based on the observation information of the visual perception model. r The angle θ between the virtual human's front and the object being repaired r , Repair tool three-dimensional spatial position vector T r and the three-dimensional spatial position vector C of the repaired parts r And define the state space as The virtual human's hand repair actions are defined as a table, and the specific actions provided to the virtual human are indexes of this table, defining an action space. Where AC0, AC1, and AC2 represent hand grasping, hand releasing, and hand movement operations, respectively;

[0053] (3) Obtain observation information for the current step based on the virtual human visual perception model. and operation target T g The virtual human selects the appropriate action to perform. The reward R is obtained through the reward and punishment function. At this point, the state of the observed information is transformed. Will T g , R and Combine into a tuple And store it in experience pool D;

[0054] (4) The virtual human completes the operation in the manner of step (3) until the action on the object is completed. A batch of data of size N is randomly sampled from the experience pool D to update the Critic network parameters ω1, ω2 and the Actor network parameters θ, and then the target network is updated.

[0055] (5) Repeat steps (3) and (4) until the iteration reaches the maximum number of steps Max-Step, then terminate the training. After training is complete, copy the model into Unity.

[0056] Preferably, the reward function R is defined as follows:

[0057]

[0058] Where, r t 'r' represents the reward value immediately gained when the moved position is closer to the target than the previous position; 'r' is a positive value. 'r' represents the reward value gained every 0.5 seconds when the virtual human is in the action state; 'r' is a negative value. a This represents the reward value immediately obtained when the virtual character is greater than the safe distance from the nearest obstacle; the reward value is defined as positive. noa This represents the reward value immediately obtained when the virtual character's distance from the nearest obstacle is less than or equal to the safe distance; the reward value is conventionally negative. o This represents the reward value immediately obtained when the captured object becomes the target of the operation; the reward value is defined as positive. noo This represents the reward value immediately obtained when the captured object is not the object to be operated on; the reward value is conventionally negative. p This represents the reward value immediately obtained when the object is moved to a specified area or into the user's hand; the reward value is defined as positive. nop This represents the reward value immediately obtained when the object being operated on is not moved to the designated area or into the user's hands, or when the operation is not completed within the specified time; the reward value is negative. f(t) d represents the position of the virtual human's hand relative to the object being manipulated during the current sampling period. f(t-1) The previous sampling period defines the position of the virtual human's hand from the object being manipulated; time represents the timing period when the virtual human begins its operation; d Smin This represents the safe distance between the virtual human's hand and the non-operational model; T represents the object the virtual human is grasping. target This indicates that the object being grabbed by the virtual human is the object being manipulated; Grab Collider The collision box representing the virtual human's hand, Target Collider NoTarget represents the collision box of the object being manipulated. Collider The collision box represents the non-operated object.

[0059] This invention provides an intelligent virtual human-assisted maintenance system based on the SAC algorithm and FSM, comprising: a processor and a memory; the memory for storing computer execution instructions; and the processor for executing the computer execution instructions, thereby causing the above-mentioned method to be executed.

[0060] Example

[0061] Figure 2 This is a schematic diagram illustrating the specific steps of this embodiment.

[0062] Step 1: 3D model building.

[0063] Step 1.1: Based on the H-Anim human modeling standard, use polygon modeling in 3DMax to model the virtual human model, use Turbo Smooth to subdivide the head, and perform detailed modeling of the human's facial features, hands, and feet.

[0064] Step 1.2: Create 4K high-definition character skin and clothing textures in Substance Painter;

[0065] Step 1.3: Uncover the UVs for the character model, add a normal map, and bake;

[0066] Step 1.4: Use the 3ds Max built-in skinning system to bind the Bipe (skeleton) to the character model, assign weights to the skeleton, enable keyframes, and set the skeleton animation;

[0067] Step 1.5: Based on the design drawings, photos and videos of the repair object, repair tools and repair scene, obtain the appearance and specific dimensions of the modeling object, use 3DMax to build a three-dimensional model and assign V-Ray material to the model;

[0068] Step 1.6: Export all models as FBX format, and check the Animation, Deformation and Geometry Smoothing groups when exporting.

[0069] Step 2: Import the model into Unity, set up NavMesh Agent for the virtual human to perform path planning, and set the maintenance operation space as the target point.

[0070] Import the FBX format virtual human model and training environment model into the 3D software Unity. Add Mesh Collider components and Box collision boxes to all models. Attach Nav Mesh Agent components to the virtual human. Click Bake in Navigation, select the block for which you want to bake the path, and set it to Static. Set the specified maintenance operation area as the target point. Update and calculate the path by calling SetDestination() and Move() functions. Call Raycast() and NavMeshHit functions to determine occlusion. Obtain the path calculation cost when crossing a specific type of area using Set&GetAreaCost(). Finally, complete the virtual human's path planning.

[0071] Step 3: Design of virtual human autonomous decision-making behavior.

[0072] Step 3.1: Add three Cube triggers to the front of the virtual human body. The middle Cube trigger is placed directly in front of the virtual human, and the two Cube triggers on both sides are placed diagonally with an angle of 60° to present a roughly 60° fan-shaped field of view. Do not check Mesh Renderer, but check Box Collider.

[0073] Step 3.2: Import the ML-Agents toolkit into Unity and configure the relevant development environment. Set the Tag of the repair tool model to Tool, the Tag of the virtual human to AgentPlayer, the Tag of the repair subject object to RepairObjects, the Tag of the repair parts to Components, and other collision-related models to Barrier.

[0074] Step 3.3: Implement basic virtual human actions based on a finite state machine. Write a state class to define the virtual human's standing, walking, running, and turning actions. In the state class, write the transition conditions from walking / running to turning. Define a state management class to manage the virtual human's states. Write scripts based on the defined state classes to implement the virtual human's standing, walking, running, and turning action logic. When the virtual human walks or runs into the designated maintenance operation space, collision detection is used to determine if Repair objects, Components, and Tools are within the range of the Cube trigger. Tag is used to determine if there are no other objects obstructing the view. This indicates whether Repair objects, Components, and Tools are within the virtual human's visual perception range. The Quaternion.Angle() method is called to calculate the angle between the virtual human and the centerline of the Cube trigger. If it is less than or equal to 30°, the Repair objects are at the optimal viewing angle, movement stops, and preparation for maintenance begins. If the angle is greater than 30°, the state is changed to turning, the angle is adjusted to less than or equal to 30°, and maintenance preparation is completed.

[0075] Step 3.4: Using the repair tools and parts within the field of vision as the operation objects, add a training script for the virtual human. The script needs to inherit the Agent class, apply the Initialize Agent() method to initialize environment parameters, add the Behavior Parameters component, modify the Space Size of this component to 30, and call the AddVectorObs method in the CollectObservations() function to add the position information of the operation object, the angle between the operation object and the center line of the Cube trigger, and the Barrier information.

[0076] Step 3.5: Call the AgentAction function to receive the environment information configured in Step 3.4 and write the reward and punishment events during the training process. During the repair process, the virtual human needs to move tools and parts to a designated area or hand them to the user. If the virtual human completes hand grasping of the repair tool and repair part, and performs the hand retraction action after completing the movement, disassembly, or assembly of some of the operation objects without colliding with other objects, then the reward is set to 12 points, the Done method is called, and the training is completed.

[0077] Step 3.6: Set the operation time. Every 0.5 seconds during the hand's grasping and retraction operations, the AddReward method will be called, deducting 0.5 points. If the operation is not completed within the time limit, 12 points will be deducted, the Done method will be called, and training will end. During the hand grasping process, if the moving position is closer to the target object than the previous position and conforms to reality, 2 points will be added; if the target object is grasped, 4 points will be added; if a non-target object is grasped, 4 points will be deducted. Moving the specified target object to the predetermined position or handing it to the user will add 4 points. If the target object collides with other non-target objects during the movement, 12 points will be deducted, Done method will be called, and training will end.

[0078] Step 3.7: Train the model using the SAC algorithm. Copy the `tainer_config.yaml` file, keeping only the default configuration. Set `batch_size` to 128, `buffer_size` to 50000, `Max-Step` to 10 million, and `num_layer` to 8 layers. Check if the number of training steps has reached the maximum number of steps (`MaxStep`). If so, terminate the training. After training, copy the model into Unity for use.

[0079] This invention uses a combination of reinforcement learning and finite state machines in its design. Simple fixed actions of the virtual human are controlled by finite state machines, while complex maintenance actions are trained using reinforcement learning. This reduces the expansion of the state machine and, to some extent, reduces the amount of interaction between the intelligent virtual human and the environment and the complexity of the samples during training, thus shortening the training time and reducing the difficulty of algorithm convergence.

[0080] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for intelligent virtual human-assisted maintenance based on SAC algorithm and FSM, characterized in that, The method includes: S1. Obtain the 3D model of the virtual human, the 3D model of the object to be repaired, the 3D model of the repair tools, and the 3D model of the repair scene. Import each 3D model into the graphical development environment so that the virtual human, the object to be repaired, and the repair tools are in the repair scene. S2. Set the virtual human's position in the maintenance scene as the starting point and the maintenance operation area as the ending point, perform path planning, and realize the virtual human's standing, walking, running, and turning actions based on FSM. The maintenance operation area is the object to be maintained and the designated space around it. S3. Following the planned path, the virtual human is moved to the maintenance operation area by walking and running. During the movement, collision detection is performed based on the visual perception model. If the detected object is an obstacle, the virtual human avoids the obstacle by turning around. After the virtual human moves to the maintenance operation area, the visual perception model is used to detect whether there is a maintenance object in the field of vision. If not, the collision detection function of the field of vision perception model is called. The virtual human first adjusts its posture by turning around until the maintenance object can be perceived in the field of vision. Then, the angle detection function is called to continue to perform the turning action to fine-tune the angle between the front of the virtual human and the maintenance object. If the object exists, the angle detection function is called directly to perform the turning action to fine-tune the angle between the front of the virtual human and the maintenance object until the angle between the front of the virtual human and the maintenance object does not exceed the preset angle, and then proceed to S4. S4. Based on the visual perception model, optimize the intelligent decision-making model using the SAC algorithm to determine the maintenance behavior of the virtual human; Step S4 includes: (1) Build a deep reinforcement learning environment using the Unity3D component ML-Agents and initialize the Critic network parameters. Actor network parameters And the experience pool D; modify the algorithm configuration file, set the maximum number of steps Max-Step to 10 million, and the network layer num_layer to 8 layers; (2) Add a base agent script (Agents) to the virtual human. The virtual human collects the three-dimensional spatial position vector based on the observation information of the visual perception model. The angle between the virtual human's front and the object being repaired , Repair tool three-dimensional spatial position vector and the three-dimensional spatial position vector of the repair parts And define the state space as Define the virtual human's hand repair actions as a table, and provide the virtual human with specific actions as indexes of this table, thus defining an action space. ,in, These represent hand grasping, hand releasing, and hand movement operations, respectively. (3) Obtain the observation information of the current step based on the virtual human visual perception model. and operational objectives The virtual human selects the appropriate action to perform. The reward R is obtained through the reward and punishment function. At this point, the observation information state is transformed. ,Will , , , R and Combine into a tuple And store it in experience pool D; (4) The virtual human completes the operation in the manner of step (3) until the action on the object is completed. A batch of data of size N is randomly sampled from the experience pool D to update the Critic network parameters. and Actor network parameters This updates the target network. (5) Repeat steps (3) and (4) until the maximum number of iterations is reached (Max-Step), then terminate the training. After training is complete, copy the model into Unity. The reward The definition is as follows: in, This indicates the reward value obtained immediately when the moved position is closer to the target than the previous position; the reward value is defined as positive. This represents the reward value that the virtual human receives every 0.5 seconds when in an operational state; the reward value is assumed to be negative. This represents the reward value obtained immediately when the virtual human is greater than the safe distance from the nearest obstacle; the reward value is defined as positive. This represents the reward value obtained immediately when the virtual human's distance from the nearest obstacle is less than or equal to the safe distance; the reward value is defined as negative. This represents the reward value obtained immediately when the captured object is the target of the operation; the reward value is defined as positive. This represents the reward value immediately obtained when the captured object is not the object to be operated on; the reward value is defined as negative. This represents the reward value obtained immediately when the object is moved to a specified area or into the user's hands; the reward value is assumed to be positive. This refers to the reward value obtained immediately if the object being operated on is not moved to the designated area or into the user's hands, or if the operation is not completed within the specified time. The reward value is negative. This indicates the position of the virtual human's hand relative to the object being manipulated during the current sampling period. This indicates the position of the virtual human's hand relative to the object being manipulated, as defined in the previous sampling period; This indicates the time taken when the virtual human begins to operate. This indicates the safe distance between the virtual human's hand and the non-operational model; This refers to the object that the virtual person is grabbing. This indicates that the object being grasped by the virtual human is the object being manipulated; The collision box representing the virtual human's hand. The collision box representing the object being manipulated. The collision box represents the non-operated object.

2. The method as described in claim 1, characterized in that, The specific implementation of the virtual human's standing, walking, running, and turning movements based on FSM is as follows: The virtual human finite state machine is represented as a quintuple. ,in, The finite set of states refers to the set of all different state sequences of a virtual human, including four states: standing, walking, running, and turning around. The triggering input events for virtual human state transitions include: perceiving obstacles, being on the planned path, arriving at the designated maintenance area, the angle between the maintenance object and the virtual human's front, and the time spent walking. This represents the state transition function, which causes the state machine to transition from its current state to another state. This represents the initial state of the virtual human state machine, where the initial state is station; A non-empty set of state transitions is defined as a quintuple. , This indicates the current state of the virtual human. This indicates the state that the virtual human will become. Represents constraints that trigger conditions; This indicates the conditions that trigger state transitions. When these conditions are met, the virtual human's state machine will undergo a state transition, including sensing an obstacle, avoiding the obstacle and returning to the planned path, the walking time reaching a preset time, the virtual human reaching the end point of the planned path, and the angle between the virtual human's front and the object being repaired being less than or equal to a preset angle. This indicates the action to be performed after a state transition.

3. The method as described in claim 2, characterized in that, When the virtual human begins to perform a task, the state machine switches to walking. Every two seconds, the state switches to running. If the visual perception model detects an obstacle while walking or running, the state switches to turning around. After avoiding the obstacle and returning to the planned route, the state switches back to walking or running. When the virtual human reaches the designated maintenance area, the state machine switches to standing and determines the angle between the virtual human's front and the maintenance object. If the angle is greater than a preset angle, the state switches to turning around. If the angle is less than or equal to the preset angle, the state switches back to standing. If the angle is less than or equal to the preset angle, the state machine remains unchanged.

4. The method as described in claim 1, characterized in that, The visual perception model includes three Cube triggers, wherein the middle Cube trigger is positioned directly in front of the virtual human, and the two Cube triggers on either side are symmetrically positioned to the left and right front of the virtual human, forming a 60° fan-shaped field of view within the trigger range of the Cube triggers; the radius of the Cube triggers is 2 meters.

5. An intelligent virtual human-assisted maintenance system based on SAC algorithm and FSM, characterized in that, include: Processor and memory; The memory is used to store computer-executed instructions; The processor is configured to execute the computer execution instructions, causing the method described in any one of claims 1 to 4 to be executed.

Images