Positioning data smoothing prediction method, device and medium for multi-person interaction in VR scene

Abstract

The application provides a positioning data smoothing prediction method for multi-person interaction in a VR scene, a device and a medium. The method comprises: obtaining first action data of a target operator at a current time and a previous period, second action data of the remaining operators in the same virtual scene as the target operator at the current time and the previous period, and a medical training task in the virtual scene; determining a first behavior action and a first operation object of the target operator at the current time according to the medical training task and the first action data; determining an interactive operator who cooperates with the target operator according to the first behavior action, the first operation object, the first action data and the second action data; and predicting a predicted pose of the target operator at a next time based on the first behavior action, the first operation object, the first action data and the second action data of the interactive operator. The application can accurately predict the pose of the operator at the next time, and improve the continuity of the operator's action.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction of electric digital data processing, and particularly relates to a positioning data smoothing prediction method for multi-person interaction in a VR scene, a device and a medium. BACKGROUND

[0002] With the development of science and technology, virtual reality (VR) technology is widely applied in medical rescue training, and emergency process drills such as rescue preparation, wounded classification and medical treatment can be realized through VR technology.

[0003] In a medical rescue scene, there are a large number of contents that need to be cooperatively operated by multiple persons in each task link. Correspondingly, in a VR system for medical rescue, the spatial positions and interactive operation data of each operator need to be synchronized in real time, so as to realize the cooperative operation of the operators. However, the network delay of data transmission in the VR system will cause the actions of the operators in the virtual space to be different from the actual actions, resulting in operation conflicts and action misplacement between different operators, and it is difficult to accurately realize the interactive cooperation of the cooperative operation, thereby affecting the cooperative efficiency and immersion experience of the virtual training. SUMMARY

[0004] The embodiments of the present application provide a positioning data smoothing prediction method for multi-person interaction in a VR scene, a device and a medium, to accurately predict the pose of an operator at the next moment, reduce the operation conflicts and action misplacement between operators caused by network delay, and improve the continuity of the actions of the operators.

[0005] In a first aspect, the embodiments of the present application provide a positioning data smoothing prediction method for multi-person interaction in a VR scene, comprising:

[0006] obtaining first action data of a target operator at a current moment and a previous period, second action data of the rest of the operators in the same virtual scene as the target operator at the current moment and the previous period, and a medical training task in the virtual scene;

[0007] determining a first behavior action and a first operation object of the target operator at the current moment according to the medical training task and the first action data;

[0008] determining an interactive operator that cooperates with the target operator according to the first behavior action, the first operation object, the first action data and the second action data;

[0009] predicting a predicted pose of the target operator at the next moment based on the first behavior action, the first operation object, the first action data and second action data of the interactive operator.

[0010] In a possible implementation, each medical training task includes a plurality of action stages, and each action stage includes a plurality of behavior actions;

[0011] According to the medical training task and the first action data, the first behavior action and the first operation object of the target operator at the current time are determined, including:

[0012] According to the first action data, the action feature of the target operator is determined;

[0013] Using the action feature, the current action stage in which the target operator is located is determined from the plurality of action stages of the medical training task;

[0014] The current pose parameter is extracted from the first pose of the target operator at the current time in the first action data;

[0015] According to the current pose parameter and the action pose parameter of each behavior action in the current action stage, the first behavior action of the target operator at the current time is determined;

[0016] Based on the first pose and the first behavior action, the first operation object of the target operator at the current time is determined.

[0017] In a possible implementation, according to the first behavior action, the first operation object, the first action data and the second action data, an interactive operator who cooperates with the target operator is determined, including:

[0018] According to the first pose of the target operator at the current time in the first action data and the second pose of the remaining operators at the current time in the second action data, the action distance between each remaining operator and the target operator is determined;

[0019] The remaining operator whose action distance is less than a preset distance threshold is determined as a first candidate operator near the position of the target operator; wherein the distance threshold is determined according to the first behavior action;

[0020] According to the action type of the first behavior action, the first action data and the second action data of the first candidate operator, a second candidate operator is determined from the first candidate operator;

[0021] Based on the first operation object and the second operation object of the second candidate operator, an interactive operator who cooperates with the target operator is determined.

[0022] In a possible implementation, the action type includes a synchronous cooperative action.

[0023] The second candidate operator is determined from the first candidate operators according to the action type of the first behavioral action, the first action data, and second action data of the first candidate operators, including:

[0024] When the action type of the first behavioral action is a synchronous cooperative action, a first time sequence feature of the target operator is extracted from the first action data, and a second time sequence feature of each first candidate operator is extracted from the second action data of the first candidate operators;

[0025] A feature similarity of the first time sequence feature and the second time sequence feature is calculated;

[0026] The first candidate operator with the feature similarity greater than a preset threshold is determined as the second candidate operator.

[0027] In a possible implementation, the action type includes a procedure coordination action.

[0028] The second candidate operator is determined from the first candidate operators according to the action type of the first behavioral action, the first action data, and second action data of the first candidate operators, including:

[0029] When the type of the first behavioral action is a procedure coordination action, a plurality of second behavioral actions of the target operator and an action time of each second behavioral action are extracted from the first action data;

[0030] A plurality of third behavioral actions of the first candidate operators and an action time of each third behavioral action are extracted from the second action data of the first candidate operators;

[0031] An action correlation between the target operator and the first candidate operators is determined according to the second behavioral actions and the third behavioral actions;

[0032] A time correlation between the target operator and the first candidate operators is determined according to the action time of the second behavioral action and the action time of the third behavioral action;

[0033] The first candidate operator with the action correlation greater than a first correlation threshold and the time correlation greater than a second correlation threshold is determined as the second candidate operator.

[0034] In a possible implementation, each medical training task includes a plurality of action stages, and each action stage includes a plurality of behavioral actions.

[0035] predict a predicted pose of the target operator at a next time based on the first behavior action, the first operation object, the first action data, and second action data of the interactive operator, including:

[0036] determine whether the first behavior action is completed according to the first action data and the first behavior action;

[0037] if the first behavior action is not completed, determine an initial pose of the target operator and the interactive operator at the next time according to the first action data, the first behavior action, second action data of the interactive operator, and a preset pose prediction model;

[0038] obtain the predicted pose of the target operator at the next time based on the first operation object and the initial pose.

[0039] In a possible implementation, after determining whether the first behavior action is completed according to the first action data and the first behavior action, the method further includes:

[0040] if the first behavior action is completed, determine a fourth behavior action after the first behavior action according to a corresponding action stage of the first behavior action in the medical training task;

[0041] determine an initial pose of the target operator and the interactive operator at the next time according to the first action data, the first behavior action, the fourth behavior action, second action data of the interactive operator, and the pose prediction model;

[0042] determine a corrected pose of the target operator and the interactive operator according to the initial pose, a first pose of the target operator at the current time in the first action data, and a second pose of the remaining operators at the current time in the second action data;

[0043] obtain the predicted pose of the target operator at the next time based on the first operation object and the corrected pose.

[0044] In a possible implementation, after predicting the predicted pose of the target operator at the next time based on the first behavior action, the first operation object, the first action data, and the second action data of the interactive operator, the method further includes:

[0045] display the predicted pose in the virtual scene when an actual pose of the target operator at the next time is not received within a preset delay time threshold;

[0046] When the actual pose is received, a plurality of transition poses are generated between the predicted pose and the actual pose; and based on the transition poses and the actual pose, display is performed in the virtual scene.

[0047] In a second aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the method according to the first aspect or any possible implementation manner of the first aspect when executing the computer program.

[0048] In a third aspect, a computer-readable storage medium is provided, which stores a computer program, and the computer program, when executed by a processor, implements the steps of the method according to the first aspect or any possible implementation manner of the first aspect.

[0049] In a fourth aspect, a computer program product is provided, which, when executed on an electronic device, causes the electronic device to perform the steps of the method according to the first aspect or any possible implementation manner of the first aspect.

[0050] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0051] The embodiments of the present application determine the first behavior action and the first operation object of the target operator at the current time by the medical training task and the first action data of the target operator at the current time and the previous period, can clearly determine the specific action of the target operator in the medical training task, so as to find the operator who cooperates with the action of the target operator subsequently. By the first behavior action, the first operation object, the first action data, and the second action data of the remaining operators in the same virtual scene as the target operator, the action and the operation object of the target operator can be used to find the operator who performs a synchronous action or a cooperative action with the target operator, so as to determine the interactive operator who cooperates with the target operator. Finally, by the first behavior action, the first operation object, and the first action data of the target operator and the second action data of the interactive operator, the reasonable change of the behavior action of the target operator can be considered, the action of the interactive operator who cooperates with the target operator, and the predicted pose of the target operator at the next time are accurately predicted, so that the predicted pose is displayed in the virtual scene when the network delay is used, the operation conflict and the action misplacement problem between operators caused by the network delay are reduced, the continuity of the action of the target operator is improved, and the cooperation efficiency and the immersion experience of the virtual training are ensured. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0053] Figure 1 is an application scenario of a VR scene multi-person interaction positioning data smoothing prediction method provided by the embodiments of the present application;

[0054] Figure 2 is an implementation flowchart of a VR scene multi-person interaction positioning data smoothing prediction method provided by the embodiments of the present application;

[0055] Figure 3 is a schematic diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0056] In the following description, specific details such as specific system structures, techniques, etc. are presented in order to thoroughly understand the embodiments of the present application, but it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits and methods are omitted to avoid unnecessary details that hinder the description of the present application.

[0057] The present inventors found that, since the VR system needs to synchronize the spatial position of the operator with the interactive operation data in real time, the non-smooth characteristics of network delay can cause the positioning data of each terminal device to be out of synchronization, and further cause operation instruction conflicts, scene rendering misalignment and other problems, which seriously affect the collaborative efficiency and immersion experience of virtual training.

[0058] In order to improve the smoothness of the operator positioning data, in the embodiments of the present application, the next time pose of the operator is accurately predicted, and the predicted pose is displayed during network delay, so as to reduce the operation conflicts and action misalignment problems between operators caused by network delay, and improve the continuity of operator actions. Among them, the behavior action and operation object of the target operator are determined through the action data of the target operator; and through the behavior action and operation object of the target operator, and the action data of the remaining operators in the same virtual scene, the interactive operator who cooperates with the target operator can be found from the remaining operators; since the action of the target operator is affected by the behavior action and the action of the interactive operator, the next time action pose of the target operator can be predicted through the action data and behavior action of the target operator, and the action data of the interactive operator.

[0059] In order to make the purpose, technical scheme and advantages of the present application clearer, specific embodiments will be described below with reference to the drawings.

[0060] Figure 1 An application scenario diagram of a positioning data smoothing prediction method for VR scene multi-person interaction provided by an embodiment of the present application, in virtual training for medical rescue, the architecture of the virtual reality-based training system mainly includes a virtual reality (VR) device and a server.

[0061] The VR device includes a data acquisition unit (such as an optical data acquisition unit and an inertial data acquisition unit), a VR wearable device (such as a head-mounted device and a motion capture device), and a computing device (such as a backpack host and a motion capture computing device), etc. Among them, the head-mounted device is used to provide immersive virtual reality experience for the user, and the backpack host can provide core computing power support for virtual scene rendering, data processing and interaction logic calculation, etc. Here, the backpack host can also be a wireless host, a cloud host and a wearable host, etc., which can be selected and set according to the actual scene.

[0062] The motion capture device is a wearable device that can capture the body movements of the trainee and transmit them to the data acquisition unit. The data acquisition unit can be set in the area where the trainee is located, used to acquire the signals of the motion tracking device, capture the movements of the trainee and transmit them to the motion capture computing device in real time for processing, at the same time, the data acquisition unit can also provide network clock synchronization function to ensure time consistency.

[0063] The motion capture computing device can capture the current position and real-time action posture of the trainee through the received data, so as to generate a virtual avatar of the user in the virtual scene. Through the body movement data, the whole body skeletal posture is reversely calculated, and accurate motion capture data is generated through action estimation and missing information speculation compensation.

[0064] The server is mainly used to realize the key tasks of data processing, information storage and system monitoring in VR technology. The server can include a computing server, a storage server and a GPU computing server, etc.

[0065] Data transmission can be performed between the VR device and the server, the server can transmit the rendered virtual scene and interaction data to the head-mounted device of each user in real time, and the motion capture computing device can transmit the current position and real-time action posture of the trainee to the server in real time to generate a virtual avatar of the user in the virtual scene to update the entire virtual scene in real time.

[0066] Figure 2 The implementation flowchart of the positioning data smoothing prediction method for VR scene multi-person interaction provided by an embodiment of the present application is described in detail as follows:

[0067] In step 201, first action data of the target operator at the current time and the previous period, second action data of the rest of the operators in the same virtual scene as the target operator at the current time and the previous period, and a medical training task in the virtual scene are obtained.

[0068] In this embodiment, the first action data includes all action-related data of the target operator in the virtual scene at the current time and the previous period, and is core information describing the action state, trend and historical trajectory. The first action data can include pose data of the target operator (such as three-dimensional coordinates, joint angles, etc. of the target operator at each time), action timing (such as action start time, action end time, action duration, speed change and acceleration change, etc.), and action trajectory (such as hand movement path and limb swing amplitude, etc.).

[0069] For example, in the surgical operation training, the action data of the target operator "transferring surgical instruments" includes the three-dimensional coordinates of the hand at the current time (x = 1.2 m, y = 0.8 m, z = 1.5 m), the elbow joint bending angle (120°), and also includes the movement trajectory (continuous path from the shoulder to the elbow) and the speed change (0.3 m / s→0.5 m / s) in the previous 3 seconds.

[0070] Here, the previous period refers to a period of time before the current time and adjacent to the current time. For example, the previous period can be 5s, 30s, 60s or 360s before the current time. Here, only examples are given, and the length of the previous period is not limited.

[0071] The second action data includes action-related data of the rest of the operators in the same virtual scene as the target operator at the current time and the previous period, which is consistent with the structure of the "first action data", and is used to analyze the coordination relationship with the target operator.

[0072] The medical training task refers to a variety of medical rescue training tasks (such as cardiopulmonary resuscitation tasks, tumor resection surgery tasks, burn emergency treatment tasks, and emergency diversion tasks, etc.) pre-set in the virtual scene. The medical training task can constrain the action logic and process of the operator, and can analyze and judge the action of the target operator.

[0073] In the medical rescue, the medical training task has strong process and strong logic. According to the characteristics of the medical training task, the actions of the medical training task can be divided. Specifically, each medical training task includes multiple action stages, and each action stage includes multiple behavior actions. For example, the burn emergency treatment task can be divided into "cleaning wound dirt (stage 1) -> wound disinfection (stage 2) -> pain management (stage 3)", and stage 2 can include taking iodophor, aiming at the wound site of the burn, and applying iodophor.

[0074] The action stage is a process step after the medical training task is disassembled, has a time sequence and a logical progressive relationship, and only after the previous action stage is completed can the next action stage be entered. Each action stage corresponds to a specific operation scene, operation object and operation target. For example, the operation object of the cleaning wound dirt stage is the burned clothing fragments of the burn personnel, the ashes and necrotic tissue of the fire scene, and the operation target is the wound part of the burn.

[0075] The behavior action is a specific operation unit in the action stage, and is the smallest action unit that can be executed and quantified. Each behavior action corresponds to a preset action pose, such as a joint angle range and a spatial coordinate range.

[0076] In step 202, the first behavior action and the first operation object of the target operator at the current time are determined according to the medical training task and the first action data.

[0077] In this embodiment, considering that the medical training task in medical rescue has strong logic, strong continuity, strong process and strong action correlation, the first action data of the target operator can be analyzed by the medical training task to determine the current behavior action and operation object of the target operator, so as to predict the pose at the next time.

[0078] Optionally, each medical training task includes multiple action stages, and each action stage includes multiple behavior actions. In this embodiment, the first behavior action and the first operation object of the target operator at the current time are determined according to the medical training task and the first action data, which can be: determining the action feature of the target operator according to the first action data; determining the current action stage in which the target operator is located from the medical training task by using the action feature; extracting the current pose parameter from the first pose of the target operator at the current time in the first action data; determining the first behavior action of the target operator at the current time according to the current pose parameter and the action pose parameter of each behavior action in the current action stage; and determining the first operation object of the target operator at the current time based on the first pose and the first behavior action.

[0079] In this embodiment, by analyzing the first action data, the action features of the target operator at the current time and the previous period can be extracted. Here, the action features can be the three-dimensional trajectory of hand movement, the relative position of the limbs to the objects in the virtual scene, the action duration (such as the arm lifting action has lasted for 1.5 seconds), the joint angle combination (such as the elbow joint is bent by 120°, and the shoulder joint is spread by 30°), etc.

[0080] By extracting the action features, the unstructured action data of the target operator can be converted into structured features for analysis, so as to match the action stage and the corresponding behavior action of the target operator in the current medical training task by using the action features.

[0081] Here, each action stage of the medical training task corresponds to different action features. For example, the action features of the wound disinfection stage can include holding iodophor with hands, moving to the body surface of the burn victim, and stable moving speed, etc.; the action features of the wound cleaning stage can include collecting the clothing fragments of the burn victim, holding scissors with hands, and holding the clothing of the burn victim, etc. By matching the action features of the target operator with the action features corresponding to each action stage, the current action stage of the target operator can be locked, the analysis range of the subsequent behavior action is narrowed, and irrelevant actions across action stages are excluded.

[0082] Optionally, the matching degree of the action features of the target operator and the action features corresponding to each action stage can be calculated, and the action stage with the highest matching degree is taken as the current action stage of the target operator.

[0083] In this embodiment, the first action data of the target operator at the current time includes the first pose of the target operator at the current time, which is the instantaneous spatial state of the target operator, and specifically includes the three-dimensional coordinates and joint angles of the target operator, etc.

[0084] Correspondingly, the extracted current pose parameters can include hand coordinates, joint angles, and relative distance to the operation object, etc.

[0085] Among them, different pose parameters can be selected for different action stages. According to the pose parameters that will be affected by the change of the behavior action in the action stage, the pose parameters of each action stage are selected to distinguish the behavior action.

[0086] In the medical training task, the standard action pose parameters of each behavior action can be preset. For example, the parameters for aligning the burn wound site are that the deviation between the hand coordinates and the wound coordinates on the body surface is ≤5cm, and the included angle between the iodophor cotton stick and the body surface is 45°±10°.

[0087] The first behavior action of the target operator at the current time can be determined by matching the current pose parameter of the target operator and the action pose parameter of each behavior action in the current action stage. The similarity score between the current pose parameter of the target operator and the action pose parameter of each behavior action can be calculated, such as calculating the spatial deviation by the Euclidean distance, and the smaller the deviation, the higher the score. The behavior action with the highest similarity is selected as the first behavior action.

[0088] The first operation object in this embodiment is the entity directly acted on by the target operator when performing the first behavior action. The operation object can be a device, an article, a human body part, etc. For example, an iodophor bottle, an iodophor cotton swab, a stretcher, and burn victim clothing, etc.

[0089] Through the first behavior action, the task target of the action currently performed by the target operator can be determined, such as the first behavior action being to disinfect the left arm burn wound, it can be determined that the target operator can be operating the iodophor cotton swab to apply to the left arm of the injured person, can be operating the iodophor bottle to open, or can be operating the iodophor bottle to close, etc. Through the spatial coordinates of the first pose, the current state of the target operator can be determined, and the action point of the target operator's limbs / tools can be located, such as the hand holding the iodophor cotton swab, and the spatial position pointed by the iodophor cotton swab. Thus, the operation object of the target operator can be obtained.

[0090] For example, the first pose is that the hand holds the iodophor cotton swab, the cotton ball of the iodophor cotton swab points to the left chest of the burn victim, and the first behavior action is that the iodophor cotton swab applies to the left chest, and the first operation object includes the iodophor cotton swab and the left chest of the burn victim.

[0091] In step 203, an interactive operator who cooperates with the target operator is determined according to the first behavior action, the first operation object, the first action data, and the second action data.

[0092] In this embodiment, it is considered that in the same virtual scene, there are operators who cooperate with the target operator, and there are also operators who do not interact with the target operator. In order to more accurately predict the pose of the target operator at the next time, the operator who cooperates with the target operator should be fully considered, and therefore, the interactive operator who cooperates with the target operator can be found from the remaining operators in the same virtual scene.

[0093] Since the interactive operator and the target operator perform collaborative actions, such as lifting the same stretcher together, treating the same burn victim together, and performing cardiac resuscitation on the same patient together, it can be seen that the target operator and the interactive operator are close in spatial position, synchronized in action, or have a cooperative relationship, and the operation objects are related. Therefore, the target operator's interactive operator can be accurately selected from the remaining operators based on the first behavior action, the first operation object, the first action data, and the second action data, with spatial closeness as the basis, action synchronization as the core, and operation object relevance as the final basis.

[0094] In step 204, the predicted pose of the target operator at the next moment is predicted based on the first behavior action, the first operation object, the first action data, and the second action data of the interactive operator.

[0095] In this embodiment, the target operator's action is limited by the behavior action being performed by the target operator and the behavior action of the interactive operator cooperating with the target operator. Based on this, the target operator's pose at the next moment is predicted by considering the first action data and the first operation object of the target operator before the current moment, to obtain the predicted pose, so that the target operator's action is smooth and coherent.

[0096] The embodiments of the present application determine the target operator's first behavior action and first operation object at the current moment through the medical training task and the target operator's first action data at the current moment and the previous period, can clearly determine the specific action performed by the target operator in the medical training task, so as to find the operator cooperating with the target operator's action subsequently. By the first behavior action, the first operation object, the first action data, and the second action data of the remaining operators in the same virtual scene as the target operator, the action performed by the target operator and its operation object can be used to find the operator performing synchronized action or cooperative action with the target operator, thereby determining the interactive operator cooperating with the target operator. Finally, by the target operator's first behavior action, first operation object, and first action data, and the second action data of the interactive operator, the reasonable change of the target operator's behavior action and the action of the interactive operator cooperating with the target operator can be considered to accurately predict the target operator's predicted pose at the next moment, so that the predicted pose is displayed in the virtual scene in the network delay, reducing the operation conflict and action misplacement problem between operators caused by network delay, improving the coherence of the target operator's action, thereby ensuring the collaborative efficiency and immersive experience of virtual training.

[0097] In some embodiments, the interactive operator who cooperates with the target operator in the action can be determined according to the first action, the first operation object, the first action data and the second action data as follows. First, the action distance between each of the remaining operators and the target operator is determined according to the first pose of the target operator in the current time in the first action data and the second poses of the remaining operators in the current time in the second action data. Then, the remaining operators whose action distance is less than a preset distance threshold are determined as first candidate operators near the position of the target operator, wherein the distance threshold is determined according to the first action. Then, the second candidate operator is determined from the first candidate operators according to the action type of the first action, the first action data and the second action data of the first candidate operator. Finally, the interactive operator who cooperates with the target operator in the action is determined based on the first operation object and the second operation object of the second candidate operator.

[0098] In the embodiment, first, the operators who have no possibility of cooperation are excluded based on the spatial position, and the candidate range is narrowed. Through the first pose of the target operator in the current time and the second poses of the remaining operators in the current time, the real-time positions of the target operator and the remaining operators can be reflected, so that the action distance between each of the remaining operators and the target operator can be calculated. The three-dimensional straight-line distance between the target operator and the remaining operators can be calculated by the Euclidean distance.

[0099] Different behavior actions correspond to different distance thresholds. For example, in the action of collecting clothes fragments of a burn victim, the operator needs to act at a close distance, and the distance threshold can be 0.5 m. In the action of lifting a stretcher by multiple people, the distance threshold can be 2 m, because multiple people need to hold the four corners of the stretcher to move the stretcher. In the action of flushing, the operator can flush at a long distance, and the distance threshold can be 3 m.

[0100] The distance threshold can be used to screen the operators who have the cooperation condition in space and exclude the operators who are far away from the target operator in space and have no possibility of cooperation, such as bystanders at the edge of the scene and operators who perform other tasks or behavior actions in the same scene.

[0101] The target operator and the interactive operator have a synchronous cooperation or process cooperation relationship, so there is also a matching relationship between the first action data of the target operator and the second action data of the interactive operator. The range of the candidate operator is further narrowed through the action data.

[0102] For example, if the target operator and the interactive operator are of the synchronous cooperation action type, the action data of the target operator and the interactive operator have similarity, such as multiple people lifting a stretcher, and the actions of the target operator and the interactive operator are performed synchronously. If the target operator and the interactive operator are of the process cooperation action type, the action data of the target operator and the interactive operator have a process connection relationship, such as multiple people cooperating in cardiac resuscitation, and the target operator and the interactive operator perform actions such as chest compression, ventilation, object transfer, and defibrillation by cooperation.

[0103] Finally, in the same virtual scene, there can be multiple groups of cooperating operators, such as multiple groups of operators lifting a stretcher, and there can be operators in the second candidate operator who match the action of the target operator but do not lift the same stretcher. Therefore, the operation object of the target operator is further used to accurately find the interactive operator from the second candidate operator, and to exclude candidate operators who are irrelevant to the operation object.

[0104] The operation objects of the target operator and the interactive operator can be the same or have a correlation. For example, in the action of lifting a stretcher, the operation objects of the target operator and the interactive operator are the same, that is, the same stretcher. In cardiac resuscitation, the operation objects of the target operator and the interactive operator can be the same or have a correlation, such as the operation object of the target operator being the chest of an injured person for chest compression, and the operation object of the interactive operator being a defibrillator for object transfer or defibrillation.

[0105] Optionally, the action type includes a synchronous cooperation action. In the synchronous cooperation action, the target operator and the interactive operator keep the action rhythm, amplitude, or trend consistent in the time dimension, and have synchronism. For example, in the action of multiple people lifting a stretcher, the target operator and the interactive operator perform the action of lifting the stretcher synchronously, and lift synchronously and move synchronously.

[0106] In this embodiment, the second candidate operator is determined from the first candidate operator according to the action type of the first behavior action, the first action data, and the second action data of the first candidate operator. The second candidate operator can be determined as follows: when the action type of the first behavior action is a synchronous cooperation action, a first time sequence feature of the target operator is extracted from the first action data, and a second time sequence feature of each first candidate operator is extracted from the second action data of the first candidate operator; a feature similarity of the first time sequence feature and the second time sequence feature is calculated; and a first candidate operator with a feature similarity greater than a preset threshold is determined as the second candidate operator.

[0107] In this embodiment, the time sequence feature reflects a feature of a time variation law of an operator action, including an action period (such as a period of 1 second for completing a small amplitude lifting in the action of lifting a stretcher), a speed or acceleration curve, a time stamp of a key action, and a joint angle change sequence.

[0108] By calculating the feature similarity of the first time sequence feature and the second time sequence feature, the matching degree of the first time sequence feature and the second time sequence feature can be determined, whether the action rhythm of the target operator and the first candidate operator is synchronized can be judged, the candidate synchronized with the action of the target operator can be accurately locked, and the operator with disengaged action rhythm can be excluded, so as to determine the second candidate operator.

[0109] For example, the similarity can be calculated by using a dynamic time warping (DTW) algorithm, the action sequences at different speeds are aligned by stretching or compressing the time axis, and the overall similarity (such as the degree of curve coincidence) is calculated.

[0110] Optionally, the action type includes a process coordination action. In the process coordination action, the actions of the target operator and the interactive operator are sequentially connected or functionally complementary according to the standardized process of the medical training task, rather than strictly synchronized, such as chest compression, ventilation, and defibrillation in cardiac resuscitation. The synergy mainly reflects the action function complementation and time sequence connection.

[0111] In this embodiment, the second candidate operator is determined from the first candidate operator according to the action type of the first behavior action, the first action data, and the second action data of the first candidate operator, which can be:

[0112] When the type of the first behavior action is a process coordination action, a plurality of second behavior actions of the target operator and the action time of each second behavior action are extracted from the first action data; a plurality of third behavior actions of the first candidate operator and the action time of each third behavior action are extracted from the second action data of the first candidate operator. Then, the action correlation between the target operator and the first candidate operator is determined according to the second behavior action and the third behavior action. The time correlation between the target operator and the first candidate operator is determined according to the action time of the second behavior action and the action time of the third behavior action. Finally, the first candidate operator with the action correlation greater than the first correlation threshold and the time correlation greater than the second correlation threshold is determined as the second candidate operator.

[0113] In this embodiment, a plurality of behavior actions in the previous period, i.e., second behavior actions, are extracted from the first action data of the target operator, and the action time of each second behavior action is recorded. A plurality of behavior actions in the previous period, i.e., third behavior actions, are extracted from the second action data of the first candidate operator, and the action time of each third behavior action is recorded.

[0114] The plurality of behavior actions can include taking an iodophor bottle, using a cotton swab to dip iodophor, aligning the wound of the wounded person, and moving the iodophor cotton swab, etc.; or can include chest compression, ventilation, article delivery, and defibrillation, etc. There is a potential process coordination relationship between these actions.

[0115] By the second behavioral action of the target operator and the third behavioral action of the first candidate operator, it can be judged whether these behavioral actions belong to adjacent process steps of the same medical training task or have functional complementarity, i.e., action correlation.

[0116] Here, the behavioral actions of the target operator and the first candidate operator can be checked according to the action stages of the medical training task and the processes of the behavioral actions in each action stage. If the actions belong to the same process and have functional complementarity, the action correlation is high, such as pressing and ventilation, which both belong to cardiopulmonary resuscitation; if the actions are irrelevant, the action correlation is low, such as moving an iodophor cotton swab and carrying a stretcher.

[0117] Among them, if the two behavioral actions belong to the same process and cooperate with each other, the correlation can be 1; if the two behavioral actions do not belong to the same process and are irrelevant, the correlation can be 0. By integrating all the second behavioral actions and the third behavioral actions, the action correlation between the target operator and each first candidate operator is obtained.

[0118] In addition, the actions of the target operator and the interactive operator also have a cooperation relationship in time, and it can be judged whether the action time of the target operator and the first candidate operator meets the time sequence logic of process connection, i.e., time correlation, such as the completion of the previous action, and the cooperative action is started within a reasonable time.

[0119] Here, the interval between the action time of the second behavioral action and the action time of the third behavioral action can be used to judge whether the two behavioral actions have a cooperation relationship in time. By the standard process duration of the behavioral actions in the medical training task, the time correlation is quantified. The closer the interval is to the standard value, the higher the time correlation is. If the interval is too long or the order is reversed, the time correlation is low.

[0120] Among them, the time interval between each two adjacent behavioral actions corresponds to a reasonable time range and an irrelevant time range respectively. If the interval between the action time of the second behavioral action and the action time of the third behavioral action is within the corresponding reasonable time range, the correlation can be 1; if the interval between the action time of the second behavioral action and the action time of the third behavioral action is within the irrelevant time range, the correlation can be 0; if the interval between the action time of the second behavioral action and the action time of the third behavioral action is not within the reasonable time range and the irrelevant time range, the correlation can be calculated according to the minimum difference between the interval and the reasonable time range, and the minimum difference between the interval and the irrelevant time range. By integrating all the second behavioral actions and the third behavioral actions, the time correlation between the target operator and each first candidate operator is obtained.

[0121] The minimum difference between the interval and the reasonable time range is the minimum difference between the difference between the interval and the upper limit of the reasonable time range and the difference between the interval and the lower limit of the reasonable time range. The minimum difference between the interval and the irrelevant time range is the minimum difference between the difference between the interval and the upper limit of the irrelevant time range and the difference between the interval and the lower limit of the irrelevant time range.

[0122] The action correlation can determine whether the behavior actions of the target operator and the first candidate operator belong to the same process and are complementary in function, and the time correlation can determine whether the time sequence of the behavior actions of the target operator and the first candidate operator conforms to the logic of process connection. Therefore, the first candidate operator that meets the requirements of both the action correlation and the time correlation is selected as the second candidate operator.

[0123] In some embodiments, each medical training task includes a plurality of action stages, and each action stage includes a plurality of behavior actions.

[0124] In this embodiment, the predicted pose of the target operator at the next moment is predicted based on the first behavior action, the first operation object, the first action data, and the second action data of the interactive operator. The predicted pose of the target operator at the next moment can be determined as follows: determining whether the first behavior action is completed according to the first action data and the first behavior action; if the first behavior action is not completed, determining the initial pose of the target operator and the interactive operator at the next moment according to the first action data, the first behavior action, the second action data of the interactive operator, and a preset pose prediction model; and obtaining the predicted pose of the target operator at the next moment based on the first operation object and the initial pose.

[0125] In this embodiment, the behavior action of the target operator is performed according to the medical training task, so that the action stages and behavior actions in the medical training task can be used as a basis for pose prediction to predict a pose that conforms to the current behavior action, thereby reducing the randomness of pose prediction.

[0126] Here, it can be determined whether the first behavior action corresponding to the current moment is completed. If the first behavior action is not completed, the target operator generally continues to perform the first behavior action. If the first behavior action is completed, the target operator can switch to the next behavior action.

[0127] The determination of whether the first behavior action is completed can be made by using the action trajectory, duration, and current pose of the target operator in the first action data, and the action characteristics of the first behavior action. For example, in the whole-body decontamination of the wounded, 2 min has been performed, the standard duration is 3 min, the current pose deviates from the end pose of the first behavior action, and the action speed is changing, so it can be determined that the first behavior action is not completed.

[0128] When the first behavior action is not completed, an initial pose of the target operator and the interactive operator can be generated according to the first action data, the first behavior action, and second action data of the interactive operator.

[0129] The first action data is used to capture the action trend of the target operator, the first behavior action is used to determine the type and parameters of the current action, and the constraint prediction direction; and the second action data of the interactive operator is used to ensure the consistency and cooperation of the cooperation. The pose prediction model can be trained by using a time series model such as a Long Short-term Memory Network (LSTM), and by fusing the above data, the continuation trend of the actions of the two is predicted, and the initial pose of the target operator and the interactive operator is output. The initial pose can include spatial coordinates, joint angles, and the like at the next moment.

[0130] Here, the pose prediction model can be obtained by training a preset time series model by using the behavior action of the target operator at the current moment, the behavior action at the next moment, the action data of the target operator and the interactive operator at the historical period, and the corresponding pose of the target operator and the interactive operator at the next moment at the historical period.

[0131] In addition, the initial pose predicted may not conform to the actual interaction logic, such as the target operator crossing the operation object for the initial pose, and the initial pose being far away from the operation object. Accordingly, the initial pose can be corrected by using the first operation object of the target operator, so as to make the initial pose conform to the actual interaction logic.

[0132] For example, the stretcher needs to maintain a horizontal state, and the inclination angle is ≤3°. If the initial pose exceeds this range, it is corrected. For another example, the operation object for applying iodophor to the left arm is the wounded left arm, and the initial pose needs to be limited within the left arm area to avoid deviating to the torso and ensure the effectiveness of the action continuation.

[0133] Optionally, after determining whether the first behavior action is completed according to the first action data and the first behavior action, if the first behavior action is completed, a fourth behavior action after the first behavior action is determined according to the corresponding action stage of the first behavior action in the medical training task; an initial pose of the target operator and the interactive operator at the next moment is determined according to the first action data, the first behavior action, the fourth behavior action, the second action data of the interactive operator, and the pose prediction model; a corrected pose of the target operator and the interactive operator is determined according to the initial pose, the first pose of the target operator at the current moment in the first action data, and the second pose of the remaining operators at the current moment in the second action data; and a predicted pose of the target operator at the next moment is obtained based on the first operation object and the corrected pose.

[0134] In the embodiment, if the first behavior action is completed, the target operator can turn to the next behavior action. Then, the fourth behavior action can be determined based on the structured framework of the medical training task.

[0135] The initial pose that cooperates with the interactive operator is generated based on the first action data, the first behavior action, the fourth behavior action, and the second action data of the interactive operator.

[0136] Here, since the target operator has completed the first behavior action, the pose of the target operator at the next moment will change like the fourth behavior action, so the target of the pose at the next moment is also needed to be determined by the fourth behavior action to constrain the prediction direction.

[0137] Correspondingly, the pose prediction model can be trained by using the behavior action of the target operator at the current moment and the behavior action of the target operator at the next moment, the action data of the target operator and the interactive operator in the historical period, and the corresponding pose of the target operator and the interactive operator at the next moment in the historical period, and a preset time sequence model.

[0138] The pose prediction model here can be the same as the pose prediction model in the previous embodiment, and the difference is that when the first behavior action is not completed, the input of the model about the behavior action only includes the first behavior action, or the first behavior action can be input twice; when the first behavior action is completed, the input of the model can include the first behavior action and the fourth behavior action.

[0139] In some embodiments, after predicting the predicted pose of the target operator at the next moment based on the first behavior action, the first operation object, the first action data, and the second action data of the interactive operator, when the actual pose of the target operator at the next moment is not received within a preset delay time threshold, the predicted pose is displayed in the virtual scene; when the actual pose is received, a plurality of transition poses are generated between the predicted pose and the actual pose; and the transition poses and the actual pose are displayed in the virtual scene.

[0140] In the embodiment, when the actual pose of the target operator at the next moment is not received within the preset delay time threshold, it indicates that the transmission of the action data or the pose data of the target operator has network delay, so when the pose data is not received, the predicted pose is directly displayed to avoid the picture from being stagnant due to the lack of the actual pose, ensure the continuous response of the action of the target operator in the virtual scene, maintain the consistency of the movement and the vision, and reduce the dizziness caused by the lag. At the same time, the interactive operator can see the smooth operation of the target operator, and the immersion of the operator is ensured.

[0141] When the actual pose of the delay is finally received, the smooth connection from the predicted pose to the actual pose can be realized through multiple frames of transition poses. Among them, multiple frames of continuous transition poses can be inserted between the predicted pose and the actual pose through an interpolation algorithm (such as linear interpolation, curve interpolation), to ensure that the change amount of each frame of pose is uniform, and to avoid picture mutation caused by direct jumping. The virtual scene is displayed frame by frame according to the transition pose first, and gradually transitions from the predicted pose to the actual pose, and finally displays the actual pose stably. This process ensures that the change from the predicted pose to the actual pose is continuous and natural, which meets the human perception expectation of smoothness of motion, and guarantees the collaborative efficiency and user experience of virtual training.

[0142] In addition, the present inventors have also found that, in addition to the pose deviation caused by network delay, the transmission time of the motion data of each operator can be different, and directly displaying the motion of the operator can cause motion conflict between each operator.

[0143] In this embodiment, the motion data of the target operator and the interactive operator is time-aligned, and the motion data of the target operator and the interactive operator is displayed by backtracking or predicting to the same time, to ensure time synchronization, reduce motion conflict and motion misplacement, and ensure the smoothness of the motion of the operator.

[0144] In some embodiments, based on the first behavior motion, the first operation object, the first motion data, and the second motion data of the interactive operator, the predicted pose of the target operator at the next time can be:

[0145] Step one, time-align the first motion data, the second motion data, and the medical training task, and determine the current time for current pose display.

[0146] In this embodiment, the first motion data of the target operator, the second motion data of the interactive operator, and the standard time axis of the medical training task are calibrated to ensure that the time stamps of the three are based on the same reference. Among them, the standard time axis of the medical training task contains the theoretical time interval of each motion stage, which can be adjusted according to the previously displayed motion data.

[0147] The time at which the virtual scene is currently performing pose display is determined, providing a unified time anchor for subsequent calculation of the time deviation of data transmission delay, and avoiding prediction errors caused by time reference confusion.

[0148] Step two, based on the aligned first motion data and second motion data, determine the first time deviation between the corresponding time of the target operator and the current time, and the second time deviation between the corresponding time of the interactive operator and the current time.

[0149] Here, the first time deviation between the time corresponding to the target operator and the current time can be determined by the data generation time corresponding to the action data of the target operator and the timestamp of the current time. The second time deviation between the time corresponding to the interactive operator and the current time can be determined by the data generation time corresponding to the action data of the interactive operator and the timestamp of the current time. The delay degree of the action data of the target operator and the interactive operator can be accurately quantified.

[0150] Step three, determining whether the first action data includes the pose corresponding to the next moment of the current moment of the target operator according to the first time deviation.

[0151] In this embodiment, the first time deviation is used to determine whether the pose corresponding to the next moment of the current moment is recorded in the first action data of the target operator.

[0152] If the data transmission delay of the target operator is low and the data acquisition frequency is high, the action data of the target operator can be faster than the current moment of the display, that is, the first action data can include the pose of the next moment.

[0153] If the data transmission delay of the target operator is high, the action data can be seriously delayed, and the action data of the target operator can be slower than the current moment of the display.

[0154] Step four, if it is included, the pose corresponding to the next moment of the current moment in the first action data is used as the predicted pose of the target operator in the next moment.

[0155] In this embodiment, if the pose corresponding to the next moment exists in the first action data, it indicates that the action data transmission delay of the target operator is low, and the pose to be displayed in the next moment has been collected relative to the current moment of the display. Therefore, it can be directly used as the predicted pose in the next moment, without using external data for pose prediction, thereby ensuring the accuracy and continuity of the action of the target operator.

[0156] Step five, if it is not included, the third action data of each interactive operator before the next moment of the current moment is extracted from the second action data according to the second time deviation; and the predicted pose of the target operator in the next moment is predicted based on the first behavior action, the first operation object, the first action data, and the third action data of the interactive operator.

[0157] In this embodiment, if the first action data does not include the pose corresponding to the next moment, it indicates that the action data transmission of the target operator has a certain delay or a higher delay.

[0158] In addition, the interactive operator who cooperates with the target operator can have a lower delay, and the second action data of the interactive operator can include the pose at the next moment. Therefore, the third action data of the interactive operator before the next moment can be extracted from the second action data. Here, the third action data includes the pose of the interactive operator at the next moment. If the interactive operator also has a higher delay, and the second action data does not include the pose at the next moment, the second action data can be directly used as the third action data.

[0159] Finally, the first behavior action, the first operation object, the first action data, and the third action data of the interactive operator are used to predict the predicted pose of the target operator at the next moment.

[0160] The third action data is used to predict the predicted pose of the target operator at the next moment. The prediction of the specific pose is only to replace the second action data with the third action data, and the specific description can be referred to the above embodiments, which will not be described here.

[0161] In the embodiment, the time alignment is used to ensure that the action data of the target and the interactive operator is displayed at the same moment, and the action conflict caused by the delay difference is reduced from the root; the action of the target operator is naturally continued by using the data of the target operator itself, and when the action data of the target operator at the next moment is missing, the action data of the interactive operator is used for position prediction, and the consistency of the multi-person action is maintained through the cooperation logic; the final predicted pose not only conforms to the action inertia of the target operator, but also matches the cooperation trend of the interactive operator, and is consistent with the process specification of the medical training task, and can achieve the dual goals of smooth display and accurate cooperation.

[0162] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0163] The following is a device embodiment of the present application. For details not described in detail, please refer to the corresponding method embodiments described above.

[0164] Figure 3 is a schematic diagram of an electronic device provided by an embodiment of the present application. As shown in Figure 3 The electronic device 30 of this embodiment includes a processor 31, a memory 32, and a computer program 33 stored in the memory 32 and executable on the processor 31. The processor 31 implements the steps in each of the above method embodiments when executing the computer program 33, for example Figure 2 Steps 201 to 204 shown in the figure.

[0165] For example, the computer program 33 can be divided into one or more modules / units, one or more modules / units are stored in the memory 32 and executed by the processor 31 to complete the present application. One or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program 33 in the electronic device 30.

[0166] The electronic device 30 can include, but is not limited to, the processor 31, the memory 32. Those skilled in the art can understand that the electronic device 30 can further include other components required for the electronic device 30 to operate, such as a bus, an input / output device, a network access device, etc. Figure 3 The electronic device 30 is only an example and does not constitute a limitation on the electronic device 30, and can include more or fewer components than the diagram, or combine certain components, or different components, for example, the electronic device can also include an input / output device, a network access device, a bus, etc.

[0167] The processor 31 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0168] The memory 32 can be an internal storage unit of the electronic device 30, such as a hard disk or a memory of the electronic device 30. The memory 32 can also be an external storage device of the electronic device 30, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 32 can include both the internal storage unit and the external storage device of the electronic device 30. The memory 32 is used to store computer programs and other programs and data required by the electronic device. The memory 32 can also be used to temporarily store data that has been output or will be output.

[0169] For the convenience and brevity of description, only the above-mentioned division of each functional module / unit is exemplified, and in actual application, the above-mentioned functions can be completed by different functional modules / units according to needs. The above-mentioned modules / units can be realized in the form of hardware, software or a combination of hardware and software.

[0170] The embodiment of the application further provides a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method in each method embodiment described above is realized.

[0171] The embodiment of the application further provides a computer program product, which comprises a computer program. When the computer program is executed by a processor, the method in each method embodiment described above is realized.

[0172] The computer program comprises computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium can comprise any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), electric carrier wave signal, telecommunication signal and software distribution medium, etc.

[0173] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in a certain embodiment can be referred to the relevant description of other embodiments. If there is no special description and logical conflict, the terms and / or descriptions of different embodiments are consistent and can be mutually referred to, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0174] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for smoothing and predicting positioning data in VR scene multi-person interaction, characterized in that, include: Acquire the target operator's first action data at the current time and in the previous time period, the second action data of other operators in the same virtual scene as the target operator at the current time and in the previous time period, and the medical training task in the virtual scene; Based on the medical training task and the first action data, determine the first action and the first target of the target operator at the current moment; Based on the first behavioral action, the first operation object, the first action data, and the second action data, determine the interactive operator who coordinates with the target operator. Based on the first action, the first operation object, the first action data, and the second action data of the interactive operator, predict the predicted pose of the target operator at the next moment; Each medical training task includes multiple action phases, and each action phase includes multiple behavioral actions; Based on the medical training task and the first action data, determine the target operator's first action and first operation object at the current moment, including: Based on the first action data, determine the action characteristics of the target operator; Using the aforementioned action characteristics, the current action phase of the target operator is determined from multiple action phases of the medical training task; Extract the current pose parameters from the first pose of the target operator at the current moment in the first action data; Based on the current pose parameters and the action pose parameters of each action in the current action phase, determine the first action of the target operator at the current moment; Based on the first pose and the first action, the first target operator is determined at the current moment.

2. The VR scene multi-person interaction positioning data smoothing prediction method according to claim 1, characterized in that, Based on the first behavioral action, the first operational object, the first action data, and the second action data, an interactive operator who coordinates with the target operator is determined, including: Based on the first pose of the target operator in the first motion data at the current moment, and the second pose of the other operators in the second motion data at the current moment, determine the motion distance between each other operator and the target operator; The remaining operators whose action distance is less than a preset distance threshold are identified as first candidate operators near the location of the target operator; wherein, the distance threshold is determined based on the first action. Based on the action type of the first action, the first action data, and the second action data of the first candidate operator, a second candidate operator is determined from the first candidate operators; Based on the first operation object and the second operation object of the second candidate operator, an interactive operator is determined to cooperate with the target operator.

3. The VR scene multi-person interaction positioning data smoothing prediction method according to claim 2, characterized in that, The action types include synchronous and coordinated actions; Based on the action type of the first action, the first action data, and the second action data of the first candidate operator, a second candidate operator is determined from the first candidate operators, including: When the action type of the first action is a synchronous collaborative action, the first temporal feature of the target operator is extracted from the first action data, and the second temporal feature of each first candidate operator is extracted from the second action data of the first candidate operators. Calculate the feature similarity between the first temporal feature and the second temporal feature; The first candidate operator whose feature similarity is greater than a preset threshold is identified as the second candidate operator.

4. The method for smoothing and predicting positioning data in VR scene multi-person interaction according to claim 2, characterized in that, The action types include process coordination actions; Based on the action type of the first action, the first action data, and the second action data of the first candidate operator, a second candidate operator is determined from the first candidate operators, including: When the type of the first action is a process coordination action, extract multiple second actions of the target operator and the action time of each second action from the first action data; Extract multiple third actions of the first candidate operator and the action time of each third action from the second action data of the first candidate operator; Based on the second and third behavioral actions, the action correlation between the target operator and the first candidate operator is determined; Based on the action time of the second action and the action time of the third action, the temporal correlation between the target operator and the first candidate operator is determined; The first candidate operator whose action correlation is greater than the first correlation threshold and whose time correlation is greater than the second correlation threshold is determined as the second candidate operator.

5. The method for smoothing and predicting positioning data in VR scene multi-person interaction according to claim 1, characterized in that, Each medical training task includes multiple action phases, and each action phase includes multiple behavioral actions; Based on the first action, the first target object, the first action data, and the second action data of the interactive operator, predict the target operator's pose at the next moment, including: Based on the first action data and the first action, determine whether the first action has been completed; If the first action is not completed, the initial pose of the target operator and the interactive operator at the next moment is determined based on the first action data, the first action, the second action data of the interactive operator, and the preset pose prediction model. Based on the first operation object and the initial pose, the predicted pose of the target operator at the next moment is obtained.

6. The method for smoothing and predicting positioning data in VR scene multi-person interaction according to claim 5, characterized in that, After determining whether the first action has been completed based on the first action data and the first action, the method further includes: If the first action is completed, then the fourth action following the first action is determined according to the action stage corresponding to the first action in the medical training task. Based on the first action data, the first behavioral action, the fourth behavioral action, the second action data of the interactive operator, and the pose prediction model, determine the initial pose of the target operator and the interactive operator at the next moment; Based on the initial pose, the first pose of the target operator in the first motion data at the current moment, and the second pose of the other operators in the second motion data at the current moment, the corrected poses of the target operator and the interactive operator are determined. Based on the first operation object and the corrected pose, the predicted pose of the target operator at the next moment is obtained.

7. The VR scene multi-person interaction positioning data smoothing prediction method according to any one of claims 1 to 6, characterized in that, After predicting the target operator's predicted pose at the next moment based on the first behavioral action, the first operating object, the first action data, and the second action data of the interactive operator, the method further includes: If the actual pose of the target operator at the next moment is not received within the preset delay time threshold, the predicted pose is displayed in the virtual scene. Upon receiving the actual pose, a multi-frame transition pose is generated between the predicted pose and the actual pose; and the transition pose and the actual pose are then displayed in the virtual scene.

8. An electronic device comprising a memory and a processor, the memory for storing a computer program, the processor for calling and running the computer program stored in the memory, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images