VR-based matching degree data processing method and system

Abstract

The invention provides a VR-based fitness data processing method and system, relates to a virtual reality technology, and can perform simulated blood drawing through a VR device, collect child sight and limb movement data through the VR device and a scene camera, and evaluate a fitness level; when a child is in a low-degree matching state, scene switching is carried out to select an adaptive scene, for example, the scene is switched into a cartoon style interesting scene, the attention of the child can be quickly attracted, fear and resistance of the child to blood drawing are dispersed, and in the process, the robot is controlled to carry out corresponding simulation operation, so that simulation is more real, and the safety of the child is improved. Therefore, the examination matching degree of the sick child is known, targeted measures are formulated, and the sick child can be encouraged to complete examination without intervention for the sick child capable of being matched; medical sedation intervention can be given to child patients who try out the system and do not cooperate with the system, and therefore relevant examinations are completed.

Description

TECHNICAL FIELD

[0001] The present application relates to virtual reality technology, in particular to a cooperation degree data processing method and system based on VR. BACKGROUND

[0002] In the pediatric medical field, blood drawing, imaging and other types of examination are key links for disease diagnosis and treatment. Especially for young children with nervous system diseases (such as epilepsy), due to their small age and limited cognitive ability, they are full of fear for the unfamiliar medical environment and complex examination process. At the same time, the mental or neurological abnormalities that may be accompanied by nervous system diseases further exacerbate their uncooperative behavior during the examination process. In actual operation, these children often show strong resistance, such as constant crying and struggling limbs, making it difficult for routine examination operations to proceed smoothly.

[0003] Currently, in order to ensure the smooth implementation of the examination, the medical sedation intervention means is generally used in the clinic, that is, the children are injected with sedatives to make them calm. However, this method has many drawbacks: on the one hand, the children and their parents have great concerns about the use of sedatives, worrying that the drugs may cause adverse reactions such as respiratory depression and allergic reactions, potentially harming the child's health, thereby producing strong rejection and resistance psychology; on the other hand, the existing scheme lacks a scientific evaluation mechanism for the cooperation degree of the children before using the sedatives, and cannot accurately judge which children really need sedation intervention, which exists the problem of over-reliance on sedatives, not only increasing the medical cost, but also possibly delaying the examination opportunity.

[0004] Therefore, how to accurately evaluate the cooperation degree of the children, effectively calm the children's emotions and reduce the dependence on sedatives, improve the understanding and acceptance of the children, and thus improve the examination efficiency, has become a problem that needs to be solved today. SUMMARY

[0005] The present application provides a cooperation degree data processing method and system based on VR, which can accurately evaluate the cooperation degree of the children, effectively calm the children's emotions and reduce the dependence on sedatives, improve the understanding and acceptance of the children, and thus improve the examination efficiency.

[0006] In a first aspect of the present application, a cooperation degree data processing method based on VR is provided, comprising: determining the frequency of visual line avoidance of the user based on the visual line tracking data collected by the simulation end; determining the frequency of action avoidance of the user according to the body posture data collected by the monitoring end; comprehensively estimating the cooperation degree level by combining the frequency of visual line avoidance and the frequency of action avoidance; when the cooperation degree level is lower than the sensitivity level of the current progress node, triggering a scene switching instruction to switch the scene elements corresponding to the current progress node.

[0007] Optionally, in a possible implementation manner of the first aspect, the gaze avoidance frequency of the user is determined based on the line-of-sight tracking data collected by the simulation terminal, and the determination includes: detecting a simulation area in the line-of-sight tracking data, the collection angle of the simulation terminal being the same as the user's angle of view; when the simulation area is not detected in the line-of-sight tracking data of a continuous preset number of frames, determining that the user avoids the gaze, recording the avoidance duration, and accumulating the number of times of gaze avoidance; when the simulation area is detected, recording the duration of staying, and accumulating the number of times of gaze avoidance when the simulation area is not detected next time; calculating the gaze avoidance frequency according to the number of times of gaze avoidance in a unit period.

[0008] Optionally, in a possible implementation manner of the first aspect, the simulation area in the line-of-sight tracking data is detected, and the detection includes: generating a detection area according to pixel points in the line-of-sight tracking data, the pixel points being in a pixel interval corresponding to the simulation area; comparing the area similarity of the simulation area and the detection area, and determining that the simulation area is detected when the area similarity is greater than or equal to a similarity threshold; determining that the simulation area is not detected when the area similarity is less than the similarity threshold.

[0009] Optionally, in a possible implementation manner of the first aspect, the action avoidance frequency of the user is determined according to the body posture data collected by the monitoring terminal, and the determination includes: detecting a key joint in the body posture data, tracking the position change of the key joint, and obtaining a body movement trajectory of the user; recognizing a movement trajectory of the execution terminal, and determining a distance change trend between the execution terminal and the user according to the movement trajectory; when the distance change trend is a decreasing trend, the interval distance between the execution terminal and the user is less than a distance threshold, and the movement trajectory is detected, obtaining a movement parameter of the body movement trajectory, the movement parameter including a movement direction and a movement amplitude; when the movement parameter meets an avoidance condition, accumulating the number of times of body avoidance, and obtaining the action avoidance frequency according to the number of times of body avoidance in a unit period.

[0010] Optionally, in a possible implementation manner of the first aspect, when the distance change trend is a decreasing trend, the interval distance between the execution terminal and the user is less than a distance threshold, and the movement trajectory is detected, the movement parameter of the body movement trajectory includes a movement direction and a movement amplitude, and the determination includes: obtaining the interval distance between the execution terminal and the user at continuous time points based on the distance change trend; When the interval distance gradually decreases, a decreasing trend is determined, and when the interval distance is less than a threshold value, a key joint connecting the previous time point to the next time point is determined as a moving direction of the limb motion trajectory; A moving amplitude of the limb motion trajectory is determined according to the displacement of the key joint from the previous time point to the next time point.

[0011] Optionally, in a possible implementation manner of the first aspect, the satisfaction of the avoidance condition comprises: The direction from the position point of the user to the position point of the execution end is determined as an approaching direction, and the avoidance condition is satisfied when an included angle between the moving direction and the approaching direction is greater than or equal to an included angle threshold value and the moving amplitude is greater than or equal to an amplitude threshold value.

[0012] Optionally, in a possible implementation manner of the first aspect, the cooperation degree level is determined by comprehensively estimating the line-of-sight avoidance frequency and the action avoidance frequency, comprising: The line-of-sight avoidance frequency and the action avoidance frequency are compared with a frequency interval corresponding to each preset cooperation degree level, and the cooperation degree level of the user is determined according to a comparison result.

[0013] Optionally, in a possible implementation manner of the first aspect, when the cooperation degree level is lower than a sensitivity level of a current progress node, a scene switching instruction is triggered to switch a scene element corresponding to the current progress node, comprising: The current progress node is determined according to an execution action of the execution end, and the scene switching instruction is triggered when the cooperation degree level is lower than a sensitivity level of a current progress node; A scene element set corresponding to the current progress node is retrieved based on the simulation end identifying a key element; According to a comparison result of the key element and the scene element set, the key element is replaced by an interactive image corresponding to a corresponding scene element, and the execution end is controlled to respond to voice data corresponding to the scene element.

[0014] Optionally, in a possible implementation manner of the first aspect, the current progress node is determined according to an execution action of the execution end, comprising: A tool held by the execution end is identified, and a plurality of progress nodes corresponding to the tool are selected; An action feature parameter of the execution action of the execution end is detected, and the action feature parameter at least comprises an action posture, a motion trajectory and a speed; The action feature parameter is compared with a configuration feature parameter of each progress node, and the current progress node is determined according to a comparison result.

[0015] The second aspect of the application provides a cooperation degree data processing system based on VR, comprising: A line-of-sight determination module is configured to determine a line-of-sight avoidance frequency of the user based on line-of-sight tracking data collected by the simulation terminal; An action determination module is configured to determine an action avoidance frequency of the user based on body posture data collected by the monitoring terminal; A level estimation module is configured to comprehensively estimate the line-of-sight avoidance frequency and the action avoidance frequency to obtain a cooperation level; A scene switching module is configured to trigger a scene switching instruction to switch a scene element corresponding to the current progress node when the cooperation level is lower than a sensitivity level of the current progress node.

[0016] The present application has the following advantages: 1. The present application determines the line-of-sight avoidance frequency by collecting image data consistent with the user's line-of-sight direction through the VR device, and determines the action avoidance frequency by monitoring the body posture data through the high-definition camera distributed around the scene, thereby comprehensively capturing the child's resistance behavior in the VR simulation blood drawing process from two dimensions of vision and body movement. The cooperation level is determined by comparing the two indicators with the preset frequency interval, which avoids the one-sidedness of single data evaluation, makes the cooperation evaluation result more accurate and comprehensive, and reflects the real state of the child, thereby providing a reliable basis for scene adjustment.

[0017] 2. When the child is in a low cooperation state according to the cooperation evaluation result, the present application can accurately determine the current progress node based on the execution action of the execution terminal and timely trigger the scene switching instruction. Seamless switching of the scene and natural transition of the interaction can be realized. This intelligent scene switching mechanism can quickly attract the child's attention, dispel his fear of blood drawing, greatly improve the child's experience in the simulation process, effectively improve the cooperation level, and ensure the smooth progress of the simulation process.

[0018] 3. The present application accurately determines the current simulation progress node by recognizing the tool taken by the execution terminal through image recognition technology and comparing the action feature parameters with the configuration feature parameters of each progress node. Based on the accurate progress node judgment, the system can accurately switch the corresponding scene elements, ensure that the scene style, props, voice, etc. are closely matched with the simulation operation, enhance the realism and coherence of the simulation process, and make the child better adapt to the simulation environment in the immersive experience, thereby laying a psychological adaptation foundation for the actual blood drawing operation and improving the effectiveness and practicality of the simulation training. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a schematic diagram of an application scene provided by an embodiment of the present application; Figure 2 is a flowchart of a cooperation degree data processing method based on VR provided by an embodiment of the present application; Figure 3is a structural schematic diagram of a VR-based cooperation degree data processing system provided by an embodiment of the present application. DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in a clear and complete manner with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0021] Referring to Figure 1 is an application scenario schematic diagram provided by an embodiment of the present application. Blood extraction examination is a common diagnostic method, but children have a fear of medical operations, and often show strong resistance, cry, and struggle during blood extraction, which not only makes the blood extraction operation difficult to proceed smoothly and increases the difficulty of the work of medical staff, but also can cause psychological trauma to children and long-term fear of hospitals and medical operations. Therefore, the present scheme can simulate blood extraction through a VR device, collect children's visual line and body movement data through the VR device and scene camera, and evaluate the cooperation degree grade; when the children are in a low cooperation state, the scene is switched to select an adaptive scene, for example, a cartoon-style interesting scene, which can quickly attract the children's attention, distract their fear and resistance to blood extraction, and in the process, a robot is also controlled to perform corresponding simulation operations, so that the simulation is more realistic, thereby understanding the examination cooperation degree of the children, and formulating targeted measures. For children who can cooperate, the children can be encouraged to complete the examination without intervention; for children who do not cooperate with the system, medical sedation intervention can be given to complete the relevant examination. The virtual end is a VR device, and the execution end is an object that performs a specific operation in the simulation scene, such as a robot that simulates blood extraction.

[0022] Referring to Figure 2 is a flowchart of a VR-based cooperation degree data processing method provided by an embodiment of the present application, Figure 2The execution subject of the method shown can be a software and / or hardware device. The execution subject of the present application can include, but is not limited to, at least one of the following: user equipment, network equipment, etc. Among them, the user equipment can include, but is not limited to, a computer, a smart phone, a personal digital assistant (PDA) and the above-mentioned electronic equipment, etc. The network equipment can include, but is not limited to, a single network server, a server group composed of multiple network servers, or a cloud composed of a large number of computers or network servers based on cloud computing, wherein cloud computing is a kind of distributed computing, which is a super virtual computer composed of a group of loosely coupled computers. The present embodiment does not make any limitation. It includes steps S1 to S4, which are as follows: S1, determining the visual line avoidance frequency of the user based on the visual line tracking data collected by the simulation end.

[0023] In the VR simulation scene, the visual line behavior of the user is a direct mapping of the psychological state. When the user has emotions such as resistance, fear or disinterest towards specific content in the simulation scene, such as simulated blood drawing operation, it will be manifested by avoiding the key area through the visual line. Determining the visual line avoidance frequency can capture this behavior feature in a quantitative way, providing key data support for evaluating the user's cooperation degree. Only by accurately obtaining this data, can the potential resistance of the user be detected in time, so as to lay a foundation for subsequent adjustment of the scene and optimization of the user experience, and ensure that the VR simulation process can proceed smoothly and achieve the expected effect.

[0024] The visual line tracking data contains image data collected by the VR device. The shooting direction of the VR device is consistent with the visual line direction of the user when collecting data, which can record the fixation point and visual line movement trajectory of the user's eyes in the VR scene. Through analysis and processing of these image data, the user's visual line staying and avoiding in different scene areas can be obtained. The visual line avoidance frequency refers to the number of times that the user's visual line avoids the key area in the VR simulation scene per unit time, which is a quantitative index for measuring the user's resistance to specific scene content.

[0025] Determining the visual line avoidance frequency based on the data collected by the VR device consistent with the user's visual line direction can accurately capture the user's visual line behavior and present the user's avoidance tendency of specific elements in the VR simulation scene with accurate quantitative data. It provides a solid data foundation for cooperation degree evaluation, so that the system can more accurately judge the user's state. When a high visual line avoidance frequency is found, the VR simulation scene can be adjusted in a targeted manner, such as switching scene style, changing element presentation method, etc., to effectively alleviate the user's resistance emotion and improve the user's cooperation degree and sense of immersion in the VR simulation process.

[0026] The specific implementation of step S1 based on the above embodiments can be: Detect the simulation area in the line-of-sight tracking data, the collection visual angle of the simulation end is the same as the user's visual angle; when the simulation area is not detected in the line-of-sight tracking data of a continuous preset number of frames, it is determined that the line-of-sight is avoided, the avoidance duration is recorded and the number of line-of-sight avoidance times is accumulated; when the simulation area is detected, the stay duration is recorded, and the number of line-of-sight avoidance times is accumulated when the simulation area is not detected next time; the line-of-sight avoidance frequency is calculated according to the number of line-of-sight avoidance times in a unit period.

[0027] The simulation area is a core area for operation in the VR simulation blood drawing scene, such as the position where the actual blood drawing operation occurs, which can be identified by adding a red rectangular marker frame.

[0028] It can be understood that in the VR simulation blood drawing scene, accurately judging whether the user's line-of-sight avoids the simulation blood drawing area is a key link for evaluating the user's cooperation degree. By setting a fixed marker frame in the simulation area and detecting its appearance in the user's visual angle picture, the line-of-sight avoidance behavior can be determined in a quantitative and standardized manner, thereby providing data support for calculating the line-of-sight avoidance frequency, and the user's cooperation degree can be comprehensively and accurately evaluated, so that the scene can be adjusted in time when the user's cooperation degree is insufficient, and the quality and effect of the VR simulation experience can be improved.

[0029] The avoidance duration is the duration of the user's line-of-sight avoiding the simulation area, the time interval between detecting that the line-of-sight starts to avoid and re-detecting the simulation area; the number of line-of-sight avoidance times is the cumulative number of times that the user's line-of-sight avoids in a certain time period; the stay duration is the time length of the user's line-of-sight staying in the simulation area, the time interval between detecting the appearance of the simulation area and again detecting that the simulation area disappears; the unit period is a fixed time interval for counting the number of line-of-sight avoidance times and calculating the line-of-sight avoidance frequency.

[0030] For example, after the simulation starts, the VR device built-in camera collects the user's visual angle image in real time at a frame rate of 60fps, which is transmitted to the visual recognition module as line-of-sight tracking data. The visual recognition module uses a target detection algorithm to detect each frame of image, and finds the red rectangular marker frame. When the simulation is performed to the 2nd minute, the marker frame is not detected in 3 consecutive frames of images, the system determines that this is a line-of-sight avoidance event, records the event timestamp, and increments the line-of-sight avoidance times N by 1. When the marker frame reappears in the image, the entry timestamp is recorded again, and then the marker frame disappears again, the exit timestamp is recorded, and the stay duration is calculated. The data is counted in 1 minute as a unit period, and in this period, a total of 8 line-of-sight avoidance events of Li Lei are detected, so the line-of-sight avoidance frequency is 8 times per minute.

[0031] Through the above steps, the user's visual line avoidance behavior in the VR simulated blood drawing scene can be accurately and efficiently detected, and corresponding visual line avoidance frequency and other key data can be calculated, which has high accuracy and stability, can effectively exclude interference factors, and accurately captures the user's visual line changes.

[0032] In some embodiments, the simulated area in the visual line tracking data can be detected by the following steps, comprising: According to the pixel points in the visual line tracking data whose pixel values are within the pixel interval corresponding to the simulated area, a detection area is generated; the area similarity of the simulated area and the detection area is compared, and when the area similarity is greater than or equal to a similarity threshold, it is determined that the simulated area is detected; and when the area similarity is less than the similarity threshold, it is determined that the simulated area is not detected.

[0033] The detection area is an area generated according to the pixel points in the visual line tracking data whose pixel values are within the pixel interval corresponding to the simulated area. The area similarity is an index for measuring the similarity between the detection area and the simulated area, reflecting the similarity between the two areas in terms of pixel value distribution, shape, etc. The similarity threshold is a pre-set value, which is used as a standard for judging whether the simulated area is detected. When the area similarity of the detection area and the simulated area is greater than or equal to the threshold, it is determined that the simulated area is detected; and when the area similarity is less than the threshold, it is determined that the simulated area is not detected.

[0034] For example, assuming that the simulated area is set as a red rectangular area, the system traverses each pixel point in the image collected by the VR device, checks whether its pixel value is within the pixel interval corresponding to the simulated area, and marks the pixel points within the pixel interval as part of the detection area. Through processing of the entire image, the detection area is finally generated. Then, an image feature-based similarity calculation algorithm (such as the structural similarity index SSIM) is used to calculate the area similarity of the detection area and the simulated area, and assuming that the calculated area similarity is 0.88. The pre-set similarity threshold is 0.8. Since the calculated area similarity 0.88 is greater than the similarity threshold 0.8, it is determined that the simulated area is detected. This means that in the current frame image, the user's visual line focuses on the simulated area. If in another frame image, the calculated area similarity is 0.72, which is less than the similarity threshold 0.8, it is determined that the simulated area is not detected, i.e., the user's visual line does not focus on the simulated area in that frame image.

[0035] In this way, it can be accurately determined whether the simulated area is within the user's visual line range, effectively avoiding misjudgment caused by subjective judgment or simple detection methods, and providing a reliable data basis for subsequent visual line avoidance analysis and cooperation degree evaluation. Moreover, this method can process the visual line tracking data collected by the VR device in real time, timely feedback the changes in the user's visual line, and enable the system to respond quickly.

[0036] S2, determine the action avoidance frequency of the user according to the limb posture data collected by the monitoring end.

[0037] The limb movement of the user in the VR simulation scene is an intuitive external manifestation of the user's inner emotions and attitudes. Especially when facing simulation operations that may produce nervous and fearful emotions, such as simulated blood drawing, the avoidance movement of the limbs can more directly reflect the degree of resistance of the user. Relying solely on gaze data cannot fully understand the true state of the user, therefore, by determining the action avoidance frequency, the cooperation evaluation data is supplemented from the perspective of limb behavior, a multi-dimensional evaluation system can be formed to more comprehensively and accurately grasp the acceptance degree of the user to the VR simulation operation, and provide a basis for more reasonably adjusting the simulation scene and the interaction mode subsequently.

[0038] The monitoring end is composed of multiple high-definition cameras distributed around the VR simulation scene, which shoot the user from different angles to ensure that the full body movement information of the user can be captured completely; the limb posture data is obtained by analyzing and processing the images collected by these cameras, mainly including the position coordinates of each key joint of the human body such as the wrist, elbow, shoulder, etc. and the change information over time; the action avoidance frequency refers to the number of times the user's limb movement shows avoidance of performing operations in the VR scene within a unit time, which reflects the user's resistance behavior to the simulation operation from the aspect of limb movement.

[0039] By determining the action avoidance frequency, important supplementary information is provided for user cooperation evaluation from the aspect of limb behavior, which makes up for the limitations of relying solely on gaze data, making the evaluation results more comprehensive and truly reflecting the actual state of the user. Based on accurate action avoidance frequency data, the system can more deeply analyze the resistance behavior pattern of the user, thereby adjusting the interaction mode, operation process, etc. in the VR simulation scene, reducing the resistance of the user, and improving the cooperation degree and participation of the user in the simulation process.

[0040] On the basis of the above embodiment, the specific implementation mode of step S2 can be: S21, detect the key joints in the limb posture data, track the position changes of the key joints, and obtain the limb movement trajectory of the user.

[0041] When analyzing the limb movement of the user to evaluate the cooperation degree, the position of the key joint and its change can provide key information about the limb movement. By detecting and tracking these key joints, the movement trajectory of the user's limbs can be accurately depicted, thereby providing a basis for subsequent judgment of whether the user has avoidance movement and analysis of the movement characteristics.

[0042] The key joint points are representative joint positions on the human body, such as the wrist, elbow, etc. Changes in their positions can reflect the main movements of the limbs, and here they can be the corresponding joint points on the arm. The limb movement trajectory is the path formed by the position changes of the key joint points in consecutive image frames, which describes the movement manner and range of the limb in space.

[0043] S22, identify the movement trajectory of the execution end, and determine the distance change trend between the execution end and the user according to the movement trajectory.

[0044] Understanding the distance change trend between the execution end, such as a robot simulating blood drawing, and the user is crucial for determining whether the user has avoidance behavior to the operation of the execution end. By identifying the movement trajectory of the execution end and determining its distance change from the user, we can more accurately capture the user's reaction and provide more comprehensive information for assessing the degree of cooperation.

[0045] The movement trajectory is the movement path of the execution end in the simulation scenario, which is obtained by tracking and recording its position in consecutive image frames. The distance change trend is the change of the distance between the execution end and the user over time, including whether the distance is increasing, decreasing, or remaining unchanged.

[0046] Identifying the movement trajectory of the execution end and determining the distance change trend can help us determine whether the user has reacted to the approach of the execution end. If the distance change trend is decreasing and the user has limb movements at the same time, further analysis is needed to determine whether these limb movements are avoidance actions, so as to more accurately assess the user's degree of cooperation.

[0047] S23, when the distance change trend is a decreasing trend and the distance between the execution end and the user is less than a distance threshold and the movement trajectory, obtain the movement parameters of the limb movement trajectory, including the movement direction and the movement amplitude.

[0048] When the execution end approaches the user and the distance is less than a certain threshold, the user's limb movements may better reflect their attitude towards the operation of the execution end. Obtaining the movement parameters of the limb movement trajectory at this time, such as the movement direction and the movement amplitude, can more detailedly describe the user's avoidance action characteristics and provide more specific basis for determining whether it is a real avoidance action.

[0049] The distance threshold is a pre-set distance value used to determine whether the distance between the execution end and the user is close enough to trigger the acquisition of the limb movement parameters. The movement parameters are parameters that describe the characteristics of the limb movement, including the movement direction and the movement amplitude. The movement direction is the direction of the limb movement. The movement amplitude is the degree of limb movement.

[0050] In some embodiments, step S23 can be implemented by the following steps: acquire the interval distance between the execution end and the user at continuous time points based on the distance change trend; when the interval distance gradually decreases, determine a decreasing trend, and when the interval distance is less than a threshold value, connect the key joint nodes at the previous time point and the next time point to obtain the moving direction of the limb movement trajectory; and obtain the moving amplitude of the limb movement trajectory according to the displacement of the key joint nodes at the previous time point and the next time point.

[0051] The threshold value is a distance value set in advance as a criterion for judging the proximity of the execution end to the user. When the interval distance is less than the threshold value, it is considered that the execution end has approached the user sufficiently.

[0052] At continuous time points, the interval distance between the execution end and the user is measured continuously. If the distance value obtained by subsequent measurement is continuously less than the previous distance value, it means that the distance between the execution end and the user is gradually decreasing over time, i.e., showing a decreasing trend. The movement direction of the limb can be reflected by the position change of the key joint nodes at different time points. In continuous image frames, connecting the key joint nodes at the previous time point and the next time point forms a line segment that approximately represents the movement direction of the joint in this time period, and thus reflects the movement direction of the limb. The displacement of the key joint nodes can directly reflect the amplitude of the limb movement, and the greater the displacement, the greater the amplitude of the limb movement in this time period.

[0053] Through the above steps, the distance change information between the execution end and the user, as well as the moving direction and amplitude of the user's limb movement trajectory, can be accurately obtained. These information are crucial for judging whether the user makes an avoidance action in response to the approach of the execution end. For example, when the execution end approaches the user, and the moving direction of the user's limb movement trajectory is opposite to the approaching direction of the execution end and the moving amplitude is large, it can be more accurately judged that the user makes an avoidance action.

[0054] S24, when the motion parameter meets the avoidance condition, accumulate the number of limb avoidance times, and obtain the action avoidance frequency according to the number of limb avoidance times in a unit period.

[0055] By setting the avoidance condition and accumulating the number of limb avoidance times, the avoidance behavior of the user can be quantified, and thus the important indicator of action avoidance frequency can be obtained. The action avoidance frequency can intuitively reflect the degree of resistance of the user to the operation of the execution end in a unit of time, and provide a quantitative basis for evaluating the cooperation degree of the user.

[0056] The avoidance condition is a set of judgment criteria set in advance for determining whether the limb movement is an avoidance action. The number of limb avoidance times is the number of limb movements that meet the avoidance condition within a certain time range.

[0057] When the motion parameter meets the following criteria, the avoidance condition is met: The direction from the position point of the user to the position point of the execution end is a close direction, and the avoidance condition is met when an included angle between the moving direction and the close direction is greater than or equal to an included angle threshold value and a moving amplitude is greater than or equal to an amplitude threshold value.

[0058] During the process of the execution end approaching the user, the user may have an avoidance behavior due to emotions such as nervousness and fear. By explicitly defining the close direction between the user and the execution end, and taking the included angle between the moving direction and the close direction and the moving amplitude as the basis for judgment, when the included angle is greater than a certain threshold value and the moving amplitude is also greater than a corresponding threshold value, the avoidance action is determined, which can accurately capture the avoidance intention of the user and avoid misjudging normal body movements as avoidance, thereby improving the accuracy and reliability of the judgment.

[0059] The close direction is a direction from the position point of the user to the position point of the execution end, which intuitively represents the direction of the execution end approaching the user. The included angle threshold value is a pre-set angle value, which is used as a standard for judging the size of the included angle between the moving direction and the close direction. When the included angle between the moving direction and the close direction is greater than or equal to the threshold value, it means that the moving direction of the user's body is greatly different from the close direction of the execution end. The amplitude threshold value is a pre-set displacement size value, which is used as a standard for judging whether the moving amplitude is large enough. When the moving amplitude is greater than or equal to the threshold value, it means that the user's body movement has a certain intensity.

[0060] For example, in a VR simulation blood drawing training scene, the user wears a VR device, and the system determines to take the wrist joint point of the user as the position point of the user, assuming that the coordinates are ((10, 10, 10), and a robot simulating blood drawing is used as the execution end, and the position point coordinates of the robot are (20, 10, 10), so the close direction is the horizontal direction from (10, 10, 10) to (20, 10, 10). During the simulation of the blood drawing operation, the robot approaches the user's wrist. At this time, it is detected through the VR device and related algorithms that the wrist joint point of the user has moved, and the displacement vector of the moving direction is calculated to be from (10, 10, 10) to (5, 10, 10), so it is determined that the moving direction is horizontal left. The system pre-sets the included angle threshold value to be 60°, and the amplitude threshold value to be 5 cm. It is calculated that the included angle between the moving direction (horizontal left) and the close direction (horizontal right) is 180°, which is obviously greater than the included angle threshold value 60°, so it can be determined that the moving direction of the user's wrist is opposite to the close direction of the execution end. At the same time, the wrist joint point of the user moves from (10, 10, 10) to (5, 10, 10), and the displacement size is calculated to be 5 cm, which is equal to the amplitude threshold value. At this time, the conditions that the included angle is greater than or equal to the included angle threshold value and the moving amplitude is greater than or equal to the amplitude threshold value are both met, and the avoidance condition is met, so the system determines that the wrist movement of the user at this time is an avoidance action, and performs corresponding recording and processing.

[0061] S3, comprehensively estimate the line-of-sight avoidance frequency and the action avoidance frequency to obtain a cooperation degree level.

[0062] The cooperation degree level refers to classifying the user's cooperation degree in the VR simulation process by comparing the user's line-of-sight avoidance frequency and action avoidance frequency with a preset corresponding relationship. It can be classified into different levels such as high cooperation, medium cooperation, and low cooperation.

[0063] Comprehensively estimating the line-of-sight avoidance frequency and the action avoidance frequency to determine the cooperation degree level can comprehensively and objectively evaluate the user's cooperation in the VR simulation process from multiple dimensions, avoiding the one-sidedness and limitations of single data evaluation. Accurate cooperation degree level evaluation provides a key basis for subsequent scene switching decisions, enabling the system to take appropriate measures such as adjusting scene style and optimizing interaction mode according to the user's actual state in a timely manner, thereby effectively improving the user's experience in the VR simulation process and improving the success rate of simulation and user satisfaction.

[0064] On the basis of the above embodiments, the specific implementation of step S3 can be: Comparing the line-of-sight avoidance frequency and the action avoidance frequency with the frequency intervals corresponding to each preset cooperation degree level, and determining the user's current cooperation degree level according to the comparison result.

[0065] The preset cooperation degree level is to divide the user's cooperation degree in the VR simulation process into different levels, such as high cooperation, medium cooperation, and low cooperation. Each cooperation degree level corresponds to certain behavior characteristics and performance. The frequency interval is the value range of the line-of-sight avoidance frequency and the action avoidance frequency set for each preset cooperation degree level. The comparison result is the result obtained by comparing the user's actual line-of-sight avoidance frequency and action avoidance frequency with the frequency intervals corresponding to each preset cooperation degree level. Through this result, it can be judged which cooperation degree level the user's avoidance behavior conforms to.

[0066] Suppose in a VR simulation blood drawing scene, the system presets the cooperation degree level and the corresponding line-of-sight avoidance frequency and action avoidance frequency interval as follows: High cooperation: line-of-sight avoidance frequency < 5 times / minute, action avoidance frequency < 3 times / minute; Medium cooperation: 5 times / minute ≤ line-of-sight avoidance frequency ≤ 10 times / minute, 3 times / minute ≤ action avoidance frequency ≤ 6 times / minute; Low cooperation: line-of-sight avoidance frequency > 10 times / minute, action avoidance frequency > 6 times / minute.

[0067] In the simulation process, after a period of monitoring, the user Xiao Wang's eye-avoidance frequency is 8 times per minute, and the action-avoidance frequency is 5 times per minute. Comparing the eye-avoidance frequency of 8 times per minute of Xiao Wang with the above-mentioned preset frequency interval, it is found that it meets the interval range of the eye-avoidance frequency in the moderate cooperation level; comparing the action-avoidance frequency of 5 times per minute also meets the interval range of the action-avoidance frequency in the moderate cooperation level. According to the comparison result, it is determined that the cooperation degree level of Xiao Wang at present is moderate cooperation.

[0068] In the above manner, the avoidance behaviors of the user in two dimensions of eye and body movement are comprehensively considered, the one-sidedness of single index evaluation is avoided, and the real cooperation degree situation of the user can be more accurately reflected.

[0069] S4, when the cooperation degree level is lower than the sensitivity level of the current progress node, triggering a scene switching instruction to switch the scene element corresponding to the current progress node.

[0070] In the VR simulation scene, different simulation progress nodes have different cooperation degree requirements for the user, and the psychological state and acceptance level of the user in different stages also differ. The sensitivity level of the current progress node is set to ensure that when the user's cooperation degree cannot meet the simulation operation requirements of the current stage, the scene switching instruction can be triggered in time, the scene element more suitable for the current state of the user is replaced, the user experience is improved, the resistance emotion of the user is reduced, and the simulation process can be smoothly pushed forward, avoiding simulation failure or bad experience of the user due to insufficient cooperation degree.

[0071] The current progress node refers to different stages of the VR simulation scene divided according to the operation process, each stage has specific operation tasks and scene elements, for example, it can include the preparation stage, blood sampling stage and blood sampling completion stage of blood sampling simulation; the sensitivity level is a cooperation degree threshold preset for each progress node, which represents the minimum cooperation degree required by the user to ensure the smooth operation of the simulation in this stage; the scene element is various elements constituting the VR simulation scene, including scene style (such as real style, cartoon style, science fiction style), props (such as blood sampling instrument, magic wand, virtual instrument), background, sound effect, etc., the experience of the user can be changed by switching these elements. The scene switching instruction is a command issued by the system to start the scene switching operation, which is triggered when the cooperation degree level is lower than the sensitivity level and other conditions are met.

[0072] When the cooperation degree level is lower than the sensitivity level of the current progress node, the scene element is switched in time, which can quickly respond to the change of the user's state and effectively alleviate the resistance emotion of the user. By switching the scene that is not suitable for the current state of the user to a scene that is more attractive and more in line with the psychological needs of the user, the cooperation degree and participation of the user can be significantly improved, and the simulation process can be smoothly carried out.

[0073] The specific implementation of step S4 based on the above embodiments can be: determine the current progress node according to the execution action of the execution end, trigger a scene switching instruction when the cooperation degree level is lower than the sensitivity level of the current progress node, identify a key element based on the simulation end, call a scene element set corresponding to the current progress node, replace the key element with an interactive image corresponding to the corresponding scene element according to the comparison result of the key element and the scene element set, and control the execution end to respond to the voice data corresponding to the scene element.

[0074] By determining the current progress node according to the action of the execution end, the progress of the simulation can be accurately grasped. When the cooperation degree level is lower than the sensitivity level of the current progress node, the scene is switched in time to improve the cooperation degree of the child when the child resists or the like. Identifying the key element, calling the corresponding scene element set, and replacing the element and controlling the execution end to respond to the voice data are to realize seamless switching of the scene and natural transition of the interaction, so that the child can better adapt to and participate in the simulation process in the new scene, and finally ensure the smooth progress of the simulation blood drawing process and prepare for the actual blood drawing operation.

[0075] The key element is various elements that have important significance and role in the current simulation scene, which may include the position and posture of the child, the key operation part of the execution end such as the simulation blood drawing needle, and the like. These elements are crucial for the switching of the scene and the coherence of the interaction. The scene element set is a set of a series of scene elements corresponding to the current progress node. It contains rich content, such as scene style (real style, cartoon style, etc.), character image (cute cartoon medical staff, real medical staff image, etc.), props (blood sampling instrument, magic wand, etc.), background (forest full of fantastic colors, real hospital environment, etc.), sound effect, etc. The interactive image is an image corresponding to the key element in the new scene, which is used to replace the key element in the original scene when the scene is switched, so as to realize the visual conversion of the scene, ensure the coherence and naturalness of the switching process, and make the child visually smoothly transition to the new scene. The voice data is voice information closely related to the new scene element, such as dialogue between cartoon characters, operation prompt voice, etc. The execution end makes corresponding voice response according to these voice data, enhances the realism and interest of the interaction, further attracts the attention of the child, and improves the participation of the child.

[0076] Suppose the simulation medical staff robot starts to pick up a blood sampling tool such as a common simulation blood drawing needle. The system accurately determines that the current progress node is in the blood sampling preparation stage according to this execution action. Through in-depth analysis of the child's line of sight and limb movement data during the simulation process, it is concluded that the child's cooperation level is low. Since the sensitivity level set in the blood sampling preparation stage is medium, the system quickly triggers the scene switching instruction because the child's cooperation level is significantly lower than the sensitivity level, and prepares for the scene conversion.

[0077] Then the key elements in the current scene are identified, including the position and posture of the child's arm, the action of the simulation medical staff robot, and the important tool of the simulation blood drawing needle picked up by the simulation medical staff robot. After receiving the scene switching instruction, the scene element set corresponding to the blood sampling preparation stage is immediately called. Suppose this time it is switched to a cartoon-style interesting scene, which contains a cute cartoon medical staff image in the set, which is cute and can effectively relieve the child's nervousness; a magic wand prop replaces the original simulation blood drawing needle, full of fantastic colors; a background image full of fantastic elements, such as a beautiful fairy tale forest, creates a relaxed and pleasant atmosphere; and voice data related to these elements, such as the cartoon medical staff's friendly greetings and operation prompt voice, etc.

[0078] Then, the system compares the key elements in the original scene, such as the simulation blood drawing needle picked up by the simulation medical staff robot, with the corresponding elements (magic wand) in the scene element set. After confirming the match, the system replaces the simulation blood drawing needle with the interactive image of the magic wand, so that the child sees the magic wand full of magic atmosphere in the new scene through the VR device, instead of the blood drawing needle that may cause fear. At the same time, the system controls the simulation medical staff robot to respond to the voice data related to the magic wand in a timely manner, in this way, the child is better integrated into the new scene, greatly reducing the fear and resistance to blood drawing operations, making the simulation process more smoothly.

[0079] By determining the progress node according to the execution end action and flexibly triggering scene switching combined with the cooperation level, the simulation scene can respond to the child's state changes in real time and accurately. Accurate identification and replacement of key elements, while controlling the execution end to respond to voice data accurately, ensures the high continuity of vision and interaction during scene switching. Children will not feel strange or confused during scene switching, and can naturally and smoothly transition from the original scene to the new scene, ensuring the smooth progress of the simulation process, making the simulation process more realistic and credible, and providing children with a good simulation experience. Replacing key elements with interactive images and making the execution end respond to lively and interesting voice data greatly enriches the interactive content of the simulation scene.

[0080] In some embodiments, the current progress node can be determined according to the execution action of the execution end by the following steps, comprising: identifying the taking tool corresponding to the execution end, and selecting a plurality of progress nodes corresponding to the taking tool; detecting action characteristic parameters of the execution action of the execution end, the action characteristic parameters at least including action posture, motion trajectory and speed; comparing the action characteristic parameters with configuration characteristic parameters of each of the progress nodes, and determining the current progress node according to the comparison result.

[0081] Accurate judgment of the progress node currently simulated is crucial for the smooth progress of the entire simulation process and timely adjustment of the scene according to the user state. By identifying the tool taken by the execution end, a plurality of possible corresponding progress nodes can be preliminarily screened, narrowing the judgment range. By detecting the action characteristic parameters of the execution end and comparing them with the configuration characteristic parameters of each progress node, the current actual progress node can be more accurately determined, thereby providing an accurate basis for scene switching, interactive control and other operations according to the progress node and user cooperation degree, ensuring that the simulation process proceeds according to the predetermined process, and being able to flexibly cope with various situations.

[0082] The taking tool is various tools used by the execution end in the simulation process, such as a simulated blood drawing needle, a disinfectant cotton ball, a tweezers, a cleaning cloth, etc. Different tools correspond to different stages in the simulation process. The progress node is a different stage divided by the VR simulated blood drawing scene according to the operation process, such as the preparation stage (including disinfection, preparation of blood collection instruments, etc.), the blood collection process, and the processing link after blood collection is completed. Each progress node has its specific operation content and characteristics. The action characteristic parameters are related parameters for describing the execution action of the execution end, at least including action posture (such as the stretching angle of the arm, the holding posture of the hand, etc.), motion trajectory (the moving path of the tool or the body part of the execution end), and speed (the degree of fast or slow of the action), which can reflect the specific action of the execution end. The configuration characteristic parameters are the action characteristic parameter range or specific value set in advance for each progress node, corresponding to the operation content of the node, which is used to compare with the actually detected execution end action characteristic parameters to determine the current progress node.

[0083] For example, a scene camera captures real-time images of a robot simulating a medical worker. Using image recognition technology, based on the slender, sharp shape and metallic sheen of the simulated blood-drawing needle, the robot determines that it is currently holding a simulated blood-drawing needle. This allows for the selection of multiple progress nodes related to the blood collection process, such as disinfection preparation for puncture, needle insertion, and needle withdrawal with pressure. Simultaneously, using camera image sequences and sensors at the robot's joints, the robot arm is detected to be in an extended position with an extension angle of approximately 135 degrees. The simulated blood-drawing needle is moving along a smooth, straight path towards the user's arm at a speed of 8 centimeters per second. Comparing these motion characteristic parameters with the preset configuration parameters for each progress node reveals a match with the "disinfection preparation for puncture" node, where the arm extension angle is between 120 and 150 degrees, the movement speed is 5 to 10 centimeters per second, and the needle moves along a straight line towards the arm. Therefore, the current progress node is determined to be the "disinfection preparation for puncture" stage.

[0084] By first identifying the tool obtained by the execution end, and then comparing the action feature parameters with the configuration feature parameters, the progress node of the current VR simulated blood drawing scene can be determined more accurately. This avoids the one-sidedness of relying solely on tools or actions for judgment, and improves the accuracy and reliability of the judgment.

[0085] See Figure 3 This is a schematic diagram of a VR-based cooperation data processing system provided in an embodiment of the present invention. The VR-based cooperation data processing system includes: The gaze determination module is used to determine the frequency of the user's gaze avoidance based on gaze tracking data collected from the analog terminal; The action determination module is used to determine the frequency of user action avoidance based on the limb posture data collected by the monitoring terminal; The level estimation module is used to comprehensively estimate the frequency of eye contact avoidance and the frequency of action avoidance to obtain the cooperation level; The scene switching module is used to trigger a scene switching command and switch the scene element corresponding to the current progress node when the cooperation level is lower than the sensitivity level of the current progress node.

[0086] Figure 3 The apparatus of the illustrated embodiment can be used to perform corresponding actions. Figure 2 The steps in the method embodiments shown are implemented in a similar manner and have similar technical effects, and will not be repeated here.

[0087] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions recorded in the above embodiments can be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A VR-based method for processing cooperation data, characterized in that, include: Determine the frequency of user eye avoidance based on eye-tracking data collected from the analog terminal; The frequency of user action avoidance is determined based on the body posture data collected by the monitoring terminal. The cooperation level is obtained by comprehensively estimating the frequency of eye contact avoidance and the frequency of movement avoidance. When the cooperation level is lower than the sensitivity level of the current progress node, a scene switching command is triggered to switch the scene element corresponding to the current progress node.

2. The method according to claim 1, characterized in that, The frequency of user eye avoidance is determined based on eye-tracking data collected from the analog terminal, including: The simulated area in the eye-tracking data is detected, and the acquisition angle of the simulated terminal is the same as that of the user. If the simulated area is not detected in the eye tracking data for a consecutive preset number of frames, it is determined as an eye avoidance, the avoidance duration is recorded and the number of eye avoidances is accumulated; When the simulated area is detected, the dwell time is recorded, and the number of eye-avoidance attempts is accumulated when the simulated area is not detected again. The frequency of visual avoidance is calculated based on the number of visual avoidances per unit cycle.

3. The method according to claim 2, characterized in that, Detecting the simulated area in the gaze tracking data includes: Based on the pixels whose pixel values ​​in the gaze tracking data are located within the pixel interval corresponding to the simulated area, a region to be inspected is generated. Compare the regional similarity between the simulated area and the area to be detected. When the regional similarity is greater than or equal to the similarity threshold, it is determined that the simulated area has been detected. When the regional similarity is less than the similarity threshold, it is determined that the simulated region has not been detected.

4. The method according to claim 1, characterized in that, The frequency of user action avoidance is determined based on body posture data collected by the monitoring terminal, including: Detect key joints in the limb posture data, track the positional changes of the key joints, and obtain the user's limb movement trajectory; Identify the movement trajectory of the execution terminal, and determine the trend of distance change between the execution terminal and the user based on the movement trajectory; When the distance change trend is decreasing and the distance between the execution terminal and the user is less than the distance threshold and the movement trajectory is obtained, the motion parameters of the limb movement trajectory are acquired, and the motion parameters include the movement direction and the movement amplitude. When the motion parameters meet the avoidance conditions, the number of limb avoidances is accumulated, and the action avoidance frequency is obtained based on the number of limb avoidances per unit cycle.

5. The method according to claim 4, characterized in that, When the distance change trend is decreasing and the distance between the execution end and the user is less than the distance threshold, and the movement trajectory is as follows, the motion parameters of the limb movement trajectory are obtained. The motion parameters include the movement direction and the movement amplitude, including: Based on the distance change trend, the interval distance between the execution terminal and the user at consecutive moments is obtained; When the interval gradually decreases, it is determined to be a decreasing trend, and when the interval is less than a threshold, the key joints from the previous moment to the next moment are connected to obtain the direction of movement of the limb movement trajectory. The range of motion of the limb movement trajectory is obtained by the displacement of key joints from the previous moment to the next moment.

6. The method according to claim 4, characterized in that, The avoidance conditions include: The direction from the user's location point to the execution end's location point is determined as the approach direction. When the angle between the movement direction and the approach direction is greater than or equal to the angle threshold and the movement amplitude is greater than or equal to the amplitude threshold, the avoidance condition is satisfied.

7. The method according to claim 1, characterized in that, By comprehensively estimating the frequency of eye contact avoidance and the frequency of motor avoidance, a cooperation level is obtained, including: By comparing the frequency of eye contact avoidance and the frequency of action avoidance with the frequency range corresponding to each preset cooperation level, the user's current cooperation level is determined based on the comparison results.

8. The method according to claim 1, characterized in that, When the cooperation level is lower than the sensitivity level of the current progress node, a scene switching command is triggered to switch the scene elements corresponding to the current progress node, including: The current progress node is determined based on the execution action of the execution end. When the cooperation level is lower than the sensitivity level of the current progress node, a scene switching command is triggered. Based on the key elements identified by the simulation terminal, the set of scene elements corresponding to the current progress node is retrieved; Based on the comparison results between the key elements and the scene element set, the key elements are replaced with the interactive images corresponding to the scene elements, and the execution terminal is controlled to respond to the voice data corresponding to the scene elements.

9. The method according to claim 8, characterized in that, The current progress node is determined based on the execution actions performed by the executor, including: Identify the retrieval tool corresponding to the execution end, and select multiple progress nodes corresponding to the retrieval tool; The motion feature parameters of the execution action of the execution end are detected, and the motion feature parameters include at least the action posture, motion trajectory and speed; The current progress node is determined by comparing the action feature parameters with the configuration feature parameters of each progress node.

10. A VR-based cooperation data processing system, characterized in that, include: The gaze determination module is used to determine the frequency of the user's gaze avoidance based on gaze tracking data collected from the analog terminal; The action determination module is used to determine the frequency of user action avoidance based on the limb posture data collected by the monitoring terminal; The level estimation module is used to comprehensively estimate the frequency of eye contact avoidance and the frequency of action avoidance to obtain the cooperation level; The scene switching module is used to trigger a scene switching command and switch the scene element corresponding to the current progress node when the cooperation level is lower than the sensitivity level of the current progress node.

Images