Multi-strategy combined object simulation method and system

Abstract

The embodiment of the invention discloses a multi-strategy combined object simulation method which comprises the following steps: acquiring first motion data of an object in any frame through an unmarked optical motion capture strategy, and making a simulation model make a corresponding posture according to the first motion data; acquiring second motion data of the reference part of the object in the same frame through a marked optical motion capture strategy, and correcting three-dimensional space information of the simulation model according to the second motion data; acquiring third motion data of the interested part of the object in the same frame through a marked optical motion capture strategy, and correcting motion information of the interested area of the simulation model according to the third motion data; wherein the region of interest is an associated region of the part of interest; and obtaining a simulation effect of the object in the frame.

Description

Technical Field

[0001] The present application relates to the field of simulation technology, and in particular to an object simulation method and system combining multiple strategies. Background Art

[0002] Human factors engineering is an indispensable verification link in many fields. By building a human-in-the-loop simulation verification environment, conducting simulation tests for typical human-computer interaction scenarios, and monitoring the behavior, physiology, and psychology of test personnel during the test, the safety, usability, and comfort of the design solution are analyzed and evaluated, thereby improving the competitiveness of the product.

[0003] For example, during the development of civil aircraft, the digital pilot driving method in the civil aviation field focuses on how to ensure that the body posture of the monitoring test personnel in the real scene is completely and accurately mapped to the digital pilot in the virtual scene in a simulation verification environment. At the same time, it is also necessary to ensure the accuracy of the mapping of the relative positions of the limb extremities and torso of the monitoring test personnel in the scene in a complex environment, thereby ensuring the reliability of the simulation.

[0004] Traditional digital human driving is often achieved with the help of motion capture systems. Common methods include: driving methods based on inertial motion capture, driving methods based on optical motion capture with markers, and driving methods based on optical motion capture without markers. The method based on inertial motion capture places IMUs on various key parts of the human body, measures the steering angles and motion accelerations of various parts of the human body, and reconstructs the movement of the human body. On the one hand, this method has problems such as position drift and insufficient end tracking accuracy due to cumulative errors. On the other hand, the work of IMU will be interfered by the electromagnetic environment. There are many metal parts and circuit components in the virtual-reality fusion cockpit simulation environment, and there is a certain amount of electromagnetic interference. In this environment, the measurement accuracy of IMU decreases, the jitter and noise of the returned data increase, and the effect of human body tracking is not ideal. The method based on optical motion capture with markers requires installing reflective markers on the surface of the human body, and arranging multiple infrared cameras in the scene. Multiple cameras shoot the markers on the human body from different angles, and calculate the three-dimensional positions of these markers to reconstruct the motion trajectory and posture of the human body parts. The degree to which this method can restore human posture depends on the number of markers fixed on the human body surface during use. If the number of markers is too small, only the motion data of some body parts can be obtained, resulting in a stiff posture of the digital human. If the number of markers is too large, the markers of different parts of the human body will interfere with each other, causing marker tracking failure or recognition errors, resulting in distortion of the digital human posture. Human-in-the-loop simulation of the virtual-reality fusion cockpit of civil aircraft is often operated by two pilots in a small space, and it is very easy for the markers of the main and co-pilots to interfere with each other. The method based on markerless optical motion capture requires the arrangement of multiple RGB cameras in the scene, extracting human contour information and key point information through neural networks, identifying key parts of the human body, and then capturing human motion in real time. The positioning accuracy error of this method is often at the centimeter level, which is slightly inferior to the tracking accuracy of optical motion capture with markers. Summary of the invention

[0005] The embodiments of this specification provide an object simulation method and system combining multiple strategies to solve the technical problem of how to improve the object simulation effect.

[0006] To solve the above technical problems, the embodiments of this specification provide the following technical solutions:

[0007] This specification provides an object simulation method combining multiple strategies, the method comprising:

[0008] Acquire the first motion data of the object in any frame through a markerless optical motion capture strategy, and make the simulation model make a corresponding gesture according to the first motion data; acquire the second motion data of the reference part of the object in the same frame through a marker optical motion capture strategy, and correct the three-dimensional spatial information of the simulation model according to the second motion data;

[0009] Acquire the third motion data of the part of interest of the object in the same frame by means of a marked optical motion capture strategy, and correct the motion information of the region of interest of the simulation model according to the third motion data; wherein the region of interest is an associated region of the part of interest;

[0010] Get the simulation effect of the object in this frame.

[0011] Optionally, according to the first motion data, causing the simulation model to make a corresponding gesture includes:

[0012] The first motion data is input into the movable joints corresponding to the simulation model to drive the simulation model to make a corresponding posture.

[0013] Optionally, correcting the three-dimensional spatial information of the simulation model according to the second motion data includes:

[0014] The second motion data is overwritten with the fourth motion data of the reference part in the same frame acquired by the marker-free optical motion capture strategy to correct the three-dimensional spatial information of the simulation model.

[0015] Optionally, correcting the motion information of the region of interest of the simulation model according to the third motion data includes:

[0016] The third motion data is used as an effector, and the motion information of the region of interest of the simulation model is solved using an inverse kinematics algorithm, and the motion information of the region of interest of the simulation model is corrected according to the solution result.

[0017] Optionally, obtaining the simulation effect of the object in the frame includes:

[0018] Performing posture detection on the object model;

[0019] If it is determined according to the posture detection result that a preset situation will occur in the target part of the simulation model, the target part is made to change its posture.

[0020] Optionally, causing the target part to change its posture includes:

[0021] Determine the expected posture effect of the target part, and drive the target part to achieve the expected posture effect.

[0022] Optionally, determining the expected posture effect of the target part includes:

[0023] The expected gesture effect is determined according to the content of the preset situation.

[0024] Optionally, the expected gesture effect is determined according to the content of the preset situation, including:

[0025] According to the content of the preset situation, a gesture effect matching the content of the preset situation is determined as the expected gesture effect.

[0026] Optionally, the method further includes:

[0027] According to the simulation effects of the object in continuous frames, the continuous simulation effect of the object is obtained.

[0028] The embodiment of this specification provides an object simulation system combining multiple strategies, the system comprising:

[0029] A markerless optical motion capture device, used to obtain first motion data of an object in any frame;

[0030] A marked optical motion capture device is used to obtain second motion data of a reference part of the object in the same frame, and to obtain third motion data of an interested part of the object in the same frame;

[0031] A simulation module is used to make the simulation model perform a corresponding posture according to the first motion data; to correct the three-dimensional space information of the simulation model according to the second motion data; to correct the motion information of the region of interest of the simulation model according to the third motion data; and to obtain the simulation effect of the object in the frame; wherein the region of interest is an associated region of the part of interest.

[0032] At least one of the above technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects:

[0033] By mixing marked optical motion capture and markerless optical motion capture, we can not only take advantage of markerless optical motion capture's ability to capture motion details to improve the naturalness of the simulation model's motion posture, but also use marked optical motion capture to improve the positioning accuracy of the object, thereby effectively improving the object simulation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the drawings required for use in the embodiments of this specification or the prior art description are briefly described below. Obviously, the following only describes the drawings required for use in some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor.

[0035] Figure 1 It is a flowchart of the object simulation method combining multiple strategies provided in the first embodiment of this specification.

[0036] Figure 2 It is a schematic diagram of the overall process of object simulation in the example provided in the first embodiment of this specification.

[0037] Figure 3 This is a schematic diagram of the multi-strategy object simulation process of the object simulation in the example provided in the first embodiment of this specification.

[0038] Figure 4 This is a schematic diagram of object motion capture in the first embodiment of this specification.

[0039] Figure 5 It is a schematic diagram of the simulation effect in the first embodiment of this specification.

[0040] Figure 6 It is a schematic diagram of the structure of the object simulation system combining multiple strategies provided in the second embodiment of this specification. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments involved in the specific implementation methods are only part of the embodiments of this application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the specific implementation methods without making creative work should belong to the scope of protection of this application.

[0042] The first embodiment of this specification (hereinafter referred to as "embodiment one") provides an object simulation method combining multiple strategies. The execution subject of embodiment one includes but is not limited to a terminal or a server or an operating system or an application, that is, the execution subject can be various and can be set, used or changed as needed. In addition, a third-party application can also assist the execution subject in executing embodiment one. For example, the method provided in embodiment one can be executed by a server, and a corresponding application can be installed on a terminal (the terminal can be held by a user), and data can be transmitted between the terminal or the application and the server, thereby assisting the server in executing the method provided in embodiment one.

[0043] refer to Figure 1 The object simulation method combining multiple strategies provided in the first embodiment includes:

[0044] S101: acquiring first motion data of an object in any frame by means of a markerless optical motion capture strategy, and making a simulation model perform a corresponding posture according to the first motion data; acquiring second motion data of a reference part of the object in the same frame by means of a marker optical motion capture strategy, and correcting the three-dimensional spatial information of the simulation model according to the second motion data;

[0045] In the first embodiment, the motion data of the object in any frame can be obtained by a markerless optical motion capture strategy, and the motion data is used as the first motion data. The object can refer to a person, an animal, or various objects. The first motion data includes but is not limited to posture information, which is not limited to the first embodiment.

[0046] Specifically, acquiring the first motion data of the object in any frame by using a marker-free optical motion capture strategy may include: acquiring the first motion data of the object in any frame by using a marker-free optical motion capture device.

[0047] According to the first motion data, the simulation model corresponding to the object (hereinafter referred to as the simulation model) can be made to make a corresponding gesture. For example, if the object makes a certain action in the frame (i.e., any of the above frames), the first motion data of the object in the frame can be obtained through the markerless optical motion capture strategy, and according to the first motion data, the simulation model can be made to make the same gesture.

[0048] Specifically, making the simulation model perform a corresponding posture according to the first motion data may include: inputting the first motion data into movable joints corresponding to the simulation model to drive the simulation model to perform a corresponding posture.

[0049] In the first embodiment, the motion data of the reference part of the object in the same frame (i.e., the same frame of any of the above frames) can also be obtained through a marked optical motion capture strategy, and the motion data is used as the second motion data. The reference part can be set as needed, including determining the reference part according to the reference system origin of the object model. For example, the object is a human body, and the simulation model is a digital pilot. Since the reference system origin of the digital pilot is at the waist, the waist of the object can be used as the reference part. The second motion data includes but is not limited to position data, which is not limited to the first embodiment.

[0050] Specifically, acquiring the second motion data of the reference part of the object in the same frame by using a marked optical motion capture strategy may include: acquiring the second motion data of the object in the same frame by using a marked optical motion capture device.

[0051] According to the second motion data, the three-dimensional spatial information of the simulation model can be corrected. Wherein, correcting the three-dimensional spatial information of the simulation model according to the second motion data includes: overwriting the motion data of the reference part in the same frame obtained by the markerless optical motion capture strategy with the second motion data (the motion data is used as the fourth motion data) to correct the three-dimensional spatial information of the simulation model. That is to say, the motion data of the reference part of the object, that is, the fourth motion data, is included in the aforementioned first motion data obtained by the markerless optical motion capture strategy. The fourth motion data includes but is not limited to position data. By overwriting the fourth motion data with the "second motion data of the reference part obtained by the marked optical motion capture strategy", the correction of the three-dimensional spatial information of the simulation model is achieved.

[0052] It should be noted that there is no absolute order between obtaining the first motion data and obtaining the second motion data, and the two can be performed in parallel.

[0053] S103: Acquire third motion data of the part of interest of the object in the same frame through a marked optical motion capture strategy, and correct motion information of the region of interest of the simulation model according to the third motion data; wherein the region of interest is an associated region of the part of interest.

[0054] In practical applications, the focus of attention, i.e., the part of interest, may be different for different objects. For example, if the object is a human body and the simulation model is a digital pilot, the hand movement of the digital pilot is the focus of the simulation. The accuracy of hand positioning directly affects the various operations of the pilot during flight, so the hand can be regarded as the part of interest.

[0055] In the first embodiment, the motion data of the part of interest can be used to perform a region correction operation on the region of interest of the simulation model. Specifically, considering the tracking accuracy of the part of interest of the object, optical markers are also added to the part of interest of the object to correct the motion information of the region of interest of the object model. For example, the daily activities of the object model mainly revolve around the upper limbs, so the positioning accuracy of the hand of the object model is particularly important. In order to ensure the accuracy of the hand tracking of the object model, optical markers are also added to the hand of the object, and the inverse kinematics algorithm is used to correct the motion of the region of interest such as the arm area or upper limb area of ​​the pilot, thereby improving the tracking accuracy of the hand and upper arm area or upper limb area.

[0056] The following further describes the content of the area correction operation:

[0057] In the first embodiment, the motion data of the part of interest of the object in the same frame can be obtained by a marked optical motion capture strategy, and the motion data is used as the third motion data, and the motion information of the region of interest of the simulation model is corrected according to the third motion data. The region of interest is an associated region or a connected region of the part of interest. For example, if the part of interest is a hand, the region of interest can be an arm region or an upper limb region associated with or connected to the hand.

[0058] Acquiring the third motion data of the part of interest of the object in the same frame by using a marked optical motion capture strategy may include: acquiring the third motion data of the part of interest of the object in the same frame by using a marked optical motion capture device.

[0059] Correcting the motion information of the region of interest of the simulation model according to the third motion data may include: using the third motion data as an effector, solving the motion information of the region of interest of the simulation model using an inverse kinematics algorithm, and correcting the motion information of the region of interest of the simulation model according to the solution result.

[0060] S105: Obtaining the simulation effect of the object in the frame.

[0061] On the basis of correcting the three-dimensional space information of the simulation model and correcting the motion information of the region of interest of the simulation model, the simulation effect of the object in the frame can be obtained, that is, the effect displayed by the simulation model.

[0062] Furthermore, in order to obtain a better simulation effect, a calibration operation may be performed. That is, obtaining the simulation effect of the object in the frame may include: performing posture detection on the object model; if it is determined according to the posture detection result that a preset situation will occur in the target part of the simulation model, the target part is caused to change its posture. This strategy of causing the target part to change its posture by detecting whether a preset situation occurs realizes posture calibration of the target part, which can be called a calibration operation for the target part.

[0063] The following further describes the contents of the calibration operation:

[0064] In the first embodiment, performing posture detection on the object model may include: detecting whether a target part of the object model will collide. Still taking the human body as the object and the digital pilot as the simulation model as an example, assuming that the environment of the digital pilot is an open cockpit simulation platform (i.e., the simulation of the human body is performed on the open cockpit simulation platform), on the basis of correcting the three-dimensional spatial information of the simulation model and correcting the motion information of the region of interest of the simulation model, the target part of the simulation model may have certain preset situations, for example, the target part model of the simulation model may collide with other models in the cockpit, and thus it is necessary to process the preset situation, including making the target part change its posture.

[0065] In the first embodiment, causing the target part to change its posture may include: determining an expected posture effect of the target part, and driving the target part to achieve the expected posture effect.

[0066] In particular, the expected posture effect of the target part may be different for different preset situations that occur. Therefore, determining the expected posture effect of the target part may include: determining the expected posture effect according to the content of the preset situation. Wherein, determining the expected posture effect according to the content of the preset situation includes: determining a posture effect that matches the content of the preset situation according to the content of the preset situation as the expected posture effect. For example, various posture effect information can be stored in a database, and the content of the preset situation can be matched with various posture effect information in the database to determine a posture effect that matches the content of the preset situation as the expected posture effect. In other words, a matching strategy can be used to determine the expected posture effect.

[0067] For example, assuming that the preset situation is that the target part of the object model will collide with other parts in the simulation environment, the content of the collision situation can be analyzed, including which parts the target part will collide with. According to the specific part that the target part collides with, the posture effect that matches the specific part is determined as the expected posture effect of the target part, and the target part is made to achieve the expected posture effect.

[0068] For any frame, the simulation effect of the object in the frame can be obtained according to the above content. Then, according to the simulation effect of the object in each consecutive frame, the continuous simulation effect of the object can be obtained, that is, the dynamic effect reflected by the simulation model following the movement of the object.

[0069] The following further illustrates the content of the first embodiment through examples:

[0070] refer to Figure 2 and Figure 3,In this example, taking the object as a human body and the simulation model as a digital pilot, ,motion tracking devices can be set on the human body and arranged in the ,simulation environment, e.g. Figure 4 As shown. The waist of the human body is used as the reference part, the hand is used as the part of interest, and the area of ​​interest is the arm area. Optical markers are installed on the waist and both hands, and the specific installation positions of the optical markers on the waist and the hands are not limited to the first embodiment.

[0071] The human body simulation is carried out in an open cockpit simulation platform. The RGB cameras of the markerless optical motion capture system and the infrared cameras of the marker optical motion capture system are arranged around the cockpit. The markerless optical motion capture system and the marker optical motion capture system are calibrated, and the simulation scene is matched with the virtual and real. The sampling frequency of the infrared camera and the RGB camera with a lower sampling frequency is used as the refresh rate of the digital pilot's movement.

[0072] First, coordinate transformation is needed to unify the human body data collected by the two sets of motion capture into the same reference system. Both marked optical motion capture and markerless optical motion capture use physical references in the real world to establish reference systems, but the physical references used by the two are different, and it is difficult to achieve accurate coordinate transformation of the two reference systems by measuring the relative positions of the physical objects. In this example, the world coordinate system is established based on the marked optical motion capture reference system, and the transformation from the world reference system P to the marked optical reference system O is Considering that the origin of the digital pilot's reference system is defined at the waist of the digital human, this example installs optical markers on the waist of the human body, and uses the tracking data of the waist to establish a medium for conversion between the marked optical reference system and the unmarked optical reference system.

[0073] It is known that the three-dimensional spatial information of the waist under the world reference frame P captured by the marked optical motion capture is in is the three-dimensional rotation information of the waist under the world reference system P, is the three-dimensional position information of the waist in the world reference system P. The three-dimensional spatial information of the waist captured by markerless optical motion capture in the markerless optical reference system N is in is the three-dimensional rotation information of the waist under the unmarked optical reference system N, is the three-dimensional position information of the waist under the markerless optical reference system N. Then the transformation from the world coordinate system to the markerless optical coordinate system is It can be defined as follows:

[0074]

[0075] in represents a rotation transformation, Represents translation transformation. This puts the marked optical reference system and the markerless optical reference system into the world reference system, making it easier to use the human motion data captured by marked optical motion capture and markerless optical motion capture to drive the digital pilot.

[0076] The first motion data of the human body in any frame is collected by a marker-free optical motion capture device (i.e., an RGB camera), and the first motion data of the frame is input into the corresponding movable joints of the digital pilot to drive the digital pilot to make the same posture.

[0077] The second motion data (including but not limited to position data or rotation data) of the human waist in the frame is collected by a marked optical motion capture device (i.e., an infrared camera), and the waist motion data of the digital pilot is updated with the second motion data, including overwriting the waist motion data (i.e., the fourth motion data) provided by the unmarked optical motion capture device, so as to correct the three-dimensional spatial information of the digital pilot.

[0078] In this example, the hand positioning accuracy of the digital pilot is very important during the simulation process, and whether various interactive simulations can be carried out normally depends on whether the hand positioning is accurate. Due to its algorithm principle, markerless optical motion capture has certain deficiencies in spatial positioning accuracy. In this example, the hand is taken as the part of interest, the arm area is taken as the area of ​​interest, and a regional correction operation is performed on the arm area. The correction strategy adopted in the regional correction operation utilizes the inverse kinematics algorithm, combines the hand optical marker points and the palm position and rotation information captured by the marked optical motion capture, and re-solves and corrects the arm area motion data captured by the markerless optical motion capture. Specifically, the regional correction operation includes obtaining the third motion data of the human hands collected by the marked optical motion capture device in the frame (including but not limited to the three-dimensional spatial information of the hand including the position data or rotation data), and using the inverse kinematics algorithm to solve the motion information of the digital pilot's arm area. The inverse kinematics algorithm uses the upper arm of the digital pilot as the root bone, the lower arm as the middle bone, the palm as the terminal bone, and the third motion data of the frame input by the marked optical motion capture device as the effector, and recalculates the rotation information of the current shoulder joint, the position and rotation information of the elbow joint, and the position and rotation information of the wrist joint (the shoulder joint, elbow joint, and wrist joint all belong to the arm area). The shoulder joint rotation information, elbow joint position and rotation information, and wrist joint position and rotation information obtained by the inverse kinematics algorithm are input into the digital pilot, and the motion information of the digital pilot's arm area (the motion information includes but is not limited to the position information) is updated and corrected (specifically, the corresponding hand motion information obtained by the markerless optical motion capture strategy (which can be included in the first motion data) is overwritten and corrected).

[0079] In actual situations, specific parts of the object model, such as hands, may collide with preset situations. For example, if the human body interacts with an interactive component, the digital pilot needs to perform interactive operations on the interactive component model accordingly, and it will be detected that the digital pilot's hand model collides with the interactive component model. In terms of gesture drive of the digital pilot, the pilot gestures that are concerned during the simulation process are often related to the operation, that is, the digital pilot focuses on the operation gestures when interacting with the equipment in the cabin, rather than the finger movements throughout the process. Therefore, in this example, the hand is used as the target part and a calibration operation is performed on the hand. The calibration operation makes comprehensive use of preset situation detection and matching strategies to make the digital pilot's hand model change according to the situation.

[0080] The calibration operation in this example includes: detecting whether the hand model of the digital pilot in the frame collides with the interactive model in the cockpit. If the detection result is that a collision will occur, the hand model is made to change its posture. Specifically, according to the content of the collision (including the specific components or models that collided, which fall within the scope of the detection results), combined with the matching strategy (because the target part in this example is the hand, the matching strategy is also called the gesture matching strategy), the posture effect that matches the collision content is determined in the digital pilot gesture database as the expected posture effect. The corresponding parts of the digital pilot (such as knuckles) are driven so that the digital pilot's hands achieve the expected posture effect (such as making corresponding gestures).

[0081] For example, when the digital pilot collides with the interactive component to perform an interactive operation, the gesture matching strategy is used to match the expected gesture effect of the digital pilot's hand in the database according to the interactive operation performed by the digital pilot, that is, to achieve the expected gesture effect of the interactive operation. In actual applications, the gesture of the digital pilot is in a natural extended state by default. When it is detected that the hand model of the digital pilot collides with the interactive component in the virtual scene, the gesture matching strategy is used to determine the gesture information matching the interactive component from the database (that is, according to the content of the preset situation, the gesture effect matching the content of the preset situation is determined). The gesture posture information matching the interactive component is called from the database, and the finger joints of the digital pilot are driven according to the gesture posture information, so that the digital pilot makes the same gesture (that is, achieves the expected gesture effect). Other parts of the digital pilot associated or connected with the hand model, such as the wrist joint, can also be rotated and moved accordingly. In this way, after the hand model of the digital pilot achieves the expected gesture effect through the calibration operation, the simulation operation effect of the interactive component is achieved.

[0082] In this example, the above detection can be performed at a certain frequency. Figure 3Where N = 2 × refresh rate, N represents the number of collision detections of the system within 2 seconds. If the data refresh rate is 30hz, then 30 detections are made per second, and N is equal to 60.

[0083] If the current hand model does not collide with any interactive component within 2 seconds, the digital pilot will restore the default gesture. When the hand model is detected to collide with a new interactive component, the new gesture information is immediately matched from the database to drive the digital pilot's fingers to achieve the expected gesture effect.

[0084] Through the above content of this example, we can get the simulation effect of the human body in this frame, for example Figure 5 The above contents can be repeatedly executed to obtain the simulation effect of the human body in each consecutive frame, thereby obtaining the continuous simulation effect of the human body and realizing the continuous movement of the digital pilot.

[0085] The first to fourth motion data mentioned above can all be three-dimensional space information, and their specific contents are not limited.

[0086] Embodiment 1 can achieve the following beneficial effects:

[0087] A hybrid motion capture driving strategy is adopted to obtain the object posture information through markerless optical motion capture and correct the spatial position of the object model in combination with marker optical motion capture. Perform regional correction operations, use the inverse kinematics algorithm and the optical marker points of the object model to solve the motion information of the object model's region of interest, and ensure the accuracy of the positioning of the part of interest and the region of interest. Perform calibration operations based on matching strategies to achieve the rationality of the posture of the target part of the object model and realize the drive of the target part in the interactive state. The following is a detailed description:

[0088] First

[0089] The first embodiment proposes that the effect of restoring the posture of an object by using marked optical motion capture depends on the number of marked points on the surface of the object. If the number of marked points is too small, only the motion data of some body parts can be obtained, and the object model will show a stiff posture. If the number of marked points is too large, the marked points of different parts of the object will interfere with each other, causing the marking point tracking failure and the digital human posture distortion. The spatial positioning accuracy of markerless optical motion capture is limited.

[0090] Embodiment 1 utilizes the first motion data obtained by the markerless optical motion capture strategy and the second motion data of the reference part obtained by the marked optical motion capture strategy, and corrects the three-dimensional spatial information of the simulation model according to the second motion data. In this way, the object model is driven to make corresponding actions at the corresponding position in the virtual scene through the hybrid motion capture driving strategy of markerless optical motion capture and marked optical motion capture. Embodiment 1 mixes marked optical motion capture and markerless optical motion capture, combines the advantages of high human body posture restoration of markerless optical motion capture and accurate spatial positioning of marked optical motion capture, uses markerless optical motion capture to track the motion posture of the object, and installs optical marker points at the reference part of the object, supplemented by marked optical motion capture to make real-time corrections to the spatial position of the object model.

[0091] The core of the hybrid motion capture driving strategy in the first embodiment is to use markerless optical motion capture to capture the body posture of the object, ensuring that the object posture is realistic and natural. The three-dimensional spatial information of the reference part provided by the marked optical motion capture is used to correct the position of the object model and improve the spatial positioning accuracy of the object model. The purpose of the hybrid motion capture driving strategy is: first, to give full play to the advantages of high posture restoration of markerless optical motion capture and strong perception of subtle movements, and to use the markerless optical motion capture strategy to capture the body posture of the object, so as to alleviate the distortion or stiffness of the body posture of the object model caused by the failure of marker point tracking or too few marker points in the marked optical motion capture scheme; second, the object model has a reference part, for example, the origin of the reference system of the digital pilot (belonging to the reference part) is at the waist, and the spatial positioning accuracy of the marked optical motion capture is extremely high, so optical motion capture tracking is used at the reference part of the object, and the second motion data of the reference part tracked by the marked optical motion capture is used to correct the three-dimensional spatial position of the object model, which solves the problem of insufficient spatial positioning accuracy of markerless optical motion capture.

[0092] Through the above-mentioned simulation method that combines multiple strategies of marked optical motion capture and markerless optical motion capture, the advantage of markerless optical motion capture in capturing the details of human body movement is brought into play to improve the naturalness of the object model's motion posture, and the marked optical motion capture is used to improve the positioning accuracy of the object and the positioning accuracy of parts of interest such as the hands. It can solve the problems of model posture distortion or stiffness, position offset, etc. existing in traditional simulation solutions, and effectively improve the object simulation effect.

[0093] second

[0094] In the first embodiment, the motion information of the region of interest of the simulation model is corrected by the third motion data of the region of interest of the object. The purpose of the region correction operation is to improve the tracking accuracy of the region of interest and the region of interest by combining the optical marker points of the region of interest with the marked optical motion capture, and effectively improve the object simulation effect.

[0095] Take the case where the object is a human body, the object model is a digital pilot, the part of interest is a hand, and the area of ​​interest is an arm area. The hand movements of the digital pilot are the focus of simulation, and the accuracy of hand positioning directly affects the various operations of the pilot during flight. However, markerless optical motion capture predicts the spatial position of the pilot's hand based on a neural network, with limited accuracy and errors often at the centimeter level, which is difficult to meet simulation requirements. In comparison, marked optical motion capture calculates the current three-dimensional spatial information of the hand through spatial geometry, with higher accuracy and errors at the sub-millimeter level. In Example 1, an inverse kinematics algorithm is used to recalculate the movement of the pilot's arm area based on the second motion data of the hand, and the motion information of the arm area is corrected to ensure that the movement of the pilot's arm area is smooth and natural, solving the problems of inaccurate drive positions of the hands and inaccurate gesture tracking, and improving the object simulation effect.

[0096] third

[0097] In the first embodiment, a calibration operation is performed on the target part of the object model, and the posture of the target part is calibrated through the calibration operation, so that the target part achieves the expected posture effect, and the simulation effect is improved. In particular, in the calibration operation, the first embodiment designs a method for determining a posture effect that matches the content of a preset situation, as a matching strategy for the expected posture effect, which can determine the reasonable expected posture effect that the target part should achieve in the database according to the content of the preset situation, including the interactive operation performed by the object model, so as to enable the target part to achieve the expected posture effect, and effectively improve the simulation effect and efficiency.

[0098] Taking the target part as the hand as an example, using additional hardware to track the gestures of the pilot is not only inconvenient to use, but also difficult to ensure the accuracy of finger tracking and gesture accuracy during the interaction process. The gesture matching strategy and calibration operation adopted in Example 1 are simpler and more accurate in gesture tracking, which can meet the simulation requirements. Compared with the hardware-based gesture tracking solution, it is more stable and convenient, and has less resource overhead while meeting the simulation requirements.

[0099] In the first embodiment, the part of interest and the target part may be the same or overlapped. The tracking accuracy of the part of interest and the region of interest may be improved through the region correction operation, and the posture accuracy of the target part may be improved through the calibration operation, thereby comprehensively improving the simulation effects of the part of interest and the target part.

[0100] Embodiment 1 has a wide range of application scenarios. For example, embodiment 1 can be used as a digital pilot driving solution for human-in-the-loop virtual simulation of civil aircraft cockpits, and can be used for digital pilot driving of human-in-the-loop simulation of civil aircraft. It can solve the problems of simulation driving of civil aviation digital pilots in the process of human factors simulation analysis, including the problems of distorted or stiff digital human posture, offset digital human position, inaccurate hand driving position and inaccurate gesture tracking in traditional human driving solutions in the field of civil aviation digital pilots. Ultimately, in a virtual-reality fusion simulation verification environment, human motion data can be accurately mapped to the digital pilot.

[0101] The second embodiment of this specification provides an object simulation system combining multiple strategies corresponding to the method described in the first embodiment. Figure 6 , the system comprising:

[0102] A markerless optical motion capture device 202, used to obtain first motion data of an object in any frame;

[0103] A marked optical motion capture device 204, used to obtain second motion data of a reference part of the object in the same frame, and to obtain third motion data of an interested part of the object in the same frame;

[0104] The simulation module 206 is used to make the simulation model perform a corresponding posture according to the first motion data; to correct the three-dimensional spatial information of the simulation model according to the second motion data; to correct the motion information of the region of interest of the simulation model according to the third motion data; and to obtain the simulation effect of the object in the frame; wherein the region of interest is an associated region of the part of interest.

[0105] The simulation module 206 may be a module on a computer.

[0106] Optionally, according to the first motion data, causing the simulation model to make a corresponding gesture includes:

[0107] The first motion data is input into the movable joints corresponding to the simulation model to drive the simulation model to make a corresponding posture.

[0108] Optionally, correcting the three-dimensional spatial information of the simulation model according to the second motion data includes:

[0109] The second motion data is overwritten with the fourth motion data of the reference part in the same frame acquired by the marker-free optical motion capture strategy to correct the three-dimensional spatial information of the simulation model.

[0110] Optionally, correcting the motion information of the region of interest of the simulation model according to the third motion data includes:

[0111] The third motion data is used as an effector, and the motion information of the region of interest of the simulation model is solved using an inverse kinematics algorithm, and the motion information of the region of interest of the simulation model is corrected according to the solution result.

[0112] Optionally, obtaining the simulation effect of the object in the frame includes:

[0113] Performing posture detection on the object model;

[0114] If it is determined according to the posture detection result that a preset situation will occur in the target part of the simulation model, the target part is made to change its posture.

[0115] Optionally, causing the target part to change its posture includes:

[0116] Determine the expected posture effect of the target part, and drive the target part to achieve the expected posture effect.

[0117] Optionally, determining the expected posture effect of the target part includes:

[0118] The expected gesture effect is determined according to the content of the preset situation.

[0119] Optionally, the expected gesture effect is determined according to the content of the preset situation, including:

[0120] According to the content of the preset situation, a gesture effect matching the content of the preset situation is determined as the expected gesture effect.

[0121] Optionally, the simulation module 206 is further configured to obtain a continuous simulation effect of the object according to the simulation effects of the object in continuous frames.

[0122] The parts of the second embodiment that are not described in detail refer to the first embodiment. The second embodiment can achieve the same beneficial effects as the first embodiment.

[0123] The above is only an embodiment of this specification and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An object simulation method combining multiple strategies, characterized in that: The method comprises: Acquire the first motion data of the object in any frame through a markerless optical motion capture strategy, and make the simulation model make a corresponding gesture according to the first motion data; acquire the second motion data of the reference part of the object in the same frame through a marker optical motion capture strategy, and correct the three-dimensional spatial information of the simulation model according to the second motion data; Acquire the third motion data of the part of interest of the object in the same frame by means of a marked optical motion capture strategy, and correct the motion information of the region of interest of the simulation model according to the third motion data; wherein the region of interest is an associated region of the part of interest; Get the simulation effect of the object in this frame.

2. The method according to claim 1, characterized in that According to the first motion data, the simulation model is made to perform a corresponding posture, including: The first motion data is input into the movable joints corresponding to the simulation model to drive the simulation model to make a corresponding posture.

3. The method according to claim 1, characterized in that Correcting the three-dimensional spatial information of the simulation model according to the second motion data includes: The second motion data is overwritten with the fourth motion data of the reference part in the same frame acquired by the marker-free optical motion capture strategy to correct the three-dimensional spatial information of the simulation model.

4. The method according to claim 1, characterized in that Correcting the motion information of the region of interest of the simulation model according to the third motion data includes: The third motion data is used as an effector, and the motion information of the region of interest of the simulation model is solved using an inverse kinematics algorithm, and the motion information of the region of interest of the simulation model is corrected according to the solution result.

5. The method according to any one of claims 1 to 4, characterized in that Obtaining the simulation effect of the object in the frame includes: Performing posture detection on the object model; If it is determined according to the posture detection result that a preset situation will occur in the target part of the simulation model, the target part is made to change its posture.

6. The method according to claim 5, characterized in that Causing the target part to change its posture includes: Determine the expected posture effect of the target part, and drive the target part to achieve the expected posture effect.

7. The method according to claim 6, characterized in that Determining the expected posture effect of the target part includes: The expected gesture effect is determined according to the content of the preset situation.

8. The method according to claim 7, characterized in that Determine the expected posture effect according to the content of the preset situation, including: According to the content of the preset situation, a gesture effect matching the content of the preset situation is determined as the expected gesture effect.

9. The method according to claim 1, characterized in that The method further comprises: According to the simulation effects of the object in continuous frames, the continuous simulation effect of the object is obtained.

10. An object simulation system combining multiple strategies, characterized in that: The system comprises: A markerless optical motion capture device, used to obtain first motion data of an object in any frame; A marked optical motion capture device is used to obtain second motion data of a reference part of the object in the same frame, and to obtain third motion data of an interested part of the object in the same frame; A simulation module is used to make the simulation model perform a corresponding posture according to the first motion data; to correct the three-dimensional space information of the simulation model according to the second motion data; to correct the motion information of the region of interest of the simulation model according to the third motion data; and to obtain the simulation effect of the object in the frame; wherein the region of interest is an associated region of the part of interest.

Images