Motion capture method, apparatus, device, and storage medium

Abstract

Embodiments of the present application disclose a motion capture method, device and equipment and a storage medium, and belong to the motion capture technical field. The method comprises: determining first skeleton data of a motion capture object in a motion capture scene based on first optical data collected by an optical motion capture camera, wherein the first optical data is obtained by collecting optical capture points arranged on the motion capture object; determining second skeleton data based on image data collected by a video analysis camera, wherein the image data contains the motion capture object in the motion capture scene, and the second skeleton data and the first skeleton data are based on the same coordinate space; and optimizing the first skeleton data based on the second skeleton data to obtain target skeleton data. By using the scheme provided in the embodiments of the present application, the target skeleton data can better represent the real action form of the motion capture object, and the quality of the target skeleton data is improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of motion capture, and particularly relate to a motion capture method, device, equipment and storage medium. BACKGROUND

[0002] With the rapid development of computer hardware and software technology and the improvement of animation production requirements, motion capture technology has also been developed accordingly.

[0003] In the related art, by setting infrared light balls on the motion actor, the spatial positions of the infrared light balls are determined by using optical motion capture cameras to obtain infrared light reflected by the infrared light balls, so that the motion capture of the motion actor is realized based on the spatial positions of the infrared light balls.

[0004] However, when there is motion occlusion of the motion actor or there is light interference that cannot be captured by the optical motion capture camera, the motion capture performed only by the optical motion capture camera will result in missing motion collection and poor motion capture data quality. SUMMARY

[0005] Embodiments of the present application provide a motion capture method, device, equipment and storage medium, which can make the target skeleton data better represent the actual motion form of the motion capture object and improve the quality of the target skeleton data. The technical solution is as follows:

[0006] On the one hand, the present application provides a motion capture method, which comprises:

[0007] determining first skeleton data of a motion capture object in the motion capture scene based on first optical data collected by the optical motion capture camera, the first optical data being obtained by collecting optical capture points set on the motion capture object;

[0008] determining second skeleton data based on image data collected by the video analysis camera, the image data containing the motion capture object in the motion capture scene, and the second skeleton data being based on the same coordinate space as the first skeleton data;

[0009] optimizing the first skeleton data based on the second skeleton data to obtain target skeleton data.

[0010] On the other hand, the present application provides a motion capture device, which comprises:

[0011] a first data determination module configured to determine first skeleton data of a motion capture object in the motion capture scene based on first optical data collected by the optical motion capture camera, the first optical data being obtained by collecting optical capture points set on the motion capture object;

[0012] a second data determination module configured to determine second skeleton data based on image data collected by the video analysis camera, the image data containing the motion capture object in the motion capture scene, the second skeleton data being based on the same coordinate space as the first skeleton data;

[0013] a data optimization module configured to optimize the first skeleton data based on the second skeleton data to obtain target skeleton data.

[0014] In another aspect, an embodiment of the present application provides a computer device, which comprises a processor and a memory, the memory storing at least one instruction, the at least one instruction being loaded and executed by the processor to implement the motion capture method according to the above aspect.

[0015] In another aspect, an embodiment of the present application provides a computer readable storage medium, which stores at least one instruction, the at least one instruction being loaded and executed by a processor to implement the motion capture method according to the above aspect.

[0016] In another aspect, an embodiment of the present application provides a computer program product, which comprises computer instructions stored in a computer readable storage medium; a processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to perform the motion capture method according to the above aspect.

[0017] In the embodiment of the present application, the computer device determines the first skeleton data of the motion capture object based on the first optical data collected by the optical motion capture camera, and determines the second skeleton data of the motion capture object based on the image data collected by the video analysis camera, wherein the second skeleton data is based on the same coordinate space as the first skeleton data, so that the computer device optimizes the first skeleton data based on the second skeleton data to obtain the target skeleton data. By using the scheme provided in the embodiment of the present application, the first skeleton data representing the accurate position of the skeleton node can be optimized based on the second skeleton data representing the better rotation state of the skeleton, so that the target skeleton data can better represent the real action form of the motion capture object, and the quality of the target skeleton data is improved. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0019] Figure 1 A schematic diagram showing an implementation environment provided by one example embodiment of the present application is shown;

[0020] Figure 2 A schematic diagram showing a video analysis camera provided by one example embodiment of the present application is shown;

[0021] Figure 3 A flowchart showing a motion capture method provided by one example embodiment of the present application is shown;

[0022] Figure 4 A schematic diagram showing generating target skeleton data provided by one example embodiment of the present application is shown;

[0023] Figure 5 A flowchart showing a motion capture method provided by another example embodiment of the present application is shown;

[0024] Figure 6 A schematic diagram showing establishing a space coordinate system provided by one example embodiment of the present application is shown;

[0025] Figure 7 A schematic diagram showing determining second skeleton data provided by one example embodiment of the present application is shown;

[0026] Figure 8 A schematic diagram showing generating target skeleton data provided by one example embodiment of the present application is shown;

[0027] Figure 9 A flowchart showing a motion capture method provided by one example embodiment of the present application is shown;

[0028] Figure 10 A structural schematic diagram of a motion capture system provided by one example embodiment of the present application is shown;

[0029] Figure 11 A flowchart showing a motion capture method provided by another example embodiment of the present application is shown;

[0030] Figure 12 A structural block diagram of a motion capture device provided by one example embodiment of the present application is shown;

[0031] Figure 13 A structural schematic diagram of a computer device provided by one example embodiment of the present application is shown. DETAILED DESCRIPTION

[0032] For the purpose of making the purpose, technical scheme and advantages of the present application more clear, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0033] Reference should be made to Figure 1Fig. 1 shows a schematic diagram of an implementation environment according to an example embodiment of the present application. The implementation environment includes a computer device 101, optical motion capture cameras 102, and video analysis cameras 103. The computer device 101 and the optical motion capture cameras 102, and the computer device 101 and the video analysis cameras 103 can communicate with each other via a communication network. The communication network can be a wired network or a wireless network, and can be at least one of a local area network, a metropolitan area network, and a wide area network.

[0034] The computer device 101 is an electronic device installed with an application having a motion capture data processing function. The motion capture data processing function can be a function of a native application or a function of a third-party application. The electronic device can be a smartphone, a tablet computer, a personal computer, a wearable device, a vehicle terminal, or the like. Figure 1 The computer device 101 is taken as a personal computer in the example embodiment, but the present application is not limited thereto.

[0035] In one possible implementation, the computer device 101, the optical motion capture cameras 102, and the video analysis cameras 103 form a motion capture system. The computer device 101 is connected to the optical motion capture cameras 102 and the video analysis cameras 103 in a motion capture scene. The computer device 101 can send a time synchronization signal to the optical motion capture cameras 102 and the video analysis cameras 103. Each optical motion capture camera 102 is placed at a fixed position in the motion capture scene. Each video analysis camera 103 can move along with a motion capture object 104 in the motion capture scene. In order to determine the spatial position of each video analysis camera 103 in the motion capture scene, at least three optical capture points that are not collinear are arranged on each video analysis camera 103. The optical motion capture cameras 102 collect optical data of the video analysis cameras 103, so that the computer device 101 determines the spatial position of each video analysis camera 103 in the motion capture scene according to the optical data.

[0036] In one possible implementation, in order to determine the six degrees of freedom information of the video analysis camera in the motion capture scene, i.e., displacement and rotation information, at least three optical capture points are arranged on the video analysis camera, and the at least three optical capture points are arranged according to the principle that three points are not collinear.

[0037] As shown in Fig. 2, according to the left and right views of the video analysis camera 201, it can be seen that the video analysis camera 201 is provided with five optical capture points 202, and the optical capture points satisfy the principle that three points are not collinear. Figure 2

[0038] ​In a possible implementation, after the motion capture object 104 provided with multiple optical capture points enters the motion capture scene, the computer device 101 respectively collects first optical data of the motion capture object 104 by the optical motion capture camera 102 and collects image data of the motion capture object 104 by the video analysis camera 103, so that after the computer device 101 receives the first optical data sent by each optical motion capture camera 102 and the image data sent by each video analysis camera 103, the computer device 101 respectively determines first skeleton data based on the first optical data and determines second skeleton data based on the image data, so that the computer device 101 optimizes the first skeleton data based on the second skeleton data to obtain target skeleton data.

[0039] Please refer to Figure 3 which shows a flowchart of a motion capture method provided by an example embodiment of the present application. The embodiment takes the method for Figure 1 The computer device 101 shown in the figure is taken as an example for illustration, and the method includes the following steps.

[0040] In step 301, first skeleton data of a motion capture object in a motion capture scene is determined based on first optical data collected by an optical motion capture camera, and the first optical data is obtained by collecting optical capture points provided on the motion capture object.

[0041] In a possible implementation, the optical motion capture camera is an optical infrared camera, and the optical motion capture camera can collect optical data by emitting infrared light. Optionally, the optical motion capture camera emits infrared light, and the infrared light can be reflected to the optical motion capture camera again after colliding with optical capture points with a reflecting effect, so that the optical motion capture camera collects the reflected infrared light to obtain optical data.

[0042] In a possible implementation, in order to avoid the infrared light reflected by the optical capture points from being disturbed by the mirror reflection of the ground or the wall, the motion capture scene can select a site with light shielding effect.

[0043] In a possible implementation, since the optical motion capture camera collects optical data of the motion capture object by collecting infrared light reflected by the optical capture points, multiple optical capture points can be provided on each part of the motion capture object to collect optical data of each part of the motion capture object, so that the optical motion capture camera collects infrared light reflected by each optical capture point to generate first optical data.

[0044] In a possible implementation, in the case that multiple optical capture points are set and the motion capture object keeps moving, the optical motion capture camera may fail to collect enough infrared light reflected by the optical capture points due to the motion blocking of the motion capture object or the angle of the optical motion capture camera, and thus the optical data may be incomplete. Therefore, in order to collect as much infrared light reflected by the optical capture points as possible, multiple optical motion capture cameras can be set in the motion capture scene to collect optical data of the optical capture points set on the motion capture object from different angles.

[0045] In a possible implementation, the computer device receives the first optical data of the motion capture object collected by each optical motion capture camera, and determines the first skeleton data of the motion capture object according to the spatial positions of the optical motion capture cameras and the corresponding first optical data. Since the first optical data is determined based on each optical capture point set on the motion capture object, the first skeleton data determined based on the first optical data can more accurately correspond to the skeleton node positions of each part of the motion capture object.

[0046] As shown in FIG. 4, Figure 4 the computer device determines the first skeleton data 402 of the motion capture object 401 in the motion capture scene based on the first optical data collected by the optical motion capture camera.

[0047] In step 302, the second skeleton data is determined based on the image data collected by the video analysis camera. The image data contains the motion capture object in the motion capture scene, and the second skeleton data is based on the same coordinate space as the first skeleton data.

[0048] In a possible implementation, in addition to the optical motion capture camera, a video analysis camera can also be set in the motion capture scene. The video analysis camera can capture the motion change process of the motion capture object in a period of time, so as to better determine the skeleton rotation information of the motion capture object and the like.

[0049] In a possible implementation, the video analysis camera is an imaging camera with a signal synchronization function and can access an external synchronization signal. According to the straight-line propagation property of light and the refraction and reflection law of light, the video analysis camera can collect the light reflected by the motion capture object to obtain the image data of the motion capture object in the motion capture scene. Further, the video analysis camera transmits the collected image data to the computer device, so that the computer device determines the second skeleton data according to the motion form of the motion capture object in the image.

[0050] Optionally, the video analysis camera can also have a night vision function to meet the motion capture scene without light.

[0051] In a possible implementation, since the optical motion capture camera and the video analysis camera capture the same motion capture object in different manners in the motion capture scene, that is, the first skeleton data and the second skeleton data both correspond to the motion state of the motion capture object at the same time in the same space, in order to facilitate comparison of the first skeleton data and the second skeleton data, the computer device determines the first skeleton data and the second skeleton data based on the same coordinate space.

[0052] As shown in FIG. 4, the computer device determines the second skeleton data 403 corresponding to the motion capture object 401 based on the image data captured by the video analysis camera. Figure 4 As shown in FIG. 4, the computer device determines the second skeleton data 403 corresponding to the motion capture object 401 based on the image data captured by the video analysis camera.

[0053] In step 303, the first skeleton data is optimized based on the second skeleton data, to obtain target skeleton data.

[0054] In a possible implementation, since the first skeleton data is determined based on the first optical data, and the first optical data is obtained by capturing infrared light reflected by optical capture points arranged on each part of the motion capture object, and the second skeleton data is determined based on the image data captured by the video analysis camera, compared with the second skeleton data, the positions of the skeleton nodes in the first skeleton data are more consistent with the positions of the skeleton nodes on the actual motion capture object; compared with the first skeleton data, the skeleton rotation states of the skeletons in the second skeleton data are more consistent with the skeleton rotation states of the skeletons on the actual motion capture object.

[0055] In a possible implementation, in order to compensate for the skeleton rotation states represented by the first skeleton data, the computer device optimizes the first skeleton data based on the second skeleton data, to obtain target skeleton data, so that the target skeleton data can represent more accurate skeleton node positions and more actual skeleton rotation states.

[0056] In a possible implementation, the computer device can directly replace the first skeleton data with the second skeleton data corresponding to the frame according to the data quality of the first skeleton data in each frame, or can partially modify and improve the first skeleton data based on the second skeleton data corresponding to the frame, and the embodiments of the present application do not limit this.

[0057] As shown in FIG. 4, the computer device determines the second skeleton data 403 corresponding to the motion capture object 401 based on the image data captured by the video analysis camera. Figure 4 As shown in FIG. 4, the computer device determines the second skeleton data 403 corresponding to the motion capture object 401 based on the image data captured by the video analysis camera.

[0058] In summary, in the embodiment of the present application, the computer device determines the first skeleton data of the motion capture object based on the first optical data collected by the optical motion capture camera, and determines the second skeleton data of the motion capture object based on the image data collected by the video analysis camera, wherein the second skeleton data and the first skeleton data are based on the same coordinate space, so that the computer device optimizes the first skeleton data based on the second skeleton data to obtain the target skeleton data. By using the scheme provided in the embodiment of the present application, the first skeleton data representing the accurate skeleton node position can be optimized by the second skeleton data representing the better skeleton rotation state, so that the target skeleton data can better represent the real action form of the motion capture object, and the quality of the target skeleton data is improved.

[0059] For example, in the case that the motion capture object performs a large-area occlusion action, or leaves the shooting range of the optical motion capture camera, or enters a closed space, since the optical motion capture camera is fixed in position, it is impossible to collect optical data of all or most of the optical capture points on the motion capture object, or in the case that the number of optical motion capture cameras is small, it is also impossible to collect optical data of all or most of the optical capture points on the motion capture object, thereby causing the problem of missing first skeleton data. The video analysis camera can move along with the motion capture object, that is, it can collect image data of the motion capture object, so that relatively complete and stable second skeleton data can be determined, and then the first skeleton data can be optimized by the second skeleton data, so that relatively complete target skeleton data can be obtained, and the data quality of the target skeleton data is ensured.

[0060] In a possible implementation, since the first skeleton data is determined based on the optical motion capture camera, and the second skeleton data is determined based on the video analysis camera, in the case of optimizing the first skeleton data based on the second skeleton data, in order to improve the accuracy of the skeleton data optimization, the computer device needs to ensure that the second skeleton data corresponds to the first skeleton data of the same time frame.

[0061] Please refer to Figure 5 which shows a flowchart of the motion capture method provided in an example embodiment of the present application. In this embodiment, the method is used for Figure 1 The computer device 101 shown in the figure is taken as an example for description, and the method includes the following steps.

[0062] Step 501, determining the spatial coordinates of the optical motion capture camera.

[0063] In a possible implementation, in order to more accurately capture the motion of the motion capture object, the computer device can establish a spatial coordinate system based on the motion capture scene. Optionally, the computer device obtains the site information of the motion capture scene, and determines the coordinate origin and unit length of the spatial coordinate system based on the site information.

[0064] Optionally, in order to ensure that multiple motion capture of the same motion capture scene can be completed based on the same space coordinates, the computer device can select the coordinate origin of the space coordinates at the center position of the motion capture scene or other position with obvious mark, so as to avoid multiple establishment of the space coordinate system for the same motion capture scene.

[0065] As shown in FIG. 6, the computer device establishes a space coordinate system 601 based on the motion capture scene, and determines the space coordinates 602 of each optical motion capture camera. Figure 6 As shown in FIG. 6, the computer device establishes a space coordinate system 601 based on the motion capture scene, and determines the space coordinates 602 of each optical motion capture camera.

[0066] In a possible implementation, after the space coordinate system is established based on the motion capture scene, in order to enable the first skeleton data to more accurately represent the action form of the motion capture object in the motion capture scene, the computer device can first determine the spatial positions of each optical motion capture camera in the motion capture scene, i.e., determine the space coordinates of each optical motion capture camera in the space coordinate system.

[0067] In step 502, the video analysis camera is externally parameter calibrated based on the second optical data collected by the optical motion capture camera and the space coordinates, to obtain the externally parameter calibration data corresponding to the video analysis camera. The second optical data is obtained by collecting the optical capture points arranged on the video analysis camera.

[0068] Further, when the image data of the motion capture object is collected by the multiple video analysis cameras to determine the second skeleton data, in order to accurately merge and de-duplicate the image data corresponding to each video analysis camera, the computer device needs to first determine the spatial positions of each video analysis camera in the motion capture scene, i.e., externally parameter calibrate each video analysis camera to determine the space coordinates of each video analysis camera in the space coordinate system.

[0069] In a possible implementation, at least one optical capture point is arranged on the video analysis camera, so that the computer device collects the second optical data of each video analysis camera in the motion capture scene by the optical motion capture camera, and externally parameter calibrates the video analysis camera based on the second optical data collected by the optical motion capture camera and the space coordinates, to obtain the externally parameter calibration data corresponding to the video analysis camera. Optionally, the externally parameter calibration data includes the spatial position, rotation direction, etc. of the video analysis camera in the space coordinate system.

[0070] In a possible implementation, the computer device determines the rotation displacement data of the video analysis camera in the motion capture scene based on the second optical data collected by the optical motion capture camera. Further, since the rotation displacement data represents the position change of the video analysis camera in the world coordinate system, and the second skeleton data determined by the video analysis camera is based on the space coordinate system, the computer device needs to perform coordinate system transformation on the rotation displacement data of the video analysis camera based on the space coordinate, so as to obtain the extrinsic calibration data corresponding to the video analysis camera.

[0071] In a possible implementation, in order to determine the six-degree-of-freedom information of the video analysis camera in the motion capture scene, that is, the displacement and rotation information, at least three optical capture points can be arranged on the video analysis camera, and the at least three optical capture points are arranged according to the principle that three points are not collinear. Therefore, the computer device collects the second optical data of the at least three optical capture points based on the optical motion capture camera, and performs extrinsic calibration on the video analysis camera based on the second optical data and the space coordinates of the optical motion capture cameras.

[0072] In a possible implementation, since a plurality of optical capture points are arranged on a video analysis camera, and the number of video analysis cameras is large and the size of the space coordinate is limited, in order to simplify the space coordinate system, the computer device can process the optical data corresponding to the plurality of optical capture points on the same video analysis camera, and create rigid body data corresponding to each optical capture point according to the space coordinates of the plurality of optical capture points in the space coordinate system, so as to determine the space coordinates of the video analysis camera in the space coordinate system based on each rigid body data. Optionally, the space coordinates of the video analysis camera in the space coordinate system can correspond to the lens imaging position of the video analysis camera in the actual motion capture scene.

[0073] In a possible implementation, the optical capture point can be an infrared reflective ball which can reflect infrared light and adjust its light frequency. In order to collect image data of the motion capture object from different directions or different angles, a plurality of video analysis cameras can be arranged, and each video analysis camera is extrinsically calibrated by arranging optical capture points. Therefore, in order to distinguish the video analysis cameras in the motion capture scene, different installation positions of the optical capture points can be arranged on different video analysis cameras, so that the computer device distinguishes the video analysis cameras based on the installation positions of the at least three optical capture points on each video analysis camera. Different light frequencies of the optical capture points can also be arranged on different video analysis cameras, so that the computer device distinguishes the video analysis cameras based on the light frequencies of the optical capture points on each video analysis camera.

[0074] In a possible implementation, according to the ideal pinhole model, the camera is too slow to expose due to the few light rays passing through the pinhole, and in actual camera use, in order to improve the speed of image generation, a lens is used, which introduces distortion, so in order to improve the accuracy of the image data collected by the video analysis camera, the internal parameters of the video analysis camera also need to be calibrated, and the main two distortions affecting the internal parameters of the video analysis camera are radial distortion and tangential distortion, wherein the main reason for radial distortion is that the light deflection is larger away from the center of the camera lens, and the main reason for tangential distortion is that the camera lens is not completely parallel to the image plane.

[0075] In an illustrative example, the internal parameter matrix of the video analysis camera can be expressed as:

[0076]

[0077] wherein f is the focal length of the video analysis camera, dx, dy is the pixel size, u0, v0 is the image center, and is the normalized focal length on the x-axis and y-axis, respectively.

[0078] Radial distortion can be corrected by the Taylor series expansion as follows.

[0079] x corrected =x(1+k1r 2 +k2r 4 +k3r 6 )

[0080] y corrected =y(1+k1r 2 +k2r 4 +k3r 6 )

[0081] Tangential distortion can be corrected by the following formula.

[0082] x corrected =x+[2p1xy+p2(r 2 +2x 2 )]

[0083] y corrected =y+[p1(r 2 +2y 2 )+2p2xy]

[0084] Thus, the distortion parameters D(k1, k2, k3, p1, p2) of the video analysis camera are obtained. Wherein (x, y) is the original position of the distorted point on the imager, r is the distance of the point from the center of the imager, (x corrected , y corrected ) is the new position after correction.

[0085] In step 503, the computer device determines first skeleton data of the motion capture object in the motion capture scene based on first optical data collected by the optical motion capture cameras, the first optical data being collected by capturing optical capture points arranged on the motion capture object.

[0086] In a possible implementation, the computer device collects optical data of optical capture points arranged on the motion capture object in the motion capture scene by using multiple optical motion capture cameras, and determines first skeleton data of the motion capture object in the motion capture scene based on first optical data collected by each optical motion capture camera and spatial coordinates corresponding to each optical motion capture camera.

[0087] In a possible implementation, data transmission between the computer device and each optical motion capture camera can be performed through a POE switch, which is not limited in the embodiments of the present application.

[0088] In a possible implementation, after determining the first skeleton data based on the first optical data, in order to distinguish each skeleton node in the first skeleton data, the computer device can mark each skeleton node according to real limb skeleton information of the motion capture object.

[0089] In step 504, the computer device determines second skeleton data based on image data collected by the video analysis cameras and extrinsic calibration data.

[0090] In a possible implementation, in order to make the second skeleton data more accurately represent the real action form of the motion capture object in the motion capture scene, the computer device collects image data of the motion capture object from different angles by using multiple video analysis cameras, and then determines the second skeleton data based on extrinsic calibration data corresponding to each video analysis camera, i.e., spatial coordinates of each video analysis camera and image data collected by each video analysis camera, so as to convert two-dimensional image data collected by the video analysis cameras into three-dimensional second skeleton data in spatial coordinates.

[0091] In a possible implementation, each video analysis camera can transmit image data to the computer device through a data acquisition card, which is not limited in the embodiments of the present application.

[0092] As shown in FIG. 7, Figure 7 the computer device analyzes image data 701 collected by the video analysis cameras and generates two-dimensional skeleton data 702, and then determines second skeleton data 703 according to extrinsic calibration data of each video analysis camera and the corresponding two-dimensional skeleton data 702.

[0093] In step 505, the computer device sends a time synchronization signal to the video analysis cameras and the optical motion capture cameras.

[0094] In a possible implementation, before the first skeleton data is optimized by the second skeleton data, since the motion capture object is in a state of action at the moment, that is, each frame of skeleton data corresponding to the motion capture object is different, and in the case that the motion capture object is in the action state, the computer device respectively collects the first optical data of the motion capture object by the optical motion capture camera and collects the image data of the motion capture object by the video analysis camera, so as to be able to optimize each frame of first skeleton data based on the second skeleton data, the computer device can send a time synchronization signal to the video analysis camera and the optical motion capture camera, so that each video analysis camera and each optical motion capture camera can synchronously collect data through the time synchronization signal.

[0095] In a possible implementation, each video analysis camera is provided with a signal access port, and each optical motion capture camera constitutes an integral optical motion capture system, the optical motion capture system performs unified signal setting on each optical motion capture camera according to the received signal instruction, and the computer device respectively sends a time synchronization signal to the optical motion capture system and each video analysis camera, so that each optical motion capture camera and the video analysis camera synchronously collect data of the motion capture object.

[0096] In a possible implementation, the time synchronization signal can be a time code generated based on a Sync synchronization technology, and the time synchronization signal can be sent in a wireless Bluetooth manner or in a wired data manner, which is not limited in the embodiment of the application.

[0097] Step 506, synchronizing the second skeleton data corresponding to the video analysis camera and the first skeleton data corresponding to the optical motion capture camera based on the time synchronization signal.

[0098] In a possible implementation, after the time synchronization signal is sent to each video analysis camera and each optical motion capture camera, the computer device receives the image data collected by each video analysis camera and the first optical data collected by each optical motion capture camera, and synchronizes the second skeleton data determined based on the image data and the first skeleton data determined based on the first optical data according to the time synchronization signal, so that each frame of first skeleton data and second skeleton data is synchronized in time.

[0099] Step 507, aligning the second skeleton data based on the first skeleton data.

[0100] In a possible implementation, considering that the positions of the skeleton nodes in the first skeleton data are more consistent with the positions of the skeleton nodes on the actual motion capture object, the computer device aligns the second skeleton data based on the first skeleton data.

[0101] In one possible implementation, since the number of bone nodes in the bone data is large, in order to improve the efficiency of bone alignment, the computer device can perform alignment processing on the second bone data by aligning key bones.

[0102] In one possible implementation, the computer device first determines the coordinates of the first bone node corresponding to the first key bone from the first skeletal data. Optionally, the key bone can be an inverse kinematic (ik) position on the motion capture object, including the center of gravity position, left and right hand positions, and left and right foot positions. Further, the computer device determines the coordinates of the second bone node of the second key bone corresponding to the first key bone from the second skeletal data. Based on the first and second bone node coordinates, the computer device aligns the second skeletal data so that the second bone node coordinates in the aligned second skeletal data correspond to the first bone node coordinates in the first skeletal data.

[0103] Indicative, such as Figure 8 As shown, the motion capture object 801 corresponds to first bone data 802 and second bone data 803. The computer device determines the head key bone 804, left hand key bone 805, right hand key bone 806, right foot key bone 807 and left foot key bone 808 based on the first bone data 802, and determines the coordinates of the first bone node based on each first key bone, as well as the coordinates of the second bone node corresponding to the second key bone in the second bone data 803, so that the computer device performs alignment processing on the second bone data 803.

[0104] Step 508: Based on the aligned second bone data, optimize the first bone data to obtain the target bone data.

[0105] In one possible implementation, since the first skeleton data is determined by acquiring optical data from optical capture points on the motion capture object using various optical motion capture cameras, and the positions of each optical motion capture camera are fixed during the motion capture object's movement, it may not be possible to acquire every optical capture point on the motion capture object. This may result in incomplete or erroneous first skeleton data. Therefore, after aligning the second skeleton data according to the skeleton node coordinates, the computer device can optimize the first skeleton data based on the second skeleton data to obtain the target skeleton data.

[0106] In one possible implementation, if there are a large number of missing or incorrect first bone data corresponding to a certain image frame, the computer device can directly replace the first bone data with the second bone data in the corresponding image frame.

[0107] In a possible implementation, in a case where there is partial sub-skeleton data error or loss in the first skeleton data corresponding to a certain image frame, the computer device can correct the first skeleton data by corresponding sub-skeleton data in the second skeleton data in the corresponding image frame.

[0108] In a possible implementation, the computer device performs data loss determination on the first skeleton data, and determines a to-be-optimized sub-skeleton in the first skeleton data according to a data loss condition of the first skeleton data, and extracts second sub-skeleton data corresponding to the to-be-optimized sub-skeleton from the aligned second skeleton data, so that the computer device replaces first sub-skeleton data corresponding to the to-be-optimized sub-skeleton with the second sub-skeleton data, to obtain optimized first skeleton data, i.e., target skeleton data.

[0109] In a possible implementation, the computer device determines a data loss condition of first sub-skeleton data corresponding to each sub-skeleton in the first skeleton data, and in a case where there is data loss in the first sub-skeleton data, the computer device determines a sub-skeleton corresponding to the first sub-skeleton data as a to-be-optimized sub-skeleton, so as to replace the first sub-skeleton data with second sub-skeleton data corresponding to the to-be-optimized sub-skeleton in the second skeleton data.

[0110] In a possible implementation, the computer device determines a to-be-optimized sub-skeleton in the first skeleton data according to a skeletal rotation degree of freedom represented by the first skeleton data, wherein the skeletal rotation degree of freedom refers to six degrees of freedom of a motion capture object in a spatial coordinate, i.e., movement degrees of freedom along x, y, and z coordinate axes and rotation degrees of freedom around the three coordinate axes, and then the computer device extracts second sub-skeleton data corresponding to the to-be-optimized sub-skeleton from the second skeleton data, and replaces first sub-skeleton data corresponding to the to-be-optimized sub-skeleton with the second sub-skeleton data, to obtain optimized first skeleton data, i.e., target skeleton data.

[0111] In a possible implementation, the computer device determines a skeletal rotation degree of freedom represented by first sub-skeleton data corresponding to each sub-skeleton in the first skeleton data, and in a case where the skeletal rotation degree of freedom is greater than a rotation degree of freedom threshold, for example, a degree of bending of an arm joint obviously exceeds a normal range, the computer device determines a sub-skeleton corresponding to the first sub-skeleton data as a to-be-optimized sub-skeleton, so as to replace the first sub-skeleton data with second sub-skeleton data corresponding to the to-be-optimized sub-skeleton in the second skeleton data.

[0112] In a possible implementation, since the number of optical capture points that can be arranged on the motion capture object is limited, even if the optical motion capture camera can collect optical data of all optical capture points, the motion state of the motion capture object represented by the first skeleton data can not be the same as the actual motion state of the motion capture object in the motion capture scene, for example, for the motion of each finger of the hand of the motion capture object, since the number of optical capture points that can be arranged on the finger part is small, the optical motion capture camera is difficult to collect the optical data corresponding to the finger part in the case that the fingers of the motion capture object are constantly moving, so that the first skeleton data lacks the skeleton data corresponding to the finger part. When the video analysis camera collects image data of the motion capture object, the video analysis camera can capture and record the finger part of the motion capture object in real time, so the second skeleton data determined according to the image data can represent the motion state of the finger part of the motion capture object, and thus the computer device can supplement the first skeleton data with the second skeleton data.

[0113] In a possible implementation, after aligning the second skeleton data and correcting the first skeleton data by using the second skeleton data to obtain target skeleton data, the computer device compares the first skeleton data with the second skeleton data, and extracts third sub-skeleton data from the aligned second skeleton data, wherein the first skeleton data does not include a sub-skeleton corresponding to the third sub-skeleton data, and then the computer device directly supplements the target skeleton data based on the third sub-skeleton data, so that the supplemented target skeleton data can more completely represent the motion state of the motion capture object.

[0114] In the above embodiments, the video analysis camera is arranged in the motion capture scene to assist the optical motion capture camera in completing the motion capture of the motion capture object, wherein the optical motion capture camera is used to perform external parameter calibration on each video analysis camera in the motion capture scene, and the second skeleton data is generated according to the external parameter calibration data and the image data collected by each video analysis camera, thereby improving the data accuracy of the second skeleton data. In addition, the time synchronization signal is sent to the optical motion capture camera and the video analysis camera, and the first skeleton data and the second skeleton data of each image frame are time synchronized, thereby improving the effectiveness and accuracy of aligning the second skeleton data based on the first skeleton data.

[0115] Moreover, in the case that the number of optical motion capture cameras is small, or the motion capture object has a large area of occlusion action, or the motion capture object enters a closed space, the first skeleton data may have data missing or the rotation degree of freedom of the skeleton represented by the first skeleton data may be incorrect. For example, at the moment when the motion capture object enters the closed space, the optical motion capture cameras cannot collect optical data of the optical capture points arranged on the legs of the motion capture object, so that the leg skeleton data in the first skeleton data is missing. Or in the case that the arms of the motion capture object are partially occluded, the optical motion capture cameras cannot collect complete optical data of each optical capture point on the arms of the motion capture object, which may cause the rotation degree of freedom of the arm part of the skeleton represented by the skeleton data in the first skeleton data to be too large, which does not conform to the fact. Therefore, the computer device judges the data missing of the first skeleton data and the rotation degree of freedom of the skeleton represented by the first sub-skeleton data, and optimizes the first skeleton data by using the second skeleton data, thereby improving the efficiency of the skeleton data optimization.

[0116] In addition, considering that the joints of the fingers of the motion capture object are thin, even if multiple optical capture points are arranged, the optical capture points cannot collect clear optical data due to the fact that the finger part action is not obvious or the optical motion capture camera has low pixels. In combination with the video analysis camera, the video analysis camera can collect video image data of the finger part action, so that after the first skeleton data is optimized to obtain the target skeleton data, the computer device can directly supplement the target skeleton data with the sub-skeleton data corresponding to the hand skeleton in the second skeleton data, thereby improving the data quality of the target skeleton data.

[0117] Please refer to Figure 9 which shows a flowchart of the motion capture method provided by an example embodiment of the present application.

[0118] In step 901, a space coordinate system is established based on the motion capture scene, and the space coordinates of the optical motion capture cameras are determined.

[0119] The computer device determines the coordinate origin and the unit length based on the motion capture scene, thereby establishing the space coordinate system and determining the space coordinates of each optical motion capture camera in the motion capture scene.

[0120] In step 902, the intrinsic parameters of the video analysis camera are calibrated.

[0121] In order to improve the accuracy of the image data collected by the video analysis camera, the computer device calibrates the intrinsic parameters of the video analysis camera to avoid tangential distortion and radial distortion.

[0122] In step 903, the extrinsic parameters of the video analysis camera are calibrated to determine the extrinsic parameter calibration data.

[0123] The computer device collects second optical data of each video analysis camera in the motion capture scene through the optical motion capture camera, and calibrates the extrinsic parameters of each video analysis camera based on the second optical data to determine extrinsic calibration data, so as to obtain the spatial coordinates of each video analysis camera.

[0124] In step 904, the time synchronization signal is sent to the video analysis camera and the optical motion capture camera.

[0125] The computer device sends the time synchronization signal to each video analysis camera and the optical motion capture camera, so that the first skeleton data and the second skeleton data in each image frame can be kept synchronized in time.

[0126] In step 905, the second skeleton data is determined based on the extrinsic calibration data of the video analysis camera and the image data.

[0127] The computer device determines the second skeleton data with three-dimensional characteristics based on the extrinsic calibration data of each video analysis camera and the image data.

[0128] In step 906, the first skeleton data is determined based on the first optical data collected by the optical motion capture camera.

[0129] The computer device collects optical data of each optical capture point arranged on the motion capture object through the optical motion capture camera to obtain the first optical data, and determines the first skeleton data based on the first optical data.

[0130] In step 907, the second skeleton data is aligned based on the first skeleton data.

[0131] According to the more accurate characteristics of the position of the bone node represented by the first skeleton data, the computer device aligns the second skeleton data based on the first skeleton data.

[0132] In step 908, the first skeleton data is optimized based on the aligned second skeleton data to obtain target skeleton data.

[0133] According to the more accurate characteristics of the bone rotation information represented by the second skeleton data, the computer device optimizes the first skeleton data based on the aligned second skeleton data, so as to obtain the target skeleton data.

[0134] In step 909, the target skeleton data is saved.

[0135] Please refer to Figure 10 which shows a structural schematic diagram of the motion capture system provided by an example embodiment of the present application.

[0136] As shown in the figure, the motion capture system can include a computer device 1001, an optical motion capture module 1002, and a video analysis module 1003, wherein the optical motion capture module 1002 includes an optical motion capture camera 1 and an optical motion capture camera 2, both of which transmit first optical data 1004 to the computer device 1001 through a POE switch; the video analysis module 1003 includes a video analysis camera 1 and a video analysis camera 2, both of which transmit image data 1005 to the computer device 1001 through a capture card, and in addition, each optical motion capture camera and video analysis camera realizes time synchronization through time code according to Sync technology.

[0137] After receiving the first optical data 1004 sent by each optical motion capture camera and the image data 1005 sent by each video analysis camera, the computer device 1001 determines second skeleton data 1007 based on the image data 1005 and the extrinsic calibration data of each video analysis camera, and at the same time, determines first skeleton data 1008 based on the first optical data 1004 and the spatial coordinates of each optical motion capture camera, further, based on the skeleton node coordinates 1006 corresponding to the first key skeleton in the first skeleton data 1008, the computer device performs alignment processing on the second skeleton data 1007, and according to the data quality of the first skeleton data 1008, optimizes the first skeleton data 1008 with the second skeleton data 1007, so as to obtain target skeleton data.

[0138] In summary of the above embodiments, in the virtual studio scenario, the process of motion capture of the computer device on the motion capture actor can include the following steps.

[0139] Step 1101, based on the studio scenario, establishing a spatial coordinate system and determining the spatial coordinates of the optical motion capture camera.

[0140] In a possible implementation, the computer device obtains scene information of the studio scenario, which can include grid marks, preset key points, etc. in the studio scenario, so that the computer device can select the center point of the ground in the studio scenario as the coordinate origin of the spatial coordinate system, and determine the unit length of the spatial coordinate system based on the grid marks in the studio scenario, and then the computer device establishes the spatial coordinate system based on the coordinate origin and the unit length.

[0141] In a possible implementation, the computer device determines the spatial coordinates of each optical motion capture camera in the spatial coordinate system according to the spatial position of the optical motion capture camera in the studio scenario.

[0142] Step 1102, calibrating the internal parameters of the video analysis camera and determining the extrinsic calibration data.

[0143] In a possible implementation, the computer device calibrates the internal parameters of the video analysis camera with radial distortion and tangential distortion based on the principle of the ideal pinhole model, and improves the imaging accuracy of the video analysis camera.

[0144] In a possible implementation, at least three optical capture points, i.e., infrared light-emitting balls, are arranged on each video analysis camera in the shooting scene, and to obtain the rotation and displacement information of each video analysis camera, the at least three infrared light-emitting balls are arranged in each video analysis camera based on the principle that three points are not collinear, and then the computer device performs second optical data acquisition on the video analysis camera by each optical motion capture camera in the shooting scene, creates a rigid body corresponding to each video analysis camera in a spatial coordinate system based on the second optical data, and performs external parameter calibration on each video analysis camera, so as to obtain external parameter calibration data of each video analysis camera, which can include spatial coordinates, displacement information, and rotation information of the video analysis camera, and the like.

[0145] Step 1103: sending a time synchronization signal to the optical motion capture camera and the video analysis camera.

[0146] In a possible implementation, to ensure that the first skeleton data corresponding to the optical motion capture camera and the second skeleton data corresponding to the video analysis camera in each image frame can correspond in time, the computer device sends a time synchronization signal to the optical motion capture camera and the video analysis camera, and controls the optical motion capture camera and the video analysis camera to simultaneously perform data acquisition on the motion capture actor by the time synchronization signal.

[0147] Step 1104: determining the second skeleton data based on the image data collected by the video analysis camera.

[0148] In a possible implementation, during the motion of the motion capture actor, the computer device receives the two-dimensional image data sent by each video analysis camera, and determines the two-dimensional skeleton data of the motion capture actor in the image based on the two-dimensional image data, so as to determine the three-dimensional second skeleton data based on the spatial coordinates of each video analysis camera in the shooting scene and the corresponding two-dimensional skeleton data.

[0149] Step 1105: determining the first skeleton data based on the first optical data collected by the optical motion capture camera.

[0150] In a possible implementation, the optical capture points are arranged on each part of the motion capture actor, including the head, limbs, chest, and back. Before the motion capture actor performs the action, the computer device can perform a first optical data collection on the motion capture actor in a static state through the optical motion capture cameras in the shooting scene. In order to ensure that all the optical capture points on the motion capture actor can be collected, the motion capture actor can keep a T-shaped posture, so that the computer device determines the first skeleton data based on the first optical data, and performs node name labeling on each skeleton node in the first skeleton data.

[0151] In a possible implementation, during the action of the motion capture actor, the computer device collects the first optical data of the optical capture points arranged on the motion capture actor through the optical motion capture cameras, and determines the first skeleton data according to the first optical data and the spatial coordinates of the optical motion capture cameras.

[0152] In step 1106, the second skeleton data is aligned based on the first skeleton data.

[0153] In a possible implementation, during the action of the motion capture actor, the optical motion capture cameras cannot collect the optical data of the optical capture points arranged on the left arm and the left leg of the motion capture actor, because the left arm and the left leg of the motion capture actor are blocked by the setting in the shooting scene, so that the sub-skeleton data corresponding to the left arm and the left leg in the first skeleton data is missing. Since the video analysis cameras can move along with the motion capture actor, the video analysis cameras can be placed in a position that is not blocked when the action of the motion capture actor is blocked by the setting, so that the computer device can collect the second optical data corresponding to the video analysis cameras through the optical motion capture cameras, and determine the spatial coordinates of the video analysis cameras. Then, the computer device can determine the second skeleton data that is more complete according to the image data of the motion capture actor collected by the video analysis cameras and the spatial coordinates of the video analysis cameras.

[0154] In a possible implementation, the computer device determines the head, right arm, and right leg of the motion capture actor in the first skeleton data as the first key skeleton, and determines the first skeleton node data corresponding to the first key skeleton. The computer device determines the left arm and the left leg of the motion capture actor in the first skeleton data as the to-be-optimized sub-skeleton. Further, the computer device determines the head, right arm, and right leg of the motion capture actor in the second skeleton data as the second key skeleton, and determines the second skeleton node coordinates corresponding to the second key skeleton. Then, the computer device aligns the second skeleton data with the first skeleton data based on the first skeleton node coordinates and the second skeleton node coordinates.

[0155] In step 1107, the first skeleton data is optimized based on the aligned second skeleton data to obtain target skeleton data.

[0156] In a possible implementation, after the second skeleton data is aligned, the computer device determines the skeleton data corresponding to the left arm and the left leg of the motion capture actor in the second skeleton data as a second sub-skeleton, and replaces the first sub-skeleton data corresponding to the to-be-optimized sub-skeleton in the first skeleton data with the second sub-skeleton, that is, the skeleton data corresponding to the left arm and the left leg of the motion capture actor in the first skeleton data, so as to obtain more complete target skeleton data.

[0157] In the above embodiment, in the virtual studio scenario, during the motion capture actor performs the action, in the case that the action of the motion capture actor is blocked by the setting in the studio, since the optical motion capture camera is fixed, the optical data collection cannot be performed on all optical capture points on the motion capture actor, and therefore the first skeleton data determined based on the first optical data has the problem of data loss. By ensuring that the video analysis camera can be completely in the shooting range of the optical motion capture camera and moving the video analysis camera, complete image data collection can be performed on the motion capture actor, so that more complete second skeleton data is determined, and then the computer device can optimize the lost part of the first skeleton data based on the second skeleton data, so that complete motion capture of the motion capture actor can be implemented, and the data quality of the motion capture is improved.

[0158] Please refer to Figure 12 which shows a structural block diagram of a motion capture device provided by an example embodiment of the present application, and the device includes:

[0159] The first data determination module 1201 is configured to determine first skeleton data of a motion capture object in the motion capture scene based on first optical data collected by the optical motion capture camera, wherein the first optical data is obtained by collecting optical capture points arranged on the motion capture object.

[0160] The second data determination module 1202 is configured to determine second skeleton data based on image data collected by the video analysis camera, wherein the image data contains the motion capture object in the motion capture scene, and the second skeleton data is based on the same coordinate space as the first skeleton data.

[0161] The data optimization module 1203 is configured to optimize the first skeleton data based on the second skeleton data to obtain target skeleton data.

[0162] Optionally, the data optimization module 1203 includes:

[0163] The data alignment unit is configured to align the second skeleton data based on the first skeleton data.

[0164] a data optimization unit, configured to optimize the first skeleton data based on the second skeleton data after alignment, to obtain the target skeleton data.

[0165] Optionally, the data alignment unit is configured to:

[0166] determine a first skeleton node coordinate of a first key skeleton in the first skeleton data;

[0167] determine a second skeleton node coordinate of a second key skeleton in the second skeleton data, the second key skeleton corresponding to the first key skeleton;

[0168] align the second skeleton data based on the first skeleton node coordinate and the second skeleton node coordinate, the second skeleton node coordinate in the second skeleton data after alignment corresponding to the first skeleton node coordinate in the first skeleton data.

[0169] Optionally, the data optimization unit is configured to:

[0170] determine a to-be-optimized sub-skeleton in the first skeleton data based on a missing condition of the first skeleton data; and / or, determine the to-be-optimized sub-skeleton in the first skeleton data based on a skeletal rotation degree of freedom represented by the first skeleton data;

[0171] extract second sub-skeleton data corresponding to the to-be-optimized sub-skeleton from the second skeleton data after alignment;

[0172] replace first sub-skeleton data corresponding to the to-be-optimized sub-skeleton in the first skeleton data with the second sub-skeleton data, to obtain the target skeleton data.

[0173] Optionally, the data optimization unit is further configured to:

[0174] determine a missing condition of first sub-skeleton data corresponding to each sub-skeleton in the first skeleton data;

[0175] in a case where the first sub-skeleton data is missing, determine the sub-skeleton corresponding to the first sub-skeleton data as the to-be-optimized sub-skeleton.

[0176] Optionally, the data optimization unit is further configured to:

[0177] determine a skeletal rotation degree of freedom represented by first sub-skeleton data corresponding to each sub-skeleton in the first skeleton data;

[0178] In a case where the rotation degree of freedom of the bone is greater than a rotation degree of freedom threshold, a sub-bone corresponding to the first sub-bone data is determined as the to-be-optimized sub-bone.

[0179] Optionally, the data optimization module 1203 is configured to:

[0180] extract third sub-bone data from the aligned second bone data, wherein the first bone data does not include a sub-bone corresponding to the third sub-bone data;

[0181] perform data supplement on the target bone data based on the third sub-bone data.

[0182] Optionally, the apparatus further includes:

[0183] a signal sending module configured to send a time synchronization signal to the video analysis camera and the optical motion capture camera;

[0184] a data synchronization module configured to synchronize the second bone data corresponding to the video analysis camera and the first bone data corresponding to the optical motion capture camera based on the time synchronization signal.

[0185] Optionally, the apparatus further includes:

[0186] a coordinate determination module configured to determine a spatial coordinate of the optical motion capture camera;

[0187] an extrinsic parameter calibration module configured to perform extrinsic parameter calibration on the video analysis camera based on second optical data collected by the optical motion capture camera and the spatial coordinate, to obtain extrinsic parameter calibration data corresponding to the video analysis camera, wherein the second optical data is obtained by collecting optical capture points arranged on the video analysis camera;

[0188] The second data determination module 1202 is configured to:

[0189] determine the second bone data based on the image data collected by the video analysis camera and the extrinsic parameter calibration data.

[0190] Optionally, the extrinsic parameter calibration module is configured to:

[0191] determine rotation displacement data of the video analysis camera based on the second optical data collected by the optical motion capture camera;

[0192] perform coordinate system transformation on the rotation displacement data based on the spatial coordinate, to obtain the extrinsic parameter calibration data corresponding to the video analysis camera.

[0193] Optionally, a plurality of video analysis cameras are arranged in the motion capture scene, and the video analysis cameras are provided with at least three optical capture points;

[0194] The apparatus further comprises:

[0195] a first camera distinguishing module, configured to distinguish the video analysis cameras based on the installation positions of the at least three optical capture points on the video analysis cameras, wherein the installation positions of the optical capture points on different video analysis cameras are different; or

[0196] a second camera distinguishing module, configured to distinguish the video analysis cameras based on the light-emitting frequencies of the optical capture points on the video analysis cameras, wherein the light-emitting frequencies of the optical capture points on different video analysis cameras are different.

[0197] In the embodiments of the present application, the computer device determines the first skeleton data of the motion capture object based on the first optical data collected by the optical motion capture camera, and determines the second skeleton data of the motion capture object based on the image data collected by the video analysis camera, wherein the second skeleton data and the first skeleton data are based on the same coordinate space, so that the computer device optimizes the first skeleton data based on the second skeleton data to obtain the target skeleton data. By using the scheme provided in the embodiments of the present application, the first skeleton data representing the accurate position of the skeleton node can be optimized by the second skeleton data representing the better rotation state of the skeleton, so that the target skeleton data can better represent the real action form of the motion capture object, and the quality of the target skeleton data is improved.

[0198] It should be noted that: the apparatus provided in the above embodiments is only exemplified by the division of the above functional modules, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules to complete all or part of the above described functions. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

[0199] Please refer to Figure 13Fig. 13 shows a schematic diagram of a computer device according to an example embodiment of the present application. Specifically, the computer device 1300 includes a central processing unit (CPU) 1301, a system memory 1304, including a random access memory 1302 and a read-only memory 1303, and a system bus 1305 that couples the system memory 1304 to the central processing unit 1301. The computer device 1300 also includes an input / output (I / O) system 1306 that helps transfer information between the various devices within the computer, and a mass storage device 1307 for storing an operating system 1313, application programs 1314, and other program modules 1315.

[0200] The I / O system 1306 includes a display 1308 for displaying information and an input device 1309, such as a mouse, keyboard, or the like, for inputting information into the computer. Both the display 1308 and the input device 1309 are connected to the central processing unit 1301 through an input / output controller 1310 that is connected to the system bus 1305. The I / O system 1306 can also include the input / output controller 1310 for receiving and processing input from a number of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1310 provides output to a display screen, printer, or other type of output device.

[0201] The mass storage device 1307 is connected to the central processing unit 1301 through a mass storage controller (not shown) that is connected to the system bus 1305. The mass storage device 1307 and its associated computer-readable media provide non-volatile storage for the computer device 1300. That is, the mass storage device 1307 can include a computer-readable medium (not shown), such as a hard disk or drive.

[0202] Without loss of generality, the computer readable medium can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes Random Access Memory (RAM), Read Only Memory (ROM), flash memory or other solid state memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. It should be understood by those skilled in the art that the computer storage media is not limited to the above-mentioned several. The system memory 1304 and the mass storage device 1307 mentioned above can be collectively referred to as memory.

[0203] The memory stores one or more programs configured to be executed by the one or more central processing units 1301, and the one or more programs contain instructions for implementing the above method. The central processing unit 1301 executes the one or more programs to implement the method provided by each method embodiment.

[0204] According to various embodiments of the present application, the computer device 1300 can also be connected to a remote computer operating on a network through a network such as the Internet. That is, the computer device 1300 can be connected to the network 1312 through the network interface unit 1311 connected to the system bus 1305, or the network interface unit 1311 can also be used to connect to other types of networks or remote computer systems (not shown).

[0205] The embodiments of the present application also provide a computer readable storage medium, which stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the action capture method described in the above embodiments.

[0206] Optionally, the computer readable storage medium can include ROM, RAM, Solid State Drives (SSD) or optical disc, etc. Among them, the RAM can include Resistance Random Access Memory (ReRAM) and Dynamic Random Access Memory (DRAM).

[0207] The embodiment of the present application provides a computer program product, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the action capture method described in the above embodiment.

[0208] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or can be instructed by a program to complete the related hardware, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk.

[0209] The above only describes optional embodiments of the present application and does not limit the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A motion capture method, characterized in that, The method is used in a computer device in a motion capture system. The computer device is connected to an optical motion capture camera and a video analysis camera in the motion capture scene. The motion capture scene is equipped with multiple video analysis cameras, and each video analysis camera is equipped with at least three optical capture points. The method includes: Based on the first optical data collected by the optical motion capture camera, the first skeletal data of the motion capture object in the motion capture scene is determined. The first optical data is obtained by collecting data from optical capture points set on the motion capture object. Determine the spatial coordinates of the optical motion capture camera; Based on the installation positions of at least three optical capture points on each of the video analysis cameras, the video analysis cameras are distinguished, wherein the installation positions of the optical capture points on different video analysis cameras are different; or, based on the emission frequency of the optical capture points on each of the video analysis cameras, the video analysis cameras are distinguished, wherein the emission frequency corresponding to the optical capture points on different video analysis cameras is different. Based on the second optical data acquired by the optical motion capture camera and the spatial coordinates, the video analysis camera is calibrated to obtain the extrinsic calibration data corresponding to the video analysis camera. The second optical data is acquired by acquiring the optical capture points set on the video analysis camera. Based on the image data acquired by the video analysis camera and the extrinsic calibration data, second skeleton data is determined. The image data includes the motion capture object in the motion capture scene. The second skeleton data and the first skeleton data are based on the same coordinate space. The first bone data is optimized based on the second bone data to obtain the target bone data.

2. The method according to claim 1, characterized in that, The optimization of the first bone data based on the second bone data to obtain the target bone data includes: The second bone data is aligned using the first bone data as a standard. Based on the aligned second bone data, the first bone data is optimized to obtain the target bone data.

3. The method according to claim 2, characterized in that, The step of aligning the second bone data with the first bone data as the standard includes: Determine the coordinates of the first bone node of the first key bone in the first skeletal data; Determine the coordinates of the second bone node of the second key bone in the second bone data, where the second key bone corresponds to the first key bone; Based on the coordinates of the first bone node and the coordinates of the second bone node, the second bone data is aligned, and the coordinates of the second bone node in the aligned second bone data correspond to the coordinates of the first bone node in the first bone data.

4. The method according to claim 2, characterized in that, The optimization of the first bone data based on the aligned second bone data to obtain the target bone data includes: Based on the missing information in the first bone data, determine the sub-bones to be optimized in the first bone data; and / or, based on the bone rotational degrees of freedom represented by the first bone data, determine the sub-bones to be optimized in the first bone data. Extract the second sub-bone data corresponding to the sub-bone to be optimized from the aligned second bone data; The first sub-bone data corresponding to the sub-bone to be optimized in the first bone data is replaced with the second sub-bone data to obtain the target bone data.

5. The method according to claim 4, characterized in that, The step of determining the sub-bones to be optimized in the first bone data based on the missing information of the first bone data includes: Determine the missing data of the first sub-bone corresponding to each sub-bone in the first skeletal data; If the first sub-bone data is missing, the sub-bone corresponding to the first sub-bone data is determined as the sub-bone to be optimized.

6. The method according to claim 4, characterized in that, The step of determining the sub-bone to be optimized in the first bone data based on the bone rotational degrees of freedom represented by the first bone data includes: Determine the degree of freedom of bone rotation represented by the first sub-bone data corresponding to each sub-bone in the first bone data; If the degree of rotational freedom of the bone is greater than the threshold of rotational degree of freedom, the sub-bone corresponding to the first sub-bone data is determined as the sub-bone to be optimized.

7. The method according to claim 2, characterized in that, The step of optimizing the first bone data based on the second bone data to obtain the target bone data further includes: Extract the third sub-bone data from the aligned second bone data, wherein the first bone data does not contain the sub-bone corresponding to the third sub-bone data; The target bone data is supplemented based on the third sub-bone data.

8. The method according to claim 1, characterized in that, Before optimizing the first bone data based on the second bone data to obtain the target bone data, the method further includes: Send time synchronization signals to the video analysis camera and the optical motion capture camera; Based on the time synchronization signal, the second skeleton data corresponding to the video analysis camera and the first skeleton data corresponding to the optical motion capture camera are synchronized.

9. The method according to claim 1, characterized in that, The step of calibrating the video analysis camera based on the second optical data acquired by the optical motion capture camera and the spatial coordinates, to obtain the extrinsic parameter calibration data corresponding to the video analysis camera, includes: Based on the second optical data acquired by the optical motion capture camera, the rotational displacement data of the video analysis camera is determined; Based on the spatial coordinates, the rotational displacement data is transformed to obtain the extrinsic parameter calibration data corresponding to the video analysis camera.

10. A motion capture device, characterized in that, The device is used as a computer device in a motion capture system. The computer device is connected to an optical motion capture camera and a video analysis camera in the motion capture scene. Multiple video analysis cameras are set in the motion capture scene, and each video analysis camera is equipped with at least three optical capture points. The device includes: The first data determination module is used to determine the first skeleton data of the motion capture object in the motion capture scene based on the first optical data collected by the optical motion capture camera. The first optical data is obtained by collecting data from optical capture points set on the motion capture object. The coordinate determination module is used to determine the spatial coordinates of the optical motion capture camera; A first camera differentiation module is used to differentiate between the video analysis cameras based on the installation positions of at least three optical capture points on each of the video analysis cameras, wherein the installation positions of the optical capture points are different on different video analysis cameras; or, The second camera differentiation module is used to differentiate each video analysis camera based on the emission frequency of the optical capture point on each video analysis camera, wherein the emission frequency corresponding to the optical capture point on different video analysis cameras is different; The extrinsic parameter calibration module is used to perform extrinsic parameter calibration on the video analysis camera based on the second optical data acquired by the optical motion capture camera and the spatial coordinates, so as to obtain the extrinsic parameter calibration data corresponding to the video analysis camera. The second optical data is obtained by acquiring optical capture points set on the video analysis camera. The second data determination module is used to determine the second skeleton data based on the image data collected by the video analysis camera and the external parameter calibration data. The image data includes the motion capture object in the motion capture scene. The second skeleton data and the first skeleton data are based on the same coordinate space. The data optimization module is used to optimize the first bone data based on the second bone data to obtain the target bone data.

11. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction, which is loaded and executed by the processor to implement the motion capture method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement the motion capture method as described in any one of claims 1 to 9.

13. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the motion capture method as described in any one of claims 1 to 9.

Images