Recording method, recording system and training method of training data set

Abstract

The invention provides a recording method, a recording system and a training method of a training data set, and solves or improves the technical problems that the preparation workload of a neural network model training data set is large and the working efficiency is reduced in the prior art. When a training data set in a gesture tracking model training process is determined, a first camera device of optical capture equipment and a second camera device in virtual helmet equipment are adopted to simultaneously and respectively shoot a first image and a second image of a hand of a user; and the optical capture equipment determines a first pose information group of the hand of the user according to the first image, and determines a second pose information group corresponding to the second camera group device according to the first pose information group and the second image, so that marking of the second image is completed. Compared with the prior art that the images are identified one by one to mark the poses of the hands corresponding to the images, the method has the advantages that the efficiency of marking the poses of the hands in the images is high, and the accuracy is high.

Description

Technical Field

[0001] The present application relates to the field of virtual reality technology, and in particular to a recording method, a recording system and a training method for a training data set. Background Art

[0002] Virtual reality (VR), also known as virtual environment, mainly uses computers to generate a three-dimensional visual virtual world, allowing users to have an immersive feeling and exploration in the virtual three-dimensional world. That is, the virtual reality helmet device can generate corresponding images according to the user's position and perspective, allowing the user to feel a sense of presence during his or her movement.

[0003] When users are immersed in the virtual reality space, the human-computer interaction module is indispensable, especially the 3D posture of the hands. Hand gesture posture recognition is very critical in virtual reality technology. At present, simple VR devices can usually only use posture sensors (gyroscopes, magnetometers, accelerometers, etc.) to provide rotational motion information. More sophisticated virtual reality helmet devices use neural network models to assist in providing accurate posture tracking information. The neural network model requires a large amount of data to train the neural network model in order to obtain a trained neural network model. At present, the training data of the neural network model is the captured picture, and the posture of the gesture in the picture is annotated, and then the picture and the corresponding gesture posture are input into the neural network model to train the neural network model. The workload of the training data set is relatively large, which reduces the work efficiency. Summary of the invention

[0004] In view of this, the present application provides a recording method, a recording system and a training method for a training data set, which solves or improves the technical problem in the prior art that the preparation workload of the neural network model training data set is large and the work efficiency is reduced.

[0005] As a first aspect of the present application, the present application provides a method for recording a data set of a gesture tracking model, which is applied to a virtual reality system and an optical motion capture device, wherein the virtual reality system includes a mobile terminal and a virtual reality helmet device; wherein the optical motion capture device includes a first camera device, and the virtual reality helmet device includes a second camera device;

[0006] Wherein, the recording method comprises:

[0007] Acquire a first position information group of a user's hand when the user uses the virtual reality system output by the optical motion capture device within a preset time period, wherein the first position information group includes a plurality of first position information arranged in chronological order, and each first position information includes: a first image of the hand, a first frame time code corresponding to the first image, and a first position of the hand;

[0008] Acquire an image information group of a user's hand output by the second camera device during the preset time period when the user uses the virtual reality system, wherein the image information group includes a plurality of second images that are continuous in time;

[0009] According to the first posture information group and the image information group, a second posture information group of the hand corresponding to the virtual reality helmet device is determined, the second posture information group includes a plurality of second posture information arranged in chronological order, the second posture information includes a second image of the hand and a second posture of the hand corresponding to the second image.

[0010] In one embodiment of the present application, determining a second posture information group of a hand corresponding to the virtual reality helmet device according to the first posture information group and the image information group includes:

[0011] Acquire a first coordinate system of the optical motion capture device and a second coordinate system of the virtual reality helmet device;

[0012] Determine a coordinate system transformation matrix according to the first coordinate system and the second coordinate system;

[0013] Based on the coordinate system conversion matrix, performing matrix conversion on the plurality of first posture information in the first posture information group respectively to generate a plurality of first target posture information;

[0014] Align the first images in the plurality of the first target pose information with the plurality of the second images to generate a second pose information group.

[0015] In one embodiment of the present application, aligning a plurality of the first target pose information with a plurality of the second images to generate a second pose information group includes:

[0016] Acquire a first shooting frame rate of the image captured by the first camera device and a second shooting frame rate of the image captured by the second camera device;

[0017] Acquire a first initial time point at which the first camera device starts working;

[0018] Calculate a first timestamp corresponding to each first image in the first pose information according to the first initial time point, the first shooting frame rate, and a first frame time code corresponding to each first image in the first target pose information;

[0019] Calculate a second timestamp corresponding to each second image according to a second initial time point at which the second camera device is started, the second shooting frame rate, and a second frame time code corresponding to each second image;

[0020] According to the second timestamp corresponding to the second image and the first timestamp corresponding to the first image in the first target pose information, the first target pose is aligned with the second image to generate a second pose corresponding to the second image.

[0021] In an embodiment of the present application, the recording method further includes:

[0022] The second camera device is controlled to send a square wave signal to the first camera device, wherein the square wave signal is used to control a plurality of the second camera devices and a plurality of the first camera devices to be exposed synchronously.

[0023] In an embodiment of the present application, the recording method further includes:

[0024] Controlling the first camera device to operate at a first preset time point after receiving the square wave signal, and controlling the second camera device to start operating at a second preset time point after sending the square wave signal;

[0025] The first preset time point is: the time point at which the first cycle starts after the square wave signal is received; the second preset time point is the time point at which the first cycle starts after the square wave signal is sent.

[0026] As a second aspect of the present application, the present application also provides a method for training a gesture tracking model, comprising:

[0027] Build an initial model for gesture tracking;

[0028] Using a training data set and a verification data set to train and verify the gesture tracking initial model to determine the gesture tracking model;

[0029] The training data set and the verification data set are both recorded using the above-mentioned method for recording a data set of a gesture tracking model.

[0030] As a third aspect of the present application, the present application also provides a system for recording a data set of a gesture tracking model, including:

[0031] A virtual reality helmet device in a virtual reality system, wherein the virtual reality helmet device comprises at least one first camera device;

[0032] An optical motion capture device, the optical motion capture device comprising at least one second camera device; and

[0033] A recording controller, the recording controller comprising:

[0034] A data acquisition unit is used to acquire a first position information group of a user's hand when the user uses the virtual reality system output by the optical motion capture device within a preset time period, wherein the first position information group includes a plurality of first position information arranged in time sequence, and each of the first position information includes: a first image of the hand, a first frame time code corresponding to the first image, and the first position of the hand; and acquire an image information group of the user's hand when the user uses the virtual reality system output by the second camera device within the preset time period, wherein the image information group includes a plurality of second images that are continuous in time;

[0035] An alignment unit is used to determine a second posture information group of a hand corresponding to the virtual reality helmet device according to the first posture information group and the image information group, wherein the second posture information group includes a plurality of second posture information arranged in chronological order, and the second posture information includes a second image of the hand and a second posture of the hand corresponding to the second image.

[0036] In an embodiment of the present application, the alignment unit includes:

[0037] A coordinate system conversion module, used to obtain a first coordinate system of the optical motion capture device and a second coordinate system of the virtual reality helmet device; and determine a coordinate system conversion matrix according to the first coordinate system and the second coordinate system;

[0038] A posture conversion module, used for performing matrix conversion on the plurality of first posture information in the first posture information group based on the coordinate system conversion matrix, so as to generate a plurality of first target posture information;

[0039] A posture alignment module is used to align the first images in the plurality of the first target posture information with the plurality of the second images to generate a second posture information group.

[0040] In one embodiment of the present application, the posture alignment module includes:

[0041] A timestamp combing module is used to obtain a first shooting frame rate of the image captured by the first camera device and a second shooting frame rate of the image captured by the second camera device; obtain a first initial time point when the first camera device starts working; calculate a first timestamp corresponding to each first image in the first posture information according to the first initial time point, the first shooting frame rate, and the first frame time code corresponding to the first image in each first target posture information; and calculate a second timestamp corresponding to each second image according to a second initial time point when the second camera device starts working, the second shooting frame rate, and the second frame time code corresponding to each second image;

[0042] A time alignment module is used to align the first target posture with the second image according to a second timestamp corresponding to the second image and a first timestamp corresponding to the first image in the first target posture information to generate a second posture corresponding to the second image.

[0043] In one embodiment of the present application, the recording controller further includes:

[0044] a time control unit, used to control the second camera device to send a square wave signal to the first camera device, wherein the square wave signal is used to control the multiple second camera devices and the multiple first camera devices to be exposed synchronously, and to control the first camera device to work at a first preset time point after receiving the square wave signal, and to control the second camera device to start working at a second preset time point after sending the square wave signal;

[0045] The first preset time point is: the time point at which the first cycle starts after the square wave signal is received; the second preset time point is the time point at which the first cycle starts after the square wave signal is sent.

[0046] The present application provides a method for recording a training data set for a gesture tracking model. When determining the training data set in the training process of the gesture tracking model, the first camera device of the optical capture device and the second camera device in the virtual helmet device are used to simultaneously respectively capture the first image and the second image of the user's hand when the user is experiencing the virtual reality system. The optical capture device determines the first position information group of the user's hand based on the first image, and confirms the second position information group corresponding to the second camera group device based on the first position information group and the second image. In this way, the marking of the second image taken by the second camera device is completed (that is, the second position of the user's hand corresponding to each second image is determined). When the built-in gesture tracking model in the virtual helmet device is trained, the second position information group can be used as a training set and a verification set. Compared with the prior art of identifying images one by one to mark the position of the hand corresponding to the image, the position of the hand in the marking image in the present application is efficient and accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0048] Figure 1 The figure is a flow chart of a method for recording a training data set of a gesture tracking model provided in one embodiment of the present application.

[0049] Figure 2 The figure is a flow chart of a method for recording a training data set of a gesture tracking model provided in another embodiment of the present application.

[0050] Figure 3 The figure is a flow chart of a method for recording a training data set of a gesture tracking model provided in another embodiment of the present application.

[0051] Figure 4 Shown is a structural block diagram of a recording controller for a training data set of a gesture tracking model provided in one embodiment of the present application.

[0052] Figure 5 Shown is a structural block diagram of a recording controller for a training data set of a gesture tracking model provided in another embodiment of the present application.

[0053] Figure 6 Shown is a structural block diagram of a recording controller for a training data set of a gesture tracking model provided in another embodiment of the present application.

[0054] Figure 7 Shown is a structural block diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0055] In the description of the application, the meaning of "multiple" is at least two, for example, two, three, etc., unless otherwise clearly and specifically limited. In the embodiment of the present application, all directional indications (such as up, down, left, right, front, back, top, bottom ...) are only used to explain the relative position relationship, motion conditions, etc. between the components under a certain specific posture (as shown in the drawings), and if the specific posture changes, the directional indication also changes accordingly. In addition, the terms "including" and "having" and any of their deformations are intended to cover non-exclusive inclusions. For example, the process, method, system, product or equipment comprising a series of steps or units is not limited to the steps or units listed, but optionally also includes the steps or units not listed, or optionally also includes other steps or units inherent to these processes, methods, products or equipment.

[0056] In addition, the reference to "embodiment" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0057] Exemplary Methods

[0058] As a first aspect of the present application, the present application provides a method for recording a training data set of a gesture tracking model, which is applicable to a virtual reality system, wherein the virtual reality system includes: the virtual reality system includes a mobile terminal, a virtual reality helmet device and an optical motion capture device.

[0059] The optical motion capture device includes a first camera device, which is used to capture images including user gestures from different angles, and calculate the skeletal position of the hand at the corresponding time point of the image through 3D reconstruction technology, so as to complete the gesture position calculation. Specifically, the number of first camera devices in the optical motion capture device can be 10-20, that is, the optical motion capture device has 10-20 built-in cameras, which capture the hand in all directions and 360 degrees, and the multi-view images are used to calculate the skeletal position of the hand at the corresponding time point of the image through 3D reconstruction technology.

[0060] The virtual reality helmet device includes a second camera device, which can take pictures including the user's hands from different angles.

[0061] like Figure 1 As shown, a method for recording a training data set of a gesture tracking model includes the following steps:

[0062] S1: obtaining a first position information group of a user's hand when the user uses the virtual reality system output by an optical motion capture device within a preset time period, wherein the first position information group includes a plurality of first position information arranged in time sequence, and each first position information includes: a first image of the hand, a first frame time code corresponding to the first image, and a first position of the hand;

[0063] Specifically, during the process of recording gesture tracking data, when the user uses a virtual reality helmet device for virtual embodiment, the optical capture device and the second camera device in the virtual reality helmet device are controlled to start working.

[0064] The optical capture device can capture the first image of the user's hand from different angles, and then calculate the first position of the hand corresponding to the first image based on the multiple first images obtained by continuous shooting. Therefore, the optical motion capture device outputs the first position information group of the hand within a preset time period, the first position information group includes multiple continuous first position information, and the first position information includes the first image and the first position corresponding to the first image.

[0065] It should be noted that the posture in this application refers to: position and posture, where the position refers to the position of the hand in three-dimensional space, such as (x, y, z) coordinates, where x, y and z represent the position of the object in three directions in the three-dimensional coordinate system. The posture refers to the direction or rotation state of the hand in space, which can be represented by a rotation matrix or three angles (such as pitch, yaw and roll).

[0066] S2: Acquire an image information group of a user's hand output by a second camera device when the user uses the virtual reality system within a preset time period, the image information group including a plurality of second images that are continuous in time;

[0067] When the optical capture device starts working, the second camera device in the virtual helmet device also starts working, and the second camera device captures a second image of the user including the user's hand when the user wears the virtual reality helmet device. When the preset time is continuously captured, multiple temporally continuous second images within the preset time can be obtained, and the multiple continuous second images constitute an image information group.

[0068] Specifically, the preset duration means that the start time and the end time of the preset duration are consistent.

[0069] S3: Determine a second posture information group of the hand corresponding to the virtual reality helmet device based on the first posture information group and the image information group, the second posture information group including a plurality of second posture information arranged in chronological order, the second posture information including a second image of the hand and a second posture of the hand corresponding to the second image.

[0070] Since the images acquired are all of the same user's hand images within a preset time period, the second image captured by the second camera device can be determined based on each first image and the corresponding first pose in the first pose information output by the optical capture device, and thus the second pose corresponding to the second image can be obtained. Then, multiple second images and the corresponding second poses can constitute a second pose information group.

[0071] After the second posture information group is determined, it is the posture information group of the virtual reality helmet device in the virtual reality system during the user's use. If the virtual reality helmet device has a built-in neural network model, the second posture information group can be used to train the neural network model to obtain a neural network model suitable for the virtual reality helmet device. When the virtual reality helmet device is used for virtual reality experience, the second camera device in the virtual reality helmet device can be used to capture the image of the user's hand during the experience, and then the image of the hand is input into the neural network model built into the virtual reality helmet device for recognition, so as to output the hand posture corresponding to the image of the hand.

[0072] When determining the training data set in the training process of the gesture tracking model, the present application uses the first camera device of the optical capture device and the second camera device in the virtual helmet device to simultaneously capture the first image and the second image of the user's hand when the user is experiencing the virtual reality system. The optical capture device determines the first position information group of the user's hand based on the first image, and confirms the second position information group corresponding to the second camera group device based on the first position information group and the second image captured by the second camera device. In this way, the marking of the second image captured by the second camera device is completed (that is, the second position of the user's hand corresponding to each second image is determined). When the built-in gesture tracking model in the virtual helmet device is trained, the second position information group can be used as a training set and a verification set. Compared with the prior art of identifying images one by one to mark the position of the hand corresponding to the image, the position of the hand in the marking image in the present application is efficient and accurate.

[0073] In one embodiment of the present application, Figure 2 As shown, the specific method for determining the second position information group of the hand corresponding to the virtual reality helmet device, that is, S3 (determining the second position information group of the hand corresponding to the virtual reality helmet device according to the first position information group and the image information group) specifically includes the following steps:

[0074] S31: Acquire a first coordinate system of the optical motion capture device and a second coordinate system of the virtual reality helmet device;

[0075] Specifically, the first coordinate system of the optical motion capture device refers to a three-dimensional space coordinate system within the capture area, which is used to determine the precise position and direction of the captured object. The motion posture of the captured object can be determined through a preset calibration method in the optical motion capture device.

[0076] After capturing a first image including a hand, the first camera device in the optical motion capture device analyzes the first image based on the first coordinate system in the optical motion capture device and a preset calibration method to obtain the posture of the hand corresponding to the first image (i.e., the posture of each skeletal joint of the hand).

[0077] Similarly, the second coordinate system of the virtual reality helmet device refers to: the three-dimensional space coordinate system within the area, which is used to determine the precise position and direction of the target object.

[0078] The optical capture device has a built-in first camera device, so the first coordinate system of the first camera device can be acquired by acquiring the parameters of the built-in first camera device.

[0079] Similarly, the second coordinate system of the second camera device may also be acquired by acquiring the parameters of the second camera device.

[0080] S32: Determine a coordinate system conversion matrix according to the first coordinate system and the second coordinate system;

[0081] S33: Based on the coordinate system conversion matrix, performing matrix conversion on the plurality of first pose information in the first pose information group respectively to generate a plurality of first target pose information;

[0082] By converting the coordinate system, the hand posture corresponding to the first image taken by the first camera device can be converted into the hand posture corresponding to the second image taken by the second camera device.

[0083] The first pose information includes the first image and the first pose corresponding to the first image. After conversion, the first image corresponding to the first image and the first target pose corresponding to the first pose can be obtained, wherein the first target pose corresponds to the first image.

[0084] The first image and the first target posture constitute the first target posture information.

[0085] After the multiple first pose information are converted, multiple first target pose information are generated.

[0086] S34: Align the first image in the plurality of first target posture information with the plurality of second images to generate a second posture information group.

[0087] The first target pose information is aligned with a plurality of second images to generate a second pose information group.

[0088] Optionally, the specific alignment method, that is, S34 (aligning the first image in the plurality of first target pose information with the plurality of second images to generate a second pose information group) specifically includes the following steps:

[0089] S341: Acquire a first shooting frame rate of an image captured by a first camera device and a second shooting frame rate of an image captured by a second camera device;

[0090] Specifically, the frame rate refers to the number of image frames that the camera device can capture and process per second. Therefore, the first shooting frame rate of the first camera device and the second shooting frame rate of the second camera device can both be obtained from the parameters of the camera devices.

[0091] Specifically, the first shooting frame rate of the first camera device and the second shooting frame rate of the second camera device may be the same (for example, both are 90fps) or different (for example, the frame rate of the first camera device is 90fps and the frame rate of the second camera device is 100fps).

[0092] S342: Obtaining a first initial time point when the first camera device starts working;

[0093] The first initial time point T1 at which the first camera device starts working can be determined by the time point at which the first camera device starts working.

[0094] S343: Determine a first timestamp corresponding to the first image in each first pose information according to the first initial time point, the first shooting frame rate, and the first frame time code corresponding to the first image in each first target pose information;

[0095] The first target posture information includes the first image and the corresponding first target posture (the first posture is obtained by transforming the transformation matrix). There is a corresponding first frame time code in the first image. Through the first initial time point and the first frame time code in each first image, the first timestamp corresponding to each first image can be determined, that is, the first time point corresponding to each first image.

[0096] S344: Determine a second timestamp corresponding to each second image according to a second initial time point at which the second camera device is started, a second shooting frame rate, and a second frame time code corresponding to the second image;

[0097] Similarly, the second initial time point T2 when the second camera device starts working, the second shooting frame rate and the second frame time encoding of each second image can calculate the second timestamp corresponding to each second image, that is, the second time point.

[0098] Specifically, the first initial time point T1 when the first camera device starts working and the second initial time point T2 when the second camera device starts working may be the same or different.

[0099] S345: Align the first image with the second image according to the second timestamp corresponding to the second image and the first timestamp corresponding to the first image in the first target pose information, and generate a second pose corresponding to the second image.

[0100] Specifically, after the second poses of the plurality of second images are determined, the second timestamps of the second images are also determined, so the second pose information includes the second images and the corresponding second poses. Then the second pose information of the plurality of second images constitutes a second pose information group, which can be used as a training data set for a gesture tracking model built into a virtual reality helmet device.

[0101] In another embodiment of the present application, Figure 3 As shown, before S1 (obtaining the first position information group of the user's hand output by the optical motion capture device when the user uses the virtual reality system within a preset time period), the gesture tracking method further includes the following steps:

[0102] S10: Control the second camera device to send a square wave signal to the first camera device, where the square wave signal is used to control multiple second camera devices and multiple first camera devices to be exposed synchronously;

[0103] A square wave signal can be used to control multiple first camera devices and multiple second camera devices to be exposed synchronously (i.e., exposed at the same time), so that the timestamps of a first image captured by the optical motion capture device at a certain moment and a second image captured by the virtual reality helmet device are consistent. In this way, the hand posture output by the optical motion capture device corresponding to the moment can be converted through a transformation matrix to obtain the hand posture (the posture of each skeletal joint) corresponding to the second image captured by the virtual reality helmet device.

[0104] S11: Control the first camera device to work at a first preset time point after receiving the square wave signal, and control the second camera device to start working at a second preset time point after sending the square wave signal, wherein the first preset time point is: the time point at which the first cycle starts after receiving the square wave signal; the second preset time point is the time point at which the first cycle starts after sending the square wave signal.

[0105] Since the time interval from the second camera device sending out the square wave signal to the first camera device receiving the square wave signal is short and can be ignored, the time point when the first cycle after the first camera device sends out the square wave signal and the time point when the first cycle after the second camera device receives the square wave signal start are the same time point. At this time, controlling the operation of the first camera device and the second camera device can make the first camera device and the second camera device work at the same time.

[0106] As a second aspect of the present application, the present application also provides a method for training a gesture tracking model, comprising the following steps:

[0107] Build an initial model for gesture tracking;

[0108] Using a training data set and a verification data set to train and verify the gesture tracking initial model to determine the gesture tracking model;

[0109] The training data set and the verification data set are both recorded using the above-mentioned method for recording a data set of a gesture tracking model.

[0110] Exemplary Recording System

[0111] As a third aspect of the present application, the present application also provides a system for recording a data set of a gesture tracking model. Figure 4 FIG. 1 is a structural block diagram of a data set recording system of a gesture tracking model provided by an embodiment of the present application. Figure 4 As shown, a recording system 1 for a data set of a gesture tracking model includes:

[0112] A virtual reality helmet device 10 in a virtual reality system, wherein the virtual reality helmet device 10 includes at least one first camera device;

[0113] An optical motion capture device 20, the optical motion capture device 20 comprising at least one second camera device; and

[0114] The recording controller 30 includes:

[0115] The data acquisition unit 301 is used to acquire a first position information group of a user's hand when the user uses the virtual reality system output by the optical motion capture device within a preset time period, the first position information group including a plurality of first position information arranged in time sequence, each first position information including: a first image of the hand, a first frame time code corresponding to the first image, and a first position of the hand; and acquire an image information group of the user's hand when the user uses the virtual reality system output by the second camera device within a preset time period, the image information group including a plurality of second images that are continuous in time;

[0116] The alignment unit 302 is used to determine a second posture information group of a hand corresponding to the virtual reality helmet device based on the first posture information group and the image information group, wherein the second posture information group includes a plurality of second posture information arranged in chronological order, and the second posture information includes a second image of the hand and a second posture of the hand corresponding to the second image.

[0117] The present application provides a recording system for a data set of a gesture tracking model. When determining a training data set in the training process of the gesture tracking model, the first camera device of the optical capture device and the second camera device in the virtual helmet device are used to simultaneously respectively capture the first image and the second image of the user's hand when the user is experiencing the virtual reality system. The optical capture device determines the first position information group of the user's hand based on the first image, and confirms the second position information group corresponding to the second camera group device based on the first position information group and the second image captured by the second camera device. In this way, the marking of the second image captured by the second camera device is completed (that is, the second position of the user's hand corresponding to each second image is determined). When the built-in gesture tracking model in the virtual helmet device is trained, the second position information group can be used as a training set and a verification set. Compared with the prior art of identifying images one by one to mark the position of the hand corresponding to the image, the position of the hand in the marking image in the present application is efficient and accurate.

[0118] In one embodiment of the present application, Figure 5 As shown, the alignment unit 302 includes:

[0119] A coordinate system conversion module 3021 is used to obtain a first coordinate system of the optical motion capture device and a second coordinate system of the virtual reality helmet device; and determine a coordinate system conversion matrix according to the first coordinate system and the second coordinate system;

[0120] Specifically, the first coordinate system of the optical motion capture device refers to a three-dimensional space coordinate system within the capture area, which is used to determine the precise position and direction of the captured object. The motion posture of the captured object can be determined through a preset calibration method in the optical motion capture device.

[0121] After capturing a first image including a hand, the first camera device in the optical motion capture device analyzes the first image based on the first coordinate system in the optical motion capture device and a preset calibration method to obtain the posture of the hand corresponding to the first image (i.e., the posture of each skeletal joint of the hand).

[0122] Similarly, the second coordinate system of the virtual reality helmet device refers to: a three-dimensional space coordinate system in the region, which is used to determine the precise position and direction of the target object. The posture conversion module 3022 is used to perform matrix conversion on multiple first posture information in the first posture information group based on the coordinate system conversion matrix to generate multiple first target posture information;

[0123] By converting the coordinate system, the hand posture corresponding to the first image taken by the first camera device can be converted into the hand posture corresponding to the second image taken by the second camera device.

[0124] The first pose information includes the first image and the first pose corresponding to the first image. After conversion, the first image corresponding to the first image and the first target pose corresponding to the first pose can be obtained, wherein the first target pose corresponds to the first image.

[0125] The first image and the first target posture constitute the first target posture information. After the first posture information is converted, multiple first target posture information are generated.

[0126] The posture alignment module 3023 is used to align the first image in the multiple first target posture information with the multiple second images to generate a second posture information group.

[0127] Optionally, the posture alignment module 3023 includes:

[0128] A timestamp combing module is used to obtain a first shooting frame rate of an image captured by a first camera device and a second shooting frame rate of an image captured by a second camera device; obtain a first initial time point at which the first camera device starts working; calculate a first timestamp corresponding to the first image in each first target posture information according to the first initial time point, the first shooting frame rate, and the first frame time code corresponding to the first image in each first target posture information; and calculate a second timestamp corresponding to each second image according to a second initial time point at which the second camera device starts working, the second shooting frame rate, and the second frame time code corresponding to each second image;

[0129] Specifically, the frame rate refers to the number of image frames that the camera device can capture and process per second. Therefore, the first shooting frame rate of the first camera device and the second shooting frame rate of the second camera device can be obtained from the parameters of the camera device. The first shooting frame rate of the first camera device and the second shooting frame rate of the second camera device can be the same (for example, both are 90fps), or different (for example, the frame rate of the first camera device is 90fps, and the frame rate of the second camera device is 100fps). The first initial time point T1 at which the first camera device starts working can be determined by the time point at which the first camera device starts working. The first target posture information includes the first image and the corresponding first target posture (the first posture is obtained by conversion matrix conversion), and there is a corresponding first frame time code in the first image. Through the first initial time point and the first frame time code in each first image, the first timestamp corresponding to each first image, that is, the first time point corresponding to each first image, can be determined.

[0130] The time alignment module is used to align the first target posture with the second image according to the second timestamp corresponding to the second image and the first timestamp corresponding to the first image in the first target posture information to generate a second posture corresponding to the second image.

[0131] Specifically, after the second poses of the plurality of second images are determined, the second timestamps of the second images are also determined, so the second pose information includes the second images and the corresponding second poses. Then the second pose information of the plurality of second images constitutes a second pose information group, which can be used as a training data set for a gesture tracking model built into a virtual reality helmet device.

[0132] In one embodiment of the present application, Figure 6 As shown, the recording controller 30 also includes:

[0133] The time control unit 303 is used to control the second camera device to send a square wave signal to the first camera device, the square wave signal is used to control the multiple second camera devices to be exposed synchronously, and to control the first camera device to work at a first preset time point after receiving the square wave signal, and to control the second camera device to start working at a second preset time point after sending the square wave signal;

[0134] The first preset time point is: the time point at which the first cycle starts after the square wave signal is received; the second preset time point is the time point at which the first cycle starts after the square wave signal is sent.

[0135] A square wave signal can be used to control multiple first camera devices and multiple second camera devices to be exposed synchronously (i.e., exposed at the same time), so that the timestamps of a first image captured by the optical motion capture device at a certain moment and a second image captured by the virtual reality helmet device are consistent. In this way, the hand posture output by the optical motion capture device corresponding to the moment can be converted through a transformation matrix to obtain the hand posture (the posture of each skeletal joint) corresponding to the second image captured by the virtual reality helmet device.

[0136] Exemplary Electronic Devices

[0137] Next, as a third aspect of the present application, the present application further provides an electronic device. Figure 7 To describe an electronic device according to an embodiment of the present application.

[0138] Figure 7 The figure illustrates a structural block diagram of an electronic device according to an embodiment of the present application.

[0139] like Figure 7 As shown, the electronic device 60 includes one or more processors 601 and a memory 602 .

[0140] The processor 601 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 60 to perform desired functions.

[0141] The memory 602 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 601 may execute the program instructions to implement the steps of the method for recording the training data set of the gesture tracking model of the present application described above and / or other desired functions.

[0142] In one example, the electronic device 60 may further include: an input device 603 and an output device 604, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0143] When the electronic device is a stand-alone device, the input device 603 may be a communication network connector, which is used to receive the collected input signals from the first device and the second device.

[0144] In addition, the input device 603 may also include, for example, a keyboard, a mouse, and the like.

[0145] The output device 604 can output various information to the outside, including the determined distance information, direction information, etc. The output device 604 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0146] Of course, to simplify, Figure 7 Only some of the components related to the present application in the electronic device 60 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 60 may also include any other appropriate components.

[0147] As a fifth aspect of the present application, the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is used to execute the steps of the method for recording a training data set of a gesture tracking model in the above-mentioned embodiments.

[0148] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0149] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program information, which, when executed by a processor, enables the processor to execute the steps of the method for recording a training data set of a gesture tracking model of each embodiment of the present application.

[0150] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0151] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0152] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0153] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

Claims

1. A method for recording a data set of a gesture tracking model, characterized in that: Applicable to a virtual reality system and an optical motion capture device, wherein the virtual reality system includes a mobile terminal and a virtual reality helmet device; wherein the optical motion capture device includes a first camera device, and the virtual reality helmet device includes a second camera device; Wherein, the recording method comprises: Acquire a first position information group of a user's hand when the user uses the virtual reality system output by the optical motion capture device within a preset time period, wherein the first position information group includes a plurality of first position information arranged in chronological order, and each first position information includes: a first image of the hand, a first frame time code corresponding to the first image, and a first position of the hand; Acquire an image information group of a user's hand output by the second camera device during the preset time period when the user uses the virtual reality system, wherein the image information group includes a plurality of second images that are continuous in time; According to the first posture information group and the image information group, a second posture information group of the hand corresponding to the virtual reality helmet device is determined, the second posture information group includes a plurality of second posture information arranged in chronological order, the second posture information includes a second image of the hand and a second posture of the hand corresponding to the second image.

2. The recording method according to claim 1, characterized in that: Determining a second posture information group of a hand corresponding to the virtual reality helmet device according to the first posture information group and the image information group includes: Acquire a first coordinate system of the optical motion capture device and a second coordinate system of the virtual reality helmet device; Determine a coordinate system transformation matrix according to the first coordinate system and the second coordinate system; Based on the coordinate system conversion matrix, performing matrix conversion on the plurality of first posture information in the first posture information group respectively to generate a plurality of first target posture information; Align the first images in the plurality of the first target pose information with the plurality of the second images to generate a second pose information group.

3. The recording method according to claim 2, characterized in that: Aligning the plurality of first target pose information with the plurality of second images to generate a second pose information group includes: Acquire a first shooting frame rate of the image captured by the first camera device and a second shooting frame rate of the image captured by the second camera device; Acquire a first initial time point at which the first camera device starts working; Calculate a first timestamp corresponding to each first image in the first pose information according to the first initial time point, the first shooting frame rate, and a first frame time code corresponding to each first image in the first target pose information; Calculate a second timestamp corresponding to each second image according to a second initial time point at which the second camera device is started, the second shooting frame rate, and a second frame time code corresponding to each second image; According to the second timestamp corresponding to the second image and the first timestamp corresponding to the first image in the first target pose information, the first target pose is aligned with the second image to generate a second pose corresponding to the second image.

4. The recording method according to claim 3, characterized in that: The gesture tracking method further includes: The second camera device is controlled to send a square wave signal to the first camera device, wherein the square wave signal is used to control a plurality of the second camera devices and a plurality of the first camera devices to be exposed synchronously.

5. The recording method according to claim 4, characterized in that: Also includes: Controlling the first camera device to operate at a first preset time point after receiving the square wave signal, and controlling the second camera device to start operating at a second preset time point after sending the square wave signal; The first preset time point is: the time point at which the first cycle starts after the square wave signal is received; the second preset time point is the time point at which the first cycle starts after the square wave signal is sent.

6. A method for training a gesture tracking model, characterized in that: include: Build an initial model for gesture tracking; Using a training data set and a verification data set to train and verify the gesture tracking initial model to determine the gesture tracking model; Wherein, the training data set and the verification data set are both recorded using the method for recording a data set of a gesture tracking model as described in any one of claims 1 to 5.

7. A system for recording a data set of a gesture tracking model, characterized in that: include: A virtual reality helmet device in a virtual reality system, wherein the virtual reality helmet device comprises at least one first camera device; An optical motion capture device, the optical motion capture device comprising at least one second camera device; as well as A recording controller, the recording controller comprising: A data acquisition unit is used to acquire a first position information group of a user's hand when the user uses the virtual reality system output by the optical motion capture device within a preset time period, wherein the first position information group includes a plurality of first position information arranged in time sequence, and each of the first position information includes: a first image of the hand, a first frame time code corresponding to the first image, and the first position of the hand; and acquire an image information group of the user's hand when the user uses the virtual reality system output by the second camera device within the preset time period, wherein the image information group includes a plurality of second images that are continuous in time; An alignment unit is used to determine a second posture information group of a hand corresponding to the virtual reality helmet device according to the first posture information group and the image information group, wherein the second posture information group includes a plurality of second posture information arranged in chronological order, and the second posture information includes a second image of the hand and a second posture of the hand corresponding to the second image.

8. The recording system according to claim 7, characterized in that: The alignment unit comprises: A coordinate system conversion module, used to obtain a first coordinate system of the optical motion capture device and a second coordinate system of the virtual reality helmet device; and determine a coordinate system conversion matrix according to the first coordinate system and the second coordinate system; A posture conversion module, used for performing matrix conversion on the plurality of first posture information in the first posture information group based on the coordinate system conversion matrix, so as to generate a plurality of first target posture information; A posture alignment module is used to align the first images in the plurality of the first target posture information with the plurality of the second images to generate a second posture information group.

9. The recording system according to claim 8, characterized in that: The posture alignment module comprises: A timestamp combing module is used to obtain a first shooting frame rate of the image captured by the first camera device and a second shooting frame rate of the image captured by the second camera device; obtain a first initial time point when the first camera device starts working; calculate a first timestamp corresponding to each first image in the first posture information according to the first initial time point, the first shooting frame rate, and the first frame time code corresponding to the first image in each first target posture information; and calculate a second timestamp corresponding to each second image according to a second initial time point when the second camera device starts working, the second shooting frame rate, and the second frame time code corresponding to each second image; A time alignment module is used to align the first target posture with the second image according to a second timestamp corresponding to the second image and a first timestamp corresponding to the first image in the first target posture information to generate a second posture corresponding to the second image.

10. The recording system according to claim 7, characterized in that: The recording controller also includes: a time control unit, used to control the second camera device to send a square wave signal to the first camera device, wherein the square wave signal is used to control the multiple second camera devices and the multiple first camera devices to be exposed synchronously, and to control the first camera device to work at a first preset time point after receiving the square wave signal, and to control the second camera device to start working at a second preset time point after sending the square wave signal; The first preset time point is: the time point at which the first cycle starts after the square wave signal is received; the second preset time point is the time point at which the first cycle starts after the square wave signal is sent.

Images