A method for constructing a high-precision piano gesture dataset based on marker removal network

Abstract

The present application relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device and medium for constructing a high-precision piano gesture data set based on a marker removal network. The present application collects the original image and initial tracking marker point position data corresponding to the gesture action when the user plays the piano, performs a preprocessing operation on the initial tracking marker point position data to obtain the target tracking marker point position data corresponding to the gesture action, inputs the original image into a pre-trained tracking marker point removal network to obtain a target bare hand image with the tracking markers removed, matches the target tracking marker point position data with the target bare hand image, and constructs a gesture data set when the user plays the piano. The present application can generate a high-precision gesture data set.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device and medium for constructing a high-precision piano gesture dataset based on a marker removal network. Background Art

[0002] Training a gesture recognition model requires preparing a gesture dataset in advance as sample data to train the relevant model in order to obtain a gesture recognition model. Currently, the way to obtain a gesture dataset is to obtain the user's gesture data by wearing a special device on the user's hand, but it is difficult to capture dexterous movements by wearing a device, resulting in the inability to generate a high-precision gesture dataset. Summary of the invention

[0003] In view of this, the embodiments of the present application at least provide a method, device, electronic device and medium for constructing a high-precision piano gesture dataset based on a marker removal network, which can generate a high-precision gesture dataset.

[0004] This application mainly includes the following aspects:

[0005] In a first aspect, an embodiment of the present application provides a method for constructing a high-precision piano gesture dataset based on a marker removal network, the method comprising:

[0006] Collecting original images and initial tracking marker position data corresponding to the hand gestures of the user playing the piano, wherein the user's hands are attached with tracking markers, the original images are data collected by a camera device, and the initial tracking marker position data are data collected by a piano gesture capture system based on the tracking markers;

[0007] Performing a preprocessing operation on the initial tracking marker point position data to obtain target tracking marker point position data corresponding to the gesture action;

[0008] Inputting the original image into a pre-trained tracking marker removal network to obtain a target bare hand image without the tracking markers;

[0009] The target tracking mark point position data is matched with the target bare hand image to construct a gesture data set of the user playing the piano.

[0010] In a second aspect, an embodiment of the present application further provides a device for constructing a high-precision piano gesture dataset based on a marker removal network, the device comprising:

[0011] The acquisition module is used to acquire the original image and initial tracking mark point position data corresponding to the hand gestures of the user playing the piano, wherein the user's hands are attached with tracking marks, the original image is data acquired by a camera device, and the initial tracking mark point position data is data acquired by the piano gesture capture system based on the tracking marks;

[0012] Preprocessing module: used to perform preprocessing operation on the initial tracking mark point position data to obtain target tracking mark point position data corresponding to the gesture action;

[0013] Removal module: used for inputting the original image into a pre-trained tracking marker removal network to obtain a target bare hand image with the tracking markers removed;

[0014] Matching module: used to match the target tracking mark point position data with the target bare hand image to construct a gesture data set of the user playing the piano.

[0015] In the third aspect, an embodiment of the present application also provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the method for constructing a high-precision piano gesture dataset based on a marker removal network described in the first aspect above.

[0016] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for constructing a high-precision piano gesture dataset based on a marker removal network described in the first aspect above are executed.

[0017] The embodiments of the present application provide a method, device, electronic device and medium for constructing a high-precision piano gesture data set based on a marker removal network, by collecting the original image and initial tracking marker position data corresponding to the gesture action when the user plays the piano, wherein the user's hand is attached with a tracking marker, the original image is data collected by a camera device, and the initial tracking marker position data is data collected by a piano gesture capture system based on the tracking marker, and a pre-processing operation is performed on the initial tracking marker position data to obtain the target tracking marker position data corresponding to the gesture action, and the original image is input into a pre-trained tracking marker removal network to obtain a target bare hand image with the tracking marker removed, and the target tracking marker position data is matched with the target bare hand image to construct a gesture data set when the user plays the piano. Since the gestures of the user are complex and dexterous when playing the piano, a high-precision gesture data set can be generated through the present application, and a gesture recognition model with higher recognition accuracy can be obtained by training a related gesture recognition model through the gesture data set.

[0018] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A flowchart of a method for constructing a high-precision piano gesture dataset based on a marker removal network provided in an embodiment of the present application is shown;

[0021] Figure 2 is a schematic diagram of an original image in an embodiment of the present application;

[0022] Figure 3 A schematic diagram of the framework of the tracking marker point removal network training process in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of a sample image of tracking marker points in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of synthesizing tracking marker point position data in an embodiment of the present application;

[0025] Figure 6A functional module diagram of a high-precision piano gesture data set construction device based on a marker removal network provided in an embodiment of the present application is shown;

[0026] Figure 7 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0027] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art, under the guidance of the content of the present application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0028] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0029] In order to enable those skilled in the art to use the contents of this application, the following implementation modes are given in combination with specific application scenarios. For those skilled in the art, the general principles defined herein can be applied to other embodiments and application scenarios without departing from the spirit and scope of this application.

[0030] The following methods, devices, electronic devices or computer-readable storage media of the embodiments of the present application can be applied to any scenario that requires the generation of gesture data based on piano performance. The embodiments of the present application are not limited to specific application scenarios. Any scheme using the method and device for constructing a high-precision piano gesture dataset based on a marker removal network provided by the embodiments of the present application is within the scope of protection of the present application.

[0031] To facilitate the understanding of the present application, the technical solution provided by the present application is described in detail below in conjunction with specific embodiments.

[0032] Figure 1 A flowchart of a method for constructing a high-precision piano gesture dataset based on a marker removal network provided in an embodiment of the present application. Figure 1 As shown, the method for constructing a high-precision piano gesture dataset based on a marker removal network provided in an embodiment of the present application includes the following steps:

[0033] S101: collecting original images and initial tracking marker position data corresponding to hand gestures when a user plays the piano, wherein the user's hands are attached with tracking markers, the original images are data collected by a camera device, and the initial tracking marker position data are data collected by a piano gesture capture system based on the tracking markers;

[0034] S102: performing a preprocessing operation on the initial tracking marker point position data to obtain target tracking marker point position data corresponding to the gesture action;

[0035] S103: inputting the original image into a pre-trained tracking marker removal network to obtain a target bare hand image without the tracking markers;

[0036] S104: Match the target tracking mark point position data with the target bare hand image to construct a gesture data set of the user playing the piano.

[0037] In this embodiment, before the user plays the piano, tracking markers are attached to the user's hands. These tracking markers can be reflective balls or sensors of special shapes, which are used to be recognized by the camera during the capture process to obtain the precise position and movement trajectory of the hands.

[0038] When a user is playing the piano, a video of the user's hands is captured by an ordinary camera device to obtain an original image. The ordinary camera device directly captures the user's gestures, such as Figure 2 , which is a schematic diagram of an original image in an embodiment of the present application, Figure 2 The gray dot on the middle hand is the tracking marker. At the same time, the piano gesture capture system based on the tracking marker (marker) is used to collect the initial tracking marker position data in conjunction with the tracking marker attached to the user's hand.

[0039] Since the collected initial tracking marker point position data may have missing marker points, it is necessary to perform a preprocessing operation on the collected initial tracking marker point position data, for example, combining a greedy matching algorithm and a manual operation to perform the preprocessing operation so that marker points of the preprocessed initial tracking marker point position data are not missed. Specifically, the preprocessing operation is performed on the initial tracking marker point position data to obtain the target tracking marker point position data corresponding to the gesture action, including:

[0040] Using a greedy matching algorithm to match the tracking mark points and the position data in the initial tracking mark point position data to obtain an initial marking result;

[0041] Performing an interpolation operation on the data of the missing tracking marking points in the initial marking result to obtain an initial marking result after the interpolation operation;

[0042] Determine whether there is defective data in the initial marking result after performing the interpolation operation;

[0043] If not, the initial marking result after the interpolation operation is performed is used as the target tracking mark point position data corresponding to the gesture action.

[0044] The tracking marker points and their position data in the initial tracking marker point position data are matched using a greedy matching algorithm, and the result is recorded as the initial marking result. If some markers are missing in the initial marking result, they can be manually marked. Due to the perfect piano gesture capture system, in practical applications, the missing markers are usually less than 0.1% of the total data volume. Then, the data of the missing tracking marker points in the initial marking result are interpolated to obtain the initial marking result after the interpolation operation. For example, the cubic spline interpolation method is used to fix occasional tracking losses, and the maximum gap is set to 10 frames. It is determined whether there is defective data in the initial marking result after the interpolation operation. If not, the initial marking result after the interpolation operation is used as the target tracking marker point position data corresponding to the gesture action.

[0045] Furthermore, the determining whether the initial marking result after the interpolation operation contains defective data further includes:

[0046] If there are defective data in the initial marking result after the interpolation operation is performed, the initial marking result is discarded, and the initial tracking mark point position data corresponding to the hand gesture of the user when playing the piano is re-collected.

[0047] If defective data still exists in the initial marking result after the interpolation operation, it means that the error of this batch of data is large, and it is necessary to discard the initial marking result and re-collect the initial tracking marker point position data corresponding to the user's hand gestures when playing the piano.

[0048] The original image is input into a pre-trained tracking marker removal network (tracking marker-removal network, MR-Net) to obtain a bare hand image with the tracking markers removed. The tracking marker removal network is trained based on the concept of the tracking marker removal network. The training process of the tracking marker removal network includes:

[0049] Step A0: Obtain a bare hand sample image and a tracking marker sample data set;

[0050] Step A1: generating a synthetic image containing synthetic tracking markers based on the bare hand sample image and the tracking marker sample data set;

[0051] Step A2: inputting the synthetic tracking marker position data into a tracking marker removal network to generate a predicted bare hand image;

[0052] Step A3: calculating the loss value of the predicted bare hand image and the real bare hand image corresponding to the synthetic tracking marker point position data through a loss function;

[0053] Step A4: Determine whether the loss value is greater than a preset threshold;

[0054] If so, adjust the parameters of the tracking marker point removal network according to the loss value, use the predicted bare hand image as the bare hand sample image, and repeat steps A1 to A4 until the loss value is less than or equal to the preset threshold, thereby obtaining a trained tracking marker point removal network.

[0055] like Figure 3 As shown, it is a schematic diagram of the framework of the tracking marker point removal network training process in the embodiment of the present application. The training process can be divided into two stages. The first stage is to generate synthetic tracking marker point position data, for example, add a marker sample to the bare hand sample image to obtain synthetic tracking marker point position data containing synthetic markers. The second stage is to remove the markers in the synthetic tracking marker point position data, input the synthetic tracking marker point position data into the tracking marker point removal network to generate a predicted bare hand image, and calculate the loss value of the predicted bare hand image and the real bare hand image (i.e., bare hand sample image) corresponding to the synthetic tracking marker point position data through the loss function. If the loss value is greater than the preset threshold, it means that the effect of removing the marker is not as expected, so the parameters of the tracking marker point removal network are adjusted by the loss value, and then the predicted bare hand image is used as the bare hand sample image again, and steps A1 to A4 are repeated until the loss value is less than or equal to the preset threshold. When the loss value is less than or equal to the preset threshold, it means that the effect of removing the marker can reach the expected effect, thereby obtaining a trained tracking marker point removal network, and inputting the original data into the trained tracking marker point removal network to accurately obtain a bare hand image with the tracking marker points removed.

[0056] Furthermore, the generating of a synthetic image containing synthetic tracking markers based on the bare hand sample image and the tracking marker sample data set includes:

[0057] Randomly selecting a preset number of tracking marker point sample images from the tracking marker point sample data set;

[0058] Using each of the tracking mark point sample images as a mark to be synthesized, and blending the mark to be synthesized into a target position of the bare hand sample image to obtain a blended image;

[0059] An image enhancement operation is performed on the mixed image to obtain synthetic tracking marker point position data containing synthetic markers.

[0060] 21 tracking marker sample images are randomly selected from the tracking marker sample dataset, and this number corresponds to the number of hand joints. Figure 4 As shown in FIG. 1 , a schematic diagram of a sample image of a tracking marker point in an embodiment of the present application is shown. Each sample image of a tracking marker point is used as a marker to be synthesized, and the marker to be synthesized is alpha-blended to the target position of the bare hand sample image to obtain a mixed image in which the marker to be synthesized is mixed on the basis of the bare hand sample image. An image enhancement operation is performed on the mixed image, for example, a maximum of 5% noise is randomly added to the bare hand sample image and the marker to be synthesized, or to the mixed image. Thus, synthetic tracking marker point position data containing the synthesized marker is obtained, such as Figure 5 , which is a schematic diagram of synthesizing tracking marker point position data in an embodiment of the present application.

[0061] Further, each of the tracking mark point sample images is used as a mark to be synthesized, and the mark to be synthesized is mixed to the target position of the bare hand sample image to obtain a mixed image, including:

[0062] Using a piano gesture feature extractor, extracting visual features of the bare hand sample image;

[0063] The marker to be synthesized is mixed into the visual feature to represent the position of the hand joint, and synthetic tracking marker point position data containing synthetic tracking marker points is generated.

[0064] Through the piano gesture feature extractor of the Mamba model (e.g., ResNet50 network), visual features for estimating the position of tracking marker points are extracted. The visual feature is a 2D vector p (size: 2x21), and then the marker to be synthesized is mixed into the visual feature to represent the position of the hand joint, thereby generating synthetic tracking marker point position data containing synthetic tracking marker points.

[0065] Further, the original image is input into a pre-trained tracking marker removal network to obtain a target bare hand image with the tracking markers removed, including:

[0066] Extracting the marker features of the original image using a piano gesture feature extractor, wherein each convolution unit extracted by the piano gesture feature extractor is added with a skip connection unit;

[0067] A deconvolution operation is performed on the marking feature to obtain a target bare hand image with the tracking marking points removed.

[0068] The marker features in the original image are extracted through the above-mentioned tracking marker point removal network. The convolution units of the network share the weights of the previous marker estimation, and then these tracking marker point marks are removed in the deconvolution stage. A jump connection unit is added after each convolution unit to better maintain the rest of the image. Finally, the target bare hand image with the tracking marker points removed can be output.

[0069] After obtaining the target bare hand image with the tracking markers removed, the target tracking marker position data is matched with the target bare hand image, thereby constructing a gesture data set when the user plays the piano. For example, the frame number of the original image and the initial tracking marker position data can be recorded, and the frame number remains unchanged after the original image is processed to obtain the target bare hand image, and the frame number remains unchanged after the initial tracking marker position data is processed to obtain the target tracking marker position data. Therefore, the target tracking marker position data can be matched with the target bare hand image to obtain a high-precision gesture data set. Since the user's gestures are complex and dexterous when playing the piano, the gesture data set obtained by this application is used to train the relevant gesture recognition model, so that a gesture recognition model with higher recognition accuracy can be obtained.

[0070] Based on the same application concept, the embodiments of the present application also provide a device for constructing a high-precision piano gesture dataset based on a marker removal network corresponding to the method for constructing a high-precision piano gesture dataset based on a marker removal network provided in the above embodiments. Since the principle of solving the problem by the device in the embodiments of the present application is similar to the method for constructing a high-precision piano gesture dataset based on a marker removal network in the above embodiments of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0071] Figure 6 This is one of the functional module diagrams of a high-precision piano gesture data set construction device 200 based on a marker removal network provided in an embodiment of the present application. Figure 6 As shown, a high-precision piano gesture data set construction device 200 based on a marker removal network comprises a collection module 210, a preprocessing module 220, a removal module 230, and a matching module 240; wherein:

[0072] The acquisition module 210 is used to acquire the original image and initial tracking mark point position data corresponding to the hand gestures of the user playing the piano, wherein the user's hands are attached with tracking marks, the original image is data acquired by a camera device, and the initial tracking mark point position data is data acquired by the piano gesture capture system based on the tracking marks;

[0073] Preprocessing module 220: used to perform preprocessing operations on the initial tracking marker point position data to obtain target tracking marker point position data corresponding to the gesture action;

[0074] Removal module 230: used for inputting the original image into a pre-trained tracking marker removal network to obtain a target bare hand image with the tracking markers removed;

[0075] Matching module 240: used to match the target tracking mark point position data with the target bare hand image to construct a gesture data set of the user playing the piano.

[0076] In one embodiment, performing a preprocessing operation on the initial tracking marker point position data to obtain target tracking marker point position data corresponding to the gesture action includes:

[0077] Using a greedy matching algorithm to match the tracking mark points and the position data in the initial tracking mark point position data to obtain an initial marking result;

[0078] Performing an interpolation operation on the data of the missing tracking marking points in the initial marking result to obtain an initial marking result after the interpolation operation;

[0079] Determine whether there is defective data in the initial marking result after performing the interpolation operation;

[0080] If not, the initial marking result after the interpolation operation is performed is used as the target tracking mark point position data corresponding to the gesture action.

[0081] In one embodiment, the determining whether the initial marking result after the interpolation operation is performed has defective data further includes:

[0082] If there are defective data in the initial marking result after the interpolation operation is performed, the initial marking result is discarded, and the initial tracking mark point position data corresponding to the hand gesture of the user when playing the piano is re-collected.

[0083] In one embodiment, the training process of the tracking marker removal network includes:

[0084] Step A0: Obtain a bare hand sample image and a tracking marker sample data set;

[0085] Step A1: generating a synthetic image containing synthetic tracking markers based on the bare hand sample image and the tracking marker sample data set;

[0086] Step A2: inputting the synthetic tracking marker position data into a tracking marker removal network to generate a predicted bare hand image;

[0087] Step A3: calculating the loss value of the predicted bare hand image and the real bare hand image corresponding to the synthetic tracking marker point position data through a loss function;

[0088] Step A4: Determine whether the loss value is greater than a preset threshold;

[0089] If so, adjust the parameters of the tracking marker point removal network according to the loss value, use the predicted bare hand image as the bare hand sample image, and repeat steps A1 to A4 until the loss value is less than or equal to the preset threshold, thereby obtaining a trained tracking marker point removal network.

[0090] In one embodiment, generating a synthetic image containing synthetic tracking markers based on the bare hand sample image and the tracking marker sample data set includes:

[0091] Randomly selecting a preset number of tracking marker point sample images from the tracking marker point sample data set;

[0092] Using each of the tracking mark point sample images as a mark to be synthesized, and blending the mark to be synthesized into a target position of the bare hand sample image to obtain a blended image;

[0093] An image enhancement operation is performed on the mixed image to obtain synthetic tracking marker point position data containing synthetic markers.

[0094] In one embodiment, the step of using each of the tracking mark point sample images as a mark to be synthesized and blending the mark to be synthesized into a target position of the bare hand sample image to obtain a blended image includes:

[0095] Using a piano gesture feature extractor, extracting visual features of the bare hand sample image;

[0096] The marker to be synthesized is mixed into the visual feature to represent the position of the hand joint, and synthetic tracking marker point position data containing synthetic tracking marker points is generated.

[0097] In one embodiment, inputting the original image into a pre-trained tracking marker removal network to obtain a target bare hand image with the tracking markers removed includes:

[0098] Extracting the marker features of the original image using a piano gesture feature extractor, wherein each convolution unit extracted by the piano gesture feature extractor is added with a skip connection unit;

[0099] A deconvolution operation is performed on the marking feature to obtain a target bare hand image with the tracking marking points removed.

[0100] Based on the same application concept, see Figure 7As shown, it is a structural diagram of an electronic device 400 provided in an embodiment of the present application, including: a processor 410, a memory 420 and a bus 430, wherein the memory 420 stores machine-readable instructions executable by the processor 410, and when the electronic device 400 is running, the processor 410 communicates with the memory 420 through the bus 430, and the machine-readable instructions are executed by the processor 410 when running to perform the steps of the method for constructing a high-precision piano gesture dataset based on a marker removal network as described in any of the above embodiments.

[0101] Specifically, when the machine-readable instructions are executed by the processor 410, the following processing may be performed:

[0102] Collecting original images and initial tracking marker position data corresponding to the hand gestures of the user playing the piano, wherein the user's hands are attached with tracking markers, the original images are data collected by a camera device, and the initial tracking marker position data are data collected by a piano gesture capture system based on the tracking markers;

[0103] Performing a preprocessing operation on the initial tracking marker point position data to obtain target tracking marker point position data corresponding to the gesture action;

[0104] Inputting the original image into a pre-trained tracking marker removal network to obtain a target bare hand image without the tracking markers;

[0105] The target tracking mark point position data is matched with the target bare hand image to construct a gesture data set of the user playing the piano.

[0106] Based on the same application concept, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for constructing a high-precision piano gesture dataset based on a marker removal network provided in the above embodiment are executed.

[0107] Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is executed, the following steps can be performed:

[0108] Collecting original images and initial tracking marker position data corresponding to the hand gestures of the user playing the piano, wherein the user's hands are attached with tracking markers, the original images are data collected by a camera device, and the initial tracking marker position data are data collected by a piano gesture capture system based on the tracking markers;

[0109] Performing a preprocessing operation on the initial tracking marker point position data to obtain target tracking marker point position data corresponding to the gesture action;

[0110] Inputting the original image into a pre-trained tracking marker removal network to obtain a target bare hand image without the tracking markers;

[0111] The target tracking mark point position data is matched with the target bare hand image to construct a gesture data set of the user playing the piano.

[0112] In the embodiment of the present application, the computer program can also execute other machine-readable instructions when run by the processor to execute other methods described in the embodiment. For the specific execution method steps and principles, please refer to the description of the embodiment, which will not be repeated here.

[0113] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0114] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0115] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0116] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0117] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0118] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for constructing a high-precision piano gesture dataset based on a marker removal network, characterized in that: The method comprises: Collecting original images and initial tracking marker position data corresponding to the hand gestures of the user playing the piano, wherein the user's hands are attached with tracking markers, the original images are data collected by a camera device, and the initial tracking marker position data are data collected by a piano gesture capture system based on the tracking markers; Performing a preprocessing operation on the initial tracking marker point position data to obtain target tracking marker point position data corresponding to the gesture action; Inputting the original image into a pre-trained tracking marker removal network to obtain a target bare hand image without the tracking markers; Matching the target tracking mark point position data with the target bare hand image to construct a gesture data set of the user playing the piano; The training process of the tracking marker removal network includes: Step A0: Obtain a bare hand sample image and a tracking marker sample data set; Step A1: generating a synthetic image containing synthetic tracking markers based on the bare hand sample image and the tracking marker sample data set; Step A2: inputting the synthetic tracking marker position data into a tracking marker removal network to generate a predicted bare hand image; Step A3: calculating the loss value of the predicted bare hand image and the real bare hand image corresponding to the synthetic tracking marker point position data through a loss function; Step A4: Determine whether the loss value is greater than a preset threshold; If so, adjust the parameters of the tracking marker point removal network according to the loss value, use the predicted bare hand image as the bare hand sample image, and repeat steps A1 to A4 until the loss value is less than or equal to the preset threshold, thereby obtaining a trained tracking marker point removal network.

2. The method for constructing a high-precision piano gesture dataset based on a marker removal network according to claim 1, characterized in that: The performing of a preprocessing operation on the initial tracking marker point position data to obtain target tracking marker point position data corresponding to the gesture action includes: Using a greedy matching algorithm to match the tracking mark points and the position data in the initial tracking mark point position data to obtain an initial marking result; Performing an interpolation operation on the data of the missing tracking marking points in the initial marking result to obtain an initial marking result after the interpolation operation; Determine whether there is defective data in the initial marking result after performing the interpolation operation; If not, the initial marking result after the interpolation operation is performed is used as the target tracking mark point position data corresponding to the gesture action.

3. The method for constructing a high-precision piano gesture dataset based on a marker removal network according to claim 2, characterized in that: The determining whether the initial marking result after the interpolation operation is performed has defect data also includes: If there are defective data in the initial marking result after the interpolation operation is performed, the initial marking result is discarded, and the initial tracking mark point position data corresponding to the hand gesture of the user when playing the piano is re-collected.

4. The method for constructing a high-precision piano gesture dataset based on a marker removal network according to claim 1, characterized in that: The generating a synthetic image containing synthetic tracking markers based on the bare hand sample image and the tracking marker sample data set includes: Randomly selecting a preset number of tracking marker point sample images from the tracking marker point sample data set; Using each of the tracking mark point sample images as a mark to be synthesized, and blending the mark to be synthesized into a target position of the bare hand sample image to obtain a blended image; An image enhancement operation is performed on the mixed image to obtain synthetic tracking marker point position data containing synthetic markers.

5. The method for constructing a high-precision piano gesture dataset based on a marker removal network according to claim 4, characterized in that: The step of taking each of the tracking mark point sample images as a mark to be synthesized, and mixing the mark to be synthesized to a target position of the bare hand sample image to obtain a mixed image includes: Using a piano gesture feature extractor, extracting visual features of the bare hand sample image; The marker to be synthesized is mixed into the visual feature to represent the position of the hand joint, and synthetic tracking marker point position data containing synthetic tracking marker points is generated.

6. A high-precision piano gesture dataset construction device based on a marker removal network, characterized in that: The device comprises: The acquisition module is used to acquire the original image and initial tracking mark point position data corresponding to the hand gestures of the user playing the piano, wherein the user's hands are attached with tracking marks, the original image is data acquired by a camera device, and the initial tracking mark point position data is data acquired by the piano gesture capture system based on the tracking marks; Preprocessing module: used to perform preprocessing operation on the initial tracking mark point position data to obtain target tracking mark point position data corresponding to the gesture action; Removal module: used for inputting the original image into a pre-trained tracking marker removal network to obtain a target bare hand image with the tracking markers removed; Matching module: used to match the target tracking mark point position data with the target bare hand image to construct a gesture data set when the user plays the piano; The training process of the tracking marker removal network includes: Step A0: Obtain a bare hand sample image and a tracking marker sample data set; Step A1: generating a synthetic image containing synthetic tracking markers based on the bare hand sample image and the tracking marker sample data set; Step A2: inputting the synthetic tracking marker position data into a tracking marker removal network to generate a predicted bare hand image; Step A3: calculating the loss value of the predicted bare hand image and the real bare hand image corresponding to the synthetic tracking marker point position data through a loss function; Step A4: Determine whether the loss value is greater than a preset threshold; If so, adjust the parameters of the tracking marker point removal network according to the loss value, use the predicted bare hand image as the bare hand sample image, and repeat steps A1 to A4 until the loss value is less than or equal to the preset threshold, thereby obtaining a trained tracking marker point removal network.

7. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate through the bus. When the processor is running, the machine-readable instructions execute the steps of the method for constructing a high-precision piano gesture dataset based on a marker removal network as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for constructing a high-precision piano gesture dataset based on a marker removal network as described in any one of claims 1 to 5 are performed.

Images