Method and device for capturing playing motion of stringed instrument based on audio and video, and electronic equipment

Abstract

The invention provides a string instrument playing motion capturing method and device based on audio and video, and electronic equipment. The method comprises the following steps: acquiring a string instrument playing video and a string instrument playing audio; acquiring a first fusion feature based on the stringed instrument playing video; obtaining a second fusion feature based on the stringed instrument playing audio; based on a pre-trained motion capture model, according to the first fusion feature and the second fusion feature, obtaining a playing motion capture result of the stringed instrument; wherein the motion capture model is obtained by training and optimizing a theoretical playing position loss function and a finger pressing force loss function. According to the method, on the premise that no optical mark or sensor is introduced, fine capturing of stringed instrument playing is achieved based on stringed instrument playing videos and stringed instrument playing audios, the result focuses on highlighting accurate hand actions, meanwhile, the stringed instrument and key interactive contact are captured, and the playing effect of the stringed instrument is improved. And high-precision modeling of the whole body skeleton, the musical instrument position and the hand fine playing action of the player in the playing process is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of string playing, and particularly relates to a stringed instrument playing motion capture method and device based on audio and video and electronic equipment. BACKGROUND

[0002] Motion capture technology, as an important research direction in the field of computer vision and human-computer interaction, has been widely applied in many fields such as film and television animation production, virtual reality, motion analysis, digital art performance, etc. According to whether it depends on external markers, the existing motion capture methods are generally divided into two categories: marker-based motion capture and markerless motion capture.

[0003] In the instrument playing scene, the existing technology generally adopts a marker-based method for motion capture. Specifically, this kind of method deploys optical marker points on the instrument body and the performer's limbs (especially the hands), and realizes high-precision motion acquisition by means of infrared sensors, electromagnetic systems or inertial measurement units and other devices. This kind of scheme has been applied in many playing forms such as guitar, piano, string quartet, etc., and can better restore the overall motion trajectory of the performer.

[0004] However, the marker-based motion capture method has many limitations. First of all, it has high requirements for the capture environment, and usually needs to be carried out under specific magnetic field, infrared field or lighting conditions, which limits its applicability in daily or unstructured environment. Secondly, since the hand movements during playing are very fine and frequent, the installation of optical markers or wearable sensors on the hands often causes interference to the playing behavior itself, and even affects the playing quality, resulting in the inability to truly restore the motion details in the natural playing state. Therefore, most current researches mainly focus on the capture of large-scale limb movements of the performer, while the fine motion recording of the hands, which is a key part of music generation, is still relatively scarce.

[0005] In recent years, with the development of deep learning and computer vision technology, human pose estimation models based on RGB cameras have rapidly emerged and made significant progress. This kind of method does not require any markers or wearable devices, and can estimate the human skeleton structure only through ordinary cameras, with the advantages of low cost, convenient deployment, little interference to users, etc. In the tasks of general motion recognition and human pose estimation, existing models have achieved high spatial restoration accuracy and robustness, providing a new technical path for motion data non-invasive collection.

[0006] However, existing human pose estimation models still have obvious deficiencies in this specific subdivision scene of instrument playing. On the one hand, most of the existing models are not specifically optimized for the instrument playing task, and lack the ability to effectively model the complex interaction between the hand and the instrument. On the other hand, the hand movements in this scene are highly detailed, and small deviations in action can cause significant changes in the sound of the instrument. In addition, the contact and occlusion problem between the performer and the instrument makes it difficult for existing methods to accurately capture key action information during the performance.

[0007] Furthermore, there is still a lack of high-quality training dataset for the instrument playing scene in the currently disclosed technical solutions, which also limits the generalization ability and performance improvement of existing human pose estimation models in this field.

[0008] As can be seen, neither the traditional method based on labeling nor the visual-based unlabeled method can achieve efficient capture of complete performance action sequences including fine hand movements and accurate recording of key interaction information between the performer and the instrument without disturbing the performer.

[0009] Therefore, how to solve the problem that the existing technology cannot effectively capture complete performance action sequences including fine hand movements without disturbing the performer is an important topic in the field of string instrument playing that needs to be solved. SUMMARY

[0010] The present application provides a string instrument playing action capture method and device based on audio and video, which overcomes the defect that the existing technology cannot effectively capture complete performance action sequences including fine hand movements without disturbing the performer, and can accurately capture fine hand movements and their interaction information with the instrument during the performance without affecting the performance behavior. On the one hand, the present application provides a string instrument playing action capture method based on audio and video, comprising: obtaining a string instrument playing video and a string instrument playing audio; obtaining a first fusion feature based on the string instrument playing video; obtaining a second fusion feature based on the string instrument playing audio; obtaining a string instrument playing action capture result based on a pre-trained action capture model according to the first fusion feature and the second fusion feature; wherein the action capture model is obtained by training and optimizing a theoretical performance position loss function and a finger pressing force loss function.

[0011] Further, the first fusion feature is obtained based on the stringed instrument performance video, including: performing human body detection on video frames in the stringed instrument performance video to obtain human body bounding box features of a performer; outputting full-body key point features of the performer according to the stringed instrument performance video based on a pre-trained time sequence human body posture estimation network; extracting visual semantic features of the video frames in the stringed instrument performance video, the visual semantic features including local region texture, posture context and a space background in which the performer is located; generating a global depth map based on a pre-trained monocular depth estimation model according to the stringed instrument performance video; and fusing the human body bounding box features, the full-body key point features, the visual semantic features and the global depth map to obtain the first fusion feature.

[0012] Further, the second fusion feature is obtained based on the stringed instrument performance audio, including: extracting mel-spectrogram features, audio semantic features and fundamental frequency features in the stringed instrument performance audio; and fusing the mel-spectrogram features, the audio semantic features and the fundamental frequency features to obtain the second fusion feature.

[0013] Further, the theoretical performance position loss function is defined as follows: ; wherein, represents a fingertip spatial coordinate of a target fingertip, , respectively represent a distal fingertip joint angle and a proximal fingertip joint angle of the target fingertip, represents a theoretical performance position.

[0014] Further, the finger pressing force loss function is defined as follows: ; wherein, represents a time frame of an actually performed string, represents a set of time frames of actually performed strings, represents a finger flexion angle threshold, represents a flexion angle of a finger at a time frame .

[0015] Further, the theoretical performance position is calculated, specifically including: extracting fundamental frequency features in the stringed instrument performance audio; and predicting a theoretical performance string based on a pre-trained audio-string prediction model according to the fundamental frequency features, wherein the audio-string prediction model is obtained by training according to audio feature samples and corresponding real string labels; and determining an actual performance position according to the theoretical performance string and a stringed instrument posture.

[0016] ​Further, the stringed instrument posture is determined, specifically comprising: performing two-dimensional capture tracking on a stringed instrument key point according to the stringed instrument performance video to obtain two-dimensional coordinates of the stringed instrument key point; and determining the stringed instrument posture under a video frame according to the two-dimensional coordinates of the stringed instrument key point and a global depth map corresponding to the video frame in the stringed instrument performance video.

[0017] In a second aspect, the present application further provides a stringed instrument performance motion capture device based on audio and video, comprising: an audio and video acquisition module configured to acquire a stringed instrument performance video and a stringed instrument performance audio; a first fusion feature acquisition module configured to acquire a first fusion feature based on the stringed instrument performance video; a second fusion feature acquisition module configured to acquire a second fusion feature based on the stringed instrument performance audio; and a stringed instrument motion capture module configured to acquire a stringed instrument performance motion capture result based on a pre-trained motion capture model according to the first fusion feature and the second fusion feature; wherein the motion capture model is obtained by training and optimization of a theoretical performance position loss function and a finger pressing force loss function.

[0018] In a third aspect, the present application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the stringed instrument performance motion capture method based on audio and video as described above.

[0019] In a fourth aspect, the present application further provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the stringed instrument performance motion capture method based on audio and video as described above.

[0020] The stringed instrument performance motion capture method based on audio and video provided by the present application acquires a stringed instrument performance video and a stringed instrument performance audio, acquires a first fusion feature based on the stringed instrument performance video, acquires a second fusion feature based on the stringed instrument performance audio, and then acquires a stringed instrument performance motion capture result based on a pre-trained motion capture model according to the first fusion feature and the second fusion feature; wherein the motion capture model is obtained by training and optimization of a theoretical performance position loss function and a finger pressing force loss function. This method realizes fine capture of stringed instrument performance based on the stringed instrument performance video combined with the stringed instrument performance audio without any optical markers or sensors, accurately restores the motion dynamics of the performer's whole body during the performance process, especially focuses on the fine motion of the performer's hands and the spatial interaction between the hands and the stringed instrument, and realizes high-precision modeling of the performer's whole body skeleton, the position of the instrument, and the fine performance motion of the hands during the performance process. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to make the technical solutions in the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0022] Figure 1 is a flowchart of a stringed instrument performance motion capture method based on audio and video provided by an embodiment of the present application.

[0023] Figure 2 is a schematic diagram of the overall flow of a stringed instrument performance motion capture method based on audio and video provided by an embodiment of the present application.

[0024] Figure 3 is a structural schematic diagram of a stringed instrument performance motion capture device provided by an embodiment of the present application.

[0025] Figure 4 is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the technical solutions in the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0027] It should be noted that, whether using markers or sensors or through a human pose estimation model, there is currently no effective method to capture fine performance motions of the hands and record key interaction information between the performer and the instrument without interfering with the performer, which seriously limits the development of motion capture technology in the field of instrument performance, making it difficult to explore new modes of instrument teaching, assist related film and television animation creation, and realize virtual digital person concerts and other wide-ranging applications.

[0028] In view of this, an embodiment of the present application provides a stringed instrument performance motion capture method based on audio and video, specifically, Figure 1 is a flowchart of a stringed instrument performance motion capture method based on audio and video provided by an embodiment of the present application.

[0029] As Figure 1As shown, the method comprises: S110, obtaining a stringed instrument performance video and a stringed instrument performance audio; S120, obtaining a first fusion feature based on the stringed instrument performance video; S130, obtaining a second fusion feature based on the stringed instrument performance audio; S140, obtaining a stringed instrument performance motion capture result according to the first fusion feature and the second fusion feature based on a pre-trained motion capture model; wherein the motion capture model is obtained by training and optimizing a theoretical performance position loss function and a finger pressing force loss function.

[0030] It is worth mentioning that in the instrument performance motion capture task, especially in the scene of stringed instruments and the like requiring high-precision finger modeling, the traditional motion capture method based on markers or sensors has the problems of high interference and high cost, and cannot realize non-invasive collection in a natural environment. As a direct product of hand movements, audio naturally contains a large amount of implicit motion features, especially when visual information is incomplete, it can serve as a powerful supplementary clue. For example, some performance actions only show slight finger movements in vision, but are accompanied by obvious sound events in audio. The temporal consistency and causal relationship between these modalities provide valuable redundant information for action restoration.

[0031] The steps S110-S140 and related steps will be described in detail below.

[0032] S110, obtaining a stringed instrument performance video and a stringed instrument performance audio.

[0033] It is easy to understand that the stringed instrument performance video and the stringed instrument performance audio in this step can be obtained by live recording or obtained from public channels.

[0034] If the stringed instrument performance video and the stringed instrument performance audio are obtained from public channels, public datasets such as URMP (University of Rochester Multi-Modal Music Performance Dataset) and GuitarSet can be used, and online platforms such as YouTube can be used. Among them, URMP is a multi-modal music performance dataset containing performance videos and audios of multiple instruments, which is suitable for the application scenario of the present embodiment. GuitarSet is a dataset specifically for guitar performance, which contains high-quality performance videos and corresponding MIDI (Musical Instrument Digital Interface) files. MIDI file is a standard format for storing music data.

[0035] If the stringed instrument performance video and the stringed instrument performance audio are obtained by live recording, a monocular camera can be used to record video and audio synchronously to ensure that the timestamps of the two are consistent.

[0036] wherein the string instrument performance video is used to provide the body posture changes of the performer over time, and the string instrument performance audio is used to provide musical semantic information such as pitch, rhythm, etc. during performance On the basis of obtaining the string instrument performance video and the string instrument performance audio in step S110, further, steps S120-S130 are performed.

[0037] S120, obtaining a first fusion feature based on the string instrument performance video.

[0038] The first fusion feature includes a human body bounding box feature, a full-body key point feature, a visual semantic feature, and a global depth map of the performer.

[0039] The first human body bounding box feature can be obtained by YOLOv8, which is used to preliminarily define the range of the performer's body in the video frames of the string instrument performance video.

[0040] The full-body key point feature can be extracted based on a human body 2D skeleton estimation network, including but not limited to head, neck, limbs, and hand key points of the performer in each video frame of the string instrument performance video.

[0041] The visual semantic feature can be obtained using a Vit model, including deep visual semantic features of the video frames as a whole in the string instrument performance video, such as local area texture, posture context, and the space background in which the performer is located, etc., which is used to capture unstructured information.

[0042] The global depth map can be generated by a pre-trained monocular depth estimation model, including the relative distance between each pixel point of the video frames in the string instrument performance video and the camera, which can provide a supplement of spatial structure information without the aid of multiple cameras or RGB-D sensors.

[0043] By fusing the human body bounding box feature, the full-body key point feature, the visual semantic feature, and the global depth map of the performer, the first fusion feature can be obtained. The fusion algorithm actually used can be set according to the actual situation, for example, direct splicing, which will not be described here.

[0044] S130, obtaining a second fusion feature based on the string instrument performance audio; The second fusion feature includes a Mel spectrum feature (Mel feature), an audio semantic feature, and a fundamental frequency feature in the string instrument performance audio.

[0045] The Mel feature is used to capture the sound energy and frequency distribution in the string instrument performance audio, which can be obtained by converting the audio into a Mel spectrum through short-time Fourier transform and Mel filter.

[0046] The audio semantic feature is used to reflect the rhythm, the contrast between strength and weakness, and the style change of the string instrument performance audio. The original string instrument performance audio can be mapped to a high-dimensional vector representation based on the semantic embedding module of a large-scale audio pre-training model to obtain the audio semantic feature.

[0047] The fundamental frequency feature is used to infer the high and low of the played notes and the actual playing area in the string instrument performance audio. The fundamental frequency feature can be obtained by extracting the pitch of the string instrument performance audio through the CREPE algorithm.

[0048] The second fusion feature can be obtained by fusing the Mel-spectral feature, the audio semantic feature, and the fundamental frequency feature in the string instrument performance audio.

[0049] It is worth mentioning that the Mel-spectral feature, the audio semantic feature, and the fundamental frequency feature in the string instrument performance audio correspond to different music dimensions of the performance, and can provide constraints for action modeling from multiple angles.

[0050] After step S120 of obtaining the first fusion feature based on the string instrument performance video and step S130 of obtaining the second fusion feature based on the string instrument performance audio, step S140 is performed.

[0051] S140, based on a pre-trained motion capture model, obtaining a string instrument performance motion capture result according to the first fusion feature and the second fusion feature; wherein the motion capture model is obtained by training and optimizing a theoretical playing position loss function and a finger pressing force loss function.

[0052] Specifically, the embodiment has a pre-trained motion capture model. The first fusion feature and the second fusion feature obtained are input into the pre-trained motion capture model, and the output string instrument performance motion capture result is obtained. The string instrument performance motion capture result here includes but is not limited to joint posture parameters, body shape parameters, global displacement parameters, and global rotation parameters of the performer.

[0053] The joint posture parameters include local joint rotation angles, such as the three-dimensional hand joint rotation angles of the performer, which are represented in the form of 6D. The body shape parameters include parameters reflecting the body shape characteristics. The global displacement parameters describe the position change of the performer's body in the world coordinates. The global rotation parameters represent the orientation adjustment of the performer.

[0054] It is worth mentioning that in the process of training the motion capture model, two specially designed loss functions, namely the theoretical playing position loss function and the finger pressing force loss function, are creatively introduced. Among them, the theoretical playing position loss function is used to guide the model to learn to generate more realistic actions, and the finger pressing force loss function is used to guide the model to learn how to generate real hand actions with natural pressing force when playing is needed.

[0055] It should be noted that the specific optimization process of the motion capture model will be expanded in the following embodiments.

[0056] The stringed instrument mentioned in the present embodiment includes but is not limited to bowed stringed instruments and plucked stringed instruments, wherein the bowed stringed instruments are, for example, violin, viola, cello, double bass and erhu, and the plucked stringed instruments are, for example, guitar, harp and pipa.

[0057] In the present embodiment, by obtaining a stringed instrument playing video and a stringed instrument playing audio, a first fusion feature is obtained based on the stringed instrument playing video, and a second fusion feature is obtained based on the stringed instrument playing audio, and then a stringed instrument playing motion capture result is obtained based on a pre-trained motion capture model according to the first fusion feature and the second fusion feature; wherein the motion capture model is obtained by training and optimization through a theoretical playing position loss function and a finger pressing force loss function. This method realizes the fine capture of stringed instrument playing based on the stringed instrument playing video combined with the stringed instrument playing audio without introducing any optical markers or sensors, and the result highlights accurate hand actions, while capturing the stringed instrument and key interactive contacts, realizing high-precision modeling of the performer's full-body skeleton, instrument position and fine playing actions during playing.

[0058] On the basis of the above-mentioned embodiments, further, the acquisition process of the first fusion feature and the second fusion feature will be described in detail as follows.

[0059] In a specific embodiment, based on the stringed instrument playing video, the first fusion feature is obtained, including: performing human body detection on the video frames in the stringed instrument playing video to obtain human body bounding box features of the performer; outputting full-body key point features of the performer according to the stringed instrument playing video based on a pre-trained time sequence human body pose estimation network; extracting visual semantic features of the video frames in the stringed instrument playing video, the visual semantic features including local area texture, posture context and the space background where the performer is located; generating a global depth map based on a pre-trained monocular depth estimation model according to the stringed instrument playing video; and fusing the human body bounding box features, the full-body key point features, the visual semantic features and the global depth map to obtain the first fusion feature.

[0060] Specifically, for video frames in a string instrument performance video, first pass through the image processing and feature extraction module, which uses the YOLOv8 model to obtain human bounding box features, preliminarily defines the range of the human body in the image, and uses a human 2D skeleton estimation network (such as OpenPose, AlphaPose) to detect the full-body key point features of the performer in each video frame, including the positions of the head, torso, limbs, and hand key points.

[0061] At the same time, the ViT (Vision Transformer) model is used to obtain deep visual semantic features of the entire video frame, including local region texture, posture context, and the spatial background in which the person is located, etc., for capturing unstructured information. To make up for the missing spatial depth information in monocular string instrument performance videos, the pre-trained monocular depth estimation model DepthAnythingV2 is introduced to generate a corresponding global depth map for each video frame. The depth map represents the relative distance between each pixel point in the video frame and the camera, and can provide a supplement of spatial structure information without the aid of multiple cameras or RGB-D sensors.

[0062] Among them, YOLOv8 is the latest iteration of a series of very popular YOLO (You Only Look Once) models in the field of target detection. YOLOv8 has the following advantages: (1) higher precision and speed balance: YOLOv8 inherits and optimizes the advantages of YOLOv7, further improves the recognition accuracy of the model in complex scenes, while maintaining high efficient inference speed; (2) enhanced backbone network architecture: YOLOv8 uses more advanced deep learning architectures, such as improved CSPNet (Cross Stage Partial Network), EfficientNet, etc., to improve feature extraction capability; (3) multi-scale detection strategy: YOLOv8 enhances the detection ability of objects of different sizes by introducing more FPN (Feature Pyramid Network) layers or other similar technologies.

[0063] The ViT model is a model that applies the successful Transformer architecture in the natural language processing field to computer vision tasks. It processes image blocks directly instead of traditional convolution operations for feature extraction and classification tasks, and demonstrates its strong performance in various visual tasks.

[0064] DepthAnythingV2 is a general depth estimation framework that supports estimating the relative depth map of a scene from any continuous frame image.

[0065] After obtaining the human body bounding box feature, the whole body key point feature, the visual semantic feature and the global depth map of the performer, the first fusion feature can be obtained by fusing these features.

[0066] In another specific embodiment, based on the string instrument performance audio, the second fusion feature is obtained, including: extracting the mel spectrum feature, the audio semantic feature and the fundamental frequency feature in the string instrument performance audio; fusing the mel spectrum feature, the audio semantic feature and the fundamental frequency feature to obtain the second fusion feature.

[0067] Specifically, by processing the string instrument performance audio through three parallel different feature extraction modules, the corresponding mel spectrum feature, audio semantic feature and fundamental frequency feature can be obtained.

[0068] In detail, firstly, the string instrument performance audio is converted into a Mel spectrum by short-time Fourier transform and a Mel filter bank, capturing the sound energy and frequency distribution; secondly, the original string instrument performance audio is mapped into a high-dimensional vector representation by a semantic embedding module based on a large-scale audio pre-training model (such as MERT), which can reflect the rhythm, contrast, style change and other deep semantic features of the music segment; thirdly, the audio pitch corresponding to each frame of the string instrument performance video in the string instrument performance audio is extracted by the CREPE algorithm, which is used to infer the high and low of the current performance note and the actual performance area. These audio features correspond to different music dimensions in performance and can provide constraints for action modeling from multiple angles.

[0069] On the basis of the above embodiments, further, the training and optimization process of the motion capture model will be described in detail.

[0070] It is easy to understand that, in training the motion capture model, the first fusion feature sample obtained based on the string instrument performance video sample and the second fusion feature sample obtained based on the string instrument performance audio sample are taken as input, the predicted string instrument performance motion capture result is taken as output, and the motion capture model is iteratively optimized by the theoretical performance position loss function and the finger pressing force loss function until convergence, obtaining the trained motion capture model.

[0071] In one specific embodiment, the calculation process of the theoretical position loss function is described in detail.

[0072] First, the string instrument posture needs to be determined, specifically including: according to the string instrument performance video, performing two-dimensional capture tracking on the string instrument key points to obtain the two-dimensional coordinates of the string instrument key points; according to the two-dimensional coordinates of the string instrument key points and the global depth map corresponding to the video frame in the string instrument performance video, determining the string instrument posture under the video frame.

[0073] Specifically, the embodiment trains an InstrumentCap model in combination with a plurality of existing public data sets and self-collected stringed instrument data under a stringed instrument playing task. The InstrumentCap model can accurately detect the two-dimensional coordinates of the key points of the stringed instrument from a single frame image in the stringed instrument playing video according to the geometric structure of the stringed instrument. The InstrumentCap model uses the part annotation data of the stringed instrument during training, combines the image with the structure prior, and effectively improves the robust recognition ability of the stringed instrument under occlusion and motion state.

[0074] Subsequently, according to the fusion of the two-dimensional coordinates of the key points of the stringed instrument output by the InstrumentCap model and the global depth map corresponding to the video frame in the stringed instrument playing video, the three-dimensional posture of the stringed instrument under the current video frame can be further recovered. The stringed instrument playing posture is not only used to judge the spatial relationship between the hand position and the stringed instrument, but also provides a geometric basis for the determination of the subsequent theoretical playing position.

[0075] After determining the three-dimensional posture of the stringed instrument, the theoretical playing position is calculated, specifically including: extracting the fundamental frequency features in the stringed instrument playing audio; based on a pre-trained audio-string prediction model, predicting the theoretical playing string according to the fundamental frequency features; wherein the audio-string prediction model is obtained by training according to the audio feature sample and the corresponding real string label; determining the theoretical playing position according to the theoretical playing string and the stringed instrument posture.

[0076] It is easy to understand that the MERT (Multimodal Emotion Representation from Text and speech) model can be used as an audio feature extractor to extract time-aligned embedding sequences from the stringed instrument playing audio, including pitch, rhythm, dynamics and other music attributes, to obtain audio features. The MERT is a multimodal language-speech joint representation model based on self-supervised learning, which is specially used to model the cross-modal semantic and emotional consistency between speech and text. The MERT combines audio signals and corresponding text content for alignment in a unified representation space.

[0077] Then, the embodiment trains an audio-string prediction model, the input of which is the audio feature at the current time, and the output is the string index of the current theoretical playing, that is, the theoretical playing string. The audio-string prediction model uses the Transformer structure to model the time sequence mapping between the audio and the string, and is trained by matching with the actual string annotation (real string label) in a supervised learning manner, to ensure that the predicted string index result is highly consistent with the real playing string.

[0078] After obtaining the theoretical playing string, the pitch of the current playing frame can be detected by the fundamental frequency detection algorithm CREPE. Then, by using the relationship between the sound pitch, the string fundamental frequency and the actual string vibration length in string instrument playing, the theoretical playing position can be determined. Specifically, the relationship between the pitch, the string fundamental frequency and the string vibration length is as follows in formula (1).

[0079] (1).

[0080] In formula (1), represents the pitch information detected by the fundamental frequency detection algorithm, represents the fundamental frequency of the target string (i.e. the theoretical playing string), represents the full length of the target string, represents the actual string vibration length. Wherein, the fundamental frequency After obtaining the theoretical playing string, the pitch of the current playing frame can be detected by the fundamental frequency detection algorithm CREPE. Then, by using the relationship between the sound pitch, the string fundamental frequency and the actual string vibration length in string instrument playing, the theoretical playing position can be determined. Specifically, the relationship between the pitch, the string fundamental frequency and the string vibration length is as follows in formula (1). , is a known quantity, and the actual string vibration length is solved.

[0081] After obtaining the actual string vibration length, in combination with the previously predicted string instrument posture, the theoretical playing position of the finger on the string, i.e. the spatial position corresponding to the actual sound position on the string, can be inferred.

[0082] Finally, according to the theoretical playing position loss function set according to the playing position, the fingertip with the minimum distance to the theoretical playing position is taken as the optimization target (target fingertip), and the finger joint angle is adjusted by the inverse dynamics method, so as to realize the minimum spatial Euclidean distance between the target fingertip and the theoretical playing position.

[0083] The theoretical playing position loss function set is as follows in formula (2).

[0084] (2).

[0085] In formula (2), represents the spatial coordinates of the fingertip of the target fingertip, , represents the initial hand posture, including the distal fingertip joint angle and the proximal fingertip joint angle of the target fingertip, represents the theoretical playing position.

[0086] It is worth mentioning that by the theoretical playing position loss function, the key sound production interaction between the performer and the string instrument in the real situation can be restored, and the accuracy of the hand motion capture can be significantly improved.

[0087] Based on the above, the corresponding theoretical performance position loss function value can be calculated at each training round of the motion capture model.

[0088] In another specific embodiment, the calculation process of the finger pressing force loss function is described in detail.

[0089] The finger pressing force loss function designed in this embodiment is used as an additional supervision signal in the training stage to guide the network to learn to generate a bent finger posture at the moment when the string should be pressed, thereby enhancing the physical reasonableness of the generated motion. Specifically, during the process of stringed instrument performance, the string pressing behavior of the left hand fingers not only depends on their position in space, but more importantly, whether they have enough pressing force, that is, whether the fingers are actually pressed on the fingerboard or string. This action is usually manifested as a change in the flexion angle of the first to third knuckles of the fingers. When the fingers press the string with force, the relevant joints will exhibit a significant flexion state, especially the turning angle value of the distal interphalangeal joint will increase significantly.

[0090] Based on this phenomenon, in the hand modeling process of this embodiment, a mapping modeling mechanism between the flexion angle of the finger joints and the pressing force is introduced. Specifically, the joint rotation angle of the left hand pressing finger in each video frame is first extracted from the stringed instrument performance motion capture result prediction, especially the joint angle in the bending direction, which is projected as a continuous quantity representing the "flexion degree" of the finger. Subsequently, according to the theoretical performance position of the current frame, it is judged whether the frame is at the moment when the string pressing behavior should occur. Once it is judged as "should press the string frame", the flexion angle of the corresponding finger in the frame is compared with an empirical physiological threshold value (representing the range of joint angles that can achieve effective string pressing), and if it is insufficient, it is determined as "insufficient pressing force".

[0091] In detail, the finger pressing force loss function defined as follows formula (3) is set.

[0092] (3).

[0093] In formula (3), represents the time frame of actually playing the string, represents the set of time frames of actually playing the string, represents the finger flexion angle threshold, represents the flexion angle of the finger at the time frame .

[0094] According to the finger pressing force loss function, when the actual flexion angle of the finger is lower than the finger flexion angle threshold, the loss function is activated and the video frame is punished. By training the loss through back propagation, the system can gradually learn how to generate a real hand motion with natural pressing force when performance is required.

[0095] In the embodiment, the theoretical playing position loss function and the finger pressing force loss function complement each other, the former ensures that the hand position is correctly on the string to be played, and the latter further ensures that the finger on the position has sufficient biomechanical state, so that the "reasonable" gesture further approaches the effective action "capable of sounding" in real playing, thereby significantly enhancing the credibility of the generated action in the two dimensions of physical consistency and playing executability.

[0096] During the whole training process of the motion capture model, an end-to-end optimization strategy is adopted, and multiple loss functions are introduced, including a pose regression loss based on SMPL-X parameters, an L2 distance loss between the predicted motion and the GT reconstruction, a time smoothing loss of motion continuity, and the above-mentioned theoretical playing position loss and finger pressing force loss. These loss terms work together to not only enable the model to output structurally reasonable and temporally coherent motion, but also ensure the consistency between the motion and the playing semantics and the reasonableness of the biomechanical state during playing. The motion capture model provided in the embodiment not only realizes end-to-end multi-modal modeling in form, but also essentially incorporates "realistic and feasible motion generation" into the learning goal, so that the finally generated motion is visually credible, physically reasonable, musically synchronized, and has high practical value. In some embodiments, Figure 2 The overall flowchart of the stringed instrument playing motion capture method provided by the embodiment of the application is shown.

[0097] As shown in Figure 2 , it mainly includes a monocular video (stringed instrument playing video) branch and an audio (stringed instrument playing audio) branch. In the monocular video branch, human body bounding box features, full-body key point features, visual semantic features, and a global depth map are obtained, and by fusing these features, a first fused feature can be obtained. In the audio branch, Mel features (Mel spectrum features), Mert features (audio semantic features), and fundamental frequency features are obtained, and by fusing these features, a second fused feature can be obtained.

[0098] Subsequently, the first fused feature obtained by the monocular video branch and the second fused feature obtained by the audio branch are sequentially spliced into a multi-modal feature sequence, which is input into a pre-trained motion capture model (i.e. Figure 2The Transformer structure utilizes a self-attention mechanism to capture long-term dependencies of action changes in the time dimension and learn interaction relationships between different modalities in the modal dimension, especially the coordination rules between audio rhythm and movement rhythm. These automatically learned cross-modal connections help to enhance the semantic consistency and physical rationality of the system, and thus obtain the string instrument motion capture results of the output, i.e., the local posture, body shape, global rotation and global displacement of the performer.

[0099] According to Figure 2 It can also be seen that the action capture model uses a theoretical performance position loss function and a finger pressing force loss function. Figure 2 The finger pressing force loss function is not shown in

[0100] It should be noted that the technical details involved in the embodiments are described in detail in the above embodiments, and will not be repeated here.

[0101] In some embodiments, the beneficial technical effects brought by the audio and video based string instrument performance motion capture method provided by the embodiments are described.

[0102] First, the audio and video based string instrument performance motion capture method provided by the embodiments does not rely on any external marker device, depth camera or inertial sensor, and only needs to input monocular video and synchronous audio to realize high-precision restoration of the whole body three-dimensional action of the string performer. Compared with traditional optical motion capture or wearable systems, the present application greatly reduces the hardware cost and use threshold, has good deployment flexibility and environmental adaptability, and is suitable for natural performance environments such as actual stage, teaching classroom or home practice.

[0103] Second, the InstrumentCap model specially designed by the present application is used to accurately estimate the two-dimensional structural key points of the instrument from the monocular image, and the pre-trained depth estimation model is further used to infer the three-dimensional instrument posture, providing a structural constraint basis for the subsequent action capture model. This modeling strategy effectively solves the problem of missing or inaccurate instrument information in traditional methods, and enhances the modeling ability of the system for the "human-instrument" interaction relationship in the performance action.

[0104] Third, the present application fully integrates audio modal information, uses the Mel spectrum, semantic embedding and fundamental frequency features of audio to guide the action generation process, and designs a theoretical performance position loss mechanism. By introducing the relationship between pitch and fret physical position into the theoretical performance position loss function, the physical connection between sound and action is established, thereby improving the consistency and naturalness of the generated action and the actual music expression.

[0105] Fourthly, the present application also innovatively introduces finger pressing force consistency loss, by analyzing the flexion angle of the finger joint, the force state of the finger in the playing process is constrained to be consistent with the actual playing behavior, and the physical feasibility and simulation accuracy of the generated result are further enhanced from the action physiology.

[0106] Fifthly, the action capture model as a whole adopts an end-to-end training strategy, fully coincides with the audio modal information, all modules are optimized in cooperation, supports multi-task joint learning, makes the system not only generate a whole body three-dimensional action with reasonable structure, but also retain the continuity and semantic accuracy of the hand details, has good generalization ability and practical popularization value.

[0107] In summary, the present application has innovation and significant improvement effect in equipment simplification, physical modeling, audio-visual fusion, precision improvement and other aspects, can be widely applied to music education, virtual performance generation, action analysis, digital human performance and other scenes, has good industrial application prospect and social benefits.

[0108] Corresponding to the stringed instrument playing action capture method based on audio and video described in the above embodiments, the present application also proposes a stringed instrument playing action capture device based on audio and video.

[0109] Specifically, Figure 3 The structure schematic diagram of the stringed instrument playing action capture device based on audio and video provided by the embodiment of the present application is shown.

[0110] As Figure 3 shown, the device comprises: an audio-video acquisition module 310, configured to acquire a stringed instrument playing video and a stringed instrument playing audio; a first fusion feature acquisition module 320, configured to acquire a first fusion feature based on the stringed instrument playing video; a second fusion feature acquisition module 330, configured to acquire a second fusion feature based on the stringed instrument playing audio; and a stringed instrument action capture module 340, configured to acquire a stringed instrument playing action capture result based on a pre-trained action capture model according to the first fusion feature and the second fusion feature; wherein the action capture model is obtained by training and optimization of a theoretical playing position loss function and a finger pressing force loss function.

[0111] In the embodiment, the string instrument performance video and the string instrument performance audio are acquired by the audio and video acquisition module 310, the first fusion feature acquisition module 320 acquires the first fusion feature based on the string instrument performance video, the second fusion feature acquisition module 330 acquires the second fusion feature based on the string instrument performance audio, and the string instrument motion capture module 340 acquires the string instrument performance motion capture result based on the pre-trained motion capture model according to the first fusion feature and the second fusion feature. The motion capture model is obtained by training and optimizing the theoretical performance position loss function and the finger pressing force loss function. The device realizes the fine capture of the string instrument performance based on the string instrument performance video combined with the string instrument performance audio without any optical marker or sensor introduction, highlights the accurate hand motion, captures the string instrument and the key interactive contact, and realizes the high-precision modeling of the whole body skeleton of the performer, the instrument position and the fine performance action of the hand during the performance.

[0112] It should be noted that the string instrument performance motion capture device based on audio and video provided by the embodiment of the application can be correspondingly referred to the string instrument performance motion capture method based on audio and video described in the above embodiments, which will not be described here.

[0113] Figure 4 An example of a schematic diagram of a physical structure of an electronic device is shown as Figure 4 The electronic device can include a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 can communicate with each other through the communications bus 440. The processor 410 can invoke the logic instructions in the memory 430 to execute the string instrument performance motion capture method based on audio and video, which includes acquiring a string instrument performance video and a string instrument performance audio, acquiring a first fusion feature based on the string instrument performance video, acquiring a second fusion feature based on the string instrument performance audio, and acquiring a string instrument performance motion capture result based on a pre-trained motion capture model according to the first fusion feature and the second fusion feature. The motion capture model is obtained by training and optimizing the theoretical performance position loss function and the finger pressing force loss function.

[0114] In addition, the logic instructions in the memory 430 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0115] In another aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the audio and video based stringed instrument playing motion capture method provided by the above method, the method comprising: obtaining a stringed instrument playing video and a stringed instrument playing audio; obtaining a first fusion feature based on the stringed instrument playing video; obtaining a second fusion feature based on the stringed instrument playing audio; obtaining a stringed instrument playing motion capture result according to the first fusion feature and the second fusion feature based on a pre-trained motion capture model; wherein the motion capture model is obtained by training and optimizing a theoretical playing position loss function and a finger pressing force loss function.

[0116] The device embodiments described above are only schematic, wherein the units illustrated as separate components can or can not be physically separated, and the components illustrated as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement without creative labor.

[0117] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0118] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An audio-video-based string instrument playing motion capture method, characterized in that, The method comprises the following steps: obtaining a stringed instrument performance video and a stringed instrument performance audio; obtaining a first fusion feature based on the stringed instrument performance video; obtaining a second fusion feature based on the stringed instrument performance audio; obtaining a stringed instrument performance motion capture result according to the first fusion feature and the second fusion feature based on a pre-trained motion capture model; wherein the motion capture model is obtained by training and optimizing a theoretical performance position loss function and a finger pressing force loss function.

2. The audio-video-based string instrument playing motion capture method according to claim 1, wherein, The step of obtaining the first fusion feature based on the stringed instrument performance video comprises the following steps: performing human body detection on video frames in the stringed instrument performance video to obtain human body bounding box features of a performer; outputting full-body key point features of the performer according to the stringed instrument performance video based on a pre-trained time-series human body pose estimation network; extracting visual semantic features of the video frames in the stringed instrument performance video, wherein the visual semantic features comprise local area texture, posture context and a space background in which the performer is located; generating a global depth map according to the stringed instrument performance video based on a pre-trained monocular depth estimation model; fusing the human body bounding box features, the full-body key point features, the visual semantic features and the global depth map to obtain the first fusion feature.

3. The audio-video based stringed musical instrument playing motion capture method of claim 1, wherein, The step of obtaining the second fusion feature based on the stringed instrument performance audio comprises the following steps: extracting mel-spectrum features, audio semantic features and fundamental frequency features in the stringed instrument performance audio; fusing the mel-spectrum features, the audio semantic features and the fundamental frequency features to obtain the second fusion feature.

4. The audio-video based stringed musical instrument playing motion capture method of claim 1, wherein, The theoretical performance position loss function is defined as follows: ； wherein, represents a fingertip space coordinate of the target fingertip, , respectively represent a distal fingertip joint angle and a proximal fingertip joint angle of the target fingertip, represents a theoretical performance position.

5. The audio-video based stringed musical instrument playing motion capture method of claim 1, wherein, The finger pressing force loss function is defined as follows: ； wherein, represents a time frame in which a string is actually played, represents a set of time frames in which a string is actually played, represents a finger flexion angle threshold value, represents a time frame finger flexion angle.

6. The audio-video based stringed musical instrument playing motion capture method of claim 4, wherein, The step of calculating the theoretical performance position comprises the following steps: extracting audio features in the stringed instrument performance audio; predicting a theoretical performance string based on the audio features based on a pre-trained audio-string prediction model; wherein the audio-string prediction model is obtained by training according to audio feature samples and their corresponding real string labels; determining the theoretical performance position according to the theoretical performance string and a stringed instrument posture.

7. The audio-video-based string instrument playing motion capture method of claim 6, wherein, The step of determining the stringed instrument posture comprises the following steps: performing two-dimensional capture tracking on stringed instrument key points according to the stringed instrument performance video to obtain two-dimensional coordinates of the stringed instrument key points; determining the stringed instrument posture under a video frame according to the two-dimensional coordinates of the stringed instrument key points and a global depth map corresponding to the video frame in the stringed instrument performance video.

8. An audio-video based stringed musical instrument playing motion capture device, characterized by, The method comprises the following steps: an audio-video acquisition module, configured to obtain a stringed instrument performance video and a stringed instrument performance audio; a first fusion feature acquisition module, configured to obtain a first fusion feature based on the stringed instrument performance video; a second fusion feature acquisition module, configured to obtain a second fusion feature based on the stringed instrument performance audio; a stringed instrument motion capture module, configured to obtain a stringed instrument performance motion capture result according to the first fusion feature and the second fusion feature based on a pre-trained motion capture model; wherein the motion capture model is obtained by training and optimizing a theoretical performance position loss function and a finger pressing force loss function.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The computer program is executed by the processor to implement the audio-video-based string instrument performance action capture method in any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the audio-video-based string instrument performance action capture method in any one of claims 1 to 6.

Images