Audio determination method, motion determination method, model training method and related equipment

By acquiring the characteristics of motion or audio, and using the extraction model to determine the matching target audio or motion, it solves the problem that users find it difficult to find suitable for sports and music matching, and improves the movement efficiency and convenience of music selection.

CN119091914APending Publication Date: 2024-12-06HANGZHOU NETEASE CLOUD MUSIC TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411116505.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-14
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

It is difficult for users to find suitable sports and music matching, resulting in low exercise efficiency and difficulty in choosing music.

Method used

By acquiring motion characteristics, the extraction model is used to determine the matching audio characteristics, so as to play the appropriate target audio, or to obtain the audio characteristics, determine the matching motion characteristics, and output the appropriate motion suggestions.

Benefits of technology

It improves the adaptability of movement and audio, improves the efficiency of movement, reduces the difficulty of finding adapted music, and saves users time.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091914A_ABST
    Figure CN119091914A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides an audio determination method, a motion determination method, a model training method and related equipment, and relates to the technical field of audio. The method comprises the steps of determining a first audio feature corresponding to a first motion feature by acquiring the first motion feature of a first user during motion, and determining an audio corresponding to the first audio feature as a target audio suitable for playing when the first user performs motion. The first audio feature can be determined based on the corresponding relationship learned from a plurality of pairs of matched motion features and audio features, and the audio corresponding to the first audio feature is determined as the target audio to be played, so that the suitability between the user's motion and the audio can be enhanced, the motion efficiency is improved, and the user experience is improved. And the difficulty of searching music matched with the exercise by the user can be reduced, and the time of the user is saved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of audio technology. More specifically, the embodiments of the present disclosure relate to an audio determination method, a motion determination method, a model training method and related devices. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the present disclosure. No description herein is admitted to be prior art by virtue of its inclusion in this section.

[0003] Nowadays, more and more people are used to listening to music while exercising. Listening to music during exercise can inspire athletes to perform better. For example, following the beat of music while running or cycling can help athletes maintain a steady rhythm and help improve exercise efficiency.

[0004] However, there are many types of sports, and each sport has its own music. It is difficult for users to find suitable sports and audio, and they often have to spend a lot of time and energy trying. Summary of the invention

[0005] The present disclosure provides an audio determination method, a motion determination method, a model training method and related devices to solve the problem of difficulty in effectively combining motion with music.

[0006] In one aspect of the embodiments of the present disclosure, there is provided an audio determination method, comprising:

[0007] Acquire a first motion feature of a first user performing a motion;

[0008] A first audio feature is determined according to the first motion feature, and audio corresponding to the first audio feature is determined as target audio matched by the motion, and the target audio is played.

[0009] In another embodiment of the present disclosure, determining a first audio feature based on the first motion feature includes: obtaining a first fusion feature output by the extraction model based on the first motion feature and an extraction model, wherein the first fusion feature is used to indicate similar features between the motion and audio matched with the motion; and determining a first audio feature that matches the first fusion feature.

[0010] In another embodiment of the present disclosure, obtaining a first fusion feature output by the extraction model according to the first motion feature and the extraction model includes:

[0011] constructing a first motion vector according to the first motion feature;

[0012] Inputting the first motion vector into the extraction model to obtain a first fused feature vector output by the extraction model, wherein the first fused feature vector is used to represent the first fused feature;

[0013] The determining a first audio feature that matches the first fusion feature, and determining the audio corresponding to the first audio feature as the target audio matched by the motion, includes:

[0014] A first audio vector matching the first fusion feature vector is determined, and audio corresponding to the first audio vector is determined as target audio matched by the motion, wherein the first audio vector is used to characterize the first audio feature.

[0015] In another embodiment of the present disclosure, before determining the first audio feature that matches the first fusion feature, the method further includes:

[0016] Acquire second audio features of multiple motion audios;

[0017] Inputting each of the second audio features into the extraction model in sequence, obtaining a second fusion feature corresponding to each of the motion audios output by the extraction model, wherein the second fusion feature is used to indicate similar features between the motion audio and the motion matching the motion audio;

[0018] saving the second fused feature as an audio feature associated with the motion audio;

[0019] The determining the audio feature that matches the first fusion feature includes:

[0020] Among the stored audio features, an audio feature matching the first fusion feature is determined.

[0021] In another embodiment of the present disclosure, obtaining a first motion feature of a first user performing a motion includes:

[0022] Acquire a first exercise video corresponding to the exercise performed by the first user;

[0023] Based on the first motion video, a first motion feature corresponding to the motion is acquired.

[0024] In another embodiment of the present disclosure, obtaining a first motion feature corresponding to the motion based on the first motion video includes:

[0025] Determining a first motion type corresponding to the motion according to the first motion video;

[0026] determining a first target parameter of the first user according to the first motion video, where the first target parameter includes at least one of a physiological parameter and a vital sign parameter;

[0027] Determining, according to the first target parameter, a first motion parameter of the first user performing the motion, wherein the first motion parameter includes at least one of a physiological parameter change and an energy consumption parameter;

[0028] A first motion feature corresponding to the motion is determined according to the first motion type and the first motion parameter.

[0029] In another aspect of the present disclosure, a motion determination method is provided, comprising:

[0030] Acquire a third audio feature corresponding to the target audio played by the second user;

[0031] Determine a second motion feature according to the third audio feature, and determine the motion corresponding to the second motion feature as the target motion matched by the target audio;

[0032] Output the prompt information of the target movement, where the prompt information is used to suggest performing the target movement under the target audio.

[0033] In one embodiment of the present disclosure, determining the second motion feature according to the third audio feature includes:

[0034] According to the third audio feature and the extraction model, obtaining a third fusion feature output by the extraction model, wherein the third fusion feature is used to indicate a similarity feature between the target audio and a motion matching the target audio;

[0035] A second motion feature that matches the third fused feature is determined.

[0036] In another embodiment of the present disclosure, obtaining a third fusion feature output by the extraction model according to the third audio feature and the extraction model includes:

[0037] Constructing a second audio vector according to the third audio feature;

[0038] Inputting the second audio vector into the extraction model to obtain a second fused feature vector output by the extraction model, wherein the second fused feature vector is used to represent the third fused feature;

[0039] The determining of the second motion feature that matches the third fusion feature, and determining the motion corresponding to the second motion feature as the target motion matched by the target audio, includes:

[0040] A second motion vector matching the third fused feature vector is determined, and a motion corresponding to the second motion vector is determined as a target motion matched by the target audio, wherein the second motion vector is used to characterize the second motion feature.

[0041] In another embodiment of the present disclosure, before determining the second motion feature that matches the third fusion feature, the method further includes:

[0042] Determining, from a plurality of second motion videos, a third motion feature corresponding to a motion in each of the second motion videos;

[0043] Inputting each of the third motion features into the extraction model in sequence, obtaining a fourth fusion feature corresponding to each second motion video output by the extraction model, wherein the fourth fusion feature is used to indicate a similarity feature between a motion in the second motion video and an audio matched with the motion in the second motion video;

[0044] saving the fourth fusion feature as a motion feature associated with motion in the second motion video;

[0045] The determining the second motion feature that matches the third fusion feature includes:

[0046] Among the saved motion features, a second motion feature matching the third fusion feature is determined.

[0047] In another embodiment of the present disclosure, obtaining a third audio feature corresponding to the target audio played by the user includes:

[0048] Extracting a first background audio, a first rhythm parameter, and a first rhythm parameter from the target audio;

[0049] A third audio feature corresponding to the target audio is determined according to the first background audio, the first rhythm parameter, and the first rhythm parameter.

[0050] In another embodiment of the present disclosure, determining a third audio feature corresponding to the target audio according to the first background audio, the first rhythm parameter, and the first rhythm parameter includes:

[0051] Acquire a first audio text in the target audio;

[0052] Determine a first reference parameter of the target audio according to the first audio text, the first reference parameter comprising at least one of an energy value and an emotion, the energy value being used to indicate how easily the target audio can mobilize the user's emotions;

[0053] A third audio feature corresponding to the target audio is determined according to the first background audio, the first rhythm parameter, the first rhythm parameter, and the first reference parameter.

[0054] In another aspect of the present disclosure, there is provided an extraction model training method, comprising:

[0055] Acquire a plurality of training samples, wherein the training samples include a fourth motion feature of a training user performing motion and a fourth audio feature corresponding to the audio played when the training user performs motion;

[0056] Training a preset model according to each of the training samples to obtain an extraction model;

[0057] Among them, the extraction model is used to output a first fusion feature corresponding to the user's movement, and the extraction model is used to output a third fusion feature corresponding to the audio played by the user, the first fusion feature is used to indicate the similarity features between the movement and the audio matching the movement, and the third fusion feature is used to indicate the similarity features between the audio and the movement matching the audio.

[0058] In one embodiment of the present disclosure, obtaining a plurality of training samples includes:

[0059] Obtain a third motion video of each of the training users; in the third motion video, obtain a fourth motion feature of the training user's motion and a fourth audio feature of the audio played in the third motion video; and construct a training sample based on the fourth audio feature and fourth motion feature corresponding to the training user.

[0060] In another embodiment of the present disclosure, constructing a training sample according to the fourth audio feature and the fourth motion feature corresponding to the training user includes:

[0061] A third audio vector is constructed according to the fourth audio feature corresponding to the training user; a third motion vector is constructed according to the fourth motion feature corresponding to the training user; and the training sample is constructed according to the third audio vector and the third motion vector, and the training sample is used for the preset model to learn similar features between the fourth audio feature and the fourth motion feature.

[0062] In another embodiment of the present disclosure, in the third motion video, obtaining a fourth motion feature of the training user performing motion includes:

[0063] Based on the third motion video, determine the second motion type corresponding to the motion in the third motion video; based on the third motion video, determine the second target parameter of the training user, the second target parameter includes at least one of a physiological parameter and a vital sign parameter; based on the second target parameter, determine the second motion parameter of the training user performing the motion, the second motion parameter includes at least one of a physiological parameter change and an energy consumption parameter; based on the second motion type and the second motion parameter, determine the fourth motion feature corresponding to the motion in the third motion video.

[0064] In another embodiment of the present disclosure, in the third motion video, obtaining a fourth audio feature of the audio played in the third motion video includes:

[0065] A second background audio, a second rhythm parameter and a second rhythm parameter are extracted from the audio in the third motion video; and a fourth audio feature is determined according to the second background audio, the second rhythm parameter and the second rhythm parameter.

[0066] In another embodiment of the present disclosure, determining a fourth audio feature according to the second background audio, the second rhythm parameter, and the second rhythm parameter includes:

[0067] Obtain a second audio text of the audio in the third motion video; determine a second reference parameter of the audio in the third motion video based on the second audio text, the second reference parameter including at least one of an energy value and an emotion, the energy value being used to indicate how easily the audio can mobilize the user's emotions; determine a fourth audio feature and the second reference parameter based on the second background audio, the second rhythm parameter, the second rhythm parameter and the second reference parameter, and determine the fourth audio feature.

[0068] In another aspect of the embodiments of the present disclosure, there is provided an audio determination device, comprising:

[0069] A first acquisition module, used to acquire a first motion feature of a first user performing motion;

[0070] The first determination module is used to determine a first audio feature according to the first motion feature, determine the audio corresponding to the first audio feature as the target audio matched by the motion, and play the target audio.

[0071] In another aspect of the present disclosure, a motion determination device is provided, comprising:

[0072] A second acquisition module, used to acquire a third audio feature corresponding to the target audio played by the second user;

[0073] A second determination module is used to determine a second motion feature according to the third audio feature, and determine the motion corresponding to the second motion feature as the target motion matched by the target audio;

[0074] An output module is used to output prompt information of the target movement, and the prompt information is used to suggest performing the target movement under the target audio.

[0075] In another aspect of the present disclosure, a model training device is provided, comprising:

[0076] A third acquisition module is used to acquire multiple training samples, where the training samples include a fourth motion feature of the training user performing exercise and a fourth audio feature corresponding to the audio played when the training user performs exercise;

[0077] A training module, used to train a preset model according to each of the training samples to obtain an extraction model;

[0078] Among them, the extraction model is used to output a first fusion feature corresponding to the user's movement, and the extraction model is used to output a third fusion feature corresponding to the audio played by the user, the first fusion feature is used to indicate the similarity features between the movement and the audio matching the movement, and the third fusion feature is used to indicate the similarity features between the audio and the movement matching the audio.

[0079] In another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method as described in any one of the above aspects is implemented.

[0080] In another aspect of the disclosed embodiments, there is provided a computing device, comprising: at least one processor;

[0081] and a memory communicatively coupled to the at least one processor;

[0082] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the computing device to perform the method as described in any one of the above aspects.

[0083] In another aspect of the embodiments of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method as described in any one of the above aspects is implemented.

[0084] According to the audio determination method, motion determination method, model training method and related devices of the embodiments of the present disclosure, by acquiring the first motion feature of the first user performing the motion, the first audio feature corresponding to the first motion feature can be determined, and the audio corresponding to the first audio feature can be determined as the target audio suitable for playing when the first user performs the motion. The first audio feature can be determined based on the correspondence learned from several pairs of matching motion features and audio features, and the audio corresponding to the first audio feature is determined as the target audio for playing, which can enhance the compatibility between the user's motion and the audio, improve the efficiency of the motion, and reduce the difficulty for the user to find music that is suitable for the motion, saving the user's time. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0086] Figure 1 A schematic diagram schematically shows an application scenario of the audio determination method according to an embodiment of the present disclosure;

[0087] Figure 2 A schematic diagram schematically shows a flow chart of an audio determination method according to an embodiment of the present disclosure;

[0088] Figure 3 A schematic diagram of a flow chart of a motion determination method according to an embodiment of the present disclosure is schematically shown;

[0089] Figure 4 The following schematically shows a flow chart of an extraction model training method according to an embodiment of the present disclosure;

[0090] Figure 5 A schematic diagram of a process flow of another extraction model training method according to an embodiment of the present disclosure is schematically shown;

[0091] Figure 6 A schematic diagram of reasoning application of an extraction model according to an embodiment of the present disclosure is schematically shown;

[0092] Figure 7 A schematic diagram schematically shows a storage medium according to an embodiment of the present disclosure;

[0093] Figure 8 The structure diagram of an audio determination device according to an embodiment of the present disclosure is schematically shown;

[0094] Fig. 9 The structure diagram of a motion determination device according to an embodiment of the present disclosure is schematically shown;

[0095] Fig.10 The structure diagram of an extraction model training device according to an embodiment of the present disclosure is schematically shown;

[0096] Fig.11 A schematic diagram of a computing device according to an embodiment of the present disclosure is schematically shown.

[0097] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0098] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0099] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0100] According to an embodiment of the present disclosure, an audio determination method, a motion determination method, a model training method and related devices are proposed.

[0101] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and corresponding operation entrances shall be provided for users to choose to authorize or refuse.

[0102] In addition, any number of elements in the drawings is for illustration and not limitation, and any naming is only for distinction and does not have any limiting meaning.

[0103] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure. SUMMARY OF THE INVENTION

[0105] Nowadays, many people like to listen to music while exercising, and the scene of listening to music while exercising is becoming more and more common, because listening to music can inspire better athletic performance. However, there are many kinds of sports, such as running, cycling, fitness, yoga, etc. It is necessary to choose the right music to match with sports, but it is difficult for users to find suitable music and sports in a short time.

[0106] The inventor therefore thought that in order to make listening to music while exercising more scientific, the motion features of the exercise and the audio features of the audio can be extracted from pre-selected exercise videos containing suitable music, and the correspondence between the motion features of different exercises and the audio features of different audios can be determined. Then, the audio features suitable for the user's current exercise can be matched according to the motion features of the user's current exercise, and the target audio corresponding to the matched audio features can be used as the audio suitable for the user's current exercise.

[0107] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0108] Application Scenario Overview

[0109] First reference Figure 1 , Figure 1 This is a schematic diagram of an application scenario provided by the present disclosure. Figure 1 As shown, if the user wants to obtain audio suitable for sports, the terminal device can shoot a sports video containing the user's sports pictures, and determine and play the audio suitable for the user to listen to during sports according to the sports video. In addition to obtaining sports videos by shooting the user's sports with the terminal device, the terminal device can also receive sports videos containing the user's sports pictures transmitted by other devices.

[0110] In some other application scenarios, the motion video can also be uploaded to the server through the terminal device. The server determines the audio to be played based on the motion video and returns the relevant information of the audio to the terminal device or other audio playback device for playback.

[0111] The application scenarios mentioned above are only some examples. Those skilled in the art may expand the application based on the interaction process, and the embodiments of the present disclosure do not impose any specific limitations on this.

[0112] Exemplary Methods

[0113] Combine the following Figure 1 For application scenarios, refer to Figure 2 To describe the audio determination method according to an exemplary embodiment of the present disclosure. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in this regard. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0114] Figure 2 The flowchart of an audio determination method according to an embodiment of the present disclosure is schematically shown. Figure 2 As shown, the method may include:

[0115] Step S201: obtaining a first motion feature of a first user performing motion.

[0116] The motion characteristics may include the type of motion performed by the user, the intensity of the motion, and the physiological parameters of the user during the motion.

[0117] Specifically, the motion characteristics of the user's exercise may be acquired through a preset terminal device.

[0118] In some possible implementations, the preset terminal device may include a pre-set visual sensor, which can capture images or videos of the first user performing exercise. For example, a camera may be placed in a suitable position to capture video images of the user performing exercise such as fitness.

[0119] In some possible implementations, the preset terminal device may be an electronic device such as a smart phone used by the first user, and the electronic device may run a preset application client, and images or pictures containing the first user performing exercise may be uploaded through the application client. In some other possible implementations, the characteristics of the first user's exercise type, exercise intensity, and physiological parameters of the user's exercise may also be directly uploaded through the application client. For example, the application client may provide a human-computer interaction interface in response to a user's input request, and the user may input characteristics such as "running" (exercise type characteristic) or "high exercise intensity" (exercise intensity characteristic) in text form.

[0120] Step S202: determine a first audio feature according to the first motion feature, determine the audio corresponding to the first audio feature as the target audio matched by the motion, and play the target audio.

[0121] The audio features may include audio characterization features and audio metadata features. The audio metadata features may include the BPM (Beat Per Minute), rhythm parameters, rhythm parameters, and lyrics energy of the audio. The rhythm parameters may represent the strength of the audio rhythm. The rhythm parameters may represent the rhythm of the audio.

[0122] Specifically, the first audio feature corresponding to the first motion feature can be determined by a preset knowledge base or model, and the knowledge base or model can be established based on a number of samples, each of which can include a motion feature with a label and an audio feature with a label. The motion feature and audio feature included in each sample can be extracted based on the matching motion video and audio.

[0123] In some possible implementations, the knowledge base or model can compare the analyzed music feature information and motion information, determine the mapping relationship between motion and audio based on algorithms such as deep learning, and based on the relationship, determine the first audio feature corresponding to the first motion feature from a number of audio features.

[0124] In some possible implementations, the knowledge base or model can record several preset audio features and the audio corresponding to each audio feature. After determining the first audio feature corresponding to the first motion feature, the audio corresponding to the first audio feature can be used as the target audio for the user's exercise and played.

[0125] In the above embodiment, by acquiring the first motion feature of the first user performing exercise, the first audio feature corresponding to the first motion feature can be determined, and the audio corresponding to the first audio feature can be determined as the target audio suitable for playing when the first user performs exercise. The first audio feature can be determined based on the correspondence learned from a plurality of pairs of matching motion features and audio features, and the audio corresponding to the first audio feature is determined as the target audio for playing, which can enhance the compatibility between the user's exercise and the audio, improve exercise efficiency, and reduce the difficulty for the user to find music suitable for exercise, saving the user's time.

[0126] In one embodiment, determining the first audio feature according to the first motion feature includes:

[0127] S2021: Obtain a first fusion feature output by the extraction model based on the first motion feature and the extraction model.

[0128] The extraction model may be obtained by training based on a number of samples containing motion features and audio features. The first fusion feature is used to indicate similar features between the motion and the audio matched with the motion.

[0129] In some possible implementations, the extraction model can learn similar features between motion and audio suitable for motion based on motion features in several motion videos and audio features in audio suitable for the motion videos, and generate fusion features corresponding to the motion features and audio features based on the learned knowledge.

[0130] In some possible implementations, a first motion vector may be constructed according to the first motion feature, and the first motion vector may be input into an extraction model to obtain a first fused feature vector output by the extraction model. The first fused feature vector is used to represent the first fused feature.

[0131] Exemplarily, the extraction model can construct vectors based on motion features and audio features in the sample, and establish a cross-modal joint vector space based on these vectors. The first motion vector is input into the extraction model, and the extraction model can find a first fused feature vector that matches the first motion vector from the joint vector space.

[0132] S2022: Determine a first audio feature that matches the first fusion feature.

[0133] In some possible implementations, a first audio vector matching the first fusion feature vector may be determined, and the audio corresponding to the first audio vector may be determined as the target audio matched by the motion. The first audio vector is used to represent the first audio feature.

[0134] Exemplarily, both the motion vector and the audio vector can be multi-dimensional vectors, and each dimension of the multi-dimensional vector can be used to represent a feature of a certain aspect. For example, the motion vector can contain element values ​​of different dimensions such as motion type and motion intensity to represent corresponding features. The feature model can generate a single-dimensional fused feature vector based on the motion vectors and audio vectors corresponding to the matching motion video and audio. When a first motion vector is input to the extraction model, the extraction model can output a first fused feature vector matching the first motion vector, and determine the corresponding audio vector as the first audio vector based on the first fused feature vector.

[0135] In one embodiment, before determining the first audio feature that matches the first fusion feature, the method further includes:

[0136] S20211, obtain second audio features of multiple motion audios.

[0137] The sports audio may be the audio played in the sports video.

[0138] S20212, input each second audio feature into the extraction model in sequence to obtain a second fusion feature corresponding to each motion audio output by the extraction model.

[0139] The second fusion feature is used to indicate similar features between the motion audio and the motion matching the motion audio.

[0140] S20213, save the second fused feature as an audio feature associated with the motion audio.

[0141] Specifically, multiple sports videos that are scientifically matched can be obtained in advance, audio features of the audio can be obtained from these sports videos and input into the extraction model in sequence, the second fusion features output by the model can be obtained and saved as audio features associated with the sports audio.

[0142] In some possible implementations, determining the audio feature that matches the first fused feature includes: determining the audio feature that matches the first fused feature among the saved audio features.

[0143] In the above embodiment, by using a model in advance to establish a knowledge base of audio features, it is possible to facilitate subsequent retrieval of motion-matched audio in the knowledge base, thereby accelerating the speed of obtaining audio.

[0144] In one embodiment, obtaining a first motion feature of a user performing a motion includes:

[0145] S2011, obtaining a first exercise video corresponding to the exercise performed by the first user.

[0146] In some possible implementations, a video of the user exercising may be captured by a pre-placed video capture device or a portable capture device on the user, and the captured video may be uploaded via an application client.

[0147] In some possible implementations, the user may also authorize the application client to use a video capture function of the terminal device to obtain a video of the user exercising through real-time capture.

[0148] S2012: Based on the first motion video, obtain a first motion feature corresponding to the motion.

[0149] The first motion feature may include the type of motion, the intensity of motion, the height and weight of the user, and the energy consumed by the user during the motion.

[0150] In some possible implementations, a machine learning model can be used to identify the user's exercise type (such as running, cycling, yoga).

[0151] In some possible implementations, the height, weight, and exercise intensity of a person exercising in a video may be identified through the video content, and the heart rate change and calorie consumption may be estimated based on the identified information.

[0152] In some possible implementations, obtaining a first motion feature corresponding to the motion based on the first motion video includes:

[0153] S20121: Determine a first motion type corresponding to the motion according to the first motion video.

[0154] For example, the first motion video may be processed using a pre-trained model, and semantic recognition may be performed using the model to determine the motion type of the person in the video. The pre-trained model may be obtained by training based on a number of videos containing person motion.

[0155] S20122: Determine a first target parameter of the first user based on the first motion video.

[0156] The first target parameter includes at least one of a physiological parameter and a vital sign parameter.

[0157] Physical sign parameters may include the user's height and weight. For example, a human target detection algorithm and a human key point recognition algorithm may be used to perform height and body shape detection and measurement to obtain the user's height and body shape information, and the user's weight may be further estimated based on the height and body shape.

[0158] Physiological parameters may include the user's heart rate. For example, the heart rate value of the user when exercising in the video may be collected by a sensing instrument such as a smart bracelet worn by the user.

[0159] S20123: Determine a first motion parameter for the first user to perform motion according to the first target parameter.

[0160] The first motion parameter includes at least one of a physiological parameter change and an energy consumption parameter.

[0161] Exemplarily, if it is determined that the user's heart rate value is a physiological parameter, the user's heart rate change value during a preset period can be calculated as the first motion parameter.

[0162] Exemplarily, the intensity of the exercise performed by the user can also be determined based on parameters such as the user's height and weight and the type of exercise performed by the user, and the energy consumption of the user per unit time can be determined as the first exercise parameter based on the exercise intensity. The unit of energy consumption can be a calorie (referred to as card). For example, a sports-related knowledge base can be established based on several sports videos, through which it can be determined that playing table tennis is a higher exercise intensity for the user's height and weight, and the energy consumption of the user at this exercise intensity can be estimated based on the knowledge base.

[0163] S20124: Determine a first motion feature corresponding to the motion according to the first motion type and the first motion parameter.

[0164] Exemplarily, parameters such as the user's weight, the type of exercise performed by the user, exercise intensity, and energy consumption per unit time may be determined as the first exercise characteristics of the user's exercise.

[0165] According to another aspect of the present disclosure, a motion determination method is provided. Figure 3 The flowchart of a motion determination method according to an embodiment of the present disclosure is schematically shown.

[0166] In one embodiment, Figure 3 As shown, the method may include:

[0167] Step S301: Acquire a third audio feature corresponding to the target audio played by the second user.

[0168] The audio features may include audio representation features and audio metadata features, etc. The audio metadata features may include the BPM, rhythm parameters, rhythm parameters, lyrics energy, etc. of the audio.

[0169] Step S302: determine a second motion feature according to the third audio feature, and determine the motion corresponding to the second motion feature as the target motion matched by the target audio.

[0170] The target display parameter includes at least one of a target display position and an interactive action.

[0171] Specifically, the second motion feature corresponding to the third audio feature can be determined by a preset knowledge base or model, and the knowledge base or model can be established based on a number of samples, each of which can include a motion feature with a label and an audio feature with a label. The motion feature and audio feature included in each sample can be extracted based on the matching motion video and audio.

[0172] In some possible implementations, the knowledge base or model can compare the analyzed music feature information and motion information, determine the mapping relationship between motion and audio based on algorithms such as deep learning, and based on the relationship, determine the second motion feature corresponding to the third audio feature from several motion features.

[0173] In some possible implementations, the knowledge base or model can record several preset motion features and the motion corresponding to each motion feature. After determining the second motion feature corresponding to the third audio feature, the motion corresponding to the second motion feature can be used as a target motion suitable for matching the target audio played by the user.

[0174] Step S303: outputting prompt information of target movement.

[0175] Among them, the prompt information is used to suggest target movement under the target audio.

[0176] Specifically, the prompt information can be output to the second user through SMS, email, application client display or voice reminder, so as to inform the user of the exercise suitable for the target audio. For example, if the user plays a soothing music style audio through the application client, and the exercise corresponding to the exercise feature determined based on the audio feature of the audio is yoga, the application client can output yoga-related prompt information to the user.

[0177] In the above embodiment, by acquiring the third audio feature of the target audio played by the second user, the second motion feature corresponding to the third audio feature can be determined, and the motion corresponding to the second motion feature can be determined as the target motion suitable for the target audio played by the second user. In this way, the motion suitable for the audio played by the user can be quickly found, saving the user's time.

[0178] In one embodiment, determining the second motion feature according to the third audio feature may include:

[0179] S3021: Obtain a third fusion feature output by the extraction model based on the third audio feature and the extraction model.

[0180] The extraction model may be obtained by training based on a plurality of samples containing motion features and audio features. The third fusion feature is used to indicate similar features between the target audio and the motion matching the target audio.

[0181] The training method of the extraction model can refer to the above-mentioned audio determination method, which will not be repeated here.

[0182] In some possible implementations, a second audio vector may be constructed based on the third audio feature, and the second audio vector is input into the extraction model to obtain a second fused feature vector output by the extraction model. The second fused feature vector is used to represent the third fused feature.

[0183] Exemplarily, the extraction model can construct vectors based on motion features and audio features in the sample, and establish a cross-modal joint vector space based on these vectors. The second audio vector is input into the extraction model, and the extraction model can find a second fused feature vector that matches the second audio vector from the joint vector space.

[0184] S3022: Determine a second motion feature that matches the third fusion feature.

[0185] In some possible implementations, a second motion vector matching the third fused feature vector may be determined, and the motion corresponding to the second motion vector may be determined as the target motion matched by the target audio. The second motion vector is used to characterize the second motion feature.

[0186] In one embodiment, before determining the second motion feature that matches the third fusion feature, the method further includes:

[0187] S30211: Determine, from the plurality of second motion videos, a third motion feature corresponding to the motion in each second motion video.

[0188] S30212, input each third motion feature into the extraction model in sequence to obtain a fourth fusion feature corresponding to each second motion video output by the extraction model.

[0189] The fourth fusion feature is used to indicate a similar feature between the motion in the second motion video and the audio matching the motion in the second motion video.

[0190] S30213, saving the fourth fusion feature as a motion feature associated with the motion in the second motion video.

[0191] Specifically, multiple motion videos that are scientifically matched can be obtained in advance, and the motion features corresponding to each video can be input into the extraction model in turn to obtain the fourth fusion feature output by the model and save it as the motion feature associated with the motion in the motion video.

[0192] In some possible implementations, determining the second motion feature that matches the third fused feature includes: determining the second motion feature that matches the third fused feature from among the saved motion features.

[0193] In the above embodiment, by using the model in advance to establish a knowledge base of motion features, it is convenient to subsequently search for motion matching the audio in the knowledge base.

[0194] In one embodiment, obtaining a third audio feature corresponding to the target audio played by the user includes:

[0195] S3011, extracting a first background audio, a first rhythm parameter, and a first rhythm parameter from the target audio.

[0196] In some possible implementations, Fourier transform may be performed on the target audio to convert the audio signal from the time domain to the frequency domain, and the background audio may be identified and separated from the target audio by analyzing the frequency spectrum.

[0197] In some possible implementations, the spectrum energy at the beat position of the target audio may be extracted for calculation, and the calculation result may be used as the first rhythm parameter to indicate the rhythm strength of the target audio.

[0198] In some possible implementations, the target audio may be processed by a Detrended Fluctuation Analysis (DFA) algorithm and the resulting image may be plotted. By observing the slope of the fitted straight line, the Hurst index of the audio signal of the target audio may be obtained, thereby analyzing its rhythmic characteristics.

[0199] S3012: Determine a third audio feature corresponding to the target audio according to the first background audio, the first rhythm parameter, and the first rhythm parameter.

[0200] In some possible implementations, feature vector encoding may be performed based on the first background audio, the first rhythm parameter, and the first rhythm parameter, and the encoding results may be combined to obtain a feature vector, which may be used to represent the third audio feature.

[0201] In some possible implementations, determining the third audio feature corresponding to the target audio according to the first background audio, the first rhythm parameter, and the first rhythm parameter further includes:

[0202] S30121, obtain the first audio text in the target audio.

[0203] For example, text information such as lyrics in the target audio may be recognized by speech recognition technology, which may include an ARS (Automatic Speech Recognition) algorithm.

[0204] S30122: Determine a first reference parameter of the target audio according to the first audio text.

[0205] The first reference parameter includes at least one of an energy value and an emotion, and the energy value is used to indicate how easily the target audio can arouse the user's emotions.

[0206] For example, the content of text information such as lyrics can be analyzed through a large language model to determine the emotion corresponding to the audio, such as relaxed or excited.

[0207] Exemplarily, the energy attributes in text information such as lyrics can also be quantified through a large language model. For example, some lyrics are relatively more passionate and easier to arouse emotions, and the degree to which the text arouses people's emotions can be quantified as a specific energy value.

[0208] S30123: Determine a third audio feature corresponding to the target audio according to the first background audio, the first rhythm parameter, the first rhythm parameter, and the first reference parameter.

[0209] Exemplarily, feature vector encoding may be performed based on the first background audio, the first rhythm parameter, the first rhythm parameter, and the first reference parameter to obtain a multi-dimensional feature vector corresponding to the third audio feature.

[0210] According to another aspect of the present disclosure, a method for training an extraction model is provided. Figure 4 A flowchart of an extraction model training method according to an embodiment of the present disclosure is schematically shown.

[0211] In one embodiment, Figure 4 As shown, the method may include:

[0212] Step S401, obtaining multiple training samples.

[0213] The training sample includes a fourth motion feature for training the user to exercise and a fourth audio feature corresponding to the audio played for training the user to exercise. The motion feature may include the type of exercise performed by the user, the exercise intensity, the height of the user, and the physiological parameters of the user during exercise. The audio feature may include an audio representation feature and an audio metadata feature. The audio metadata feature may include the BPM, rhythm parameters, rhythm parameters, and lyrics energy of the audio.

[0214] Specifically, information extraction can be performed on a pre-prepared motion video to obtain a fourth motion feature and a fourth audio feature. The motion video contains images of a user exercising and the motion video is accompanied by background audio played when the user is exercising. The background audio in the video is appropriately matched with the motion performed by the user.

[0215] Step S402: training the preset model according to each training sample to obtain an extraction model.

[0216] Among them, the extraction model is used to output the first fusion feature corresponding to the user's movement, and the extraction model is used to output the third fusion feature corresponding to the audio played by the user. The first fusion feature is used to indicate the similarity between the movement and the audio matching the movement, and the third fusion feature is used to indicate the similarity between the audio and the movement matching the audio.

[0217] Specifically, the fourth motion feature and the fourth audio feature in the training sample can be input into a preset model. The preset network can include a multi-layer neural network structure. The relationship between the motion feature and the audio feature can be learned by adjusting the neuron parameters and weights of the preset model, so that the model can output corresponding fusion features based on the input motion features or audio features, and the extraction model can be obtained by controlling the training loss of the preset model.

[0218] In some possible implementations, the preset model for training the extraction model may be a Siamese neural network. A Siamese neural network, also known as a "twin neural network" or "twin neural network", can learn similar features between two inputs.

[0219] In one example, taking the twin network as the preset model, training the extraction model based on the training samples may include: inputting the motion features and audio features in the same training sample into the twin network for comparative learning training, optimizing the training loss based on the cross entropy function and the comparative learning loss function NT-Xent (Normalized Temperature-scaled Cross Entropy Loss), and when the preset training loss conditions are met, the extraction model can be obtained.

[0220] In the above embodiment, matching motion features and audio features can be obtained based on training samples, and the preset model can learn the relationship between the motion features and the audio features to obtain an extraction model, which can be applied to the above audio determination method or motion determination method.

[0221] In one embodiment, obtaining a plurality of training samples may include:

[0222] S4011, obtaining the third motion video of each training user.

[0223] S4012: In the third sports video, obtain a fourth sports feature for training the user to perform sports and a fourth audio feature of the audio played in the third sports video.

[0224] In some possible implementations, the motion video can be input into a preset motion parameter analysis engine to obtain motion features output by the motion parameter analysis engine. The motion parameter analysis engine can integrate algorithms such as semantic recognition and human target detection to implement functions such as identifying the type of motion performed by the user and determining the height of the person in the video.

[0225] In some possible implementations, the audio signal of the motion video can be input into a preset music analysis engine to obtain the audio features output by the music parameter analysis engine. The music analysis engine can use an autoencoder structure to train audio representation features based on a transformer structure, and the trained structure can be used to extract audio representations. In some examples, the input audio can be converted into a [1×768] array vector.

[0226] S4013: Construct a training sample according to the fourth audio feature and the fourth motion feature corresponding to the training user.

[0227] In some possible implementations, motion features and audio features extracted from the same motion video may be combined into one training sample.

[0228] In one embodiment, in the third motion video, obtaining a fourth motion feature for training a user to perform motion includes:

[0229] S40121: Determine, according to the third motion video, a second motion type corresponding to the motion in the third motion video.

[0230] S40122: Determine a second target parameter for training the user based on the third motion video.

[0231] The second target parameter includes at least one of a physiological parameter and a vital sign parameter.

[0232] S40123: Determine a second motion parameter for training the user to perform exercise according to the second target parameter.

[0233] The second motion parameter includes at least one of a physiological parameter change and an energy consumption parameter.

[0234] S40124: Determine a fourth motion feature corresponding to the motion in the third motion video according to the second motion type and the second motion parameter.

[0235] The process of obtaining the fourth motion feature in this embodiment may refer to the process of obtaining the first motion feature in the above embodiment, and will not be described in detail here.

[0236] In one embodiment, in the third motion video, obtaining a fourth audio feature of the audio played in the third motion video includes:

[0237] S40124, extracting a second background audio, a second rhythm parameter, and a second rhythm parameter from the audio in the third motion video.

[0238] S40124: Determine a fourth audio feature according to the second background audio, the second rhythm parameter, and the second rhythm parameter.

[0239] In some possible implementations, a second audio text of the audio in the third motion video may be obtained; a second reference parameter of the audio in the third motion video may be determined based on the second audio text; a fourth audio feature and the second reference parameter may be determined based on the second background audio, the second rhythm parameter, the second rhythm parameter, and the second reference parameter, and a fourth audio feature may be determined. The second reference parameter includes at least one of an energy value and an emotion, and the energy value is used to indicate how easily the audio can mobilize the user's emotions.

[0240] The process of obtaining the fourth audio feature in this embodiment can refer to the process of obtaining the third audio feature in the above embodiment, and the process of determining the fourth audio feature according to the second background audio, the second rhythm parameter and the second rhythm parameter can refer to the process of determining the third audio feature in the above embodiment, which will not be repeated here.

[0241] In one embodiment, the fourth motion characteristic may include the training user's height, weight, the type of motion performed by the training user in the motion video, the motion intensity, and the energy consumption.

[0242] The height and weight features can be obtained by using a human target detection algorithm and a human key point recognition algorithm to determine the height and body shape of the training user, and the weight of the training user can be determined based on the height and body shape of the training user.

[0243] Obtaining the sport type may include: labeling the sport category by manual labeling or other methods, obtaining labels such as playing basketball, badminton, running, yoga, etc., and developing a video information classification algorithm based on these sport category labels using a transformer-based VIT model to identify the sport category in the third sport video. The VIT (Vision Transformer Computer Vision) model is an image classification model based on the Transformer architecture.

[0244] Obtaining the motion intensity may include: designing a motion intensity algorithm training set, dividing the motion intensity into low intensity, medium intensity, medium-high intensity, and high intensity. Using this training set, the algorithm uses a VIT text-image cross-modal algorithm, and uses a deep learning classification algorithm in conjunction with a motion category recognition algorithm to identify the motion intensity in the third motion video.

[0245] Obtaining energy consumption may include: using a preset calorie calculation formula to calculate the actual exercise calories of the training user in the third exercise video by identifying the training user's height, weight, exercise type and exercise intensity information, and using the calculated calorie value as a parameter to characterize energy consumption.

[0246] In one embodiment, the fourth audio feature may include audio characterization features of the audio in the third motion video, and may also include audio-related metadata features such as audio BPM, rhythm parameters, rhythm parameters, and lyrics energy.

[0247] The audio representation feature can be extracted from the audio in the third motion video based on the pre-trained transformer network structure, and an audio can be converted into an array vector of [1×768].

[0248] The audio metadata features can be obtained by extracting the audio in the third motion video through a preset audio processing algorithm. For example, a transformer-based beat transformer (beat conversion) model can be used to extract audio BPM. This model inputs the prepared music data and corresponding labels (i.e., rhythm information) into the model for training. During the training process, an appropriate loss function (such as mean square error) is used to measure the difference between the model output and the true rhythm, and the model parameters are optimized by the back propagation algorithm. Stochastic Gradient Descent (SGD) or other optimization algorithms can be used to update the parameters. The rhythm information is mainly the beat point label: whether there is a beat point on each time step. It can be represented by a binary label, 1 means there is a beat point, and 0 means no beat point. Use this label to train the model, and finally statistically output the beat point to output the rhythm BPM information.

[0249] In one embodiment, constructing a training sample according to the fourth audio feature and the fourth motion feature corresponding to the training user includes:

[0250] S40131, construct a third audio vector according to the fourth audio feature corresponding to the training user.

[0251] Specifically, the features of the audio signal in the third motion video can be extracted through the music parameter analysis engine, and a third audio vector can be constructed based on these features. The third audio vector can include five dimensions: audio characterization features, audio BPM, rhythm parameters, rhythm parameters and lyrics energy.

[0252] S40132: Construct a third motion vector according to the fourth motion feature corresponding to the training user.

[0253] Specifically, the features in the third motion video can be extracted through the motion parameter analysis engine, and a third motion vector can be constructed based on these features. The third motion vector can include five dimensions: motion type, motion intensity, energy consumption, user height and user weight.

[0254] S40133: Construct a training sample according to the third audio vector and the third motion vector.

[0255] The training samples are used to preset a model to learn similar features between the fourth audio feature and the fourth motion feature.

[0256] Specifically, the third audio vector and the third motion vector may be paired so that the third audio vector and the third motion vector from the same third motion video are combined into a feature vector of a single dimension, and each combined feature vector may be used as a training sample.

[0257] The extraction model can be obtained through the extraction model training method in each of the above embodiments, and the extraction model can be applied to the above audio determination method or motion determination method.

[0258] Figure 5 A flowchart of another extraction model training method according to an embodiment of the present disclosure is schematically shown.

[0259] like Figure 5 As shown, a third motion video containing training user motion can be obtained, and the audio and video images in the video can be input into the music parameter analysis engine and the motion parameter analysis engine respectively. The music parameter analysis engine can extract fourth audio features such as audio characterization features and audio metadata features from the audio, and the motion parameter analysis engine can extract fourth motion features such as user metadata information (such as training user height and weight), motion type, exercise intensity and energy consumption from the video image. These extracted features are then input into a model (such as a twin model) for correlation training to learn the relationship between audio features and motion features to obtain a trained extraction model.

[0260] Figure 6 A schematic diagram of reasoning application of an extraction model according to an embodiment of the present disclosure is schematically shown.

[0261] like Figure 6 As shown, in getting Figure 5 After the extraction model is built, first, an audio feature library can be pre-established based on the audio in the music library, specifically including using the music parameter analysis engine to extract the audio features of the audio in the music library and inputting them into the extraction model, and then establishing the audio feature library based on the output of the extraction model. Then, the video image provided by the user is obtained, the motion features are extracted by the motion parameter analysis engine and input into the extraction model, and the pre-established audio feature library is searched based on the output of the extraction model, and the audio corresponding to the search result is played as the target audio.

[0262] In the above embodiment, the user can record his or her own exercise process through a camera device. The video information of the exercise process is used to extract the cross-modal features of the exercise through a motion parameter analysis engine and an extraction model. The feature is then used to search the audio feature library, and the most suitable song is matched and played. In this way, suitable sports songs are matched for the user.

[0263] Exemplary Media

[0264] After introducing the method of the exemplary embodiment of the present disclosure, next, refer to Fig.10 A storage medium according to an exemplary embodiment of the present disclosure is described.

[0265] refer to Figure 7As shown, the storage medium 70 stores a program product for implementing the above method according to an embodiment of the present disclosure, which can be a portable compact disk read-only memory (CD-ROM) and includes program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto.

[0266] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0267] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable signal medium may also be any readable medium other than a readable storage medium.

[0268] Program code for performing the disclosed operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN).

[0269] Exemplary Devices

[0270] After introducing the medium of the exemplary embodiment of the present disclosure, next, reference is made to Figure 8 An audio determination device according to an exemplary embodiment of the present disclosure is described, which is used to implement the audio determination method in any of the above method embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0271] like Figure 8 As shown, the audio determination device 800 may include:

[0272] The first acquisition module 801 is used to acquire a first motion feature of a first user performing a motion.

[0273] The first determination module 802 is used to determine the first audio feature according to the first motion feature, determine the audio corresponding to the first audio feature as the target audio matched by the motion, and play the target audio.

[0274] In one embodiment, the first determination module 802 is further used to: obtain a first fusion feature output by the extraction model based on the first motion feature and the extraction model, wherein the first fusion feature is used to indicate similar features between the motion and the audio matched with the motion; and determine the first audio feature that matches the first fusion feature.

[0275] In one embodiment, the first determination module 802 is also used to: construct a first motion vector based on the first motion feature; input the first motion vector into the extraction model to obtain a first fusion feature vector output by the extraction model, and the first fusion feature vector is used to characterize the first fusion feature; determine a first audio vector that matches the first fusion feature vector, and determine the audio corresponding to the first audio vector as the target audio matched by the motion, and the first audio vector is used to characterize the first audio feature.

[0276] In one embodiment, the first determination module 802 is also used to: obtain second audio features of multiple motion audios; input each second audio feature into the extraction model in sequence to obtain a second fusion feature corresponding to each motion audio output by the extraction model, the second fusion feature being used to indicate similar features between the motion audio and the motion matching the motion audio; save the second fusion feature as an audio feature associated with the motion audio; and determine the audio feature that matches the first fusion feature among the saved audio features.

[0277] In one embodiment, the first acquisition module 801 is further used to: acquire a first motion video corresponding to the motion performed by the first user; and acquire a first motion feature corresponding to the motion based on the first motion video.

[0278] In one embodiment, the first acquisition module 801 is also used to: determine a first motion type corresponding to the motion based on a first motion video; determine a first target parameter of the first user based on the first motion video, the first target parameter including at least one of a physiological parameter and a vital sign parameter; determine a first motion parameter of the first user performing the motion based on the first target parameter, the first motion parameter including at least one of a change in a physiological parameter and an energy consumption parameter; determine a first motion feature corresponding to the motion based on the first motion type and the first motion parameter.

[0279] Next, refer to Fig. 9The motion determination device of the exemplary embodiment of the present disclosure is described, which is used to implement the motion determination method in any of the above method embodiments. Its implementation principle and technical effect are similar and will not be repeated here.

[0280] like Fig. 9 As shown, the motion determination device 900 may include:

[0281] The second acquisition module 901 is used to acquire a third audio feature corresponding to the target audio played by the second user.

[0282] The second determination module 902 is configured to determine a second motion feature according to the third audio feature, and determine the motion corresponding to the second motion feature as the target motion matched by the target audio.

[0283] The output module 903 is used to output prompt information of the target movement, where the prompt information is used to suggest performing the target movement under the target audio.

[0284] In one embodiment, the second determination module 902 is also used to: obtain a third fusion feature output by the extraction model based on the third audio feature and the extraction model, the third fusion feature being used to indicate similar features between the target audio and the motion matching the target audio; and determine the second motion feature matching the third fusion feature.

[0285] In one embodiment, the second determination module 902 is also used to: construct a second audio vector based on the third audio feature; input the second audio vector into the extraction model to obtain a second fusion feature vector output by the extraction model, and the second fusion feature vector is used to characterize the third fusion feature; determine a second motion vector that matches the third fusion feature vector, and determine the motion corresponding to the second motion vector as the target motion matched by the target audio, and the second motion vector is used to characterize the second motion feature.

[0286] In one embodiment, the second determination module 902 is also used to: determine the third motion feature corresponding to the motion in each second motion video from multiple second motion videos; input each third motion feature into the extraction model in turn to obtain a fourth fusion feature corresponding to each second motion video output by the extraction model, the fourth fusion feature being used to indicate the similarity between the motion in the second motion video and the audio matching the motion in the second motion video; save the fourth fusion feature as a motion feature associated with the motion in the second motion video; and determine the second motion feature that matches the third fusion feature among the saved motion features.

[0287] In one embodiment, the second acquisition module 901 is also used to: extract the first background audio, the first rhythm parameter and the first rhythm parameter from the target audio; and determine the third audio feature corresponding to the target audio based on the first background audio, the first rhythm parameter and the first rhythm parameter.

[0288] In one embodiment, the second acquisition module 901 is also used to: acquire a first audio text in the target audio; determine a first reference parameter of the target audio based on the first audio text, the first reference parameter including at least one of an energy value and an emotion, the energy value being used to indicate how easily the target audio can mobilize user emotions; determine a third audio feature corresponding to the target audio based on the first background audio, the first rhythm parameter, the first rhythm parameter and the first reference parameter.

[0289] Next, refer to Fig.10 An extraction model training device according to an exemplary embodiment of the present disclosure is described, which is used to implement the extraction model training method in any of the above method embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0290] like Fig.10 As shown, the extraction model training device 1000 includes:

[0291] The third acquisition module 1001 is used to acquire multiple training samples, where the training samples include a fourth motion feature of a training user performing exercise and a fourth audio feature corresponding to the audio played when the training user performs exercise.

[0292] The training module 1002 is used to train the preset model according to each training sample to obtain an extraction model; wherein the extraction model is used to output a first fusion feature corresponding to the movement performed by the user, and the extraction model is used to output a third fusion feature corresponding to the audio played by the user, the first fusion feature is used to indicate the similarity feature between the movement and the audio matched with the movement, and the third fusion feature is used to indicate the similarity feature between the audio and the movement matched with the audio.

[0293] In one embodiment, the third acquisition module 1001 is also used to: obtain a third motion video of each training user; in the third motion video, obtain the fourth motion feature of the training user's exercise and the fourth audio feature of the audio played in the third motion video; and construct a training sample based on the fourth audio feature and the fourth motion feature corresponding to the training user.

[0294] In one embodiment, the third acquisition module 1001 is also used to: construct a third audio vector based on the fourth audio feature corresponding to the training user; construct a third motion vector based on the fourth motion feature corresponding to the training user; and construct a training sample based on the third audio vector and the third motion vector, and the training sample is used to preset the model to learn the similar features between the fourth audio feature and the fourth motion feature.

[0295] In one embodiment, the third acquisition module 1001 is also used to: determine the second motion type corresponding to the motion in the third motion video based on the third motion video; determine the second target parameters for training the user based on the third motion video, the second target parameters including at least one of physiological parameters and vital sign parameters; determine the second motion parameters for training the user to exercise based on the second target parameters, the second motion parameters including at least one of the change in physiological parameters and energy consumption parameters; determine the fourth motion feature corresponding to the motion in the third motion video based on the second motion type and the second motion parameters.

[0296] In one embodiment, the third acquisition module 1001 is further used to: extract the second background audio, the second rhythm parameter and the second rhythm parameter from the audio in the third motion video; and determine the fourth audio feature according to the second background audio, the second rhythm parameter and the second rhythm parameter.

[0297] In one embodiment, the third acquisition module 1001 is further used to: acquire a second audio text of the audio in the third motion video;

[0298] Determine a second reference parameter of the audio in the third motion video according to the second audio text, the second reference parameter comprising at least one of an energy value and an emotion, the energy value being used to indicate how easily the audio can mobilize the user's emotions;

[0299] According to the second background audio, the second rhythm parameter, the second rhythm parameter and the second reference parameter, a fourth audio feature and the second reference parameter are determined to determine the fourth audio feature.

[0300] Exemplary Computing Devices

[0301] After introducing the method, medium and apparatus of the exemplary embodiments of the present disclosure, next, reference is made to Fig.11 A computing device according to an exemplary embodiment of the present disclosure is described.

[0302] Fig.11 The computing device 110 shown is only an example and should not bring any limitation to the functionality and scope of use of the embodiments of the present disclosure.

[0303] like Fig.11 As shown, the computing device 110 is in the form of a general computing device. The components of the computing device 110 may include, but are not limited to: at least one processing unit 1101, at least one storage unit 1102, and a bus 1103 connecting different system components (including the processing unit 1101 and the storage unit 1102). Among them, at least one storage unit 1102 stores computer-executable instructions; at least one processing unit 1101 includes a processor, and the processor executes the computer-executable instructions to implement the method described above.

[0304] The bus 1103 includes a data bus, a control bus, and an address bus.

[0305] The storage unit 1102 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 11021 and / or a cache memory 11022 , and may further include a readable medium in the form of a non-volatile memory, such as a read-only memory (ROM) 11023 .

[0306] The storage unit 1102 may also include a program / utility 11025 having a set (at least one) of program modules 11024, such program modules 11024 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0307] The computing device 110 may also communicate with one or more external devices 1104 (e.g., keyboards, pointing devices, etc.). Such communication may be performed via an input / output (I / O) interface 1105. In addition, the computing device 110 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 1106. Fig.11 As shown, the network adapter 1106 communicates with other modules of the computing device 110 via the bus 1103. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the computing device 110, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0308] It should be noted that, although several units / modules or sub-units / modules of the interactive device are mentioned in the above detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above may be embodied in one unit / module. Conversely, the features and functions of one unit / module described above may be further divided into multiple units / modules for embodiment.

[0309] In addition, although the operations of the disclosed method are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0310] Although the spirit and principle of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.

Claims

1. An audio determination method, It is characterized in that include: Acquire a first motion feature of a first user performing a motion; A first audio feature is determined according to the first motion feature, and audio corresponding to the first audio feature is determined as target audio matched by the motion, and the target audio is played.

2. The audio determination method according to claim 1, It is characterized in that The determining a first audio feature according to the first motion feature includes: Acquire, according to the first motion feature and the extraction model, a first fusion feature output by the extraction model, wherein the first fusion feature is used to indicate a similarity feature between the motion and the audio matched with the motion; A first audio feature that matches the first fused feature is determined.

3. The audio determination method according to claim 2, It is characterized in that The step of acquiring, according to the first motion feature and the extraction model, a first fusion feature output by the extraction model comprises: constructing a first motion vector according to the first motion feature; Inputting the first motion vector into the extraction model to obtain a first fused feature vector output by the extraction model, wherein the first fused feature vector is used to represent the first fused feature; The determining a first audio feature that matches the first fusion feature, and determining the audio corresponding to the first audio feature as the target audio matched by the motion, includes: A first audio vector matching the first fusion feature vector is determined, and audio corresponding to the first audio vector is determined as target audio matched by the motion, wherein the first audio vector is used to characterize the first audio feature.

4. The audio determination method according to claim 2, It is characterized in that Before determining the first audio feature that matches the first fusion feature, the method further includes: Acquire second audio features of multiple motion audios; Inputting each of the second audio features into the extraction model in sequence, obtaining a second fusion feature corresponding to each of the motion audios output by the extraction model, wherein the second fusion feature is used to indicate similar features between the motion audio and the motion matching the motion audio; saving the second fused feature as an audio feature associated with the motion audio; The determining the audio feature that matches the first fusion feature includes: Among the stored audio features, an audio feature matching the first fusion feature is determined.

5. The audio determination method according to any one of claims 1 to 4, It is characterized in that The obtaining of a first motion feature of a first user performing a motion comprises: Acquire a first exercise video corresponding to the exercise performed by the first user; Based on the first motion video, a first motion feature corresponding to the motion is acquired.

6. A method for determining motion, It is characterized in that include: Acquire a third audio feature corresponding to the target audio played by the second user; Determine a second motion feature according to the third audio feature, and determine the motion corresponding to the second motion feature as the target motion matched by the target audio; Output the prompt information of the target movement, where the prompt information is used to suggest performing the target movement under the target audio.

7. A method for training an extraction model, It is characterized in that include: Acquire a plurality of training samples, wherein the training samples include a fourth motion feature of a training user performing motion and a fourth audio feature corresponding to the audio played when the training user performs motion; Training a preset model according to each of the training samples to obtain an extraction model; Among them, the extraction model is used to output a first fusion feature corresponding to the user's movement, and the extraction model is used to output a third fusion feature corresponding to the audio played by the user, the first fusion feature is used to indicate the similarity features between the movement and the audio matching the movement, and the third fusion feature is used to indicate the similarity features between the audio and the movement matching the audio.

8. An audio determination device, It is characterized in that include: A first acquisition module, used to acquire a first motion feature of a first user performing motion; The first determination module is used to determine a first audio feature according to the first motion feature, determine the audio corresponding to the first audio feature as the target audio matched by the motion, and play the target audio.

9. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 7 is implemented.

10. A computing device, It is characterized in that include: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the computing device to perform the method as claimed in any one of claims 1 to 7.