Classical music multi-person interactive experience method and system based on image recognition technology

Abstract

The invention discloses a classical music multi-person interactive experience method based on an image recognition technology, and belongs to the field of music interactive experience, and the method comprises the following steps: S1, obtaining a musical instrument playing action; s2, generating a dance action, and searching an optimal combination of the dance action and the playing action; s3, generating a virtual scene model; s4, performing multi-modal fusion on the optimal combination of the dance actions and the playing actions and the virtual scene model to generate a three-dimensional model, and storing the three-dimensional model into a database; and S5, capturing the dance movement of the user by using a camera, inputting the dance movement into the ResNet50 model to generate a human skeleton model, and matching and integrating the generated human skeleton model with the standard skeleton stored in the three-dimensional model to realize interactive experience. According to the classical music multi-user interactive experience method and system based on the image recognition technology, the user can be helped to more comprehensively understand and visually contact classical music culture, the classification accuracy of the model is improved, and better audio-visual experience is provided for the user.

Description

Technical Field

[0001] The present invention relates to the field of music interactive experience, and in particular to a method and system for multi-person interactive experience of classical music based on image recognition technology. Background Art

[0002] Music is an art form and cultural activity whose medium is time-organized, regular sound waves (a type of mechanical wave). The creation, performance, meaning and even definition of music vary according to cultural and social contexts.

[0003] At present, the display and dissemination of classical music mainly rely on cultural exhibitions and performances, such as chime bells and chime stones, so that the audience can intuitively understand their appearance and historical background, in order to enhance the public's understanding and interest in classical music. However, the traditional way of disseminating classical music culture has the following problems:

[0004] 1) The display and dissemination of classical music mainly rely on cultural relic exhibitions and performances. These methods are limited by factors such as time, place and personnel. The scale and speed of inheritance are slow and it is difficult to adapt to the rapid development and dissemination needs of modern society;

[0005] 2) The contents of ancient books and music scores are obscure and difficult to understand. Without knowing much about classical music culture, the audience lacks the motivation to learn more about classical music culture.

[0006] 3) Feeling and understanding classical music requires a certain level of music foundation and cultural knowledge, but individuals differ in their musical talent and understanding, resulting in uneven learning outcomes;

[0007] 4) The existing communication methods lack interactive and systematic learning mechanisms. The audience mainly accepts passively and has no personal experience, so it is difficult to form a deep understanding and mastery. This can easily lead to misunderstandings and fail to fully convey the true emotions and profound connotations of classical music.

[0008] To solve the above problems, existing technologies have been combined with Internet technologies to develop music interaction technologies based on machine learning and artificial intelligence, which have shown the following potential in promoting the inheritance and development of classical music culture:

[0009] 1) Using artificial intelligence and big data analysis, audio, video and text materials related to classical music can be systematically organized and stored for retrieval and in-depth analysis. However, this method lacks practical application, cannot replace actual performance and interactive experience, and cannot truly simulate the actual application and performance of classical music, thus limiting users' comprehensive understanding and true feelings of classical music culture;

[0010] 2) Using deep learning technology, pre-training neural networks to extract the musical features of classical music and generate new musical works. This method is difficult to fully capture the creativity and emotional expression of humans in music creation and performance, and cannot replace the unique creativity and emotional depth embodied in human creation and performance;

[0011] 3) Through artificial intelligence and motion capture technology, dance movements are captured and analyzed in detail. It has a strong dependence on hardware. At the same time, the complex operation process and large amounts of data processing may make the learning process cumbersome and time-consuming. Beginners need to spend more time to adapt and master the technology, which affects the overall learning effect;

[0012] 4) Through the combination of artificial intelligence and virtual interaction technology, users can play in a virtual reality environment and provide real-time feedback and guidance. It usually relies on hardware devices such as head displays and sensors. However, these devices sometimes cause users to feel dizzy, thus affecting the user experience;

[0013] 5) When using deep neural networks to process complex music data, if gradient explosion occurs, the model may be unable to effectively learn and generate high-quality music content, thus affecting the user's interactive experience. Summary of the invention

[0014] To solve the above problems, the present invention provides a method and system for multi-person interactive experience of classical music based on image recognition technology, which can help users to more comprehensively understand and intuitively contact classical music culture, improve the classification accuracy of the model, and provide users with a better audio-visual experience.

[0015] To achieve the above object, the present invention provides a method for multi-person interactive experience of classical music based on image recognition technology, comprising the following steps:

[0016] S1. Based on the audio data of classical music, the Mel-frequency cepstrum coefficients are used to extract the timbre characteristics of various musical instruments and obtain the instrument playing movements;

[0017] S2, based on classical music video data, use the SVM model to extract video features, generate dance movements, and search for the best combination of dance movements and performance movements;

[0018] S3. Generate a virtual scene model using the ResNet50 model based on historical information data;

[0019] S4, using a multimodal deep learning algorithm to perform multimodal fusion on the optimal combination of dance movements and performance movements and the virtual scene model, generate a three-dimensional model, and store the three-dimensional model in a database;

[0020] S5. Use the camera to capture the user's dance movements and input them into the ResNet50 model to generate a human skeleton model. Match and integrate the generated human skeleton model with the standard skeleton stored in the three-dimensional model to achieve an interactive experience.

[0021] The present invention has the following beneficial effects:

[0022] 1. By setting up a deep learning model, users can experience the charm of classical music culture in an immersive state, which can help users understand and experience classical music culture more comprehensively and improve the classification accuracy of the model, thereby providing users with a better audio-visual experience;

[0023] 2. By setting up a multimodal data fusion module and a real-time feedback and guidance module, it provides users with a systematic and digital learning method, enabling them to achieve both entertainment and learning in an immersive experience, thereby better spreading classical music culture;

[0024] 3. Through the time synchronization mechanism or the use of music beats to unify the rhythm of all users' movements, combined with real-time correction and feedback of postures and movements, it ensures that the dance movements of multiple users in the virtual scene are highly consistent; and in terms of personalized adjustment of individual interaction effects, the system will dynamically adjust the audio and video effects according to the real-time performance of each user;

[0025] 4. Use ResNet50 convolutional neural network to identify different locations of users, and reduce the number of model parameters and computational complexity through neural network structure design and model pruning technology;

[0026] 5. Use human skeleton key point technology to compare the user's skeleton point connection line with the standard skeleton point connection line, making real-time feedback and guidance more accurate and smooth;

[0027] 6. Enhanced robustness against noise and interference to ensure stable operation of the system under various circumstances.

[0028] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A flowchart of a method for multi-person interactive experience of classical music based on image recognition technology of the present invention;

[0030] Figure 2 This is a diagram of the human body key point detection and analysis results of the present invention;

[0031] Figure 3 It is a Mel spectrum analysis diagram of the present invention;

[0032] Figure 4This is the ResNet50 image classification result diagram of the present invention;

[0033] Figure 5 The PCA distribution diagram for video image feature extraction of the present invention;

[0034] Figure 6 It is a matching diagram of the dancing posture characteristics of the music figurines of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical scheme and advantages disclosed in the embodiments of the present invention clearer, the embodiments of the present invention are further described in detail in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the embodiments of the present invention and are not intended to limit the embodiments of the present invention. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions.

[0036] It should be noted that the terms "include", "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or devices.

[0037] Like reference numerals and letters denote similar items in the following drawings, and thus, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0038] like Figure 1 As shown, a method for multi-person interactive experience of classical music based on image recognition technology includes the following steps:

[0039] S1. Based on classical music audio data, the timbre characteristics of various musical instruments are extracted using Mel-frequency cepstral coefficients (MFCCs extract the spectral characteristics of audio by simulating the human ear's perception of sound) to obtain the instrument playing movements;

[0040] Step S1 specifically includes the following steps:

[0041] S11, collecting audio data of classical music;

[0042] S12, data preprocessing: the audio data is cropped, segmented and denoised in sequence, wherein the denoising process first uses high-pass and low-pass filters to denoise, and then uses fast Fourier transform to remove background noise;

[0043] S13, feature extraction: extracting timbre features in audio data using Mel frequency cepstrum coefficients;

[0044] Step S13 specifically includes the following steps:

[0045] S131, dividing the audio data into several short frames, each short frame is 20ms-40ms, and applying a Hamming window or a Hanning window to reduce spectrum leakage;

[0046] S132, applying short-time Fourier transform to each short frame to convert the time domain signal into the frequency domain to obtain a spectrum diagram;

[0047] Among them, the short-time Fourier transform formula is as follows:

[0048]

[0049] In the formula, x(t) represents the time domain signal, and X(f) represents the frequency domain signal;

[0050] S133, decomposing the short-time Fourier transform result into an amplitude spectrum |X(f)| and a phase spectrum arg(X(f)), and using the amplitude spectrum |X(f)| to analyze the frequency components in the spectrum graph, and identifying the rhythm pattern and the emotional characteristics (temperature attributes) of the music;

[0051] Rhythm and beat: The system uses audio feature extraction technology to analyze the rhythm and beat of Chu music. By detecting the frequency and pitch changes in the audio signal, the system can determine the rhythm pattern and main beat of the music. For example, through spectrum analysis (such as Fourier transform), the beat characteristics of the music can be extracted and the rhythm model of the music can be constructed.

[0052] Emotional characteristics: The system also analyzes the emotional characteristics of the music, such as the emotional tone of the music (such as cheerful, heavy) and the dynamic range (such as the change of strength). This information helps to understand the overall atmosphere of the music performance and guide the performance style of the dance movements.

[0053] The result of Fourier transform X(f) is a complex number, which can be decomposed into amplitude spectrum and phase spectrum. The amplitude spectrum |X(f)| describes the intensity of the signal at different frequencies and is often used to analyze the frequency components of the signal. The phase spectrum arg(X(f)) describes the phase information of each frequency component.

[0054] S134, converting the spectrum graph to a Mel frequency scale, which simulates the nonlinear perception of human ears to frequencies, thereby emphasizing the frequency region to which the human ears are sensitive, obtaining a Mel spectrum, and performing a logarithmic operation on the Mel spectrum to convert the amplitude characteristics into energy characteristics;

[0055] S135, obtaining Mel frequency cepstral coefficients by discrete cosine transform. In this embodiment, the first 13 coefficients are retained as audio features to reduce the dimension.

[0056] Among them, the calculation formula of Mel frequency cepstrum coefficient is as follows:

[0057]

[0058] In the formula, FFT k represents the frequency domain signal; h mn represents the coefficients of the Mel filter bank; w m represents weight, K represents the number of frequency bands; M represents the number of Mel filters; MFCC n represents the Mel-frequency cepstral coefficient;

[0059] S136, using Mel frequency cepstral coefficients MFCC n Captures the timbre characteristics of audio data, including pitch (for identifying pitch changes and analyzing melody), beat (for identifying the rhythm and speed of music), and Mel-spectrogram (for analyzing the structure and characteristics of music); Mel-spectrogram can intuitively display the frequency content of audio signals over time, providing deep understanding of harmonic and percussive elements.

[0060] S14, using a support vector machine to classify the timbre features according to the material in combination with the type of musical instrument being played, and storing the classification results in a database.

[0061] Step S14 specifically includes the following steps:

[0062] S141, performing standardization processing on the extracted timbre features to ensure that all timbre features are compared on the same scale;

[0063] S142, dividing the standardized timbre feature data into a training set and a test set;

[0064] S143, using the training set data to train the SVM model, and optimizing the penalty parameter C and kernel function parameters of the SVM model through cross-validation and grid search;

[0065] Among them, the SVM model formula is as follows:

[0066]

[0067] In the formula, α i represents the Lagrange multiplier; y i represents the label; K(x i ,x) represents the kernel function; b represents the bias;

[0068] S144. Input the test set into the trained SVM model for classification. The goal of SVM is to find a hyperplane to maximize the interval between samples of different categories. The training process includes adjusting hyperparameters, such as the regularization parameter (C) and the parameters of the kernel function (such as γ of the RBF kernel), and calculating the accuracy, precision, recall and F1-score to evaluate the classification effect of the SVM model.

[0069] SVM model classification can efficiently analyze and classify Chu music audio data, automatically identify different timbres, and provide accurate data support for various music analysis applications.

[0070] In this embodiment, the SVM (support vector machine) model is used to classify music. Combined with the types of instruments used in Chu music, the music is divided into eight categories according to the material (gold, stone, earth, leather, silk, wood, gourd, and bamboo). Finally, the processed and classified data is stored in the Chu music database.

[0071] S2, based on classical music video data, use the SVM model to extract video features, generate dance movements, and search for the best combination of dance movements and performance movements;

[0072] Step S2 specifically includes the following steps:

[0073] S21. Collecting video data of classical music

[0074] S22, data preprocessing: performing cropping, segmentation and denoising on the video data in sequence;

[0075] S23, reading the video frame by frame, and converting the video into continuous frame images;

[0076] S24, extracting optical flow features to obtain dynamic features: by calculating the optical flow between frames, the amplitude and direction of the dancer's movement in the image are obtained, and the amplitude and direction data are converted into histogram features;

[0077] Optical flow feature extraction is a technique used to capture the motion changes between video frames. Optical flow can help identify the direction and speed of movement of body parts in dance, so the design uses optical flow to extract the motion features between each frame and its subsequent frames. By calculating the optical flow between frames, the amplitude and direction of the movement can be obtained, and then the above information is converted into histogram features for subsequent classification processing. It can be seen that optical flow features are particularly suitable for capturing fast-changing movements in dance, such as dancers' rotations and jumps.

[0078] S25, using a corner point detection algorithm to extract the number and distribution of corner points in each frame of the image, and using the number and distribution of corner points as key point features to obtain static features of the dancer's dance movements in the image;

[0079] Key point feature extraction is a method used to identify important visual features in a frame. By detecting key points in a frame (such as corners and edges), it helps identify the dancer's posture, gestures, and movement starting points, thereby obtaining static features of dance movements.

[0080] S26, integrating optical flow features and key point features into a video feature vector to provide rich information for classification;

[0081] S27, using the SVM model to classify the integrated video feature vectors and generate dance movements;

[0082] In this embodiment, human posture estimation can also be performed: through video processing technology, the system uses a human posture estimation algorithm to extract key point information of dance movements. These key points include major joints and limb positions, such as shoulders, elbows, knees, etc.

[0083] Movement rhythm and emotional expression: The system analyzes the user's movement rhythm and emotional expression. By comparing the key points and speed of the dance movements, the system can evaluate whether the user's movements match the rhythm of the music. At the same time, the system also analyzes whether the user's emotional expression matches the emotional tone of the music.

[0084] S28. Calculate the accuracy, precision, recall and F1-score to evaluate the classification effect of the SVM model.

[0085] S3. Generate a virtual scene model using the ResNet50 model based on historical information data;

[0086] Step S3 specifically includes the following steps:

[0087] S31, collecting historical image information and historical text information as historical information;

[0088] S32, parsing the historical text information using semantic analysis, and identifying element information in the historical image information using a ResNet50 model, so that the historical text information matches the historical image information;

[0089] The element information includes scene layout, architectural style, clothing patterns, and musical instrument types, and the ResNet50 model formula is as follows:

[0090] Output=Activation(Convolution(Input×Kernel)+Bias) (4)

[0091] In the formula, Output represents output; Convolution represents convolution operation; Activation represents activation function; Input represents input; Kernel represents convolution kernel, which is used to extract features from historical image information; Bias represents bias term.

[0092] Therefore, the convolutional layer of ResNet50 can be used to extract low-level features in the image (such as edges and textures), while subsequent layers extract higher-level features (such as joint features, movement details, and the spatial layout of the scene). On this basis, the system uses ResNet50 to identify and track the user's dance posture in real time. The deep network structure of ResNet50 enables it to capture complex dance movement features, including the speed, posture, and rhythm of the movement. Through the deep network structure of ResNet50, the positions of various key points of the human body (such as the head, shoulders, elbows, knees, etc.) are captured. The human body features extracted by ResNet50 are used to analyze the user's movement changes and rhythm, and the user's real-time movements are combined with the virtual model in the database for matching and integration.

[0093] In the construction of the database, the system not only relies on image analysis, but also conducts in-depth analysis of historical documents through semantic analysis technology to ensure that the information in the images and documents can accurately correspond. For example, the system performs natural language processing (NLP) on the descriptive text in the documents and matches it with the visual elements in the images, thereby building a richer database of Chu music culture. This semantic correspondence mechanism allows the system to refer not only to images when generating virtual scenes, but also to combine the historical background and description of the documents, ensuring that the generated music figurines and scene interaction interfaces are both authentic and in line with historical culture.

[0094] In this embodiment, the extracted features are used to construct a 3D model data storing various Chu music tomb scenes and Chu people's costumes (the database includes the historical background of Chu music, environmental details and unique designs of costumes). These 3D models are reconstructed and annotated with high-precision data to generate realistic Chu music culture scenes, which not only show the history and culture of Chu music, but also provide users with an immersive audio-visual experience, allowing them to learn and explore in the environment.

[0095] The system also integrates the information in the above historical images and documents in a multimodal manner to enrich the content of the database and provide strong data support for the subsequent dissemination and research of Chu music culture. Through the establishment of this database, users can not only experience realistic virtual scenes during the interaction process, but also have a deeper understanding of the historical background and artistic value of Chu music culture through the diverse content provided by the system.

[0096] S4, using a multimodal deep learning algorithm to perform multimodal fusion on the optimal combination of dance movements and performance movements and the virtual scene model, generate a three-dimensional model, and store the three-dimensional model in a database;

[0097] In step S4, the multimodal fusion formula is:

[0098] CombinedFeature=Concatenate(AudioFeatures,VideoFeatures) (5)

[0099] In the formula, CombinedFeature represents the multimodal fusion result; AudioFeatures represents the video features of user feedback; VideoFeatures represents the timbre features of user feedback; Concatenate represents the concatenation of the timbre features VideoFeatures of user feedback and the video features AudioFeatures of user feedback.

[0100] like Figure 2 As shown, S5 uses a camera to capture the user's dance movements and inputs them into a ResNet50 model to generate a human skeleton model. The generated human skeleton model is matched and integrated with the standard skeleton stored in the three-dimensional model to achieve an interactive experience.

[0101] In this example, ResNet50: main class represents the entire ResNet50 network, including input, output, and core components.

[0102] ResidualBlock: Represents a residual block, which includes a convolutional layer, a batch normalization layer, a ReLU activation function, and a skip connection.

[0103] ConvolutionalLayer: represents the convolutional layer.

[0104] BatchNormalization: represents the batch normalization layer.

[0105] ReluActivation: represents the ReLU activation function.

[0106] SkipConnection: represents a skip connection, which is used for residual learning.

[0107] The deep network structure of the ResNet50 model can be used to capture complex dance movement features, including speed, posture, and rhythm.

[0108] In step S5, when the user is a single user;

[0109] It specifically includes the following steps:

[0110] S51, using a camera to collect dance movement images of the user, and inputting the dance movement images into a ResNet50 model;

[0111] S52. In the ResNet50 model, the human skeleton key point detection method is used to compare the user's skeleton point connection line with the standard skeleton point connection line. The human skeleton key point detection formula is as follows:

[0112] Keypoints = fImage) (6)

[0113] In the formula, Keypoints represents the key points extracted from the user's dance action image; f represents the posture estimation model; Image represents the input user's dance action image;

[0114] Among them, multiple music data and dance movements in the database are combined in pairs, and the optimal combination is searched out as the standard skeleton;

[0115] S53, generating a human skeleton model using the extracted human skeleton key points;

[0116] S54, scoring the user's dance movements based on a comprehensive score of accuracy, fluency, and coordination by comparing the generated human skeleton model with a standard skeleton of the human skeleton model;

[0117] Among them, accuracy evaluates whether the user's movements are consistent with standard movements. This includes whether the position of the skeleton key points meets the expected standards and whether the movements are completed within the specified time frame. The system evaluates accuracy by comparing the user's real-time key point data with the standard skeleton data in the database. For example, if the user's arm does not reach the standard position when making a waving motion, the system will give a low accuracy score.

[0118] Skeletal key point position: The system scores the user's skeletal key points based on how well they match the standard position. The deviation of each key point is calculated as the Euclidean distance between the actual position and the standard position. The smaller the deviation, the higher the score. The specific accuracy calculation formula is as follows:

[0119]

[0120] Where N represents the number of key points of human skeleton; d i Represents the deviation of the i-th human skeleton key point; Z is a constant used to adjust the scoring range;

[0121] In this embodiment, the system calculates the deviation between the time the user completes the action and the standard action time. If the action is completed within the specified time, full marks are awarded; otherwise, marks are deducted.

[0122] Time deviation score: 100 points. When the deviation exceeds the set threshold, points will be deducted according to a linear relationship.

[0123] Fluency evaluates the continuity and naturalness of the movements. This involves whether the transitions between movements are smooth, without noticeable interruptions or jumps. The system uses motion trajectory analysis of key points to measure the smoothness of the movements. If the user exhibits jerky or incoherent changes when performing a series of movements, the system will identify these issues and reflect them in the fluency score.

[0124] Action continuity: The system analyzes the motion trajectory of key points and evaluates the smoothness of the action. Therefore, by calculating the rate of change of the trajectory curve, the system can determine whether the action is smooth. The specific smoothness calculation formula is as follows:

[0125]

[0126] Where M represents the number of segments of the dance trajectory; Δv j represents the speed change of the j-th dance trajectory; L is a constant;

[0127] In this embodiment, transition smoothness: By analyzing the changes in speed and acceleration between actions, the system evaluates the natural transition of actions. The smoother the transition, the higher the score.

[0128] Smoothness deviation score: 100 points, with deductions based on drastic changes in speed and acceleration.

[0129] Coordination evaluates how well different parts of the user's body work together when performing an action. The system analyzes the synchronization of each joint and limb to ensure that the overall performance of the action is as expected. For example, when performing a rotation, the user's body should maintain a consistent rotation speed and direction. If the user's arms and legs are not synchronized, the coordination score will be affected.

[0130] Limb synchronization: The system evaluates coordination by calculating the synchronization of movements of different body parts, and the higher the synchronization, the higher the score. The specific coordination calculation formula is as follows:

[0131]

[0132] Where P represents the number of body part pairs; S p represents the synchronization error of the pth pairing; C is a constant;

[0133] Consistency of rotational movement in this embodiment: When performing a rotational movement, the system monitors the consistency of the rotational speed and direction of each part, and the smaller the deviation, the higher the score.

[0134] Rotation consistency score: 100 points, deviation is deducted according to the linear relationship.

[0135] The comprehensive score calculation formula is as follows:

[0136] Total score = 0.4 × accuracy score + 0.3 × fluency score + 0.3 × coordination score;

[0137] S55. Provide user feedback and visual guidance based on the comprehensive score.

[0138] Instant feedback means that when the system detects a deviation in the user's movements, it will immediately display specific error information through the interface. For example, if the user's posture is incorrect, the system will point out the specific joint position and adjustment suggestions.

[0139] Visual guidance is to show the difference between the user's action and the standard action through a graphical interface, for example, using animation or marking to show the deviation of the user's key point position from the standard position. In this way, the user can intuitively see the part that needs to be improved.

[0140] This embodiment also provides action correction tips: providing correction suggestions for specific errors, such as how to adjust the arm angle, foot position, etc. The system can provide specific steps and techniques based on the type and severity of the error to help users make adjustments and improvements.

[0141] In this embodiment, a phased goal setting function is also provided: the system also includes a phased goal setting function, which is designed to set specific and measurable goals for users to help them gradually improve their dancing skills. Goals are divided into three categories: short-term, medium-term, and long-term, such as completing a certain number of fluency exercises within a week, or improving the accuracy of a specific action to more than 90% within a month. Long-term goals may include reaching the level of advanced dance skills within three months and being able to complete the entire dance smoothly. By setting these phased goals, users can clearly see their progress and continue to improve their dancing skills driven by their goals.

[0142] This embodiment also has a training data analysis function: the system analyzes the recorded training data of the user, including the score of each action, the progress curve, and the training frequency, etc. It can identify the user's long-term trends and room for improvement, and generate a detailed training report. For example, the system will show how much the accuracy has improved and how the fluency has improved in the past month. The report also includes the identification of room for improvement. Based on the data analysis results, the system will provide specific improvement suggestions, such as suggesting the addition of specific types of training, so that users can improve in a targeted manner.

[0143] In this embodiment, a skill improvement function is also provided: the system provides a skill improvement strategy to help users improve their dance skills through a series of learning resources. These resources include tutorials, video demonstrations, and expert advice, which can recommend the most appropriate learning content based on the specific needs of users. For example, for users with insufficient fluency, the system may recommend expert video demonstrations on how to keep movements smooth. In addition, the system encourages users to combine theoretical learning with practical practice, watch tutorial videos, and imitate movements to achieve better learning results. Through these strategies, the system can fully support the user's dance learning process and improve their skill level and learning experience.

[0144] In step S55, in order to cope with the challenges in multi-person interaction, a fault tolerance mechanism is also set up, and the setting process is as follows:

[0145] First, set the key point position formula of the dance action as follows:

[0146] Keypoints = PoseEstimation(I t ) (10)

[0147] In the formula, Keypoints represents the set of key points of the human body; I t represents the input image sequence at time step t; PoseEstimation represents the human pose estimation algorithm;

[0148] Then, by analyzing the key point trajectory, the long short-term memory network (LSTM) is used to evaluate the user's action performance, and the fault tolerance mechanism calculation formula is set as follows:

[0149] Score = LSTM (Keypoints) (11)

[0150] In the formula, Score represents the scoring result; LSTM represents the long short-term memory network algorithm;

[0151] In step S55, task allocation is also set. First, the system will dynamically allocate tasks according to each user's performance, skill level and current status. For example, after analyzing each user's learning progress and action performance in real time, the system will allocate tasks of moderate difficulty to suitable users to ensure that each user's role and task in the interaction can match their ability. And the task allocation formula is as follows:

[0152] Task i,j =AssignTask(Skil i ,Performance i ,Status i ) (12)

[0153] In the formula, Task i,jrepresents the jth task assigned to user i; Skil i is the skill level of user i; Performance i It represents the real-time comprehensive score of user i; Status i represents the current state of user i (such as fatigue, concentration, etc.); AssignTask represents the task assignment algorithm based on three parameters;

[0154] In the system's multi-person interaction function, task allocation and fault tolerance mechanisms are important links to ensure a smooth collaborative experience.

[0155] In step S55, during the task allocation process, the system will consider the synergy between multiple users to ensure the balance and rationality of task allocation. If it is detected that a user has too much task or performs poorly, the system will dynamically adjust the task load to balance the workload of the entire team. The formula for task load adjustment is:

[0156] LoadAdjustment i =AdjustLoad(TaskLoad i ,TeamPerformance) (13)

[0157] In the formula, LoadAdjustment i Indicates the task load adjustment for user i; TaskLoad i represents the task load of user i; TeamPerformance represents the overall comprehensive score of the entire team; AdjustLoad represents the task load adjustment algorithm based on the task load and the team comprehensive score;

[0158] In step S55, a delay process is also provided to eliminate the delay by using an adaptive filtering technique, and the delay process formula is:

[0159] FilteredSignal i =AdaptiveFilter(Signal i ,Delay i ) (14)

[0160] Where FilteredSignal i Represents the signal data of user i after adaptive filtering; Signal i Represents the original signal data of user i; Delay i Represents the detected delay time; AdaptiveFilter represents an adaptive filter function, which is used for delay processing based on the original signal and the delay time;

[0161] In step S55, redundancy detection is also provided to deal with delay, noise and inconsistent motion problems to ensure the smoothness of user experience, and the formula for redundancy detection is expressed as:

[0162]

[0163] In the formula, RedundantDetection i Represents the redundant detection result of user i; KeypointDetection i,k is the kth key point detection result of user i; n is the total number of key points;

[0164] By detecting and cross-validating multiple key points, the system is able to reduce the impact of a single false detection, thereby improving the overall reliability of detection.

[0165] In step S55, personalized coordination adjustment is also provided. According to the user's learning status, the system can provide customized musical instrument and dance coordination suggestions. Using collaborative filtering technology, the formula for setting personalized coordination adjustment is expressed as:

[0166] Feedback=ColaborativeFiltering(UserHistory) (16)

[0167] In the formula, UserHistory represents the user's historical learning data, and Feedback represents the personalized learning suggestions provided after the user finishes learning;

[0168] For example, when a user performs poorly on a specific dance move, the system can adjust the rhythm of the music or provide guidance on modifying the moves to help the user gradually improve. In addition, the system also adjusts the practice content and difficulty based on the user's learning progress. For beginners, the system may choose relatively simple music and dance moves for coordinated training, while for advanced users, more complex rhythms and movement requirements will be introduced to increase the challenge of practice.

[0169] In step S55, during the user learning process, a process adjustment is also provided, and the formula is as follows:

[0170] Guidance=GuidanceModel(Keypoints,UserProgress) (17)

[0171] In the formula, Guidance represents the personalized guidance suggestions provided during the user's learning process; UserProgress represents the user's learning progress;

[0172] For example, the system's personalized training suggestion function is designed to automatically identify the user's weaknesses and provide targeted training programs based on their performance in the movement scoring. For example, when a user scores low in the fluency score, the system will recommend a series of training programs focused on improving fluency, including specific movement exercises, rhythm exercises, and continuity training for the entire set of movements. If the user's coordination is insufficient, the system will recommend synchronization training to improve coordination by focusing on the synchronous movements of the arms and legs. For accuracy issues, the system may recommend exercises that gradually decompose the movements to ensure that each key movement is accurate. Thereby helping users make progress in specific areas.

[0173] In step S55, reinforcement learning is also used to customize the optimal strategy for the user according to the user's current skill level and learning progress:

[0174]

[0175] In the formula, π * (s) represents the optimal strategy; Q(s,a) represents the state-action value function; r represents the reward; a represents the learning rate; γ represents the discount factor;

[0176] In step S55, historical learning data is analyzed to identify the user's error patterns, and an improvement strategy is customized using a cluster analysis method:

[0177] Clusters=KMeans(HistoricalData) (19)

[0178] In the formula, Clusters represents the recognized action mode; HistoricalData represents the user's historical learning data;

[0179] In step S55, the adaptive signal processing technology is also used to analyze the user's performance and learning status in real time and dynamically adjust the music parameters:

[0180] MusicParameters=AdaptiveAdjustment(Keypoints,Score) (20)

[0181] Where MusicParameters represents the adjusted music parameters; Keypoints and Score represent the real-time key point data and action score respectively;

[0182] In step S55, the optimal learning strategy combination is searched by Bayesian optimization method according to the user's feedback and performance:

[0183]

[0184] R(θ)~GaussianProcess(m(θ),k(θ,θ')) (21)

[0185] In the formula, θ * represents the optimal learning strategy combination parameter; R(θ) represents the strategy return; m(θ) and k(θ,θ') represent the mean and kernel function of the Gaussian process respectively;

[0186] For example, by analyzing historical learning data, the system can identify common error patterns of users and provide targeted improvement strategies accordingly. The system also continuously optimizes and updates the cooperation strategy based on user feedback and performance to ensure that users can get the most effective learning experience.

[0187] In step S55, based on the dance key point information, the user's learning progress and performance trend are tracked through the long short-term memory network, where the long short-term memory network formula is as follows:

[0188] Calculation of the forget gate: f t =σ(W f ·[h t-1 ,x t ]+b f )

[0189] Calculation of input gate: i t =σ(W i ·[h t-1 ,x t ]+b i )

[0190] Update candidate cell status:

[0191] Update cell status:

[0192] Output gate calculation: o t =σ(W o ·[h t-1 ,x t ]+b o )

[0193] Calculation of hidden state: h t =o t tanh(C t )

[0194] In the formula, f t represents the output of the forget gate, whose value is between 0 and 1, indicating how much of the cell state at the previous moment should be retained at the current moment; σ represents the Sigmoid function, which is used to compress the output value to the range of [0,1]; Wf represents the weight matrix of the forget gate; h t-1 represents the hidden state of the previous moment, which contains the input information of the previous moment; x t Represents the input data at the current moment; b f Represents the bias term of the forget gate; i t Represents the output of the input gate, which is used to indicate the importance of the input information at the current moment, and its value is between 0 and 1; W i represents the weight matrix of the input gate; b i represents the bias term of the input gate; represents the candidate cell state, which is used to represent the candidate memory content at the current moment; W C The weight matrix representing the candidate cell state; b C represents the bias term of the candidate cell state; C t , C t-1 Respectively represent the cell state at the current moment (which combines the cell state at the previous moment and the input information at the current moment) and the cell state at the previous moment; o t represents the output of the output gate, which is used to control the influence of the current cell state on the hidden state ht; b o Represents the bias term of the output gate.

[0195] In step S55 of the present embodiment, a synchronization and coordination mechanism is also introduced, through a time synchronization mechanism, such as network clock synchronization technology, or using music beats to unify the action rhythm of all users. This mechanism also combines real-time correction and feedback of posture and action to ensure that the dance movements of multiple users in the virtual scene can be highly consistent. In addition, in terms of personalized adjustment of individual interaction effects, the system will dynamically adjust the audio and video effects according to the real-time performance of each user. For example, when a user's performance is outstanding, the system will allocate more virtual instruments to him or her, or let him or her occupy a more important position in the scene to enhance the individual's sense of participation.

[0196] In this embodiment, the system also uses collaborative filtering or content recommendation algorithms to provide users with personalized special practice suggestions based on the user's historical performance and real-time data. When the system detects that the user's performance on a certain action is poor, the corresponding reinforcement practice resources in the database will be retrieved and recommended to the user. In addition, the system provides an intelligent retrieval function that allows users to query relevant learning content based on personal interests and needs. Through natural language processing (NLP) technology, the system can understand the user's retrieval intention and generate a personalized training plan based on the user's choice to ensure that the recommended content matches the user's learning goals and interests. In this way, the system not only helps users improve their skills in specific actions, but also provides a flexible learning path, enhancing the sense of participation and achievement in learning.

[0197] In step S11, the audio data is recognized using a generative adversarial network, and the generative adversarial network expression is as follows:

[0198] Loss D =-[log(D(x))+log(1-D(G(z)))] (22)

[0199] Loss G = -log(D(G(z))) (23)

[0200] In the formula, D represents the discriminator, G represents the generator, x represents the real sample, and z represents random noise;

[0201] Generative Adversarial Networks (GANs) are an unsupervised learning method consisting of a generator (G) and a discriminator (D). GANs continuously optimize the model through an adversarial process to generate more realistic samples. Its goal is to make the data generated by the generator look as real as possible to deceive the discriminator; while the discriminator is committed to distinguishing between real samples and generated samples, thereby enhancing the robustness of the model by generating audio data with different noise levels and interference.

[0202] In step S21, Q-Learning is used to recognize video data, and the Q-Learning expression is as follows:

[0203]

[0204] In the formula, a represents the learning rate; r represents the reward; γ represents the discount factor; s and s' represent the current state and the next state respectively; a and a' both represent actions; Q(s,a) represents the Q value of executing action a in the current state s; It represents the maximum Q value of all possible actions a' in the next state s', which is used to represent the expected return that can be obtained by taking the best action in the next state s', reflecting the estimation of future benefits.

[0205] When applying Q-Learning (reinforcement learning) for dance movement recognition, the task is first defined as a reinforcement learning environment, and the goal of the system is to optimize the accuracy and fluency of dance movements. The reward mechanism is designed to encourage the model to maintain the stability of dance movements under noise and interference conditions. The reward may be evaluated based on the accuracy, stability, and synchronization with the music rhythm of the movement. The model continuously interacts with the environment, receives feedback (rewards or penalties), and adjusts its strategy accordingly to optimize the performance of dance movements. Through this continuous learning and strategy adjustment, the reinforcement learning model can adapt to different environmental conditions and ensure the stability and accuracy of the movement under interference.

[0206] At the same time, in this embodiment, the system has the ability to automatically adjust dance movements according to the rhythm and intensity of the music to ensure the coordination of dance and musical instruments. That is, through deep learning models such as recurrent neural networks (RNNs) or long short-term memory networks (LSTMs), the above model can analyze the rhythm changes of Chu Music in real time, predict future beats, and adjust dance movements according to the dynamics of the music. At the same time, the database stores a variety of combinations of music and dance movements, from which the system can extract and match the most appropriate movements to match the music rhythm. Through this adaptive adjustment mechanism, the system can provide a more smooth and natural interactive experience, ensuring performance stability and coordination in various scenarios and environments.

[0207] The system of the classical music multi-person interactive experience method based on image recognition technology includes:

[0208] An information collection module, used to collect audio data, video data and historical information of classical music;

[0209] An image evaluation and classification module for generating dance moves based on the collected video data;

[0210] A timbre feature extraction module, used to generate timbre features based on the collected audio data;

[0211] A virtual scene generation module is used to generate virtual scenes based on the collected historical information (e.g., by analyzing the distribution of ancient scenes and costumes of characters in Chu music performances, generating corresponding music figurines and scene interaction interfaces;

[0212] Multimodal data fusion module, used to perform multimodal fusion of dance movements, timbre features and virtual scenes to generate a three-dimensional model;

[0213] A database for storing audio data, video data, historical information, and 3D models of classical music;

[0214] In this embodiment, MongoDB is selected to store data, and the index structure is used to improve the retrieval efficiency of data. The MongoDB database not only stores audio and video files, but also stores related classification information and feature data. Using the MongoDB database, further analysis and matching can be performed, and music recommendations and similarity analysis can be performed through similarity calculation. This ensures efficient management and utilization of the database, providing a reliable foundation for subsequent music research and application.

[0215] Camera, used to monitor the user's dance movements in real time;

[0216] A user motion recognition module, used to recognize the user's human skeleton based on the user's dance motion;

[0217] The real-time feedback and guidance module is used to provide feedback and guide the user's actions based on the comparison results between the user's human skeleton and the standard skeleton in the database.

[0218] Simulation experiment

[0219] In this simulation experiment, a comprehensive Chu music database is established. The audio and video data of Chu music are collected from various sources, including recordings, live performances, and traditional music materials, to ensure the comprehensiveness and diversity of the database. The sources of the above data are rich and diverse, covering the performances of Chu music in different regions, eras, and styles.

[0220] After data collection, the collected audio and video data was subjected to a series of preprocessing tasks, including cropping and segmentation, to improve data processability and storage efficiency. By splitting long clips into manageable small clips, subsequent analysis can be more efficient. In addition, the resolution of the video data was also adjusted to reduce storage space requirements and speed up processing.

[0221] In order to further improve the quality of audio and video data in the Chule database, a variety of denoising techniques are used to improve audio clarity and video visual effects. In terms of audio denoising, high-pass and low-pass filters are used to remove low-frequency and high-frequency noise, such as wind noise, microphone noise, and current noise. At the same time, spectral subtraction, as a core technology, accurately estimates and subtracts background noise through fast Fourier transform (FFT) to ensure the purity of the audio signal.

[0222] In view of the unique rhythm, timbre and dance performance of Chu music, multi-level special technical processing was carried out. Chu music is famous for its deep drum sounds, melodious orchestral sounds and highly penetrating sheng sounds. In order to preserve these characteristics, the frequency bands where the drum beats and sheng sounds are located were accurately retained during the denoising process. At the same time, the relevant video clips were personalized according to the unique movement rhythm in the Chu music dance performance. For example, the slow and powerful gestures and steps common in Chu music dance are enhanced in smoothness and coherence through time domain filtering technology. In addition, when processing the group arrangement and overall formation in the dance, special attention is paid to the clarity and consistency of the picture to ensure the synchronization of multiple movements and the restoration of performance details.

[0223] like Figure 3 As shown in the figure, in terms of audio processing, special attention is paid to the subtle timbre changes in Chu music, such as the alternation between high and low frequencies of bamboo flute and Xiao. Through adaptive filtering technology, the denoising intensity can be dynamically adjusted to retain these subtle timbre changes and ensure the complete presentation of Chu music.

[0224] Through these sophisticated collection, preprocessing, denoising and special technical processing based on the characteristics of Chu music and dance, not only the quality of audio and video data in the Chu music database is improved, but also the diversity, accuracy and artistic expression of the data are ensured, providing a reliable high-quality data foundation for subsequent music and dance analysis research.

[0225] Then the algorithm of the present invention is used to process the collected audio data, video data ("Chu Music and Dance") and historical information. In this process, Figure 4 As shown in the figure, ResNet50 is used to process the input image to identify and classify different elements in the image (such as architectural style, clothing, etc.). The deep network of ResNet50 can capture complex features, such as the spatial layout of the scene and the details of the clothing. During the recognition process, the system matches and integrates the user's dance posture features with the scene and virtual music model in the database to provide personalized virtual background and interactive content.

[0226] In the analysis of the Chu Music Dance video, the evaluation results can help understand the model's ability to recognize different action categories. The design uses confusion matrices and classification reports to show the details of the classification results. In addition, through dimensionality reduction and visualization techniques (such as principal component analysis PCA), the high-dimensional feature space can be mapped to Figure 5 The two-dimensional plane shown is convenient for visually observing the feature distribution and classification results.

[0227] The video classification labels are as follows: 1032 still images were successfully captured from the video, with slow-paced low motion, slow-paced high motion, fast-paced low motion, and fast-paced high motion accounting for 19%, 30%, 28%, and 23% respectively. This enables efficient identification of the complex dance movements in the "Chu Music Dance" video, providing a practical and effective technical solution for dance analysis and classification.

[0228] By using the method of the present invention, users can select the difficulty level of the dance recommended by the system according to their own situation. For beginners, the system recommends relatively simple music and dance movements for coordinated training; for advanced users, the system introduces more complex rhythms and movement requirements to increase the challenge. Once the user makes a choice, the system matches and integrates the user's dance posture characteristics with the scenes in the database and the Chu State Music Figurine Model, providing Figure 6 The personalized virtual background and interactive content shown in the video enable users to experience the environment and cultural charm of ancient music performance in an immersive way.

[0229] User performance analysis and recommendation: Using this database, the system can conduct further analysis and matching, and perform music recommendations and similarity analysis through similarity calculations.

[0230] Therefore, the present invention adopts the above-mentioned classical music multi-person interactive experience method and system based on image recognition technology, which can help users more comprehensively understand and intuitively contact classical music culture, improve the classification accuracy of the model, and provide users with a better audio-visual experience.

[0231] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for multi-person interactive experience of classical music based on image recognition technology, characterized in that: The following steps are involved: S1. Based on the audio data of classical music, the Mel-frequency cepstrum coefficients are used to extract the timbre characteristics of various musical instruments and obtain the instrument playing movements; S2, based on classical music video data, use the SVM model to extract video features, generate dance movements, and search for the best combination of dance movements and performance movements; S3. Generate a virtual scene model using the ResNet50 model based on historical information data; S4, using a multimodal deep learning algorithm to perform multimodal fusion on the optimal combination of dance movements and performance movements and the virtual scene model, generate a three-dimensional model, and store the three-dimensional model in a database; S5. Use the camera to capture the user's dance movements and input them into the ResNet50 model to generate a human skeleton model. Match and integrate the generated human skeleton model with the standard skeleton stored in the three-dimensional model to achieve an interactive experience.

2. The method for multi-person interactive experience of classical music based on image recognition technology according to claim 1, characterized in that: Step S1 specifically includes the following steps: S11, collecting audio data of classical music; S12, data preprocessing: the audio data is cropped, segmented and denoised in sequence, wherein the denoising process is to first use high-pass and low-pass filters to denoise, and then use fast Fourier transform to remove background noise; S13, feature extraction: extracting timbre features in audio data using Mel frequency cepstrum coefficients; S14, using a support vector machine to classify the timbre features according to the material in combination with the type of musical instrument being played, and storing the classification results in a database.

3. The method for multi-person interactive experience of classical music based on image recognition technology according to claim 2, characterized in that: Step S13 specifically includes the following steps: S131, dividing the audio data into several short frames, each short frame is 20ms-40ms; S132, applying short-time Fourier transform to each short frame to convert the time domain signal into the frequency domain to obtain a spectrum diagram; Among them, the short-time Fourier transform formula is as follows: In the formula, x(t) represents the time domain signal, and X(f) represents the frequency domain signal; S133, decomposing the short-time Fourier transform result into an amplitude spectrum |X(f)| and a phase spectrum arg(X(f)), and using the amplitude spectrum |X(f)| to analyze the frequency components in the spectrum graph, and identifying the rhythm pattern and the emotional characteristics of the music; S134, converting the spectrum graph to a Mel frequency scale to obtain a Mel spectrum, and performing a logarithmic operation on the Mel spectrum to convert the amplitude characteristic into an energy characteristic; S135, obtaining Mel-frequency cepstrum coefficients by discrete cosine transform; Among them, the calculation formula of Mel frequency cepstrum coefficient is as follows: In the formula, FFT k represents the frequency domain signal; h mn represents the coefficients of the Mel filter bank; w m represents weight, K represents the number of frequency bands; M represents the number of Mel filters; MFCC n represents the Mel-frequency cepstral coefficient; S136, using Mel frequency cepstral coefficients MFCC n Capturing the timbre characteristics of audio data, where the timbre characteristics include pitch, beat, and Mel-spectrogram; Step S14 specifically includes the following steps: S141, performing standardization processing on the extracted timbre features to ensure that all timbre features are compared on the same scale; S142, dividing the standardized timbre feature data into a training set and a test set; S143, using the training set data to train the SVM model, and optimizing the penalty parameter C and kernel function parameters of the SVM model through cross-validation and grid search; Among them, the SVM model formula is as follows: In the formula, α i represents the Lagrange multiplier; y i represents the label; K(x i ,x) represents the kernel function; b represents the bias; S144. Input the test set into the trained SVM model for classification, and calculate the accuracy, precision, recall and F1-score to evaluate the classification effect of the SVM model.

4. The method for multi-person interactive experience of classical music based on image recognition technology according to claim 3 is characterized by: Step S2 specifically includes the following steps: S21. Collecting video data of classical music S22, data preprocessing: performing cropping, segmentation and denoising on the video data in sequence; S23, reading the video frame by frame, and converting the video into continuous frame images; S24, extracting optical flow features to obtain dynamic features: by calculating the optical flow between frames, the amplitude and direction of the dancer's movement in the image are obtained, and the amplitude and direction data are converted into histogram features; S25, using a corner point detection algorithm to extract the number and distribution of corner points in each frame of the image, and using the number and distribution of corner points as key point features to obtain static features of the dancer's dance movements in the image; S26, integrating the optical flow features and the key point features into a video feature vector; S27, using the SVM model to classify the integrated video feature vectors and generate dance movements; S28. Calculate the accuracy, precision, recall and F1-score to evaluate the classification effect of the SVM model.

5. The method for multi-person interactive experience of classical music based on image recognition technology according to claim 4 is characterized in that: Step S3 specifically includes the following steps: S31, collecting historical image information and historical text information as historical information; S32, parsing the historical text information using semantic analysis, and identifying element information in the historical image information using a ResNet50 model, so that the historical text information matches the historical image information; The element information includes scene layout, architectural style, clothing patterns, and musical instrument types, and the ResNet50 model formula is as follows: Output=Activation(Convolution(Input×Kernel)+Bias) (4) In the formula, Output represents output; Convolution represents convolution operation; Activation represents activation function; Input represents input; Kernel represents convolution kernel, which is used to extract features from historical image information; Bias represents bias term.

6. The method for multi-person interactive experience of classical music based on image recognition technology according to claim 5, characterized in that: In step S4, the multimodal fusion formula is: CombinedFeature=Concatenate ( AudioFeatures,VideoFeatures ) (5) In the formula, CombinedFeature represents the multimodal fusion result; AudioFeatures represents the video features of user feedback; VideoFeatures represents the timbre features of user feedback; Concatenate represents the concatenation of the timbre features VideoFeatures of user feedback and the video features AudioFeatures of user feedback.

7. The method for multi-person interactive experience of classical music based on image recognition technology according to claim 6, characterized in that: In step S5, when the user is a single user; It specifically includes the following steps: S51, using a camera to collect dance movement images of the user, and inputting the dance movement images into a ResNet50 model; S52. In the ResNet50 model, the human skeleton key point detection method is used to compare the user's skeleton point connection line with the standard skeleton point connection line. The human skeleton key point detection formula is as follows: Keypoints=fImage ) (6) In the formula, Keypoints represents the key points extracted from the user's dance action image; f represents the posture estimation model; Image represents the input user's dance action image; Among them, multiple music data and dance movements in the database are combined in pairs, and the optimal combination is searched out as the standard skeleton; S53, generating a human skeleton model using the extracted human skeleton key points; S54, scoring the user's dance movements based on a comprehensive score of accuracy, fluency, and coordination by comparing the generated human skeleton model with a standard skeleton of the human skeleton model; The accuracy calculation formula is as follows: Where N represents the number of key points of human skeleton; d i Represents the deviation of the key point of the i-th human skeleton; Z is a constant; The formula for calculating fluency is as follows: Where M represents the number of segments of the dance trajectory; Δv j represents the speed change of the j-th dance trajectory; L is a constant; The coordination degree calculation formula is as follows: Where P represents the number of body part pairs; S p represents the synchronization error of the pth pairing; C is a constant; The comprehensive score calculation formula is as follows: Total score = 0.4 × accuracy score + 0.3 × fluency score + 0.3 × coordination score; S55. Provide user feedback and visual guidance based on the comprehensive score.

8. The method for multi-person interactive experience of classical music based on image recognition technology according to claim 6, characterized in that: In step S55, a fault tolerance mechanism is also provided, and the setting process is as follows: First, set the key point position formula of the dance action as follows: Keypoints=PoseEstimation(I t ) (10) In the formula, Keypoints represents the set of key points of the human body; I t represents the input image sequence at time step t; PoseEstimation represents the human pose estimation algorithm; Then set the fault tolerance mechanism calculation formula as follows: Score=LSTM(Keypoints) (11) In the formula, Score represents the scoring result; LSTM represents the long short-term memory network algorithm; In step S55, task allocation is also provided, and the task allocation formula is as follows: Task i,j =AssignTask(Skill i ,Performance i ,Status i ) (12) In the formula, Task i,j represents the jth task assigned to user i; Skil i is the skill level of user i; Performance i It represents the real-time comprehensive score of user i; Status i represents the current state of user i; AssignTask represents the task assignment algorithm based on three parameters; In step S55, task load adjustment is also provided, and the formula for task load adjustment is: LoadAdjustment i =AdjustLoadTaskLoad i ,TeamPerformance) (13) In the formula, LoadAdjustment i Indicates the task load adjustment for user i; TaskLoad i represents the task load of user i; TeamPerformance represents the overall comprehensive score of the entire team; AdjustLoad represents the task load adjustment algorithm based on the task load and the team comprehensive score; In step S55, a delay process is also provided, and the formula for the delay process is: FilteredSignal i =AdaptiveFilter(Signal i ,Delay i ) (14) Where FilteredSignal i Represents the signal data of user i after adaptive filtering; Signal i Represents the original signal data of user i; Delay i Represents the detected delay time; AdaptiveFilter represents an adaptive filter function, which is used for delay processing based on the original signal and the delay time; In step S55, a redundancy check is also provided, and the formula of the redundancy check is expressed as: In the formula, RedundantDetection i represents the redundant detection result of user i; KeypointDetection i,k is the kth key point detection result of user i; n is the total number of key points; In step S55, a personalized matching adjustment is also provided, and the formula of the personalized matching adjustment is expressed as: Feedback=ColaborativeFiltering ( UserHistory ) (16) In the formula, UserHistory represents the user's historical learning data, and Feedback represents the personalized learning suggestions provided after the user finishes learning; In step S55, during the user learning process, a process adjustment is also provided, and the formula is as follows: Guidance=GuidanceModel(Keypoints,UserProgress ) (17) In the formula, Guidance represents the personalized guidance suggestions provided during the user's learning process; UserProgress indicates the user's learning progress; In step S55, reinforcement learning is also used to customize the optimal strategy for the user according to the user's current skill level and learning progress: In the formula, π * (s) represents the optimal strategy; Q(s,a) represents the state-action value function; r represents the reward; a represents the learning rate; γ represents the discount factor; In step S55, historical learning data is analyzed to identify the user's error patterns, and an improvement strategy is customized using a cluster analysis method: Clusters=KMeans(HistoricalData) (19) In the formula, Clusters represents the recognized action mode; HistoricalData represents the user's historical learning data; In step S55, the user's performance and learning status are analyzed in real time, and the music parameters are adjusted dynamically: MusicParameters=AdaptiveAdjustment(Keypoints,Score) (20) Where MusicParameters represents the adjusted music parameters; Keypoints and Score represent the real-time key point data and action score respectively; In step S55, the optimal learning strategy combination is searched by Bayesian optimization method according to the user's feedback and performance: R(θ)~GaussianProcess(m(θ),k(θ,θ')) (21) In the formula, θ * represents the optimal learning strategy combination parameter; R(θ) represents the strategy return; m(θ) and k(θ,θ') represent the mean and kernel function of the Gaussian process respectively; In step S55, based on the dance key point information, the user's learning progress and performance trend are tracked through the long short-term memory network, where the long short-term memory network formula is as follows: Calculation of the forget gate: f t =σ(W f ·[h t-1 ,x t ]+b f ) Calculation of input gate: i t =σ(W i ·[h t-1 ,x t ]+b i ) Update candidate cell status: Update cell status: Output gate calculation: o t =σ(W o ·[h t-1 ,x t ]+b o ) Calculation of hidden state: h t =o t tanh(C t ) In the formula, f t represents the output of the forget gate; σ represents the Sigmoid function; W f represents the weight matrix of the forget gate; h t-1 Indicates the hidden state of the previous moment; x t Represents the input data at the current moment; b f Represents the bias term of the forget gate; i t represents the output of the input gate; W i represents the weight matrix of the input gate; b i represents the bias term of the input gate; represents the candidate cell state; W C The weight matrix representing the candidate cell state; b C represents the bias term of the candidate cell state; C t , C t-1 Respectively represent the cell state at the current moment and the cell state at the previous moment; o t represents the output of the output gate; b o Represents the bias term of the output gate.

9. The method for multi-person interactive experience of classical music based on image recognition technology according to claim 8, characterized in that: In step S11, the audio data is recognized using a generative adversarial network, and the generative adversarial network expression is as follows: Loss D =-[log(D(x))+log(1-D(G(z)))] (22) Loss G =-log(D(G(z))) (23) In the formula, D represents the discriminator, G represents the generator, x represents the real sample, and z represents random noise; In step S21, Q-Learning is used to recognize video data, and the Q-Learning expression is as follows: In the formula, a represents the learning rate; r represents the reward; γ represents the discount factor; s and s' represent the current state and the next state respectively; a and a' both represent actions; Q(s,a) represents the Q value of executing action a in the current state s; It represents the maximum Q value of all possible actions a' in the next state s'.

10. A system for the method for multi-person interactive experience of classical music based on image recognition technology according to any one of claims 1 to 9, characterized in that: include: An information collection module, used to collect audio data, video data and historical information of classical music; An image evaluation and classification module for generating dance moves based on the collected video data; A timbre feature extraction module, used to generate timbre features based on the collected audio data; A virtual scene generation module, used to generate a virtual scene based on the collected historical information; Multimodal data fusion module, used to perform multimodal fusion of dance movements, timbre features and virtual scenes to generate a three-dimensional model; A database for storing audio data, video data, historical information, and 3D models of classical music; Camera, used to monitor the user's dance movements in real time; A user motion recognition module, used to recognize the user's human skeleton based on the user's dance movements; The real-time feedback and guidance module is used to provide feedback and guide the user's actions based on the comparison results between the user's human skeleton and the standard skeleton in the database.

Images