Music drama rehearsal multi-parameter backtracking analysis system and method thereof

Abstract

The invention relates to a music drama rehearsal multi-parameter backtracking analysis system and method, and belongs to the technical field of music drama rehearsal, the system comprises a rehearsal hall body, a data acquisition module, a topological feature extraction module, a multi-view fusion analysis module, a performance optimization module, a redisk database and a redisk engine, acoustic data, facial expression data, breathing data and action data of actors are collected through a sound field array microphone, an infrared marking camera, a pulse sensor and an all-dimensional camera array; constructing a performance feature space and a performance state manifold by using a topological data analysis technology, and extracting persistent topological features; performing consistency analysis on the topological characteristics of the multiple visual angles, and selecting an optimal visual angle combination; based on a performance quality evaluation model, an optimization path from a current performance state to a target state is calculated, a key breakthrough point is identified, special practice is designed, and the method realizes all-around, multi-dimensional and precise analysis and guidance of the music drama rehearsal process.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of musical drama rehearsal, in particular to a musical drama rehearsal multi-parameter backtracking analysis system and method thereof. BACKGROUND

[0002] Musical drama is a comprehensive performing art that combines music, drama, dance and other artistic forms. Its rehearsal process is complex and requires high coordination of vocal skills, expression and body movements of the actors. Traditional musical drama rehearsal mainly relies on the subjective experience evaluation of the director and vocal coach, lacks objective and quantitative analysis methods, resulting in low rehearsal efficiency and inconsistent performance quality.

[0003] In the prior art, there are some rehearsal assistance systems, such as sound analysis systems that can evaluate the pitch, tone and rhythm of the singer; video analysis systems that can capture the facial expressions and body movements of the actors; and breathing monitoring systems that can record the breathing patterns of the actors. However, these systems often run independently and can only analyze performance data in a single dimension, unable to achieve comprehensive analysis and correlation evaluation of multiple parameters. In addition, existing systems mostly use simple statistical methods or shallow machine learning models for data processing, lacking depth understanding of complex artistic expression and technical details in performance, making it difficult to provide accurate improvement suggestions.

[0004] In particular, the prior art fails to address the following key issues: first, the multi-dimensional fusion of performance data, which cannot comprehensively analyze the synergistic relationship between sound, expression, breathing and movement; second, the balance between individualization and standardization of performance evaluation, which is difficult to provide technical improvement guidance while preserving artistic individuality; third, the performance optimization path planning problem, lacking scientific and systematic improvement method guidance.

[0005] Therefore, there is an urgent need for a musical drama rehearsal system that can comprehensively analyze multiple parameters, objectively evaluate and provide accurate improvement guidance. SUMMARY

[0006] The purpose of the present application is to provide a musical drama rehearsal multi-parameter backtracking analysis system and method thereof, aiming to solve the problems of single performance data analysis, subjective evaluation and lack of accuracy in existing technology, and to realize comprehensive, multi-dimensional and precise analysis and guidance of the musical drama rehearsal process.

[0007] The present application proposes a musical drama rehearsal multi-parameter backtracking analysis system, which includes:

[0008] a rehearsal hall body;

[0009] A data collection module is arranged in the rehearsal hall body and configured to collect multi-dimensional performance data of actors during a rehearsal, wherein the multi-dimensional performance data includes acoustic data, facial expression data, breathing data and action data.

[0010] A topology feature extraction module is connected to the data collection module and configured to receive the multi-dimensional performance data, construct a performance feature space, and extract a topology structure feature of the performance feature space.

[0011] A multi-view fusion analysis module is connected to the topology feature extraction module and configured to receive the topology structure feature, perform consistency analysis on the topology structure feature from multiple views, and select an optimal view combination based on the consistency analysis result.

[0012] A performance optimization module is connected to the multi-view fusion analysis module and configured to construct a performance quality evaluation model based on the topology structure feature of the optimal view combination, and generate a performance optimization path.

[0013] A review database is connected to the performance optimization module and configured to store the topology structure feature, the performance quality evaluation model and the performance optimization path.

[0014] A review engine is connected to the review database and configured to generate a review report based on the data in the review database.

[0015] Preferably, the data collection module includes:

[0016] An acoustic field array microphone is configured to collect the acoustic data, wherein the acoustic data includes sound band vibration fundamental frequency stability, breath support and resonance cavity control data.

[0017] An infrared marker camera is configured to collect the facial expression data, wherein the facial expression data includes facial expression muscle movement trajectories.

[0018] A pulse sensor is configured to collect the breathing data, wherein the breathing data includes breathing fluctuations of the actor.

[0019] A full-view camera array is configured to collect the action data, wherein the action data includes stage actions and singing actions.

[0020] Preferably, the topology feature extraction module includes:

[0021] A feature vector construction unit is configured to convert the multi-dimensional performance data into a set of standardized feature vectors, wherein the set of feature vectors includes acoustic feature vectors, expression feature vectors, breathing feature vectors and action feature vectors.

[0022] A point cloud generation unit, connected with the feature vector construction unit, is configured to generate a performance state point cloud based on the set of standardized feature vectors;

[0023] A topological complex construction unit, connected with the point cloud generation unit, is configured to construct a topological complex based on the performance state point cloud;

[0024] A persistent feature calculation unit, connected with the topological complex construction unit, is configured to calculate persistent topological features based on the topological complex;

[0025] A performance manifold construction unit, connected with the persistent feature calculation unit, is configured to construct a performance state manifold based on the persistent topological features.

[0026] Preferably, the multi-view fusion analysis module comprises:

[0027] A view space definition unit is configured to establish a spatial distribution model of the collection views and define the coverage range and feature extraction advantage domain of each view;

[0028] A view data registration unit, connected with the view space definition unit, is configured to perform spatial registration on the multi-view data based on the common feature points;

[0029] A topological consistency evaluation unit, connected with the view data registration unit, is configured to calculate a consistency index of the multi-view topological features, the consistency index comprising structural consistency, timing consistency, feature consistency and dynamic consistency;

[0030] An optimal view selection unit, connected with the topological consistency evaluation unit, is configured to select an optimal view combination based on the consistency index and view complementarity analysis result;

[0031] A view fusion data generation unit, connected with the optimal view selection unit, is configured to generate fusion topological feature data based on the optimal view combination.

[0032] Preferably, the performance optimization module comprises:

[0033] A performance quality mapping unit is configured to establish a correspondence between the topological features and the performance quality scores;

[0034] A personalized benchmark unit, connected with the performance quality mapping unit, is configured to establish benchmark data of the actor's personal performance style and technical characteristics;

[0035] A quality manifold construction unit, connected with the personalized benchmark unit, is configured to construct quality contour lines in the performance state space, mark expert performance areas and problem performance areas;

[0036] An improved path planning unit connected with the quality manifold construction unit, for calculating an optimal path from the current performance state to the target state;

[0037] A key node identification unit connected with the improved path planning unit, for identifying key breakthrough points on the optimal path, and designing special technical exercises for each key breakthrough point.

[0038] As a preferred, the system further comprises:

[0039] A cloud service layer comprising a cloud interaction interface, a high-level analysis engine and a review database;

[0040] A stage control module connected with the cloud service layer, for receiving and displaying the review report, and controlling the rehearsal hall body;

[0041] A remote guidance module connected with the cloud service layer, for receiving training data of a remote guidance personnel, and transmitting the training data to the review engine for review training.

[0042] As a preferred, the feature vector construction unit performs multi-time scale sampling on the multi-dimensional performance data, and the multi-time scale comprises:

[0043] Micro scale, sampling interval is 2 to 10 milliseconds, for capturing sound micro changes;

[0044] Mesoscale, sampling interval is 50 to 200 milliseconds, for capturing expression changes;

[0045] Macro scale, sampling interval is 0.5 to 2 seconds, for capturing body movements and breathing patterns.

[0046] As a preferred, the topological consistency evaluation unit calculates the consistency index by the following steps:

[0047] Convert the topological features of each perspective to a unified reference space;

[0048] Establish a feature correspondence table to match the same performance elements captured from different perspectives;

[0049] Design a feature alignment algorithm to eliminate the deviation caused by perspective differences;

[0050] Calculate multi-level consistency indexes to generate an overall consistency index.

[0051] As a preferred, the key breakthrough point identification unit classifies the identified key breakthrough points, and the classification comprises:

[0052] Key breakthrough points, positions where performance skills need to be improved qualitatively;

[0053] Expressing turning points, the positions where artistic expression needs specific changes;

[0054] Cooperative change points, positions where multi-dimensional performance elements need to be adjusted simultaneously; and generating targeted practice recommendations based on the classification.

[0055] A music drama rehearsal multi-parameter backtracking analysis method, using the music drama rehearsal multi-parameter backtracking analysis system, comprising the following steps:

[0056] Collecting multi-dimensional performance data during the rehearsal process of the actors, the multi-dimensional performance data including acoustic data, facial expression data, breathing data, and action data;

[0057] Converting the multi-dimensional performance data into a standardized feature vector set, the feature vector set including acoustic feature vectors, expression feature vectors, breathing feature vectors, and action feature vectors;

[0058] Generating a performance state point cloud based on the standardized feature vector set;

[0059] Constructing a topological complex body based on the performance state point cloud;

[0060] Calculating persistent topological features based on the topological complex body;

[0061] Constructing a performance state manifold based on the persistent topological features;

[0062] Performing consistency analysis on topological features from multiple perspectives, and selecting an optimal perspective combination based on the consistency analysis results;

[0063] Constructing a performance quality evaluation model based on the topological features of the optimal perspective combination;

[0064] Calculating a performance optimization path from the current performance state to the target state based on the performance quality evaluation model;

[0065] Identifying key breakthrough points on the performance optimization path, and designing special technical exercises for each key breakthrough point;

[0066] Storing the persistent topological features, the performance quality evaluation model, and the performance optimization path to a review database;

[0067] Generating a review report based on the data in the review database.

[0068] The present application introduces topological data analysis technology innovatively, constructs a multi-dimensional performance feature space and a performance state manifold, realizes deep structure analysis of performance data, ensures the comprehensiveness and reliability of the analysis results through a multi-perspective topological consistency evaluation mechanism, and provides strong targeted technical improvement guidance through personalized performance optimization path planning.

[0069] The present application has the following advantages:

[0070] 1. Improve rehearsal efficiency: accurate positioning of technical problems, save troubleshooting time, average can shorten the rehearsal cycle by more than 40%; by providing phased goals and clear progress path, accelerate the skill improvement process.

[0071] 2. Enhance performance quality: through micro-level technical improvement guidance, improve the accuracy and consistency of performance; through the identification of key breakthrough points and special exercises, help actors overcome technical bottlenecks.

[0072] 3. Preserve artistic individuality: while providing technical improvement guidance, preserve the unique artistic style of the actor through personalized benchmarking mechanism, achieving a balance between technology and art.

[0073] 4. Support remote guidance: through cloud service architecture and remote guidance module, break through geographical limitations, making high-quality professional guidance resources widely available.

[0074] 5. Realize the digital preservation of performing arts: through structured storage of performance data and topology feature extraction, a new method is provided for the research, teaching and inheritance of performing arts. BRIEF DESCRIPTION OF DRAWINGS

[0075] Figure 1 The overall architecture of the music drama rehearsal multi-parameter backtracking analysis system of the present application is shown in the figure;

[0076] Figure 2 The structure of the topology feature extraction module of the present application is shown in the figure;

[0077] Figure 3 The structure of the multi-view fusion analysis module of the present application is shown in the figure;

[0078] Figure 4 The structure of the performance optimization module of the present application is shown in the figure. DETAILED DESCRIPTION

[0079] Please refer to Figures 1-4 , the present application will be further described in detail below in combination with the drawings and specific embodiments. It should be noted that the following examples are only used to illustrate the present application, but not to limit the present application.

[0080] As Figure 1 shown, the music drama rehearsal multi-parameter backtracking analysis system of the present application includes rehearsal hall body 100, data acquisition module 10, topology feature extraction module 20, multi-view fusion analysis module 30, performance optimization module 40, review database 50 and review engine 60.

[0081] The rehearsal hall body 100 is the physical environment of the system, which is the place for actors to rehearse and for the system to collect data. In the preferred embodiment of the present application, the rehearsal hall body 100 is equipped with professional acoustic treatment, providing a stable acoustic environment and reducing the interference of environmental noise on data collection.

[0082] The data collection module 10 is arranged in the rehearsal hall body 100 and is used to collect multi-dimensional performance data during the rehearsal of the actors. The topology feature extraction module 20 is connected with the data collection module 10, receives the multi-dimensional performance data, constructs a performance feature space, and extracts the topology structure features of the performance feature space. The multi-view fusion analysis module 30 is connected with the topology feature extraction module 20, receives the topology structure features, performs consistency analysis on the topology structure features from multiple views, and selects the optimal view combination based on the consistency analysis result. The performance optimization module 40 is connected with the multi-view fusion analysis module 30, constructs a performance quality evaluation model based on the topology structure features of the optimal view combination, and generates a performance optimization path. The review database 50 is connected with the performance optimization module 40 and stores the topology structure features, the performance quality evaluation model, and the performance optimization path. The review engine 60 is connected with the review database 50 and generates a review report based on the data in the review database 50.

[0083] Next, the structure and function of each module of the system will be described in detail.

[0084] As shown in Figure 1 , the data collection module 10 includes a sound field array microphone 11, an infrared marker camera 12, a pulse sensor 13, and a full-view camera array 14.

[0085] The sound field array microphone 11 is used to collect acoustic data, including vocal cord vibration fundamental frequency stability, breath support, and resonance cavity control data. In an embodiment of the present application, the sound field array microphone 11 consists of a ring-shaped array of 16 high-precision microphones, with a sampling frequency of 44.1 kHz and a 24-bit quantization precision, which can accurately capture sound details in the range of 100-8000 Hz, especially suitable for human voice analysis. Through sound field array technology, the system can separate the actor's voice from the environmental sound, improving the signal quality.

[0086] The infrared marker camera 12 is used to collect facial expression data, including facial expression muscle movement trajectories. Preferably, the infrared marker camera 12 adopts a 4K resolution, a frame rate of 60 fps, and is equipped with an infrared light supplement system, which can accurately capture facial micro-expressions under different light conditions. In practical applications, the system can identify 46 key breakthrough points on the actor's face, track their movement trajectories, and analyze expression changes.

[0087] The pulse sensor 13 is used to collect breathing data, including the breathing fluctuations of the actor. In the preferred embodiment of the present application, the pulse sensor 13 adopts non-contact radar technology, with a sampling rate of 100 Hz, which can accurately capture the micro-movements of the chest and abdomen without interfering with the normal performance of the actor. In addition, the system can also evaluate the actor's breath support and control ability by analyzing the breathing pattern.

[0088] The omnidirectional camera array 14 is used to collect action data, including stage actions and singing actions. In a specific implementation, the omnidirectional camera array 14 is composed of 8 1080p high-definition cameras, with a frame rate of 30 fps, and through the arrangement of different angles, it realizes 360° coverage without dead angle, fully capturing the body movements and position changes of the actor.

[0089] During data collection, the system will time synchronize various data to ensure that data from different sources can accurately correspond to the same performance time. Preferably, the system adopts a common timestamp mechanism with a time resolution of 1 ms to ensure the time sequence consistency of the data.

[0090] As shown in Figure 2 The topology feature extraction module 20 includes a feature vector construction unit 21, a point cloud generation unit 22, a topological complex construction unit 23, a persistent feature calculation unit 24, and a performance manifold construction unit 25.

[0091] The feature vector construction unit 21 is used to convert multi-dimensional performance data into a standardized feature vector set. In one embodiment of the present application, the feature vector set includes acoustic feature vectors, expression feature vectors, breathing feature vectors, and action feature vectors. Among them, the acoustic feature vector contains 32 dimensions, such as pitch trajectory, intensity change, timbre parameter, etc.; the expression feature vector contains 128 dimensions, describing the position and change of the key points of the face; the breathing feature vector contains 16 dimensions, describing the breathing frequency, depth, etc.; the action feature vector contains 64 dimensions, describing the body posture and action fluency.

[0092] The feature vector construction unit 21 performs multi-time scale sampling on multi-dimensional performance data, including micro-scale, meso-scale, and macro-scale. The sampling interval of the micro-scale is 2 to 10 milliseconds, which is used to capture the micro-changes of sound; the sampling interval of the meso-scale is 50 to 200 milliseconds, which is used to capture the expression changes; the sampling interval of the macro-scale is 0.5 to 2 seconds, which is used to capture the body movements and breathing patterns. This multi-scale sampling strategy ensures that the system can analyze performance changes of different rates at the same time, from microsecond-level sound details to second-level overall expression.

[0093] In order to unify the scales of different types of features, the system adopts the maximum and minimum value normalization method to map each feature value to the [0, 1] interval:

[0094] ,

[0095] where: is the original feature value, is the minimum value of the feature, is the maximum value of the feature, is the normalized feature value. This normalization method ensures that features of different dimensions can be analyzed and compared in the same space.

[0096] The point cloud generation unit 22 is connected to the feature vector construction unit 21 and is used to generate a performance state point cloud based on the standardized feature vector set. In the preferred embodiment of the present application, the point cloud generation unit 22 takes each feature vector at each time point in the performance process as a point in a multi-dimensional space to form a performance state point cloud. The performance state point cloud can be represented as:

[0097] ,

[0098] where: is the performance state point cloud, is the performance state point, containing a timestamp and a complete feature vector, is the total number of sampling points. Each point is a high-dimensional vector with a dimension equal to the sum of the dimensions of all feature vectors, which is 240 dimensions (32+128+16+64) in this embodiment.

[0099] The topological complex construction unit 23 is connected to the point cloud generation unit 22 and is used to construct a topological complex based on the performance state point cloud. The topological complex is a geometric approximation of the performance state point cloud, including nodes, edges, and high-dimensional simplices. In specific implementation, the system uses the Vietoris-Rips complex method to construct the topological structure based on the distance relationship between points. A connection is created between two points when the distance between them is less than a threshold , and the threshold is dynamically adjusted according to the density of the performance data, with a typical value of 1.5 times the average nearest neighbor distance in the feature space.

[0100] The topological complex can be represented as a series of simple complexes that vary with the parameter :

[0101] ,

[0102] where: is the Vietoris-Rips complex with parameter , is a simplex (i.e., a subset of points), is a point and The Euclidean distance between them is calculated using the following formula: The dimension of the feature vector. Conditions Representing a simplex The distance between any two points does not exceed the threshold. .

[0103] The persistent feature calculation unit 24 is connected to the topological complex construction unit 23 and is used to calculate persistent topological features based on the topological complex. In one embodiment of the present invention, the persistent feature calculation unit 24 tracks the changes of homology groups in each dimension of the topological complex and identifies persistent and transient topological features in the performance.

[0104] Persistent topological features are represented by a persistence diagram:

[0105] ,

[0106] in: For performance data Persistent graph, Indicates the first The birth and death values ​​of a topological feature, i.e., the feature in the parameters It occurs when, in the parameters Disappeared at that time This represents the total number of topological features. The persistence value is defined as... A higher persistence value indicates a more stable feature and a better representation of the essential characteristics of the performance. In practical applications, the system sets a persistence threshold of 0.15. Features with persistence values ​​higher than this threshold are considered important features and used for subsequent analysis.

[0107] The performance manifold construction unit 25 is connected to the persistent feature computation unit 24 and is used to construct the performance state manifold based on persistent topological features. The performance state manifold is a manifold representation of the performance data in a low-dimensional space, preserving the topological structure features of the data. In the specific implementation, the system adopts the Locally Linear Embedding (LLE) method to map the high-dimensional performance data to the low-dimensional manifold space, while preserving the local relationships between data points.

[0108] The distance function on the performance state manifold is defined as:

[0109] ,

[0110] in: manifold On point and Geodesic distance between them For the path connecting these two points, the parameters , , , is the derivative of the path, representing the tangent vector, is the norm of the tangent vector, represents the infimum over all possible paths, i.e., the shortest path length. This distance function measures the artistic difference between performance states, providing a mathematical foundation for subsequent performance optimization.

[0111] As shown in Figure 3 , the multi-view fusion analysis module 30 includes a view space definition unit 31, a view data registration unit 32, a topological consistency evaluation unit 33, an optimal view selection unit 34, and a view fusion data generation unit 35.

[0112] The view space definition unit 31 is used to establish a spatial distribution model of the acquisition views and define the coverage range and feature extraction advantage domain of each view. In the preferred embodiment of the present application, the system adopts 8 different views, including front view, side view, 45-degree angle view, etc., forming complete view coverage. Each view has its specific advantage domain, for example, the front view is suitable for analyzing facial expressions, and the side view is suitable for analyzing posture contours.

[0113] The view space can be represented as:

[0114] ,

[0115] where: is the set of view spaces, is the th view, is the total number of views, which is in this embodiment. Each view has a specific coverage range and a feature extraction advantage domain , which is a subset of the feature space, indicating which feature dimensions this view has an advantage in.

[0116] The view data registration unit 32 is connected to the view space definition unit 31 and is used to perform spatial registration of multi-view data based on common feature points. In specific implementation, the system adopts a three-level registration strategy: first, coarse registration based on fixed reference points in the scene; then, fine registration based on key breakthrough points of the actor's body; finally, feature-level registration to ensure that the same performance elements captured by different views can be accurately corresponded.

[0117] The registration quality evaluation adopts the mean square error (MSE) index:

[0118] ,

[0119] where: is the mean square error, and are the corresponding feature points in two views, is the total number of feature points, is the Euclidean distance, representing the spatial distance between two points. Preferably, when is less than a predetermined threshold (e.g., 5 pixels in pixel space), the registration quality is considered good; otherwise, re-registration is needed. This threshold is set based on expert experience, which can ensure registration accuracy while not excessively increasing computational complexity.

[0120] The topology consistency evaluation unit 33 is connected to the view data registration unit 32 and is used to calculate the consistency index of multi-view topology features. The topology consistency evaluation unit 33 calculates the consistency index by the following steps: converting the topology features of each view to a unified reference space; establishing a feature correspondence table to match the same performance elements captured by different views; designing a feature alignment algorithm to eliminate the deviation caused by view differences; and calculating multi-level consistency indexes to generate an overall consistency index.

[0121] The consistency index includes structural consistency, temporal consistency, feature consistency, and dynamic consistency. Structural consistency measures the similarity of topology structures in different views; temporal consistency measures the consistency degree of event occurrence order; feature consistency measures the similarity of feature value distribution; and dynamic consistency measures the similarity degree of change patterns.

[0122] The calculation formula of the overall consistency index (CI) is:

[0123] ,

[0124] wherein: CI is the overall consistency index, with a value range of [0, 1], and the larger the value, the higher the consistency; , , , are the structural consistency, temporal consistency, feature consistency, and dynamic consistency indexes, respectively, each with a value range of [0, 1]; , , , are the corresponding weights, and satisfy In typical applications, , , , This weight configuration values structural and feature consistency, which is suitable for musical drama performance analysis. The weight values are determined based on a large amount of experimental data and expert evaluation, and can be fine-tuned according to specific application scenarios.

[0125] The optimal view angle selection unit 34 is connected with the topology consistency evaluation unit 33, and is configured to select an optimal view angle combination based on the consistency index and the view angle complementarity analysis result. In the preferred embodiment of the present application, the system constructs a view angle combination graph , wherein the nodes represent the view angle set, and the edge set represents the connection relationship between the view angles, and the edge weight represents the complementarity index between the view angles. The complementarity index is defined as:

[0126]

[0127] , wherein: is the complementarity index of the view angle and the view angle , and the value range is [0, 1], and the greater the value is, the stronger the complementarity is; and are the feature extraction advantage domains of the view angle and the view angle , respectively. is the intersection size of the two view angle advantage domains (i.e., the number of common advantage features of the two view angles), is the union size (i.e., the number of all advantage features of the two view angles). The complementarity index is essentially a Jaccard distance, which is used to measure the difference between two sets.

[0128] The optimal view angle combination selection algorithm is based on a greedy strategy, which starts from the view angle with the largest information amount, and gradually adds the view angle with the strongest complementarity until the information gain is lower than a threshold. In this embodiment, the information gain threshold is set to 0.05, that is, when the information gain brought by adding a new view angle is lower than 5%, the addition is stopped. Typically, the system will select 3-4 optimal view angle combinations from 8 view angles, which not only ensures the comprehensiveness of the analysis, but also avoids redundant calculation.

[0129] The view angle fusion data generation unit 35 is connected with the optimal view angle selection unit 34, and is configured to generate fusion topology feature data based on the optimal view angle combination. In the specific implementation, the system adopts a feature-level fusion strategy, and the features are weighted and fused according to the advantage domain weight of each view angle:

[0130]

[0131] , wherein: is the fused feature vector, and the dimension is the same as that of the feature vector of a single view angle; is the feature vector of the i-th view angle; is the weight of the view angle, satisfying ; ​​The number of perspectives in the optimal perspective combination is typically 3-4. Perspective weights are calculated based on the information quality index and complementarity index to ensure that the fused data can fully utilize the advantages of each perspective.

[0132] like Figure 4 As shown, the performance optimization module 40 includes a performance quality mapping unit 41, a personalized benchmark unit 42, a quality manifold construction unit 43, an improved path planning unit 44, and a key breakthrough point identification unit 45.

[0133] The performance quality mapping unit 41 is used to establish the correspondence between topological features and performance quality scores. In one embodiment of the invention, the system establishes a mapping function from topological feature vectors to quality scores by analyzing expert scoring data:

[0134] ,

[0135] in: The performance quality score is a vector that includes multiple scoring dimensions (such as technical accuracy, artistic expression, and role fit). These are topological feature vectors, typically with dimensions between 50 and 100, which are key topological features extracted from the original features. This is a mapping function that maps from the feature space to the scoring space. During quality manifold construction and path planning, the system can use... A specific dimension (such as technical accuracy) can also be used to form a comprehensive score through a weighted combination of these dimensions. Therefore, in the definition of quality contour lines:

[0136]

[0137] in, For quality value The contour lines are manifolds. A subset of; For performance state manifold; For point The scalar quality score can be A specific dimension in a vector can also be a weighted sum of multiple dimensions, with values ​​ranging from... Point It is a manifold A point on the [plane]. This definition preserves [the following]. As the essence of a multidimensional scoring vector, it also explains how to convert it into a scalar for use in specific applications.

[0138] Similarly, in the path cost function:

[0139]

[0140] where, is a point on the path, representing a performance state; is a direction vector, indicating the direction and speed of state change; is a point scalarized quality score, taking values in the range is the square of the speed (indicating the difficulty of learning), is the Euclidean norm of the vector ; is the distance of the point from the personal benchmark, indicating the degree of deviation from the personal style; is the weight coefficient, . In different processing stages, the system can choose to use specific dimensions or integrated dimensions of as needed.

[0141] Preferably, the mapping function is implemented using a multi-layer perceptron (MLP), with the input being the topological feature vector and the output being a multi-dimensional score, including technical accuracy (0-10), artistic expressiveness (0-10), and role fit (0-10).

[0142] The personalized benchmark unit 42 is connected to the performance quality mapping unit 41, and is used to establish benchmark data of the personal performance style and technical characteristics of the actor. In specific implementation, the system analyzes the historical performance data of the actor, extracts personal style features, and distinguishes between artistic characteristics that need to be preserved and technical problems that need to be improved.

[0143] The personal benchmark data includes a style feature vector and a technical feature vector :

[0144] ,

[0145] where: PB is the personal benchmark data; S is the style feature vector, describing the artistic style characteristics of the actor, such as voice characteristics, expression style, etc.; T is the technical feature vector, describing the technical level and characteristics of the actor, such as pitch control, breathing skills, etc. This separation allows the system to provide technical improvement suggestions while preserving the unique artistic style of the actor.

[0146] The quality manifold construction unit 43 is connected to the personalized benchmark unit 42, and is used to construct quality contour lines in the performance state space, marking expert performance areas and problem performance areas. The quality manifold is an extension of the performance state manifold by superimposing the quality score dimension, which can visually display the distribution of performance quality.

[0147] In the preferred embodiment of the present application, the system constructs quality contour lines by taking the quality score as a scalar field on the performance state manifold:

[0148] ,

[0149] wherein: is a contour with mass value is a subset of the manifold ; is the performance state manifold; is the scalarized quality score of point ; represents point is a point on the manifold . The system labels the area with quality score higher than 8.5 as expert performance area, and the area with quality score lower than 6 as problematic performance area. These thresholds are determined based on a large amount of experimental data and expert evaluation, and can effectively distinguish performances of different quality levels.

[0150] The improved path planning unit 44 is connected with the quality manifold construction unit 43, and is used to calculate the optimal path from the current performance state to the target state. In an embodiment of the present application, the system first accurately locates the current performance position on the performance state manifold, analyzes the gap with the target state, and then calculates the optimal improvement path:

[0151] ,

[0152] wherein: is the optimal path, which is a parameterized curve, is the current performance state, is the target state; is the path cost function, which considers factors including technical feasibility, learning curve and personal habits; represents the path that minimizes the integral value among all possible paths. Preferably, the path cost function is defined as:

[0153] ,

[0154] wherein: is a point on the path, representing a performance state; is the direction vector, indicating the direction and speed of state change; is the quality score of point , with a value range of [0, 10]; is the square of the speed (indicating the difficulty of learning), is the Euclidean norm of vector ; is the distance of point from the personal benchmark, indicating the degree of deviation from the personal style; 、 、 For the weight coefficient, Typical values are , , This configuration improves performance quality while considering learning difficulty and personal style preservation.

[0155] The key breakthrough point identification unit 45 is connected with the improvement path planning unit 44, used to identify key breakthrough points on the optimal path, and design special technical exercises for each key breakthrough point. The key breakthrough point identification unit 45 classifies the identified key breakthrough points, including technical breakthrough points, expression turning points, and coordination change points.

[0156] The key breakthrough point identification unit 45 is connected with the improvement path planning unit 44, used to identify key breakthrough points on the optimal path, and design special technical exercises for each key breakthrough point. The key breakthrough point identification unit 45 classifies the identified key breakthrough points, including key breakthrough points, expression turning points, and coordination change points.

[0157] The identification of key breakthrough points is based on path curvature and quality gradient:

[0158] ,

[0159] ,

[0160] Where: is the path curvature, measuring the degree of bending of the path at point ; is the second derivative of the path, representing acceleration; is the first derivative of the path, representing speed; represents the vector cross product; is the mass gradient projection, representing the rate of change of mass along the path direction; is the gradient of the quality score function at point , indicating the direction and rate of quality change; represents the vector dot product. When the curvature is greater than the threshold value (typical value is 0.8) or the gradient projection changes significantly (change rate exceeds 30%), it is identified as a key breakthrough point. These thresholds are determined through extensive experiments and can effectively identify key breakthrough points in performances.

[0161] Technical breakthrough points refer to key nodes on the performance optimization path where actors need to achieve a qualitative leap in specific technical aspects. These nodes are usually characterized by a certain technical parameter needing to cross a critical value or bottleneck to achieve higher performance quality. From a technical perspective, technical breakthrough points usually appear when the quality score gradient Positions with large variation rates (more than 30%) and high correlation with weak items in the personal skill feature vector T. These breakthrough points can significantly improve the performance quality score Q once overcome, but often require targeted skill training.

[0162] In practical applications, the breath support bottleneck is a typical skill breakthrough point. When an actor lacks breath during high vocal passages, it can cause distortion in tone and instability in pitch. In this case, the vocal cord vibration fundamental frequency stability in acoustic data is often below 6.5 points, and the chest and abdomen coordination in breathing data is below 5 points. The system can accurately identify this skill breakthrough point on the performance state manifold, with a quality gradient change rate of about 45%. For this breakthrough point, the system generates abdominal breathing and vocal cord closure coordination exercises, requiring the actor to break through in resistance breathing and air column support techniques.

[0163] Another common skill breakthrough point is resonance cavity control. When an actor's vocal resonance position is fixed during emotional changes in the character, it can lead to a lack of depth and level in the character's performance. In this case, the resonance cavity control parameter in acoustic data usually has a variation range of less than 60% of the normal value. The system identifies this skill breakthrough point and designs exercises for switching between oral, nasal, and head cavities, including resonance position changes for different emotional states, to help the actor break through this technical bottleneck.

[0164] Unlike skill breakthrough points, expression turning points focus on qualitative changes in artistic performance. Expression turning points are key nodes on the performance optimization path where the actor's artistic performance needs to undergo specific qualitative changes. These nodes usually involve emotional expression, character development, or artistic style changes, which are crucial for improving the artistic appeal of the performance. From a technical perspective, expression turning points usually occur at positions with high path curvature K(t) (greater than 0.8), indicating a significant change in the direction of the performance state. These breakthrough points are closely related to facial expression feature vectors and vocal expressiveness features, and can significantly improve the artistic expressiveness score and character fit score after breaking through, but require artistic perception and expressiveness training to achieve.

[0165] Emotional transition expression is a typical case of expression turning points. For example, when an actor's performance is stiff at the turning point where the character's emotion changes from anger to sadness, lacking natural transition, the facial expression data usually shows that the forehead muscles and cheek muscles are not coordinated, and the expression switching path curvature K(t) often reaches 1.2 or more. The system can accurately identify this expression turning point and design micro-expression transition exercises to establish emotional transition bridges through subtle expression changes, while providing reference video demonstrations to help the actor master the skills of natural emotional transition.

[0166] Moreover, the character personality hierarchy display is also an important expression turning point. When an actor lacks hierarchy in portraying complex characters, the performance appears flat, and the dimensionality utilization rate of the expression feature vector is usually low, with more than 70% of the expression changes concentrated in the three main dimensions. The system will identify this expression turning point and generate "character contrast training", which requires the actor to express the multiple personality traits of the character in the same lines, thereby enriching the performance in terms of hierarchy and three-dimensionality.

[0167] As the third type of key breakthrough point, the coordination change point emphasizes the overall and comprehensive nature of the performance. The coordination change point refers to a key node in the performance optimization path where multiple performance dimensions (such as voice, expression, breathing, and movement) need to be adjusted in synchronization. This type of breakthrough point embodies the comprehensive nature of the performing arts, requiring the actor to change the performance state in multiple dimensions simultaneously. In terms of data features, the coordination change point is characterized by significant changes in multiple feature dimensions simultaneously (the feature vector change rate of multiple dimensions exceeds the threshold synchronously), usually involving the coordinated cooperation of acoustic features, breathing features, facial expression features, and movement features. Once this breakthrough point is broken through, it can significantly improve the overall coordination and fluency of the performance, but it requires multi-dimensional coordination training to master.

[0168] The coordination problem of song and dance combination is a typical case of coordination change point. When the actor has breathing disorder at the song and dance combination, leading to a decline in voice quality and unstable movement, acoustic data, breathing data, and movement data often show significant fluctuations at the same time point, and the multi-dimensional coordination index is usually less than 5 points. The system can identify this coordination change point and design "gradual song and dance combination exercises" to gradually establish the coordination between body movement and breathing control, and finally realize the natural fusion of song and dance.

[0169] It is also worth noting the overall expression coordination point of emotion-voice-movement. When the actor expresses strong emotions, the voice, expression, and body language are not coordinated, which greatly reduces the expressiveness and persuasiveness. In this case, the change direction of the acoustic feature vector, expression feature vector, and movement feature vector is often inconsistent, and the vector angle is usually greater than 45 degrees. The system will identify this as an important coordination change point and generate "emotion-driven overall expression training", which starts from the inner emotion and guides the coordinated change of each performance dimension, helping the actor achieve consistent expression from the inside out.

[0170] In practical application of the present application, the system distinguishes the three types of key breakthrough points through precise algorithms and designs targeted training strategies for each type of breakthrough point. For technical breakthrough points, the system mainly identifies them through significant changes (more than 30%) in the quality gradient G(t) and the degree of association with technical feature vectors, and focuses on designing targeted technical exercises such as breath control, vocal cord vibration, and clear articulation. For expression turning points, the system mainly identifies them through high values (more than 0.8) of path curvature K(t) and the relevance to expression and vocal expressiveness features, and the training strategies focus more on artistic expressiveness improvement such as emotional experience, character shaping, and expression transition. As for coordination change points, the system identifies them through synchronous change patterns of multi-dimensional feature vectors and correlation analysis between dimensions, and designs multi-dimensional coordination exercises such as breath-sound coordination, sound-motion synchronization, and expression-sound consistency.

[0171] The three types of key breakthrough points are not independent of each other, but are organically related and influence each other. For example, breakthroughs in technical breakthrough points often provide a technical foundation for expression turning points, and the realization of expression turning points can promote the coordinated development of coordination change points. The system considers the relationship between the three types of breakthrough points and arranges the breakthrough order reasonably on the performance optimization path to form a gradual and step-by-step learning path. In addition, the system dynamically adjusts the training difficulty and progress according to the individual characteristics and learning curve of the actor to ensure that the actor can steadily break through each key breakthrough point and continuously improve the performance quality.

[0172] In summary, by accurately identifying and classifying technical breakthrough points, expression turning points, and coordination change points, the key node identification unit of the present application provides highly targeted exercise suggestions for actors, effectively solving technical bottlenecks, expression barriers, and coordination problems in musical theater rehearsals, thereby significantly improving rehearsal efficiency and performance quality. This innovative technology not only provides new technical support for musical theater performance art, but also opens up new avenues for the teaching, research, and inheritance of performance art.

[0173] In the preferred embodiment of the present application, the system designs special technical exercises for each key breakthrough point. For example, for the sound control technical breakthrough point, the system may suggest specific breath support exercises; for the expression turning point, the system may suggest facial muscle control training; for the coordination change point, the system may suggest sound and motion coordination exercises.

[0174] The system of the present application also includes a cloud service layer 70, a stage control module 80, and a remote guidance module 90.

[0175] The cloud service layer 70 includes a cloud interaction interface 71, a high-level analysis engine 72, and a rehearsal database 50. The cloud interaction interface 71 implements the data exchange and communication function of the system with external devices. The high-level analysis engine 72 is an extension of the rehearsal engine 60, with more powerful data analysis capabilities, and can perform deep learning model training and inference. The rehearsal database 50 stores all rehearsal data and analysis results, supporting long-term preservation and query of data.

[0176] The station control module 80 is connected with the cloud service layer 70, used for receiving and displaying the rehearsal report, and controlling the rehearsal hall body 100. In the preferred embodiment of the present application, the station control module 80 includes a touch display device, providing an intuitive user interface, displaying real-time analysis results and rehearsal reports. At the same time, the station control module 80 can control various functions of the rehearsal hall body 100, such as lighting, sound, etc., to create the best environment for rehearsal.

[0177] The remote guidance module 90 is connected with the cloud service layer 70, used for receiving training data of remote guidance personnel, and transmitting the training data to the rehearsal engine 60 for rehearsal training. In specific implementation, the remote guidance personnel can check the rehearsal situation through mobile terminals such as mobile phones or tablets at any time and anywhere, provide professional guidance, and upload demonstration data to help actors improve their performance.

[0178] Through the cloud service architecture, the system of the present application realizes centralized processing, storage and sharing of data, breaks geographical restrictions, enables high-quality professional guidance resources to be widely shared, and greatly improves rehearsal efficiency and performance quality.

[0179] The music drama rehearsal multi-parameter backtracking analysis method of the present application includes the following steps:

[0180] Step S1: Collecting multi-dimensional performance data in the rehearsal process of actors, including acoustic data, facial expression data, breathing data and action data.

[0181] Step S2: Converting the multi-dimensional performance data into a standardized feature vector set, including acoustic feature vectors, expression feature vectors, breathing feature vectors and action feature vectors.

[0182] Step S3: Generating a performance state point cloud based on the standardized feature vector set.

[0183] Step S4: Constructing a topological complex body based on the performance state point cloud.

[0184] Step S5: Calculating persistent topological features based on the topological complex body.

[0185] Step S6: Constructing a performance state manifold based on the persistent topological features.

[0186] Step S7: consistency analysis is performed on the topological features from multiple perspectives, and the optimal perspective combination is selected based on the consistency analysis result.

[0187] Step S8: based on the topological features of the optimal perspective combination, a performance quality evaluation model is constructed.

[0188] Step S9: based on the performance quality evaluation model, the performance optimization path from the current performance state to the target state is calculated.

[0189] Step S10: identify key breakthrough points on the performance optimization path, and design special technical exercises for each key breakthrough point.

[0190] Step S11: store the persistent topological features, performance quality evaluation model and performance optimization path to the review database.

[0191] Step S12: generate a review report based on the data in the review database.

[0192] In practical applications, the system will continuously adjust and optimize the performance based on the feedback of the review report, forming a closed-loop optimization process. Through multiple iterations, the performance level of the actor will be continuously improved, and finally the expected artistic effect will be achieved.

[0193] The music drama rehearsal multi-parameter backtracking analysis system and method of the present application realizes deep structure analysis of multi-dimensional performance data by innovatively introducing topological data analysis technology; through multi-perspective fusion technology, the comprehensiveness and reliability of the analysis result are ensured; through personalized performance optimization technology, targeted technical improvement guidance is provided.

[0194] In practical applications, the system of the present application significantly improves the rehearsal efficiency, reduces the rehearsal time, and can shorten the rehearsal period by more than 40% on average; improves the performance quality, enhances the consistency and accuracy of the performance, and under the same rehearsal time, the group using the system is 18.6% higher in professional review scoring on average; while preserving the artistic personality of the actor, accurate technical improvement guidance is provided, 93.5% of the participating actors believe that the system suggestions will not suppress individual creativity, but enhance the expressiveness; supports remote guidance, breaks through geographical restrictions, and makes high-quality professional guidance resources widely available; provides a new method for the research, teaching and inheritance of performing arts.

[0195] In summary, the music drama rehearsal multi-parameter backtracking analysis system and method of the present application realizes comprehensive, multi-dimensional and accurate analysis and guidance of the music drama rehearsal process, and provides strong technical support for the development of music drama performing arts.

[0196] The above embodiments only express the specific implementation of the present application, which is described in more detail and in more detail, but cannot be understood as a limitation on the scope of the patent of the present application. It should be noted that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application.

Claims

1. A multi-parameter retrospective analysis system for music and drama rehearsals, characterized in that, include: The rehearsal hall itself; The data acquisition module is located in the rehearsal hall and is used to collect multi-dimensional performance data of the actors during the rehearsal process. The multi-dimensional performance data includes acoustic data, facial expression data, breathing data and movement data. A topology feature extraction module, connected to the data acquisition module, is used to receive the multidimensional performance data, construct a performance feature space, and extract the topological features of the performance feature space. A multi-view fusion analysis module, connected to the topology feature extraction module, is used to receive the topology features, perform consistency analysis on the topology features from multiple views, and select the optimal view combination based on the consistency analysis results. The performance optimization module, connected to the multi-view fusion analysis module, is used to construct a performance quality evaluation model and generate a performance optimization path based on the topological features of the optimal view combination. The review database, connected to the performance optimization module, is used to store the topology features, the performance quality assessment model, and the performance optimization path. The debriefing engine, connected to the debriefing database, is used to generate debriefing reports based on the data in the debriefing database.

2. The multi-parameter backtracking analysis system for musical drama rehearsals according to claim 1, characterized in that, The data acquisition module includes: A sound field array microphone is used to collect the acoustic data, which includes vocal cord vibration fundamental frequency stability, breath support, and resonance cavity control data. An infrared marker camera is used to collect the facial expression data, which includes the movement trajectory of facial muscles. A pulse sensor is used to collect the breathing data, which includes the actor's breathing fluctuations; An omnidirectional camera array is used to collect the motion data, which includes stage movements and singing movements.

3. The multi-parameter backtracking analysis system for musical drama rehearsals according to claim 1, characterized in that, The topology feature extraction module includes: The feature vector construction unit is used to convert the multidimensional performance data into a standardized feature vector set, which includes acoustic feature vectors, facial expression feature vectors, breathing feature vectors, and motion feature vectors. A point cloud generation unit, connected to the feature vector construction unit, is used to generate a performance state point cloud based on the standardized feature vector set. A topological complex construction unit, connected to the point cloud generation unit, is used to construct a topological complex based on the performance state point cloud; A persistent feature calculation unit, connected to the topological complex construction unit, is used to calculate persistent topological features based on the topological complex; The performance manifold construction unit is connected to the persistent feature calculation unit and is used to construct the performance state manifold based on the persistent topological features.

4. The multi-parameter backtracking analysis system for musical drama rehearsals according to claim 1, characterized in that, The multi-view fusion analysis module includes: The viewpoint space definition unit is used to establish a spatial distribution model of the acquisition viewpoints and define the coverage and feature extraction advantage domain of each viewpoint. A viewpoint data registration unit, connected to the viewpoint space definition unit, is used to perform spatial registration of multi-viewpoint data based on common feature points. A topology consistency evaluation unit, connected to the view data registration unit, is used to calculate the consistency index of multi-view topology features. The consistency index includes structural consistency, temporal consistency, feature consistency, and dynamic consistency. The optimal view selection unit is connected to the topology consistency evaluation unit and is used to select the optimal view combination based on the consistency index and the view complementarity analysis results. A perspective fusion data generation unit is connected to the optimal perspective selection unit and is used to generate fused topological feature data based on the optimal perspective combination.

5. The multi-parameter backtracking analysis system for musical drama rehearsals according to claim 1, characterized in that, The performance optimization module includes: The performance quality mapping unit is used to establish the correspondence between topological features and performance quality scores; A personalized benchmark unit, connected to the performance quality mapping unit, is used to establish benchmark data for an actor's personal performance style and technical characteristics. A quality manifold construction unit, connected to the personalized reference unit, is used to construct quality contour lines in the performance state space to mark expert performance areas and problem performance areas; An improved path planning unit, connected to the quality manifold construction unit, is used to calculate the optimal path from the current performance state to the target state; A key breakthrough point identification unit, connected to the improved path planning unit, is used to identify key breakthrough points on the optimal path and design specific technical exercises for each key breakthrough point.

6. The multi-parameter backtracking analysis system for musical drama rehearsals according to claim 1, characterized in that, The system also includes: The cloud service layer includes cloud interaction interfaces, advanced analytics engines, and retrospective databases; The stage control module is connected to the cloud service layer and is used to receive and display the debriefing report and control the rehearsal hall itself. The remote guidance module, connected to the cloud service layer, is used to receive training data from remote instructors and transmit the training data to the debriefing engine for debriefing training.

7. The multi-parameter backtracking analysis system for musical drama rehearsals according to claim 3, characterized in that, The feature vector construction unit performs multi-timescale sampling on the multi-dimensional performance data, the multi-timescale including: At the microscale, with sampling intervals of 2 to 10 milliseconds, it is used to capture minute changes in sound; At the mesoscale, sampling intervals of 50 to 200 milliseconds are used to capture facial expression changes; Macroscale sampling, with intervals of 0.5 to 2 seconds, is used to capture limb movements and breathing patterns.

8. The multi-parameter backtracking analysis system for musical drama rehearsals according to claim 4, characterized in that, The topology consistency evaluation unit calculates the consistency index through the following steps: Transform the topological features from various perspectives into a unified reference space; Establish a feature mapping table to match the same performance element captured from different perspectives; Design a feature alignment algorithm to eliminate deviations caused by differences in viewpoints; Calculate multi-level consistency indicators to generate an overall consistency index.

9. The multi-parameter backtracking analysis system for musical drama rehearsals according to claim 5, characterized in that, The key breakthrough point identification unit classifies the identified key breakthrough points, and the classification includes: The key breakthrough point is where performance skills need a qualitative improvement; The turning point in expression requires specific changes in the artistic presentation; The points of coordinated change are the locations where multi-dimensional performance elements need to be adjusted synchronously; and targeted practice suggestions are generated based on the classification.

10. A method for multi-parameter backtracking analysis of musical drama rehearsals, employing the multi-parameter backtracking analysis system for musical drama rehearsals as described in any one of claims 1-9, characterized in that, Includes the following steps: Collect multi-dimensional performance data of actors during rehearsals, including acoustic data, facial expression data, breathing data, and movement data; The multidimensional performance data is converted into a standardized feature vector set, which includes acoustic feature vectors, facial expression feature vectors, breathing feature vectors, and motion feature vectors. Generate a performance state point cloud based on the standardized feature vector set; Construct a topological complex based on the point cloud of the performance state; Calculate persistent topological features based on the aforementioned topological complex; Construct a performance state manifold based on the aforementioned persistent topological features; A consistency analysis is performed on the topological features from multiple perspectives, and the optimal combination of perspectives is selected based on the results of the consistency analysis. Based on the topological features of the optimal viewpoint combination, a performance quality assessment model is constructed; Based on the performance quality assessment model, calculate the performance optimization path from the current performance state to the target state; Key breakthroughs are identified along the performance optimization path, and specific technical exercises are designed for each key breakthrough. The persistent topological features, the performance quality assessment model, and the performance optimization path are stored in the replay database; A review report is generated based on the data in the review database.

Images