Three-dimensional reconstruction and optimization method for dance movement track

Abstract

The invention discloses a dance movement trajectory three-dimensional reconstruction and optimization method, and relates to the technical field of dance trajectory reconstruction, and the method comprises the steps: obtaining a multi-dancer time sequence three-dimensional human body key point position set, each position correlation observation visual angle, observation reliability and dancer identity identification in group dance; identifying a shielding candidate period based on the key point positions of adjacent time points and the observation reliability, and screening a temporary contact shielding period violating at least one constraint of multi-view observation consistency, human skeleton length consistency or key point motion continuity in combination with the view angle and the key point positions before and after the candidate period; maintaining the identity and complementing the low-reliability key points in the shielding time period, redistributing the identity after the shielding is finished, and generating a complete three-dimensional dance movement track of each dancer; the method has the beneficial effects that three-dimensional trajectory reconstruction of continuous and non-jumping dance trajectories of dancers can be realized under the condition that limbs of the dancers are shielded at high frequency in group dance performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of dance trajectory reconstruction, and in particular to a dance motion trajectory three-dimensional reconstruction and optimization method. BACKGROUND

[0002] With the increasing demand for digitalization and intelligent analysis of dance performances, human motion capture and three-dimensional trajectory reconstruction technology based on video are widely used in dance teaching, dance evaluation, virtual stage presentation, and performance motion archiving. The existing technology usually collects dancer performance videos through monocular or multi-camera devices, detects and identifies key points of the human body in the video, and combines multi-view geometric relationships or temporal constraints to realize the reconstruction of dancer limb key points in three-dimensional space and trajectory generation.

[0003] In the case of a single person or a small number of dancers with less occlusion, the existing three-dimensional reconstruction technology can obtain relatively continuous motion trajectories. However, in large-scale stage group dance performances, due to the large number of dancers, strong motion synchronization, and frequent spatial position interlacing, the arms, legs, and other limb parts of dancers are easily occluded or even partially overlapped for a long time, making it difficult for camera devices to continuously and stably observe the complete limb structure of each dancer. To address this occlusion problem, the existing technology usually uses interpolation completion, multi-view matching, or prediction methods based on historical motion trends to generate trajectory data during occlusion.

[0004] However, in practical applications of group dance with high-frequency occlusion, since the occlusion often lasts for a long time and the limbs of multiple dancers are highly close or even overlapping in space, traditional interpolation or prediction methods cannot accurately determine which dancer the reappearing limb after occlusion belongs to, which may cause problems such as trajectory breakage, limb misconnection, or dancer identity confusion. At the same time, the existing technology usually treats occlusion as simply missing information, without effectively maintaining the continuity of dancer identity and limb attribution during occlusion, thereby affecting the accuracy and reliability of three-dimensional trajectory reconstruction results in group dance scenarios.

[0005] Therefore, a dance motion trajectory three-dimensional reconstruction and optimization method is proposed. SUMMARY

[0006] In view of the above existing technical conditions, the present application is proposed. The embodiments of the present application provide a dance motion trajectory three-dimensional reconstruction and optimization method, which can realize continuous and non-jumping three-dimensional trajectory reconstruction in large-scale stage group dance performance scenarios.

[0007] According to an aspect of the present application, a dance motion trajectory three-dimensional reconstruction and optimization method is provided, comprising: obtaining a set of three-dimensional human key point positions of a plurality of dancers in a group dance performance in a time sequence, each human key point position being associated with an observation view angle, an observation reliability and an identity of a corresponding dancer; based on the human key point positions and the corresponding observation reliabilities of adjacent time points in the time sequence, identifying candidate time periods in which mutual occlusion occurs between dancers; based on the observation view angle and the human key point positions before and after the candidate time periods, identifying candidate time periods that violate at least one of multi-view observation consistency, human skeleton length consistency and key point motion continuity as temporary contact occlusion time periods with identity confusion risk; in the temporary contact occlusion time period, maintaining the identity of the human key points of the dancers involved in the occlusion unchanged, and predicting and completing human key point positions with observation reliability lower than a preset threshold to obtain completion data; after the temporary contact occlusion time period ends, according to the human key point positions after the end, the completion data and the constraints of multi-view observation consistency, human skeleton length consistency and key point motion continuity, reassigning the identity of each human key point position to restore the correct association between human key points and identities; and generating complete three-dimensional dance motion trajectories of each dancer according to the association between the restored human key points and identities.

[0008] According to another aspect of the present application, a dance motion trajectory three-dimensional reconstruction and optimization system is provided, comprising: a data acquisition module configured to acquire a set of three-dimensional human key point positions of a plurality of dancers in a group dance performance over a time sequence, each human key point position being associated with an observation view angle, an observation reliability and an identity of a corresponding dancer; a candidate occlusion period identification module configured to identify a candidate period of mutual occlusion between dancers based on human key point positions and corresponding observation reliabilities of adjacent time points in the time sequence; a temporary contact occlusion identification module configured to identify a candidate period of temporary contact occlusion with a risk of identity confusion by violating at least one constraint of multi-view observation consistency, human skeleton length consistency and key point motion continuity based on the observation view angle and human key point positions before and after the candidate period; an identity maintenance and key point completion module configured to maintain the identity of human key points of dancers involved in occlusion unchanged and to predict and complete human key point positions with observation reliability lower than a preset threshold to obtain completion data in the temporary contact occlusion period; an identity reassignment module configured to reassign the identity of each human key point position to restore the correct association between human key points and identities according to human key point positions after the temporary contact occlusion period, the completion data and the constraints of human skeleton length consistency and key point motion continuity; and a dance trajectory generation module configured to generate complete three-dimensional dance motion trajectories of each dancer according to the restored association between human key points and identities.

[0009] According to another aspect of the present application, an electronic device is provided, comprising a memory and a processor, the memory being configured to store computer executable instructions, and the processor being configured to execute the computer executable instructions, which, when executed by the processor, implement the steps of the method described above.

[0010] According to another aspect of the present application, a computer storage medium is provided, having stored thereon computer executable instructions, which, when executed by a processor, implement the steps of the method described above.

[0011] Compared with the prior art, the dance motion trajectory three-dimensional reconstruction and optimization method according to the embodiments of the present application can distinguish between ordinary occlusion and high-risk identity confusion occlusion under the condition of high-frequency occlusion of dancer's limbs in group dance performance by introducing temporary contact occlusion period judgment and comprehensive constraints based on multi-view observation consistency, human skeleton length consistency and key point motion continuity, avoid the problem of identity jump of human key points caused by traditional interpolation completion, and thus stably restore the correct association between human key points and identities of each dancer after the occlusion is removed, thereby improving the continuity, accuracy and reliability of three-dimensional dance motion trajectory reconstruction in a group dance scene. BRIEF DESCRIPTION OF DRAWINGS

[0012] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0013] Figure 1 This is a flowchart of the method for three-dimensional reconstruction and optimization of dance movement trajectories according to the present invention.

[0014] Figure 2 This is a block diagram of the dance movement trajectory three-dimensional reconstruction and optimization system of the present invention.

[0015] Figure 3 This is a block diagram of an electronic device according to the present invention. Detailed Implementation

[0016] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0017] Exemplary method:

[0018] Figure 1 The illustration shows a method for three-dimensional reconstruction and optimization of dance movement trajectories according to an embodiment of this application, including steps S1 to S6.

[0019] like Figure 1 As shown, in step S1, a set of three-dimensional human body key point positions of multiple dancers in a group dance performance is obtained in time series. Each human body key point position is associated with the observation viewpoint, observation reliability and the identity identifier of the corresponding dancer.

[0020] The input for this step is the human body key point data that has already undergone preliminary 3D reconstruction. In practice, this data can be obtained by acquiring group dance performance scenes through a multi-view camera system, followed by conventional processing such as human body detection, 2D key point extraction, and multi-view triangulation. The location of each human body key point... This represents the three-dimensional coordinates of the i-th dancer at the j-th joint at time t, along with the observation reliability of that keypoint. (This can be calculated by considering factors such as the number of visible cameras, 2D detection confidence, and reprojection error) and the corresponding dancer identification identifier i.

[0021] This step yields a set of key human body locations containing complete information such as identity identifiers, spatial location, and observation quality, providing a data foundation for subsequent occlusion detection and identity maintenance.

[0022] In step S2, based on the positions of human key points at adjacent time points in the time series and the corresponding observation reliability, candidate time periods in which dancers occlude each other are identified.

[0023] In large-scale stage group dance performances, multiple dancers move intermittently in space, inevitably resulting in situations where their bodies approach, obstruct each other, or even briefly touch. If these obstruction events are not identified and handled specially, they can easily lead to confusion in identification in subsequent steps. Therefore, the purpose of this step is to identify potential obstruction time periods as early as possible by observing changes in spatial distance and observation reliability, thus providing a candidate set for further in-depth determination.

[0024] Identifying candidate time periods where dancers may be occluding each other involves the following steps:

[0025] First, at adjacent time points, based on the positions of the key points of each dancer's body, the spatial distance between the key points of different dancers is calculated. Specifically, for any two dancers i and k, at time t, the minimum Euclidean distance between their key joints (e.g., wrist, ankle, knee, elbow, etc.) is calculated:

[0026] ;

[0027] in, and These are the joint indices for dancers i and k, respectively. and Let the i-th and k-th dancers be at time t. and The three-dimensional coordinates of each joint.

[0028] Then, it is determined whether the spatial distance is less than the preset occlusion distance threshold. If so, it is considered that the two dancers are close enough in space to potentially occlude each other, and the corresponding time period is identified as a candidate time period in which the dancers might occlude each other.

[0029] In addition to the methods mentioned above, changes in observation reliability can be used as an auxiliary factor in the determination: when the observation reliability of certain joints of a dancer changes... A sudden drop below the threshold, and the persistence of this low reliability state for a certain period of time, can also be used as a criterion for determining occlusion candidates.

[0030] This step constructs a set of occlusion candidate time periods, with each candidate time period recording the dancers involved, the joints involved, and the start and end times. , which provides a basis for further determining which candidate time period belongs to the "temporary contact occlusion state".

[0031] In step S3, based on the observation view angle and the human body key point position before and after the candidate time period, the candidate time period that violates at least one of the multi-view observation consistency and the human body skeleton length consistency and the key point motion continuity is identified as a temporary contact occlusion time period with identity confusion risk.

[0032] The traditional method simply regards occlusion as "key point disappearance or loss", and after interpolation completion, it often leads to identity label jump. The present application redefines the occlusion as "temporary contact / overlap state between dancers", and determines whether there is an identity confusion risk through multi-dimensional consistency test. Only when the candidate time period not only has spatial proximity, but also is accompanied by obvious observation abnormality, skeleton distortion or motion mutation, it is determined as a temporary contact occlusion time period that needs special processing. This determination mechanism fundamentally distinguishes "ordinary short occlusion" from "high-risk identity confusion occlusion", and provides accurate trigger conditions for subsequent identity freezing and regression distribution.

[0033] Specifically, for each candidate time period identified in step S2, whether it is a temporary contact occlusion time period is determined through multi-view observation consistency, human body skeleton length consistency and key point motion continuity consistency test.

[0034] The reason for performing multi-view observation consistency test is that in a multi-view system, the three-dimensional positions of the same human body key point reconstructed under different camera views should basically coincide. If the reconstructed positions of a key point under multiple views deviate greatly, it means that the observation data of the key point may be affected by occlusion or other interference, leading to inconsistent observation of different views. Specifically, the multi-view observation consistency test includes:

[0035] First, the deviation value of the human body key point position of the same human body key point under different observation view angles is calculated. Specifically, for the joint j of the dancer i at time t, the joint may be observed under multiple camera view angles. For any two observation view angles c1 and c2, the spatial distance between the three-dimensional positions of the joint reconstructed under the two view angles is calculated as the deviation value under the view angle pair:

[0036] ;

[0037] Wherein, and represent the three-dimensional position of the joint reconstructed based on view angle c1 and view angle c2, respectively.

[0038] ​Then, the maximum deviation value of all observable visual angles of the joint j is calculated as the deviation value of the joint at time t :

[0039] ;

[0040] Finally, it is determined whether the deviation value is greater than a preset deviation threshold , if so, it is determined that the observation visual angle consistency is violated.

[0041] The human skeleton length consistency test is performed because the lengths of the segments of the human skeleton should remain stable in a short period of time. If the skeleton length of a dancer suddenly changes abnormally during the occlusion period, it is likely that the human key points of the dancer are incorrectly associated with other dancers, resulting in a skeleton that does not conform to the human structure. Specifically, the human skeleton length consistency test includes:

[0042] First, the human key point positions of the dancers at multiple time points before and after the candidate period are extracted as sample key point positions based on the human key point position set. Specifically, a stable interval before the start of the candidate period is selected (e.g., 10-30 frames), in which the observation reliability of the dancer is high and there is no occlusion interference.

[0043] Then, based on the sample key point positions at multiple time points, the average skeleton length between the connected human key points is calculated as the reference value of the human skeleton of the dancer. For example, for the upper arm skeleton of dancer i (shoulder joint a to elbow joint b), the reference length is:

[0044] ;

[0045] The reference length of all skeleton segments is established .

[0046] Next, within the candidate period, the current skeleton length is calculated based on the human key point positions of each dancer. For time t, the difference between the current skeleton length and the human skeleton reference value is calculated :

[0047] ;

[0048] Finally, it is determined whether the difference between the current skeleton length of each dancer and the human skeleton reference value is greater than a preset length threshold, if so, it is determined that the human skeleton length consistency is violated.

[0049] If the difference between the current skeleton length and the human skeleton reference value continuously exceeds the threshold value in the candidate period, it indicates that the key point association relationship of the dancer may have been wrong, and there is identity confusion. For example, if the upper arm of dancer A is 30 cm long before the occlusion, and suddenly becomes 45 cm during the occlusion, it is likely that the elbow joint of the upper arm is incorrectly associated to dancer B who is taller, resulting in skeleton stretching distortion.

[0050] The key point motion continuity test is performed because the motion of the dancer has inertia and continuity, and the speed direction and size of the key point will not suddenly change dramatically in a short time (unless there is a clear action transition). If the motion trend of a key point of a dancer is obviously discontinuous before and after the candidate period, it indicates that the identity attribution of the key point may have been incorrectly switched during the occlusion. Specifically, the key point motion continuity test includes:

[0051] First, based on the human key point positions of each dancer at adjacent time points, the motion trend of each human key point before and after the candidate period is calculated. Specifically, the velocity trend is estimated in the time window before the candidate period :

[0052] ;

[0053] Similarly, the velocity trend is estimated in the time window after the candidate period .

[0054] Then, according to the human key point positions of each dancer at each time point in the candidate period, the motion trend of each dancer in the candidate period is calculated (if the key point is still partially observable during the occlusion);

[0055] Finally, it is determined whether the deviation between the motion trend in the candidate period and the motion trend before and after the candidate period is greater than a preset motion deviation threshold, and if so, it is determined that the key point motion continuity is violated. The motion continuity deviation :

[0056] ;

[0057] wherein, is a set of key joints (such as wrist, ankle, etc. joints with larger motion amplitude), is the cosine similarity of the velocity direction, is the weight coefficient of the velocity size difference. It is determined whether the deviation between the motion trend in the candidate period and the motion trend before and after the candidate period is greater than a preset motion deviation threshold , if so, it is determined that the key point motion continuity is violated.

[0058] It should be noted that in the present application, the "multi-view observation consistency check" is taken as a necessary condition for determining the temporary contact occlusion period, and the "human skeleton length consistency check" and the "key point motion continuity check" are set as auxiliary determination conditions selected from two, which are determined based on the occurrence mechanism of different abnormal phenomena in the group dance occlusion scene and the influence degree of the identity reliability:

[0059] Firstly, the multi-view observation consistency reflects the reliability of human key points in the spatial observation level. In the multi-view imaging system, the same human key points should be consistently reconstructed to similar three-dimensional positions under different observation angles. When the consistency is destroyed, it usually means that the key points are occluded, overlapped or severely disturbed in at least part of the view, resulting in unreliable observation data. Therefore, the multi-view observation consistency anomaly is a basic condition for judging whether there is a real occlusion event. If the multi-view consistency is still good in the candidate period, even if the dancers are close in space, it does not necessarily cause the identity confusion risk.

[0060] Secondly, the human skeleton length consistency and the key point motion continuity reflect whether the identity identification has been incorrectly associated from two different dimensions of human structure constraint and time motion constraint. In actual group dance scenes, different occlusion situations do not completely affect the human key points in the same way: in some cases, occlusion directly leads to the incorrect splicing of human key points to other dancers, resulting in abnormal changes in the skeleton length; in other cases, the skeleton structure is still approximately reasonable in value, but the motion direction or speed of the key points changes obviously before and after the occlusion, reflecting the switching of the key point identity in the time dimension.

[0061] Therefore, the present application does not require the human skeleton length consistency and the key point motion continuity to be destroyed at the same time. When the candidate period is under the premise of multi-view observation consistency anomaly, as long as there is a significant abnormality in either the human skeleton length consistency or the key point motion continuity, it can be determined that there is a high risk of identity confusion in the candidate period, which needs to be treated as a temporary contact occlusion period for subsequent identity freezing and regression allocation processing. Through the above determination mechanism, it can effectively avoid frequent triggering of complex processing due to short-time observation noise or slight posture changes, while ensuring timely and accurate intervention in the occlusion scene where there is a real risk of identity confusion.

[0062] Return Figure 1 In step S4, during the temporary contact occlusion period, the identity identification of the human key points corresponding to the dancers involved in the occlusion is maintained, and the human key point positions with observation reliability lower than a preset threshold are predicted and completed to obtain the completed data.

[0063] The traditional method usually adopts interpolation or prediction to complete the missing key points during occlusion, although a visually continuous trajectory curve can be obtained, the identity attribution of these interpolated points is often wrong, resulting in the "jump-in" phenomenon after the occlusion is removed. The present application adopts the strategy of "identity freezing + limited prediction": during the temporary contact occlusion period, the identity label is explicitly maintained unchanged, ensuring that each key point always belongs to its original dancer; at the same time, instead of free interpolation for invisible key points, limited prediction is performed based on the posture before occlusion and the motion trend, and strict skeleton constraints and motion constraints are imposed to prevent prediction drift. The core purpose of this strategy is not to generate "beautiful curves", but to maintain the correctness of identity attribution, and to assign accurate reference information for identity regression after the occlusion is removed.

[0064] During the temporary contact occlusion period, identity freezing is performed on the involved dancers: the identity labels of the human key points corresponding to the dancers involved in the occlusion are maintained unchanged. Even if some key points of a dancer decrease in observation reliability or even become completely invisible during this period, these key points will not be re-assigned to other dancers, and the visible key points of other dancers will not be incorrectly associated with this dancer. This mechanism ensures that the identity of each dancer remains stable during the occlusion, providing a basis for correct attribution regression later.

[0065] Specifically, the prediction completion is as follows:

[0066] First, during the temporary contact occlusion period, for human key point positions with observation reliability below a preset threshold, the human key point positions and key point motion trends of the corresponding dancer before the start of the temporary contact occlusion period are obtained. Specifically, for joint j of dancer i, if its observation reliability at time t is less than the prediction trigger threshold , the human key point position at the last stable time before the start of the temporary contact occlusion period is obtained as an anchor point, and the key point motion trend in the time window before the temporary contact occlusion period is obtained for short-term prediction. Then, based on the human key point position and the key point motion trend, the position of the human key point in the temporary contact occlusion period is predicted to obtain the predicted key point position, which can be predicted by linear extrapolation or the like.

[0067] It should be noted that during the prediction process, the predicted key point position is subjected to the human skeleton length constraint and the key point motion continuity constraint of the corresponding dancer to limit the change range of the predicted key point position. The human skeleton length constraint is used to ensure that the distance between the predicted key point and the adjacent key point conforms to the reference length of the skeleton of the dancer . For example, if the elbow joint position is predicted, it needs to satisfy:

[0068] ;

[0069] wherein, is the allowed deviation range of the skeleton length (e.g. 5%-10% of the skeleton reference length). If the prediction result does not satisfy the constraint, the predicted position is corrected by projection so as to fall within the feasible region satisfying the skeleton length constraint.

[0070] The key point motion continuity constraint is used to limit the predicted human key point motion state to be consistent with the true motion state of the key point before and after the occlusion occurs when predicting and completing the position of the human key point during the temporary contact occlusion period, thereby avoiding sudden changes in the position of the human key point that do not conform to the human motion law due to occlusion.

[0071] Finally, the predicted key point position satisfying the human skeleton length constraint and the key point motion continuity constraint is used as the completion data obtained by prediction and completion. For visible key points (key points whose observation reliability is still higher than the threshold), the observation values are directly retained; for invisible key points, the result of the above limited prediction is used as the completion data.

[0072] Return Figure 1 In step S5, after the temporary contact occlusion period ends, the identity of each human key point position is re-assigned to restore the correct association between the human key point and the identity according to the human key point position after the end, the completion data, and the constraints of multi-view observation consistency, human skeleton length consistency, and key point motion continuity.

[0073] After the occlusion ends, the human key points of each dancer gradually recover visibility. At this time, the core problem is how to correctly assign these newly observed key points to each dancer. The traditional method often uses a simple "nearest neighbor matching", which is prone to errors in multi-person occlusion removal. The present application uses a "comprehensive scoring + global optimal allocation" strategy: instead of relying on a single spatial distance, it considers multi-dimensional information such as multi-view consistency, skeleton consistency, and motion continuity, and uses a global optimization algorithm to ensure the optimal overall allocation result, thereby fundamentally avoiding mismatch caused by local greed.

[0074] The re-assignment of the identity of each human key point position includes:

[0075] First, according to the set of human key point positions, the human key point positions of each dancer after the end of the temporary contact occlusion period are extracted as candidate key point positions. Specifically, during the end time of the temporary contact occlusion period and the subsequent recovery time window, human detection and key point extraction are performed on all observation views to obtain a set of newly observed key points. The identity of these observed key points is not yet clear and needs to be determined by matching and allocation.

[0076] Then, based on the candidate keypoint positions and the completion data, a matching cost between each candidate keypoint position and each dancer is calculated, including at least a multi-view observation consistency cost, a human skeleton length consistency cost, and a keypoint motion continuity cost. The "matching cost" can be understood as "how much inconsistency will be caused if candidate keypoint j' is assigned to dancer i". The lower the cost, the more reasonable the assignment.

[0077] Specifically, for the candidate matching "assigning candidate keypoint j' to joint j of dancer i", the following cost components are calculated:

[0078] Firstly, the multi-view observation consistency cost

[0079]

[0080] wherein, is the set of multi-view cameras, is the projection function of camera c, is the 2D observation of the candidate keypoint under camera c. The smaller the re-projection error, the more consistent the candidate keypoint is with the observation under multi-view, and the higher the credibility of the assignment.

[0081] Secondly, the human skeleton length consistency cost:

[0082]

[0083] wherein, is the joint adjacent to joint j, is the skeleton reference length of dancer i. The smaller the skeleton length residual error, the more consistent the assignment is with the human structure.

[0084] Finally, the keypoint motion continuity cost

[0085]

[0086] wherein, is the predicted position of joint j of dancer i at time in step S4, is the predicted velocity trend, is the observed velocity of the candidate keypoint (calculated based on adjacent frames). This cost measures whether the candidate keypoint "continues" the motion trajectory before occlusion.

[0087] For the matching "assigning candidate keypoint j' to joint j of dancer i", the comprehensive cost is:

[0088] ;​​​​​

[0089] wherein, , , are weight coefficients, which can be adjusted according to the actual scene to adjust the importance of different consistency constraints.

[0090] Returning to the reassignment of the identity label, next, according to the matching cost corresponding to the position of each candidate key point, the identity label assignment result satisfying the global consistency constraint is determined.

[0091] In this process, instead of using the greedy "one-by-one nearest neighbor matching" method, a global optimal assignment strategy is used to optimize the assignment relationship between all candidate key points and dancer identities at the same time point as a whole, so that the total matching cost is minimized under the premise of satisfying each candidate key point being assigned to only one dancer and the key point set corresponding to each dancer being consistent as a whole.

[0092] The global optimal assignment strategy can be realized by a global optimization algorithm, such as the Hungarian algorithm or an equivalent minimum cost perfect matching algorithm, thereby avoiding human key point mismatching or identity jumping caused by local optimization or point-by-point matching in the multi-person unocclusion scene, and ensuring the consistency and stability of the identity label reassignment result as a whole.

[0093] Finally, the identity label assignment result is taken as the reassignment result of each human key point position to restore the correct association relationship between the human key point position and the corresponding dancer identity label.

[0094] Finally, through the above global optimal assignment, the correct identity attribution of each candidate key point after unocclusion is obtained, and these assignment results are updated to the human key point position set to complete the recovery of the identity label.

[0095] Returning to Figure 1 In step S6, according to the association relationship between the recovered human key points and the identity labels, the complete three-dimensional dance motion trajectory of each dancer is generated.

[0096] After the processing of the foregoing steps, the identity attribution of the key points of each dancer over the entire time sequence has been correctly maintained:

[0097] In the non-occlusion period, the key point observation is reliable, and the identity label is clear;

[0098] In the temporary contact occlusion period, through the identity freezing mechanism, it is ensured that the identity label does not switch incorrectly;

[0099] After the temporary contact occlusion period ends, the identity attribution of each key point is correctly recovered through the identity consistency regression assignment.

[0100] Therefore, the key point positions at each time can be concatenated according to the dancer identity to form a continuous three-dimensional trajectory of each dancer, and the output trajectory data can be used for subsequent motion analysis, choreography assistance, virtual reality reproduction and the like.

[0101] Compared with the traditional method, the trajectory output by the present application has the following advantages:

[0102] Firstly, the trajectory of the same dancer remains continuous before and after occlusion, and the error of "the hand of dancer A suddenly becomes the hand of dancer B" does not occur.

[0103] Secondly, the trajectory always satisfies the skeleton length constraint and the joint angle constraint, and distortion such as "arm lengthening" or "joint reverse bending" does not occur.

[0104] In summary, the dance motion trajectory three-dimensional reconstruction and optimization method proposed in the present application realizes continuous and non-jumping three-dimensional trajectory reconstruction in a large stage group dance performance scene compared with the prior art.

[0105] An exemplary system is shown in the following.

[0106] Figure 2 The dance motion trajectory three-dimensional reconstruction and optimization system according to an embodiment of the present application is shown in the following, which comprises: a data acquisition module for acquiring a set of three-dimensional human body key point positions of multiple dancers in a group dance performance in a time sequence, each human body key point position being associated with an observation view angle, an observation reliability and an identity of a corresponding dancer; a candidate occlusion period identification module for identifying a candidate period of mutual occlusion between dancers based on human body key point positions and corresponding observation reliabilities of adjacent time points in the time sequence; a temporary contact occlusion identification module for identifying a temporary contact occlusion period with an identity confusion risk based on the observation view angle and human body key point positions before and after the candidate period, which violates at least one constraint of multi-view observation consistency, human skeleton length consistency and key point motion continuity; an identity maintenance and key point completion module for maintaining the identity of the human body key point of the dancer involved in the occlusion unchanged and predicting and completing human body key point positions with an observation reliability lower than a preset threshold to obtain completion data within the temporary contact occlusion period; an identity reassignment module for reassigning the identity of each human body key point position to restore the correct association between the human body key point and the identity after the temporary contact occlusion period ends according to the human body key point position after the end, the completion data and the human skeleton length consistency and key point motion continuity constraints; and a dance trajectory generation module for generating a complete three-dimensional dance motion trajectory of each dancer according to the association between the human body key point and the identity after the restoration.

[0107] In one example, the identifying the candidate period in which mutual occlusion occurs between dancers includes: in adjacent time points, calculating spatial distances of human body key points between different dancers based on human body key point positions of the dancers; determining whether the spatial distances are less than a preset occlusion distance threshold, and if so, identifying the corresponding time period as a candidate period in which mutual occlusion occurs between dancers.

[0108] In one example, the multi-view observation consistency identification includes: calculating a deviation value of human body key point positions of the same human body key point under different observation angles; determining whether the deviation value is greater than a preset deviation threshold, and if so, determining that the observation angle consistency is violated.

[0109] In one example, the human body skeleton length consistency identification includes: extracting human body key point positions of each dancer at multiple time points before and after the candidate period as sample key point positions according to the human body key point position set; calculating an average value of skeleton lengths between human body key points connected to each other as a human body skeleton reference value of the dancer based on the sample key point positions at the multiple time points; calculating a current skeleton length based on the human body key point positions of each dancer within the candidate period; determining whether a difference between the current skeleton length of each dancer and the human body skeleton reference value is greater than a preset length threshold, and if so, determining that the human body skeleton length consistency is violated.

[0110] In one example, the key point motion continuity identification includes: calculating motion trends of each human body key point before and after the candidate period based on human body key point positions of each dancer at adjacent time points; calculating motion trends of each dancer within the candidate period according to human body key point positions of each dancer at each time point within the candidate period; determining whether a deviation between the motion trends within the candidate period and the motion trends before and after the candidate period is greater than a preset motion deviation threshold, and if so, determining that the key point motion continuity is violated.

[0111] In one example, the prediction completion includes: within the temporary contact occlusion period, for human body key point positions with observation reliability lower than a preset threshold, obtaining human body key point positions and key point motion trends of the corresponding dancer before the start of the temporary contact occlusion period; predicting positions of the human body key point within the temporary contact occlusion period based on the human body key point positions and the key point motion trends, to obtain predicted key point positions; in the prediction process, imposing a human body skeleton length constraint and a key point motion continuity constraint of the corresponding dancer on the predicted key point positions to limit a change range of the predicted key point positions; taking the predicted key point positions that satisfy the human body skeleton length constraint and the key point motion continuity constraint as completion data obtained by the prediction completion.

[0112] In one example, the re-allocating the identities of the human key point positions comprises: extracting, according to the set of human key point positions, human key point positions of each dancer after the temporary contact occlusion period ends as candidate key point positions; calculating, based on the candidate key point positions and the completion data, a matching cost between each candidate key point position and each dancer, the matching cost at least including a multi-view observation consistency cost, a human skeleton length consistency cost, and a key point motion continuity cost; determining, according to the matching cost corresponding to each candidate key point position, an identity allocation result that satisfies a global consistency constraint; and taking the identity allocation result as the re-allocated result of the human key point positions, to restore the correct association relationship between the human key point positions and the corresponding dancer identities.

[0113] An example electronic device:

[0114] Figure 3 A block diagram of an electronic device according to embodiments of the present application is shown.

[0115] As shown in Figure 3 , the electronic device includes one or more processors and a memory.

[0116] The processor can be a central processing unit (CPU) or other form of processing unit having data processing and / or instruction execution capabilities, and can control other components in the electronic device to perform desired functions.

[0117] The memory can include one or more computer program products that can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory, for example, can include random access memory (RAM), cache, and / or the like. Non-volatile memory, for example, can include read-only memory (ROM), hard disk, flash memory, and / or the like.

[0118] In one example, the electronic device can further include an input device and an output device, which are interconnected through a bus system and / or other forms of connection mechanism (not shown).

[0119] Of course, in order to simplify, Figure 3 only some of the components in the electronic device related to the present application are shown, and components such as buses, input / output interfaces, and the like are omitted. In addition, depending on the specific application, the electronic device can also include any other appropriate components.

[0120] An example computer-readable medium:

[0121] Embodiments of the present application can also be computer readable storage media storing computer program instructions thereon which, when executed by a processor, cause the processor to perform the steps described in the above "Exemplary Methods" section of this specification according to various embodiments of the present application.

[0122] The computer readable storage medium can be any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0123] The above describes the basic principles of the present application in conjunction with specific embodiments, but it should be noted that the advantages, benefits, effects and the like mentioned in the present application are only examples and are not limiting, and these advantages, benefits, effects and the like cannot be considered as necessary for each embodiment of the present application. In addition, the above specific details are only for the purpose of example and understanding, and are not limiting, and the above details do not limit the present application to the above specific details.

[0124] The block diagrams of the devices, apparatuses, equipment, systems involved in the present application are only illustrative examples and are not intended to require or imply the connection, arrangement, configuration shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, equipment, systems can be connected, arranged, configured in any manner. Words such as "include", "contain", "have" and the like are open-ended words, meaning "including but not limited to", and can be used interchangeably. The words "or" and "and" used herein mean the word "and / or", and can be used interchangeably unless the context clearly indicates otherwise. The word "such as" used herein means the phrase "such as but not limited to", and can be used interchangeably.

[0125] It should also be noted that in the devices, apparatuses and methods of the present application, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions of the present application.

[0126] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other aspects without departing from the scope of the application. Thus, the present application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0127] The above description has been presented to enable any person skilled in the art to make or use the application. Numerous modifications and alterations to this application will be readily apparent to those skilled in the art, and it is intended to include all such modifications and alterations insofar as they come within the scope of the general concepts defined herein.

Claims

1. A method for three-dimensional reconstruction and optimization of dance movement trajectories, characterized in that, include: Obtain a set of three-dimensional human key point positions of multiple dancers in a group dance performance over time. Each human key point position is associated with the observation perspective, observation reliability, and the corresponding dancer's identity identifier. Based on the location of human key points at adjacent time points in the time series and the corresponding observation reliability, candidate time periods in which dancers occlude each other are identified. Based on the observation perspective and the positions of key human body points before and after the candidate time period, candidate time periods that violate the consistency of multi-view observation and the consistency of human skeleton length and the continuity of key point movement are identified as temporary contact occlusion time periods with the risk of identity confusion. During the temporary contact occlusion period, the identity identifiers of the human body key points corresponding to the dancers participating in the occlusion remain unchanged, and the positions of human body key points with observation reliability lower than a preset threshold are predicted and completed to obtain completed data. After the temporary contact occlusion period ends, based on the human body key point positions after the end, the supplementary data, and the constraints of multi-view observation consistency, human skeleton length consistency, and key point movement continuity, the identity labels of each human body key point position are redistributed to restore the correct association between human body key points and identity labels. Based on the correlation between the restored human body key points and identity markers, complete three-dimensional dance movement trajectories for each dancer are generated.

2. The method for three-dimensional reconstruction and optimization of dance movement trajectories according to claim 1, characterized in that, The candidate time periods for identifying mutual occlusion between dancers include: At adjacent time points, the spatial distance between the human body key points of different dancers is calculated based on the corresponding human body key point positions of each dancer. Determine whether the spatial distance is less than a preset occlusion distance threshold. If so, identify the corresponding time period as a candidate time period in which dancers may occlude each other.

3. The method for three-dimensional reconstruction and optimization of dance movement trajectories according to claim 1, characterized in that, The identification of consistency in multi-view observations includes: Calculate the deviation of the position of the same human body key point under different observation angles; Determine whether the deviation value is greater than a preset deviation threshold. If so, it is determined to be a violation of the consistency of the observation perspective.

4. The method for three-dimensional reconstruction and optimization of dance movement trajectories according to claim 1, characterized in that, The identification of the consistency of human skeletal length includes: Based on the set of human body key point locations, extract the human body key point locations of each dancer at multiple time points before and after the candidate time period as sample key point locations; Based on the sample key point positions at multiple time points, the average skeletal length between interconnected human key points is calculated as a reference value for the human skeleton of the dancer. During the candidate time period, the current skeleton length is calculated based on the positions of the human body key points corresponding to each dancer; Determine whether the difference between the current skeleton length of each dancer and the reference value of the human skeleton is greater than a preset length threshold. If so, it is determined to be a violation of the consistency of human skeleton length.

5. The method for three-dimensional reconstruction and optimization of dance movement trajectories according to claim 1, characterized in that, The identification of the motion continuity of the key points includes: Before and after the candidate time period, based on the positions of the human body key points of each dancer at adjacent time points, the movement trend of each human body key point before and after the candidate time period is calculated; Based on the positions of key points on the human body of each dancer at each time point within the candidate time period, calculate the movement trend of each dancer within the candidate time period. Determine whether the deviation between the motion trend within the candidate time period and the motion trend before and after the candidate time period is greater than a preset motion deviation threshold. If so, it is determined to be a violation of the motion continuity of key points.

6. The method for three-dimensional reconstruction and optimization of dance movement trajectories according to claim 1, characterized in that, The prediction completion includes: During the temporary contact occlusion period, for the human body key point positions where the observation reliability is lower than the preset threshold, the corresponding dancer's human body key point positions and key point movement trends before the start of the temporary contact occlusion period are obtained. Based on the location and movement trend of the key points of the human body, the location of the key points of the human body during the temporary contact occlusion period is predicted to obtain the predicted key point location. During the prediction process, constraints on the length of the dancer's human skeleton and the continuity of key point movement are applied to the predicted key point positions to limit the range of variation of the predicted key point positions. The predicted keypoint positions that satisfy the constraints on human skeleton length and keypoint motion continuity are used as the completion data obtained from the prediction completion.

7. The method for three-dimensional reconstruction and optimization of dance movement trajectories according to claim 1, characterized in that, The reassignment of identity markers for key points on each human body includes: Based on the set of human body key point locations, extract the human body key point locations of each dancer after the temporary contact occlusion period ends as candidate key point locations. Based on the candidate keypoint locations and the completion data, the matching cost between each candidate keypoint location and each dancer is calculated. The matching cost includes at least the multi-view observation consistency cost, the human skeleton length consistency cost, and the keypoint motion continuity cost. Based on the matching cost corresponding to the location of each candidate key point, the identity identifier allocation result that satisfies the global consistency constraint is determined; The identity identifier allocation result is used as the redistribution result of each human body key point position to restore the correct association between the human body key point position and the corresponding dancer identity identifier.

8. A three-dimensional reconstruction and optimization system for dance movement trajectories, characterized in that, include: The data acquisition module is used to acquire a set of three-dimensional human key point positions of multiple dancers in a group dance performance over time. Each human key point position is associated with the observation perspective, observation reliability, and the corresponding dancer's identity identifier. The candidate occlusion period identification module is used to identify candidate periods in which dancers occlude each other based on the positions of human key points at adjacent time points in the time series and the corresponding observation reliability. The temporary contact occlusion recognition module is used to identify, based on the observation angle and the positions of key human body points before and after the candidate time period, the candidate time period that violates at least one of the constraints of multi-view observation consistency, human skeleton length consistency and key point movement continuity as a temporary contact occlusion time period with the risk of identity confusion. The identity maintenance and key point completion module is used to maintain the identity identifier of the human body key points corresponding to the dancers involved in the temporary contact occlusion during the temporary contact occlusion period, and to predict and complete the positions of human body key points with observation reliability lower than a preset threshold to obtain completed data. The identity reassignment module is used to reassign the identity identifiers of each human key point position after the temporary contact occlusion period ends, based on the human key point position after the end, the supplementary data, and the constraints of human skeleton length consistency and key point movement continuity, so as to restore the correct association between human key points and identity identifiers. The dance trajectory generation module is used to generate complete 3D dance movement trajectories for each dancer based on the correlation between the restored human body key points and identity identifiers.

9. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method as described in any one of claims 1 to 7.

10. A computer storage medium storing computer-executable instructions thereon, characterized in that: When the computer-executable instructions are executed by a processor, they implement the steps of the method as described in any one of claims 1 to 7.

Images