Human motion optimization method and system for multi-lens video, terminal and storage medium

Abstract

The invention discloses a human motion optimization method and system for a multi-lens video, a terminal and a storage medium, and relates to the technical field of computer vision. The method comprises the following steps: performing shot switching point detection and multi-level segmentation on a multi-shot video frame sequence to obtain a target video frame sequence; calculating a target camera track and a human body motion parameter according to the target video frame sequence; and obtaining a target human body motion parameter of the multi-shot video frame sequence based on the target camera track and the human body motion parameter. The method comprises the following steps: firstly, segmenting a multi-shot video frame sequence into a plurality of target video frame sequences without shot switching points according to the shot switching points; and calculating a target camera track and a human body motion parameter according to each target video frame sequence. And finally, the human body motion state in the video is analyzed by combining the target camera track and the human body motion parameters, so that target human body motion parameters with high precision, continuity and physical consistency are obtained. The method is suitable for various application scenes such as motion capture, virtual reality and sports analysis.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, system, terminal and storage medium for optimizing human motion in multi-lens video. Background Art

[0002] With the rapid development of computer vision and 3D reconstruction technology, 3D human posture and motion recovery has been widely used in many fields, such as motion capture, virtual reality, film and television production, sports analysis, and human-computer interaction.

[0003] Existing research mainly focuses on human motion recovery from single-lens videos. Although it is possible to accurately reconstruct the 3D posture of the human body from a single camera perspective, it still has the following shortcomings when processing videos containing multi-lens switching: 1. The switching of lenses often leads to discontinuity in human posture and direction, affecting the overall motion recovery effect; 2. The interference of dynamic objects on camera trajectory estimation is very obvious in some scenarios, especially when humans occupy most of the image, it is difficult to accurately extract a stable camera trajectory; 3. Foot sliding and trajectory jitter are common motion recovery problems, especially in dynamic environments where human motion may be affected by ground conditions or gait changes.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the invention

[0005] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a method, system, terminal and storage medium for optimizing human motion in multi-lens video are provided, aiming to solve the problem that the existing human motion recovery technology for videos is mainly applied to single-lens video, and when applied to multi-lens video, it is easy to cause discontinuous motion due to lens switching.

[0006] The technical solution adopted by the present invention to solve the problem is as follows: In a first aspect, an embodiment of the present invention provides a method for optimizing human motion in a multi-lens video, the method comprising: Obtain a multi-lens video frame sequence; Performing shot switching point detection on the multi-shot video frame sequence, and performing multi-level segmentation on the multi-shot video frame sequence according to the detected shot switching points to obtain a target video frame sequence; Calculating target camera trajectory and human motion parameters according to the target video frame sequence; The target human motion parameters of the multi-lens video frame sequence are obtained based on the target camera trajectory and the human motion parameters.

[0007] In one implementation, performing shot switching point detection on the multi-shot video frame sequence, and performing multi-level segmentation on the multi-shot video frame sequence according to the detected shot switching points to obtain a target video frame sequence includes: Performing shot switching point detection on the multi-shot video frame sequence to obtain a first shot switching point video frame, a second shot switching point video frame, and a third shot switching point video frame respectively; The multi-lens video frame sequence is multi-level segmented according to the first lens switching point video frame, the second lens switching point video frame and the third lens switching point video frame to obtain a first video frame sequence, a second video frame sequence and the target video frame sequence respectively.

[0008] In one implementation, performing shot switching point detection on the multi-shot video frame sequence to obtain a first shot switching point video frame, a second shot switching point video frame, and a third shot switching point video frame respectively includes: Determining a first shot switching point video frame by detecting a scene change of each video frame in the multi-shot video frame sequence; Determine a second shot switching point video frame by detecting a human body boundary frame of each video frame in the multi-shot video frame sequence; The third shot switching point video frame is determined by detecting the human body key points of each video frame in the multi-shot video frame sequence.

[0009] In one implementation, the multi-shot video frame sequence is segmented at multiple levels according to the first shot switching point video frame, the second shot switching point video frame, and the third shot switching point video frame to obtain a first video frame sequence, a second video frame sequence, and the target video frame sequence, respectively, including: Segment the multi-shot video frame sequence according to the first shot switching point video frame to obtain the first video frame sequence, where the first video frame sequence includes a plurality of first video frame subsequences; The first video frame sequence is divided according to the second shot switching point video frame to obtain the second video frame sequence, where the second video frame sequence includes a plurality of second video frame subsequences; The second video frame sequence is segmented according to the third shot switching point video frame to obtain the target video frame sequence, where the target video frame sequence includes a plurality of target video frame subsequences.

[0010] In one implementation, determining a first shot switching point video frame by detecting a scene change of each video frame in the multi-shot video frame sequence includes: Taking each non-first frame in the multi-lens video frame sequence as a current video frame, calculating the background difference between the current video frame and the previous video frames, wherein the background difference is used to reflect the degree of scene change; Determine a first shot switching point video subframe based on the background difference and a preset difference threshold; The first shot switching point video frame is obtained according to all the first shot switching point video sub-frames.

[0011] In one implementation, determining the second shot switching point video frame by detecting a human body bounding box of each video frame in the multi-shot video frame sequence includes: For any two adjacent video frames in any of the first video frame subsequences, calculating the degree of overlap of human body bounding boxes of the two adjacent video frames; Determining a second shot switching point video subframe based on the overlap degree and a preset overlap threshold; The second shot switching point video frame is obtained according to all the second shot switching point video sub-frames.

[0012] In one implementation, determining a third shot switching point video frame by detecting a human body key point of each video frame in the multi-shot video frame sequence includes: For any two adjacent video frames in any of the second video frame subsequences, obtain true values ​​and predicted values ​​of human body key points of the two adjacent video frames; Based on the real values ​​and predicted values ​​of the human body key points of the two adjacent video frames, a third shot switching point video subframe is obtained; The third shot switching point video frame is obtained according to all the third shot switching point video sub-frames.

[0013] In one implementation, obtaining the true value and the predicted value of the human body key points of the two adjacent video frames includes: Detecting the human key points of the two adjacent video frames by a human key point detection algorithm to obtain the true values ​​of the human key points of the two adjacent video frames; The human body key points of the two adjacent video frames are predicted by a preset filtering algorithm to obtain predicted values ​​of the human body key points of the two adjacent video frames.

[0014] In one implementation, obtaining a third shot switching point video subframe based on the real values ​​and predicted values ​​of the human body key points of the two adjacent video frames includes: Calculate the matching value according to the real value and the predicted value of the human body key point of the two adjacent video frames; The matching value is compared with a preset matching threshold, and a later video frame of the two adjacent video frames is selected as the third shot switching point video subframe.

[0015] In one implementation, calculating the target camera trajectory and human motion parameters according to the target video frame sequence includes: Calculating a target camera trajectory according to the target video frame sequence; Based on the target video frame sequence and the target camera trajectory, a human body motion parameter of the target video frame sequence is obtained.

[0016] In one implementation, calculating a target camera trajectory according to the target video frame sequence includes: The target video frame sequence includes a plurality of target video frame subsequences; Selecting a target pixel point of each video frame according to a gradient map of each video frame in any of the target video frame subsequences; Tracking the target pixel points in each video frame by a preset tracking algorithm to obtain a target motion trajectory corresponding to the target pixel points; Acquire an initial camera trajectory of any of the target video frame subsequences, and optimize the initial camera trajectory according to the target motion trajectory of the target pixel point to obtain a target camera trajectory of the target video frame subsequence; According to the target camera trajectories in all target video frame subsequences, the target camera trajectory of the target video frame sequence is obtained.

[0017] In one implementation, optimizing the initial camera trajectory according to the target motion trajectory of the target pixel point to obtain the target camera trajectory of the target video frame subsequence includes: Selecting a number of video frames to be optimized from the target video frame subsequence; Optimizing the camera extrinsic parameters according to the target motion trajectory of the target pixel point in each of the video frames to be optimized to obtain optimized camera extrinsic parameters; The target camera trajectory of the target video frame subsequence is calculated according to the optimized camera extrinsic parameters.

[0018] In one implementation, optimizing the camera extrinsic parameters according to the target motion trajectory of the target pixel point in each of the video frames to be optimized to obtain the optimized camera extrinsic parameters includes: Calculating the reprojection error of the target motion trajectory of the target pixel point in each of the video frames to be optimized; The optimized camera extrinsic parameters are obtained according to the reprojection error and a preset error threshold.

[0019] In one implementation, obtaining the target human motion parameter of the multi-lens video frame sequence based on the target camera trajectory and the human motion parameter includes: Inputting the target video frame sequence and the target camera trajectory of the target video frame sequence into a preset first model to obtain human body motion parameters of the target video frame sequence; The human body bounding box of each video frame in the multi-lens video frame sequence, the target camera trajectory and the human body motion parameters are input into a preset second model to obtain the target human body motion parameters of the multi-lens video frame sequence.

[0020] In one implementation, inputting the human body bounding box of each video frame in the multi-lens video frame sequence, the target camera trajectory, and the human body motion parameters into a preset second model to obtain the target human body motion parameters of the multi-lens video frame sequence includes: De-noising the human body motion parameters to obtain de-noised human body motion parameters; The human body bounding box of each video frame in the multi-lens video frame sequence, the target camera trajectory and the denoised human body motion parameters are input into a preset second model, and the denoised human body motion parameters are smoothed to obtain the target human body motion parameters of the multi-lens video frame sequence.

[0021] In one implementation, obtaining the human motion parameters of the target video frame sequence further includes: According to any two adjacent target video frame subsequences in the target video frame sequence, a plurality of adjacent target video frame pairs are obtained; For any pair of adjacent target video frames, calculating a camera rotation difference value of the pair of adjacent target video frames; Correcting the human body direction of the subsequent video frame in the adjacent target video frame pair according to the camera rotation difference value and a preset difference threshold value to obtain a corrected human body direction of the adjacent target video frame pair; Based on all corrected human body directions of the adjacent target video frame pairs, human body motion parameters of the target video frame sequence are obtained.

[0022] In one embodiment, the method further comprises: Calculating foot contact data of each video frame according to the target human body motion parameter of each video frame in the multi-lens video frame sequence; Calculating the human body motion speed of each video frame based on the target human body motion parameter and the foot contact data of each video frame in the multi-lens video frame sequence; The human body motion speed of each video frame in the multi-lens video frame sequence is accumulated to obtain the global motion trajectory of the human body.

[0023] In one embodiment, calculating the human body motion speed of each video frame based on the target human body motion parameter and the foot contact data of each video frame in the multi-lens video frame sequence includes: The target human motion parameters and the foot contact data of each video frame in the multi-lens video frame sequence are respectively input into a preset neural network to obtain the human motion speed of each video frame.

[0024] In a second aspect, an embodiment of the present invention further provides a human motion optimization system for multi-lens video, the system comprising: A video acquisition module, used to acquire multi-lens video frame sequences; A video segmentation module, used to detect shot switching points of the multi-shot video frame sequence, and perform multi-level segmentation on the multi-shot video frame sequence according to the detected shot switching points to obtain a target video frame sequence; A video analysis module, used for calculating a target camera trajectory and human motion parameters according to the target video frame sequence; A motion optimization module is used to obtain target human motion parameters of the multi-lens video frame sequence based on the target camera trajectory and the human motion parameters.

[0025] In a third aspect, an embodiment of the present invention further provides a terminal, comprising a memory and one or more processors; the memory stores one or more programs; the program comprises instructions for executing a human motion optimization method for multi-lens video as described in any one of the above; and the processor is used to execute the program.

[0026] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a plurality of instructions are stored, wherein the instructions are suitable for being loaded and executed by a processor to implement the steps of the method for optimizing human motion in multi-lens videos as described in any of the above.

[0027] Beneficial effects of the present invention: The embodiment of the present invention obtains a multi-lens video frame sequence; performs lens switching point detection on the multi-lens video frame sequence, and performs multi-level segmentation on the multi-lens video frame sequence according to the detected lens switching points to obtain a target video frame sequence; calculates the target camera trajectory and human motion parameters according to the target video frame sequence; and obtains the target human motion parameters of the multi-lens video frame sequence based on the target camera trajectory and the human motion parameters. The present invention first divides the multi-lens video frame sequence into a number of target video frame sequences that do not contain lens switching points according to the lens switching points. Then, the target camera trajectory and human motion parameters are calculated according to each target video frame sequence. Finally, the target camera trajectory and human motion parameters are combined to analyze the human motion state in the video, thereby obtaining target human motion parameters with high precision, continuity and physical consistency. It is suitable for a variety of application scenarios such as motion capture, virtual reality, and sports analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1 It is a flow chart of a method for optimizing human motion in multi-lens video provided by an embodiment of the present invention.

[0030] Figure 2 It is a complete flow chart of the method for optimizing human motion in multi-lens video provided by an embodiment of the present invention.

[0031] Figure 3 It is a schematic diagram of data flow of each module in the human motion optimization method for multi-lens video provided by an embodiment of the present invention.

[0032] Figure 4 Schematic diagram of a human body trajectory fine-tuning module provided in an embodiment of the present invention.

[0033] Figure 5 It is a module schematic diagram of a human motion optimization system for multi-lens video provided by an embodiment of the present invention.

[0034] Figure 6 It is a principle block diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The present invention discloses a method, system, terminal and storage medium for optimizing human motion in multi-lens video. In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0036] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0037] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as herein.

[0038] In view of the above-mentioned defects of the prior art, the present invention provides a method for optimizing human motion in multi-lens video, the method comprising: obtaining a multi-lens video frame sequence; performing lens switching point detection on the multi-lens video frame sequence, and performing multi-level segmentation on the multi-lens video frame sequence according to the detected lens switching points to obtain a target video frame sequence; calculating a target camera trajectory and human motion parameters according to the target video frame sequence; and obtaining the target human motion parameters of the multi-lens video frame sequence based on the target camera trajectory and the human motion parameters. The present invention firstly segments the multi-lens video frame sequence into a plurality of target video frame sequences excluding lens switching points according to the lens switching points. Then, the target camera trajectory and human motion parameters are calculated according to each target video frame sequence. Finally, the human motion state in the video is analyzed by combining the target camera trajectory and human motion parameters, so as to obtain target human motion parameters with high precision, continuity and physical consistency. It is suitable for a variety of application scenarios such as motion capture, virtual reality, and sports analysis.

[0039] like Figure 1 As shown, the method specifically comprises the following steps: Step S100: Acquire a multi-lens video frame sequence.

[0040] Step S200: performing shot switching point detection on the multi-shot video frame sequence, and performing multi-level segmentation on the multi-shot video frame sequence according to the detected shot switching points to obtain a target video frame sequence.

[0041] First, a multi-lens video frame sequence is input to obtain a basic data source for subsequent processing. The multi-lens video frame sequence contains multiple video frames, wherein the video frames in the multi-lens video frame sequence have perspective or scene switching, but the movements of people in the video are continuous before and after the video scene switching. This embodiment requires segmentation processing of the multi-lens video frame sequence. Specifically, the multi-lens video frame sequence can be subjected to lens switching point detection by a lens switching detector. The lens switching point refers to the timestamp of the previous video frame in two adjacent video frames with significant differences in picture content. The video frame where each lens switching point is located can be identified by lens switching point detection. The multi-lens video frame sequence is segmented at multiple levels through the video frames where these lens switching points are located to obtain a target video frame sequence that does not contain lens switching points, and the target video frame sequence needs to be processed subsequently.

[0042] In one implementation, the multi-lens video frame sequence is subjected to shot switching point detection, and the multi-lens video frame sequence is subjected to multi-level segmentation according to the detected shot switching points to obtain a target video frame sequence, including: Performing shot switching point detection on the multi-shot video frame sequence to obtain a first shot switching point video frame, a second shot switching point video frame, and a third shot switching point video frame respectively; The multi-lens video frame sequence is multi-level segmented according to the first lens switching point video frame, the second lens switching point video frame and the third lens switching point video frame to obtain a first video frame sequence, a second video frame sequence and the target video frame sequence respectively.

[0043] Specifically, in a multi-lens video frame sequence, the previous video frame in the adjacent video frames where the picture content changes significantly is the video frame corresponding to the lens switching point. This embodiment will identify video frames corresponding to lens switching points of multiple different levels through lens switching point detection, that is, obtain a first lens switching point video frame, a second lens switching point video frame and a third lens switching point video frame, wherein the first lens switching point video frame includes at least one first lens switching point video subframe, the second lens switching point video frame includes at least one second lens switching point video subframe, and the third lens switching point video frame includes at least one third lens switching point video subframe. The multi-lens video frame sequence is segmented by obtaining the first lens switching point video frame, the second lens switching point video frame and the third lens switching point video frame to obtain video frame sequences of different levels and multiple length ranges, namely, a first video frame sequence, a second video frame sequence and a target video frame sequence. Among them, the target video frame sequence is the most refined segmentation result, and the target video frame sequence does not contain lens switching points.

[0044] In one implementation, performing shot switching point detection on the multi-shot video frame sequence to obtain a first shot switching point video frame, a second shot switching point video frame, and a third shot switching point video frame respectively includes: Determining a first shot switching point video frame by detecting a scene change of each video frame in the multi-shot video frame sequence; Determine a second shot switching point video frame by detecting a human body boundary frame of each video frame in the multi-shot video frame sequence; The third shot switching point video frame is determined by detecting the human body key points of each video frame in the multi-shot video frame sequence.

[0045] like Figure 2As shown, this embodiment adopts a combination strategy of scene change detection, human bounding box tracking and key point tracking, and realizes shot switching point detection through a cascade cut-shot detector. The identified multiple shot switching points can segment the multi-shot video frame sequence into multiple video frames, each of which contains at least one video frame, and finally realize multi-level segmentation of the multi-shot video frame sequence. Specifically, by analyzing the background difference between consecutive video frames in the multi-shot video frame sequence, the video frame with significant scene changes is identified as the first shot switching point video subframe. All the first shot switching point video subframes of the identified multi-shot video frame sequence are aggregated to obtain the first shot switching point video frame, and the first shot switching point video frame contains at least one first shot switching point video subframe. In each first video frame subsequence, the human bounding box (Bounding Box) is obtained by detecting the position and size of the person in each video frame in the first video frame sequence, and then the video frame with significant changes in the person position is identified to obtain the second shot switching point video subframe. All the second shot switching point video subframes are aggregated to obtain the second shot switching point video frame, and the second shot switching point video frame contains at least one second shot switching point video subframe. In each second video frame subsequence, by detecting key points of the human body, a video frame in which the action or posture of the person changes significantly is identified to obtain a third shot switching point video subframe. All third shot switching point video subframes are aggregated to obtain a third shot switching point video frame, and the third shot switching point video frame includes at least one third shot switching point video subframe.

[0046] In one implementation, the multi-shot video frame sequence is multi-level segmented according to the first shot switching point video frame, the second shot switching point video frame, and the third shot switching point video frame to obtain a first video frame sequence, a second video frame sequence, and the target video frame sequence, respectively, including: Segment the multi-shot video frame sequence according to the first shot switching point video frame to obtain the first video frame sequence, where the first video frame sequence includes a plurality of first video frame subsequences; The first video frame sequence is divided according to the second shot switching point video frame to obtain the second video frame sequence, where the second video frame sequence includes a plurality of second video frame subsequences; The second video frame sequence is segmented according to the third shot switching point video frame to obtain the target video frame sequence, where the target video frame sequence includes a plurality of target video frame subsequences.

[0047] Specifically, the first level segmentation is implemented based on the scene change detection technology: obtaining the timestamps of all first shot switching point video frames, wherein the first shot switching point video frames are video frames in which the scene changes significantly in the multi-shot video frame sequence, segmenting the multi-shot video frame sequence by the timestamps of all first shot switching point video frames, obtaining a plurality of first video frame subsequences, and summarizing all the first video frame subsequences to obtain a first video frame sequence. The second level segmentation is implemented based on the target detection algorithm and the preset filtering algorithm: obtaining the timestamps of all second shot switching point video frames, wherein the second shot switching point video frames in the first video frame sequence are video frames in which the position of the characters changes significantly, segmenting the first video frame sequence by the timestamps of all second shot switching point video frames, obtaining a plurality of second video frame subsequences, and summarizing all the second video frame subsequences to obtain a second video frame sequence. The third level segmentation is implemented based on the human key point algorithm: obtaining the timestamps of all third shot switching points, wherein the third shot switching point video frames in the second video frame sequence are video frames in which the action or posture of the characters changes significantly, segmenting the second video frame sequence by the timestamps of all third shot switching points, and obtaining a plurality of target video frame subsequences. All target video frame subsequences are aggregated to obtain a target video frame sequence.

[0048] In one implementation, determining a first shot switching point video frame by detecting a scene change of each video frame in the multi-shot video frame sequence includes: Taking each non-first frame in the multi-lens video frame sequence as a current video frame, calculating the background difference between the current video frame and the previous video frames, wherein the background difference is used to reflect the degree of scene change; Determine a first shot switching point video subframe based on the background difference and a preset difference threshold; The first shot switching point video frame is obtained according to all the first shot switching point video sub-frames.

[0049] Specifically, the input multi-lens video frame sequence is obtained, and the background difference between each video frame and the previous several video frames is calculated by the scene change detection algorithm. The larger the background difference, the higher the degree of scene change between the video frame and the previous video frame. When the background difference of the video frame is greater than the preset difference threshold, it means that there is a significant scene change in the video frame, and the video frame is used as a first lens switching point video subframe, and the timestamp of the first lens switching point video frame is recorded. Repeat the above steps, identify all the first lens switching point video subframes in turn, and form the first lens switching point video frame. In actual application scenarios, scene changes can be detected by background subtraction algorithms and optical flow methods of background modeling. Background modeling algorithms (such as Gaussian mixture models) can effectively extract background and foreground differences, and the optical flow law detects scene changes by calculating the motion of adjacent frames.

[0050] In one implementation, determining the second shot switching point video frame by detecting a human body bounding box of each video frame in the multi-shot video frame sequence includes: For any two adjacent video frames in any of the first video frame subsequences, calculating the degree of overlap of human body bounding boxes of the two adjacent video frames; Determining a second shot switching point video subframe based on the overlap degree and a preset overlap threshold; The second shot switching point video frame is obtained according to all the second shot switching point video sub-frames.

[0051] Specifically, the human body bounding box of each video frame in each first video frame subsequence is tracked by the target detection algorithm. Taking a first video frame subsequence as an example, for any two adjacent video frames in the first video frame subsequence, after the human body bounding boxes in the two adjacent video frames are detected by the target tracking algorithm, the overlap degree of the two human body bounding boxes of the two adjacent video frames is evaluated. For example, the intersection over union (IoU) of the two human body bounding boxes can be calculated, and the overlap degree of the two human body bounding boxes can be quantified by the intersection over union value. If the overlap degree is less than the preset overlap threshold, it means that the position of the human body has changed significantly, then the latter video frame of the two adjacent video frames is used as a second shot switching point video subframe, and the timestamp of the second shot switching point video subframe is recorded. All the second shot switching point video subframes are aggregated to obtain the second shot switching point video frame. It should be noted that the first video frame sequence includes multiple first video frame subsequences, and calculating the overlap degree of the human body bounding boxes of any two adjacent video frames in the first video frame sequence actually means: calculating the overlap degree of the human body bounding boxes for any two adjacent video frames in any first video frame subsequence. In actual application scenarios, human bounding box tracking can be achieved through target detection algorithms such as SORT and DeepSORT. Among them, the SORT algorithm uses Kalman filtering and Hungarian algorithm to track targets, and the DeepSORT algorithm introduces deep learning features to enhance tracking accuracy in complex environments. These target detection algorithms can detect obvious changes in perspective between video frames, thereby achieving human bounding box tracking in video frames, effectively solving problems such as lens switching and dynamic object interference, and thus facilitating subsequent posture correction and motion optimization.

[0052] In one implementation, determining a third shot switching point video frame by detecting a human body key point of each video frame in the multi-shot video frame sequence includes: For any two adjacent video frames in any of the second video frame subsequences, obtain true values ​​and predicted values ​​of human body key points of the two adjacent video frames; Based on the real values ​​and predicted values ​​of the human body key points of the two adjacent video frames, a third shot switching point video subframe is obtained; The third shot switching point video frame is obtained according to all the third shot switching point video sub-frames.

[0053] Specifically, for any two adjacent video frames in any second video frame subsequence, the true values ​​of the human body key points in the two adjacent video frames are detected, and the predicted values ​​of the human body key points are calculated at the same time. By analyzing the true values ​​and predicted values ​​of the human body key points of two adjacent videos, the video frame in which the character's action or posture changes significantly is identified, and the video frame is used as a third lens switching point video subframe, and the timestamp of the video frame is recorded. All third lens switching point video subframes are aggregated to obtain a third lens switching point video frame. It should be noted that the second video frame sequence includes multiple second video frame subsequences, and calculating the true values ​​and predicted values ​​of the human body key points of any two adjacent video frames in the second video frame sequence actually means: calculating the true values ​​and predicted values ​​of the human body key points for any two adjacent video frames in any second video frame subsequence.

[0054] For example, taking a second video frame subsequence as an example, select any two adjacent video frames, calculate the true value and predicted value of the human body key points of the two adjacent video frames, and by comparing the true value and predicted value of the human body key points, determine whether the character action or posture in the latter video frame has changed significantly relative to the previous video frame. If the character action or posture has changed significantly, the latter video frame is used as a third lens switching point video subframe, and the timestamp of the video frame is recorded. This embodiment accurately locates the tiny third lens switching point video subframe by analyzing the movement of human body key points (such as joint positions) between adjacent frames. The recognition accuracy of the lens switching point is improved, especially in scenes with complex human movements.

[0055] In one implementation, obtaining the true value and the predicted value of the human body key points of the two adjacent video frames includes: Detecting the human key points of the two adjacent video frames by a human key point detection algorithm to obtain the true values ​​of the human key points of the two adjacent video frames; The human body key points of the two adjacent video frames are predicted by a preset filtering algorithm to obtain predicted values ​​of the human body key points of the two adjacent video frames.

[0056] Specifically, for each second video frame subsequence, the true value of the human key point in each video frame is detected by a human key point detection algorithm, and the human key points in adjacent video frames are predicted by a preset filtering algorithm (such as a Kalman filtering algorithm) to obtain the predicted value of the human key point. By comparing the true value and the predicted value of the human key point, the motion trajectory of the human key point between two adjacent video frames can be analyzed, and then the tiny third lens switching point video subframe can be accurately located. In actual application scenarios, the threshold and motion model of the human key point detection algorithm can be adaptively adjusted so that the human key point detection algorithm can effectively identify lens switching in high dynamic scenes, avoiding the omission of switching points due to scene complexity.

[0057] In one implementation, obtaining a third shot switching point video subframe based on the real values ​​and predicted values ​​of the human body key points of the two adjacent video frames includes: Calculate the matching value according to the real value and the predicted value of the human body key point of the two adjacent video frames; The matching value is compared with a preset matching threshold, and a later video frame of the two adjacent video frames is selected as the third shot switching point video subframe.

[0058] Specifically, for any two adjacent video frames in each second video frame subsequence, the predicted values ​​and true values ​​of the human body key points of the two adjacent video frames are matched to obtain a matching value. The motion trajectory of the human body key points between the two adjacent video frames can be analyzed through the matching value, and then the motion similarity and displacement between the human body key points can be analyzed. If the matching value is less than a preset threshold, it means that the character action or posture of the latter video frame in the adjacent video frames has changed significantly, then the latter video frame is used as a third lens switching point video subframe. The human body key point detection algorithm combines image features and motion patterns to maintain high-precision switching point positioning in the case of complex actions and multiple perspective switching.

[0059] Step S300: Calculate the target camera trajectory and human motion parameters according to the target video frame sequence.

[0060] Specifically, the target camera trajectory is first obtained by analyzing the pixel motion in the target video frame sequence, and then the human body motion parameters are obtained to provide basic data for subsequent posture correction and human body parameter optimization.

[0061] In one implementation, calculating the target camera trajectory and human motion parameters according to the target video frame sequence includes: Calculating a target camera trajectory according to the target video frame sequence; Based on the target video frame sequence and the target camera trajectory, a human body motion parameter of the target video frame sequence is obtained.

[0062] Specifically, the camera trajectory is estimated by analyzing the pixel motion in the target video frame sequence to obtain the target camera trajectory. Then, the target video frame sequence and the target camera trajectory are used to jointly analyze the human motion state in the target video frame sequence, that is, to obtain the human motion parameters.

[0063] In one implementation, calculating a target camera trajectory according to the target video frame sequence includes: The target video frame sequence includes a plurality of target video frame subsequences; Selecting a target pixel point of each video frame according to a gradient map of each video frame in any of the target video frame subsequences; Tracking the target pixel points in each video frame by a preset tracking algorithm to obtain a target motion trajectory corresponding to the target pixel points; Acquire an initial camera trajectory of any of the target video frame subsequences, and optimize the initial camera trajectory according to the target motion trajectory of the target pixel point to obtain a target camera trajectory of the target video frame subsequence; According to the target camera trajectories in all target video frame subsequences, the target camera trajectory of the target video frame sequence is obtained.

[0064] Specifically, the target video frame sequence includes multiple target video frame subsequences. Taking a target video frame subsequence as an example, by calculating the gradient map of each video frame, the local feature change information of each video frame can be obtained. The gradient map of each video frame is combined with the adaptive algorithm, and multiple target pixels that meet the pixel change requirements in the gradient map of each video frame can be selected. Through a preset tracking algorithm, such as an optical flow method or a KLT tracking algorithm, multiple target pixels in each frame of the video image are tracked, and these target pixels are transferred from one frame to the next frame to obtain the target motion trajectory of the target pixel. The camera trajectory of the target video frame subsequence is randomly initialized to obtain the initial camera trajectory of the target video frame subsequence. The target motion trajectory of the target pixel point is used to analyze the movement of the target pixel point between consecutive video frames, and the initial camera trajectory of the camera is optimized with the target motion trajectory of the target pixel point as a guide, and the camera posture is adjusted, thereby improving the accuracy of the camera's trajectory point, more in line with the real movement of the human body, and the target camera trajectory of the target video frame subsequence is obtained after optimization.

[0065] For example, in an actual application scenario, the pixel change requirement may be that the pixel value changes significantly, for example, the pixel value change is greater than a preset threshold. In addition, it may be further required that the confidence of the selected pixel point needs to reach a preset confidence threshold, ensuring that the selected target pixel point is a pixel point with a significant pixel value change and a high confidence, and providing reliable basic data for subsequent human motion recovery. Furthermore, it is also possible to repeatedly perform the steps of calculating the gradient map of each video frame, selecting a number of target pixels according to the gradient map, tracking each target pixel point through a preset tracking algorithm, and obtaining the motion trajectory corresponding to each target pixel point, until the preset number of iterations is reached; the motion trajectory of each target pixel point obtained in the last time is used as the target motion trajectory of each target pixel point. By iteratively calculating the gradient map of the video frame, screening the target pixel points, tracking the target pixel points, etc., the accuracy of the trajectory points can be improved, and the motion trajectory of each target pixel point can be optimized. In this embodiment, the motion trajectory of the target pixel point after optimization is defined as the target motion trajectory, and the trajectory optimization can ensure stable tracking and provide high-quality feature points in complex environments (such as lens switching or dynamic object interference).

[0066] In one implementation, optimizing the initial camera trajectory according to the target motion trajectory of the target pixel point to obtain the target camera trajectory of the target video frame subsequence includes: Selecting a number of video frames to be optimized from the target video frame subsequence; Optimizing the camera extrinsic parameters according to the target motion trajectory of the target pixel point in each of the video frames to be optimized to obtain optimized camera extrinsic parameters; The target camera trajectory of the target video frame subsequence is calculated according to the optimized camera extrinsic parameters.

[0067] Specifically, taking a target video frame subsequence as an example, first, a sliding window strategy can be used to select a preset number (for example, 20 frames) of video frames in the target video frame subsequence for optimization. The selection criteria can be determined based on the number of matched feature points to reduce the computational complexity and maintain real-time performance. In this embodiment, the selected video frames are defined as video frames to be optimized. The camera extrinsic parameters describe the position and posture of the camera relative to the world coordinate system, and the actual motion trajectory of the object in the video frame has a certain correlation with the camera extrinsic parameters. Therefore, in complex dynamic scenes, the camera trajectory can be estimated by the target motion trajectory of the target pixel points in the video frame to be optimized, and then the camera extrinsic parameters can be extracted and optimized to more accurately reflect the position and posture of the camera. The camera extrinsic parameters include the rotation and translation parameters of the camera, which can provide a basis for subsequent posture correction and human body parameter optimization. In actual application scenarios, dynamic mask technology (Masked LEAP-VO) can be used in combination with the target motion trajectory of the target pixel points to realize segment-by-segment camera parameter estimation (such as Figure 2 as shown).

[0068] In one implementation, optimizing the camera extrinsic parameters according to the target motion trajectory of the target pixel point in each of the video frames to be optimized to obtain the optimized camera extrinsic parameters includes: Calculating the reprojection error of the target motion trajectory of the target pixel point in each of the video frames to be optimized; The optimized camera extrinsic parameters are obtained according to the reprojection error and a preset error threshold.

[0069] In summary, for each target video frame subsequence, this embodiment uses an improved visual odometer estimation algorithm (Masked LEAP-VO) to estimate the camera trajectory of the target video frame subsequence. The improved visual odometer estimation algorithm is based on the minimization of bundle adjustment. On the basis of the traditional method, it combines the dynamic object masking technology to improve the estimation accuracy of the camera rotation (R) and translation (T) parameters by filtering the dynamic points in the human body area. In a multi-lens scene, the camera trajectory of each lens is optimized separately by bundle adjustment, and the interference of dynamic objects is effectively reduced by dynamically adjusting the weights of feature points, and finally a more consistent and smooth camera trajectory is obtained. Specifically, for the target motion trajectory of the target pixel points in each optimized video frame calculated, it is compared with the projection trajectory estimated based on the camera parameters and the target pixel points to obtain a reprojection error. The optimization goal is to make the reprojection error converge to a preset error threshold, so as to obtain the optimized camera extrinsic parameters.

[0070] For example, after calculating the reprojection error, if the reprojection error at this time is greater than the error threshold, the camera extrinsic parameters need to be adjusted by the reprojection error to converge the reprojection error. When the reprojection error converges to the error threshold, the camera extrinsic parameters at this time are used as the optimized camera extrinsic parameters. The camera trajectory is estimated by the optimized camera extrinsic parameters to ensure the accuracy of the trajectory.

[0071] Step S400: obtaining target human motion parameters of the multi-lens video frame sequence based on the target camera trajectory and the human motion parameters.

[0072] Specifically, this embodiment further optimizes the human body motion in the multi-lens video frame sequence through the target camera trajectory and human body motion parameters obtained above, so as to obtain the target human body motion parameters in the multi-lens video frame sequence in the world coordinate system.

[0073] In one implementation, obtaining the target human motion parameter of the multi-lens video frame sequence based on the target camera trajectory and the human motion parameter includes: Inputting the target video frame sequence and the target camera trajectory of the target video frame sequence into a preset first model to obtain human body motion parameters of the target video frame sequence; The human body bounding box of each video frame in the multi-lens video frame sequence, the target camera trajectory and the human body motion parameters are input into a preset second model to obtain the target human body motion parameters of the multi-lens video frame sequence.

[0074] Specifically, in order to realize human body modeling, this embodiment needs to obtain a preset first model for human body modeling. For each target video frame subsequence in the target video frame sequence, the human motion parameters in the target video frame subsequence are initialized according to the target video frame subsequence and its optimized target camera trajectory by the preset first model, including but not limited to posture (θ), shape (β), direction (Γ) and global position (τ). In order to further enhance the smoothness and consistency of the motion, this embodiment also establishes a multi-lens human motion recovery module (ms-HMR) by presetting a second model. After the multi-lens video frame sequence is segmented at multiple levels, in addition to the target video frame sequence with the finest segmentation, a first video frame sequence segmented by scene changes is obtained, and the first video frame sequence contains at least one first video frame subsequence. The human motion parameters can be optimized according to the first video frame subsequence as a processing unit to obtain the target human motion parameters of each video frame. Taking a first video frame subsequence as an example, the human bounding box of each video frame in the first video frame subsequence is obtained according to the human bounding box of each video frame in the multi-lens video frame sequence. The target camera trajectory of the first video frame subsequence is obtained according to the target camera trajectory of each corresponding target video frame subsequence, for example, it can be obtained by splicing the target camera trajectory of each corresponding target video frame subsequence. The human motion parameters of the first video frame subsequence are obtained according to the human motion parameters of each corresponding target video frame subsequence, for example, it can be obtained by splicing the human motion parameters of each corresponding target video frame subsequence. By presetting the second model, according to the human body boundary box, the target camera trajectory and the human body motion parameters of each video frame in the first video frame subsequence, the motion relationship between different lenses is captured to obtain the target human body motion parameters of the first video frame subsequence. Then, according to the target human body motion parameters of each first video frame subsequence, the target human body motion parameters of the first video frame sequence are obtained, and then according to the target human body motion parameters of each first video frame sequence, the target human body motion parameters of the multi-lens video frame sequence are obtained. Thereby, the jitter and discontinuity problems caused by lens switching are reduced.

[0075] For example, in actual application scenarios, the GVHMR model (a method for realizing human motion recovery based on video) can be used as the preset first model to realize segment-by-segment human body parameter initialization (such as Figure 2As shown). Secondly, a deep learning model (Transformer) based on the self-attention mechanism can be used as the preset second model, and the Transformer encoder can use the self-attention mechanism to model and optimize the human motion parameters in each first video frame subsequence to obtain the target human motion parameters of each first video frame subsequence. The Transformer model can use the self-attention mechanism to capture the temporal context relationship, effectively handle the impact of the shot switching point, and ensure the motion consistency between shots.

[0076] In one implementation, the human body motion parameters of the target video frame sequence are obtained, and then the method further includes: According to any two adjacent target video frame subsequences in the target video frame sequence, a plurality of adjacent target video frame pairs are obtained; For any pair of adjacent target video frames, calculating a camera rotation difference value of the pair of adjacent target video frames; Correcting the human body direction of the subsequent video frame in the adjacent target video frame pair according to the camera rotation difference value and a preset difference threshold value to obtain a corrected human body direction of the adjacent target video frame pair; Based on all corrected human body directions of the adjacent target video frame pairs, human body motion parameters of the target video frame sequence are obtained.

[0077] Since camera switching may introduce significant discontinuities in posture and direction, human motion needs to be further adjusted to ensure overall smoothness. Figure 2 As shown, after the segment-by-segment human body parameter initialization is realized, the present embodiment will also detect whether the posture between adjacent target video frame subsequences is continuous. Taking a pair of adjacent target video frame subsequences as an example, the last frame of the previous target video frame subsequence and the first frame of the next target video frame subsequence are defined as an adjacent target video frame pair. The camera rotation difference of the two video frames in the adjacent target video frame pair is calculated, and the posture continuity is judged by the camera rotation difference. If the camera rotation difference is greater than the preset difference threshold, it means that the posture parameter is discontinuous, then the optimization is performed segment by segment for different individuals, and targeted posture adjustments are performed, such as correction of the human body direction. If the camera rotation difference is not greater than the difference threshold, it means that the posture parameter is continuous, then the subsequent optimization process is entered, and the method of multi-segment parameter joint optimization is adopted to achieve the global consistency of the motion trajectory. Specifically, according to the camera rotation difference of the two video frames in the adjacent target video frame pair, the human body direction of the next video frame is corrected to make it consistent with the human body direction of the previous video frame, thereby realizing the alignment and correction of the cross-lens motion and ensuring the global consistency of the motion trajectory.

[0078] For example, this embodiment calibrates the optimized target camera trajectory by correcting the human body direction of the next video frame in the adjacent target video frame pair. Specifically, the camera rotation difference between the previous video frame and the next video frame in the adjacent target video frame pair is calculated, and the human body direction is corrected by the following formula so that the next video frame is rotated to the human body direction of the previous video frame: ; in, represents the camera rotation difference; Indicates the direction of the human body in the world coordinate system before correction of the next video frame; It represents the human body direction in the world coordinate system after correction in the next video frame. The corrected human body direction in the next video frame is aligned with the human body direction in the previous video frame, thereby achieving motion continuity in the shot switching.

[0079] In one implementation, inputting a human body bounding box of each video frame in the multi-lens video frame sequence, the target camera trajectory, and the human body motion parameters into a preset second model to obtain the target human body motion parameters of the multi-lens video frame sequence includes: De-noising the human body motion parameters to obtain de-noised human body motion parameters; The human body bounding box of each video frame in the multi-lens video frame sequence, the target camera trajectory and the denoised human body motion parameters are input into a preset second model, and the denoised human body motion parameters are smoothed to obtain the target human body motion parameters of the multi-lens video frame sequence.

[0080] This embodiment uses the preset second model in the global motion optimization stage, and the global optimization stage will achieve trajectory fine-tuning in the world coordinate system (such as Figure 2As shown in the figure, the trajectory is smoothed. In addition, the human motion parameters (such as root rotation, speed, posture, etc.) are optimized to obtain the target human body parameters in the world coordinate system, so that the motion of the characters between different lens segments is consistent. Specifically, before modeling, the human motion parameters are denoised to obtain denoised human motion parameters, for example, by low-pass filtering. After the multi-lens video frame sequence is segmented at multiple levels, a first video frame sequence segmented by scene changes is obtained, and the first video frame sequence includes at least one first video frame subsequence. In the global motion optimization stage, the first video frame subsequence can be used as a processing unit. Taking a first video frame subsequence in the first video frame sequence as an example, the human body bounding box, target camera trajectory and denoised human body motion parameters of each video frame in the first video frame subsequence are input into the preset second model. In the modeling process, the smoothed human body motion parameters are calculated according to the denoised human body motion parameters, and the smoothed human body motion parameters are used as the target human body motion parameters of the first video frame subsequence. In this way, the motion trajectories of all first video frame subsequences are optimized as a whole, and the trajectories are further smoothed and adjusted to a unified world coordinate system at the same time, so as to ensure the matching of the final motion with the real physical environment. In short, this embodiment can further reduce the jitter of the motion trajectory and ensure the natural transition of the motion by presetting the second model to achieve denoising and / or smoothing operations. The target human motion parameters finally obtained can eliminate the jitter and discontinuity problems caused by lens switching and improve the stability and accuracy of human posture estimation in multi-lens videos.

[0081] For example, on the one hand, the human motion parameters can be denoised by low-pass filtering. On the other hand, the L2 regularization loss function can be introduced to smooth the denoised human motion parameters to obtain the smoothed human motion parameters.

[0082] In one implementation, the method further includes: Calculating foot contact data of each video frame according to the target human body motion parameter of each video frame in the multi-lens video frame sequence; Calculating the human body motion speed of each video frame based on the target human body motion parameter and the foot contact data of each video frame in the multi-lens video frame sequence; The human body motion speed of each video frame in the multi-lens video frame sequence is accumulated to obtain the global motion trajectory of the human body.

[0083] After the multi-lens video frame sequence is segmented at multiple levels, a first video frame sequence segmented by scene changes is obtained, and the first video frame sequence includes at least one first video frame subsequence. In the global motion optimization stage, the first video frame subsequence can be used as a processing unit. Specifically, taking a first video frame subsequence in the first video frame sequence as an example, the target human motion parameter of the first video frame subsequence includes the foot posture of the human body, so the foot contact data of each video frame can be calculated according to the foot posture of the target human motion parameter of each video frame in the first video frame subsequence, and whether the human foot is in contact with the ground can be known through the foot contact data. Then, according to the target human motion parameter and foot contact data of each video frame in the first video frame subsequence, the human motion speed in each video frame is calculated, and the human motion speed in each video frame is accumulated to obtain the global motion trajectory of the human body in the first video frame subsequence.

[0084] For example, this embodiment can construct an additional post-processing module to solve the problem of foot sliding and trajectory jitter. The post-processing module optimizes the global motion trajectory by predicting the foot contact probability (pc) and the root speed (v) and combining the image features. Specifically, the label of whether each video frame in each first video frame subsequence is grounded is calculated according to the foot posture in the target human motion parameters, and then the foot contact data of each video frame is obtained. According to the target human motion parameters and the foot contact data of each video frame, the human motion speed in each video frame is calculated. The human motion speed in each video frame in each first video frame subsequence is accumulated to obtain the global motion trajectory of the human body in the first video frame subsequence. This embodiment combines the foot contact probability and the root speed to optimize the trajectory, and smoothes the motion trajectory, which can eliminate jitter and discontinuity, and ensure the consistency and natural fluency of the global motion trajectory. The post-processing module effectively eliminates the problem of foot sliding and trajectory jitter, and improves the accuracy and stability of multi-lens human posture estimation. It is suitable for scenes such as motion capture, virtual reality, and sports analysis.

[0085] In one implementation, calculating the human body motion speed of each video frame based on the target human body motion parameter and the foot contact data of each video frame in the multi-lens video frame sequence includes: The target human motion parameters and the foot contact data of each video frame in the multi-lens video frame sequence are respectively input into a preset neural network to obtain the human motion speed of each video frame.

[0086] Specifically, after the multi-lens video frame sequence is segmented at multiple levels, a first video frame sequence segmented by scene changes is obtained, and the first video frame sequence includes at least one first video frame subsequence. In the global motion optimization stage, the first video frame subsequence can be used as a processing unit. Taking a first video frame subsequence in the first video frame sequence as an example, the target human motion parameters and foot contact data of each video frame in the first video frame subsequence are respectively input into the preset neural network as a time series, and the preset neural network is used to capture the dynamic changes of foot contact and root speed to obtain the human motion speed in each video frame. After training with training data, the preset neural network can effectively predict the foot contact state and speed, thereby identifying and correcting the foot sliding phenomenon. The human motion speed in each video frame is accumulated to obtain the global motion trajectory of the human body in the first video frame subsequence.

[0087] For example, a bidirectional recurrent neural network (bidirectional LSTM) can be used to achieve global optimization. In actual application scenarios, a human trajectory fine-tuning module can be built to perform the trajectory optimization process to improve the efficiency of trajectory optimization. This module further optimizes the global motion trajectory of the human body by extracting the image features of each video frame in each first video frame subsequence using a convolutional neural network (CNN) and fusing them with the foot contact state and human motion speed predicted by the LSTM model.

[0088] In order to facilitate understanding of the technical solution of the present invention, Figure 3 As shown in Figure 1, the complete process of recovering human 3D motion from multi-lens video is demonstrated: First, human key points and human bounding boxes are extracted through VITPose (a model based on visual Transformer). Specifically, through a shot switching detector including scene switching detection, bounding box mutation detection and key point mutation detection, a cascade detection method combining scene detection, bounding box tracking and posture tracking is used to detect the shot switching point, and the first shot switching point video frame determined based on the scene change, the second shot switching point video frame determined based on the human bounding box and the third shot switching point video frame determined based on the human key points are obtained. The video is divided into multiple sub-shot segments through the first shot switching point video frame, the second shot switching point video frame and the third shot switching point video frame, that is, multiple video frame sub-sequences such as the first video frame sequence, the second video frame sequence and the target video frame sequence are obtained.

[0089] Then, the human body pose parameters are estimated segment by segment for the target video frame sequence. Specifically, in each shot of the target video frame sequence, the camera pose parameters are estimated by dynamic masking technology (i.e. Masked LEAPVO, also known as dynamic object masking technology), and the camera trajectory is optimized to obtain the target camera trajectory. The human body pose, shape and position parameters are initialized by the GVHMR model to obtain the initialized human body motion parameters.

[0090] Subsequently, the human body orientation across lenses is aligned through camera correction. For the human body posture parameters of multi-lens switching, the human body parameters of different sequences are aligned through camera extrinsic calibration and multi-lens human body motion recovery module, thereby optimizing the global posture and trajectory consistency and obtaining the target human body motion parameters.

[0091] Finally, the motion decoder of a bidirectional recurrent neural network (e.g., bidirectional long short-term memory network) is used to reduce foot sliding and optimize the smoothness of the motion trajectory, and the human trajectory fine-tuning module (e.g., Figure 4 The human motion trajectory is further refined to obtain the global motion trajectory of the human body, thereby ensuring that the final output motion data has high accuracy, global consistency and physical authenticity.

[0092] The entire process effectively solves the problems of discontinuous motion and uneven trajectory in multi-lens videos, providing a complete solution for high-quality motion restoration. It has stronger adaptability in complex scenes and is suitable for diverse scenes such as sports events and film and television production.

[0093] The present invention significantly improves the continuity and accuracy of human motion recovery in multi-lens videos through technologies such as lens switching detection, dynamic object masking, parameter joint optimization, and trajectory fine-tuning, and effectively solves the problem of motion discontinuity caused by lens switching. The advantages of the present invention are specifically: 1. The present invention adopts a global motion tracking algorithm to reconstruct the posture and direction through a unified global reference system when switching multiple lenses, avoiding the accumulation of local errors under the perspective of a single lens, and ensuring that the posture and direction of the human body maintain a smooth transition when switching lenses. In addition, the camera calibration data is combined with lens switching detection to automatically adjust the coordinate system of each lens to maintain consistency during the global motion recovery process and avoid sudden changes in posture and direction. The problem of discontinuity in posture and direction caused by lens switching is solved.

[0094] 2. The present invention introduces dynamic object separation and deep learning enhanced camera trajectory estimation. The dynamic object masking technology significantly improves the robustness of camera trajectory estimation, effectively avoiding the interference of dynamic objects on the estimation. Convolutional neural network (CNN) and self-supervised learning methods are used to separate dynamic objects and static scenes to avoid interference with camera trajectory estimation. By identifying and eliminating the influence of dynamic objects, the stability of camera trajectory estimation is ensured, especially when people occupy most of the image, high-precision motion recovery can still be maintained. The problem of interference of dynamic objects on camera trajectory estimation is solved.

[0095] 3. The present invention adopts contact-sensing trajectory optimization technology to reduce foot sliding and trajectory jitter through foot contact prediction and root speed adjustment mechanism. During the motion recovery process, the contact state of the foot and the ground is detected and predicted in real time, and the root speed and direction are dynamically adjusted to ensure accurate swing of the footsteps and smooth motion trajectory. Combined with deep learning models and physical constraints, it can effectively reduce trajectory instability caused by uneven ground or gait changes, and improve the authenticity of motion recovery. The problem of foot sliding and trajectory jitter is solved.

[0096] 4. The present invention can realize modular design and segmented processing framework, thereby improving computing efficiency and reducing the computational complexity of overall optimization.

[0097] Based on the above embodiments, the present invention also provides a human motion optimization system for multi-lens video, such as Figure 5 As shown, the system comprises: Video acquisition module 01, used to acquire multi-lens video frame sequences; The video segmentation module 02 is used to detect shot switching points of the multi-shot video frame sequence, and perform multi-level segmentation on the multi-shot video frame sequence according to the detected shot switching points to obtain a target video frame sequence; A video analysis module 03, used for calculating the target camera trajectory and human motion parameters according to the target video frame sequence; The motion optimization module 04 is used to obtain the target human motion parameters of the multi-lens video frame sequence based on the target camera trajectory and the human motion parameters.

[0098] Based on the above embodiment, the present invention further provides a terminal, whose principle block diagram can be shown as follows: Figure 6As shown. The terminal includes a processor, a memory, a network interface, and a display screen connected through a system bus. Among them, the processor of the terminal is used to provide computing and control capabilities. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a human body motion optimization method for multi-lens video is implemented. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.

[0099] Those skilled in the art will understand that Figure 6 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0100] In one implementation, the terminal has one or more programs stored in its memory and is configured to be executed by one or more processors. The one or more programs include instructions for performing a method for optimizing human motion in multi-lens video.

[0101] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0102] In summary, the present invention discloses a method, system, terminal and storage medium for optimizing human motion in multi-lens video, and relates to the field of computer vision technology. The method comprises: obtaining a multi-lens video frame sequence; performing lens switching point detection on the multi-lens video frame sequence, and performing multi-level segmentation on the multi-lens video frame sequence according to the detected lens switching points to obtain a target video frame sequence; calculating a target camera trajectory and human motion parameters according to the target video frame sequence; and obtaining the target human motion parameters of the multi-lens video frame sequence based on the target camera trajectory and the human motion parameters. The present invention first divides the multi-lens video frame sequence into a plurality of target video frame sequences that do not contain lens switching points according to the lens switching points. Then, the target camera trajectory and human motion parameters are calculated according to each target video frame sequence. Finally, the human motion state in the video is analyzed by combining the target camera trajectory and human motion parameters, so as to obtain the target human motion parameters with high precision, continuity and physical consistency. It is suitable for a variety of application scenarios such as motion capture, virtual reality, and sports analysis.

[0103] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for optimizing human motion in multi-lens video, characterized in that: The method comprises: Obtain a multi-lens video frame sequence; Performing shot switching point detection on the multi-shot video frame sequence, and performing multi-level segmentation on the multi-shot video frame sequence according to the detected shot switching points to obtain a target video frame sequence; Calculating target camera trajectory and human motion parameters according to the target video frame sequence; The target human motion parameters of the multi-lens video frame sequence are obtained based on the target camera trajectory and the human motion parameters.

2. The method for optimizing human motion in multi-lens video according to claim 1, characterized in that: The method includes: detecting shot switching points on the multi-shot video frame sequence, and performing multi-level segmentation on the multi-shot video frame sequence according to the detected shot switching points to obtain a target video frame sequence, including: Performing shot switching point detection on the multi-shot video frame sequence to obtain a first shot switching point video frame, a second shot switching point video frame, and a third shot switching point video frame respectively; The multi-lens video frame sequence is multi-level segmented according to the first lens switching point video frame, the second lens switching point video frame and the third lens switching point video frame to obtain a first video frame sequence, a second video frame sequence and the target video frame sequence respectively.

3. The method for optimizing human motion in multi-lens video according to claim 2, characterized in that: Performing shot switching point detection on the multi-shot video frame sequence to obtain a first shot switching point video frame, a second shot switching point video frame, and a third shot switching point video frame respectively includes: Determine a first shot switching point video frame by detecting a scene change of each video frame in the multi-shot video frame sequence; Determine a second shot switching point video frame by detecting a human body bounding box of each video frame in the multi-shot video frame sequence; The third shot switching point video frame is determined by detecting the human body key points of each video frame in the multi-shot video frame sequence.

4. The method for optimizing human motion in multi-lens video according to claim 3, characterized in that: The multi-shot video frame sequence is segmented at multiple levels according to the first shot switching point video frame, the second shot switching point video frame, and the third shot switching point video frame to obtain a first video frame sequence, a second video frame sequence, and the target video frame sequence, respectively, including: Segment the multi-shot video frame sequence according to the first shot switching point video frame to obtain the first video frame sequence, where the first video frame sequence includes a plurality of first video frame subsequences; The first video frame sequence is divided according to the second shot switching point video frame to obtain the second video frame sequence, where the second video frame sequence includes a plurality of second video frame subsequences; The second video frame sequence is segmented according to the third shot switching point video frame to obtain the target video frame sequence, where the target video frame sequence includes a plurality of target video frame subsequences.

5. The method for optimizing human motion in multi-lens video according to claim 4, characterized in that: Determining a first shot switching point video frame by detecting a scene change of each video frame in the multi-shot video frame sequence includes: Taking each non-first frame in the multi-lens video frame sequence as a current video frame, calculating the background difference between the current video frame and the previous video frames, wherein the background difference is used to reflect the degree of scene change; Determine a first shot switching point video subframe based on the background difference and a preset difference threshold; The first shot switching point video frame is obtained according to all the first shot switching point video sub-frames.

6. The method for optimizing human motion in multi-lens video according to claim 4, characterized in that: Determining a second shot switching point video frame by detecting a human body boundary frame of each video frame in the multi-shot video frame sequence includes: For any two adjacent video frames in any of the first video frame subsequences, calculating the degree of overlap of human body bounding boxes of the two adjacent video frames; Determining a second shot switching point video subframe based on the overlap degree and a preset overlap threshold; The second shot switching point video frame is obtained according to all the second shot switching point video sub-frames.

7. The method for optimizing human motion in multi-lens video according to claim 4, characterized in that: Determining a third shot switching point video frame by detecting a human body key point of each video frame in the multi-shot video frame sequence includes: For any two adjacent video frames in any of the second video frame subsequences, obtain true values ​​and predicted values ​​of human body key points of the two adjacent video frames; Based on the real values ​​and predicted values ​​of the human body key points of the two adjacent video frames, a third shot switching point video subframe is obtained; The third shot switching point video frame is obtained according to all the third shot switching point video sub-frames.

8. The method for optimizing human motion in multi-lens video according to claim 7, characterized in that: Obtaining the true value and predicted value of the human body key points of the two adjacent video frames, including: Detecting the human key points of the two adjacent video frames by a human key point detection algorithm to obtain the true values ​​of the human key points of the two adjacent video frames; The human body key points of the two adjacent video frames are predicted by a preset filtering algorithm to obtain predicted values ​​of the human body key points of the two adjacent video frames.

9. The method for optimizing human motion in multi-lens video according to claim 8, characterized in that: Based on the real values ​​and predicted values ​​of the human body key points of the two adjacent video frames, a third shot switching point video subframe is obtained, including: Calculate the matching value according to the real value and the predicted value of the human body key point of the two adjacent video frames; The matching value is compared with a preset matching threshold, and a later video frame of the two adjacent video frames is selected as the third shot switching point video subframe.

10. The method for optimizing human motion in multi-lens video according to claim 1, characterized in that: Calculating the target camera trajectory and human motion parameters according to the target video frame sequence includes: Calculating a target camera trajectory according to the target video frame sequence; Based on the target video frame sequence and the target camera trajectory, a human body motion parameter of the target video frame sequence is obtained.

11. The method for optimizing human motion in multi-lens video according to claim 10, characterized in that: Calculating a target camera trajectory according to the target video frame sequence includes: The target video frame sequence includes a plurality of target video frame subsequences; Selecting a target pixel point of each video frame according to a gradient map of each video frame in any of the target video frame subsequences; Tracking the target pixel points in each video frame by a preset tracking algorithm to obtain a target motion trajectory corresponding to the target pixel points; Acquire an initial camera trajectory of any of the target video frame subsequences, and optimize the initial camera trajectory according to the target motion trajectory of the target pixel point to obtain a target camera trajectory of the target video frame subsequence; According to the target camera trajectories in all target video frame subsequences, the target camera trajectory of the target video frame sequence is obtained.

12. The method for optimizing human motion in multi-lens video according to claim 11, characterized in that: The initial camera trajectory is optimized according to the target motion trajectory of the target pixel point to obtain the target camera trajectory of the target video frame subsequence, including: Selecting a number of video frames to be optimized from the target video frame subsequence; Optimizing the camera extrinsic parameters according to the target motion trajectory of the target pixel point in each of the video frames to be optimized to obtain optimized camera extrinsic parameters; The target camera trajectory of the target video frame subsequence is calculated according to the optimized camera extrinsic parameters.

13. The method for optimizing human motion in multi-lens video according to claim 12, characterized in that: Optimizing the camera extrinsic parameters according to the target motion trajectory of the target pixel point in each of the video frames to be optimized to obtain the optimized camera extrinsic parameters, including: Calculating the reprojection error of the target motion trajectory of the target pixel point in each of the video frames to be optimized; The optimized camera extrinsic parameters are obtained according to the reprojection error and a preset error threshold.

14. The method for optimizing human motion in multi-lens video according to claim 1, characterized in that: Obtaining target human motion parameters of the multi-lens video frame sequence based on the target camera trajectory and the human motion parameters includes: Inputting the target video frame sequence and the target camera trajectory of the target video frame sequence into a preset first model to obtain human body motion parameters of the target video frame sequence; The human body bounding box of each video frame in the multi-lens video frame sequence, the target camera trajectory and the human body motion parameters are input into a preset second model to obtain the target human body motion parameters of the multi-lens video frame sequence.

15. The method for optimizing human motion in multi-lens video according to claim 14, characterized in that: Inputting the human body bounding box of each video frame in the multi-lens video frame sequence, the target camera trajectory, and the human body motion parameters into a preset second model to obtain the target human body motion parameters of the multi-lens video frame sequence, including: De-noising the human body motion parameters to obtain de-noised human body motion parameters; The human body bounding box of each video frame in the multi-lens video frame sequence, the target camera trajectory and the denoised human body motion parameters are input into a preset second model, and the denoised human body motion parameters are smoothed to obtain the target human body motion parameters of the multi-lens video frame sequence.

16. The method for optimizing human motion in multi-lens video according to claim 10, characterized in that: The human body motion parameters of the target video frame sequence are obtained, and then the following steps are further included: According to any two adjacent target video frame subsequences in the target video frame sequence, a plurality of adjacent target video frame pairs are obtained; For any pair of adjacent target video frames, calculating a camera rotation difference value of the pair of adjacent target video frames; Correcting the human body direction of the subsequent video frame in the adjacent target video frame pair according to the camera rotation difference value and a preset difference threshold value to obtain a corrected human body direction of the adjacent target video frame pair; Based on all corrected human body directions of the adjacent target video frame pairs, human body motion parameters of the target video frame sequence are obtained.

17. The method for optimizing human motion in multi-lens video according to any one of claims 1 to 16, characterized in that: The method further comprises: Calculating foot contact data of each video frame according to the target human body motion parameter of each video frame in the multi-lens video frame sequence; Calculating the human body motion speed of each video frame based on the target human body motion parameter and the foot contact data of each video frame in the multi-lens video frame sequence; The human body motion speed of each video frame in the multi-lens video frame sequence is accumulated to obtain the global motion trajectory of the human body.

18. The method for optimizing human motion in multi-lens video according to claim 17, characterized in that: Calculating the human body motion speed of each video frame based on the target human body motion parameter and the foot contact data of each video frame in the multi-lens video frame sequence includes: The target human motion parameters and the foot contact data of each video frame in the multi-lens video frame sequence are respectively input into a preset neural network to obtain the human motion speed of each video frame.

19. A human motion optimization system for multi-lens video, characterized in that: The system comprises: A video acquisition module, used to acquire multi-lens video frame sequences; A video segmentation module, used to detect shot switching points of the multi-shot video frame sequence, and perform multi-level segmentation on the multi-shot video frame sequence according to the detected shot switching points to obtain a target video frame sequence; A video analysis module, used for calculating a target camera trajectory and human motion parameters according to the target video frame sequence; A motion optimization module is used to obtain target human motion parameters of the multi-lens video frame sequence based on the target camera trajectory and the human motion parameters.

20. A terminal, characterized in that: The terminal includes a memory and one or more processors; the memory stores one or more programs; the program contains instructions for executing the human motion optimization method for multi-lens video as described in any one of claims 1-18; and the processor is used to execute the program.

21. A computer-readable storage medium having a plurality of instructions stored thereon, characterized in that: The instructions are suitable for being loaded and executed by a processor to implement the steps of the human motion optimization method for multi-lens video as described in any one of claims 1-18.

Images