A method and system for locating vocalization actions based on inflection point flow

Abstract

This invention discloses a method and system for temporal localization of vocal actions based on inflection point flow. First, a silent video containing vocal actions is read. Then, inflection point flow analysis is performed according to kinematic laws to obtain bidirectional motion flow and bidirectional inflection point flow. Subsequently, feature extraction is performed to obtain image context features, bidirectional motion features, and bidirectional velocity inflection point features. Next, cross-kinematic features from image to motion and from image to inflection point are concatenated to obtain cross-kinematic aggregate features. Simultaneously, a discriminant map is obtained from the cross-kinematic aggregate features. On the discriminant map, cross-video comparative learning is performed on moving and non-moving regions, and intra-video smoothing constraints are applied to obtain activated motion region features. Finally, spatiotemporal feature fusion is performed, and frame-by-frame classification prediction is conducted to identify whether each image frame is a vocal action frame. This invention achieves frame-level collision vocal action frame localization and can accurately identify the video frame corresponding to a collision vocal action.

Description

Technical Field

[0001] This invention belongs to the technical field of computer vision and video processing, specifically relating to a method and system for locating the timing of vocalization actions based on inflection point flow. Background Technology

[0002] With the booming development of social media platforms, video content has become an indispensable part of daily life, especially on popular video sharing platforms. For visual media such as movies and TV series, dubbing is a crucial part of the production process. For dubbing actions involving collisions (such as musical instruments colliding), this process relies on professionals to determine the precise timing of audio insertion, making it particularly complex and challenging.

[0003] Techniques specifically designed for video clip dubbing applications, particularly for accurately locating the temporal position of vocal actions within a video, are still in their early stages of development. Recent research focuses on temporal action localization methods, which automatically identify specific actions occurring in a video and accurately mark their temporal boundaries. These techniques aim to classify action instances and determine their start and end times within the video stream. However, existing temporal action localization methods primarily focus on detecting the overall process of a single action, such as the entire cycle of a kicking motion. Their focus is mainly on the overall semantic aspects of the action rather than its specific kinematic features. This emphasis leads to the neglect of internal changes within the action, which are crucial for identifying and distinguishing the timing of vocalization. Furthermore, these methods do not consider the explicit spatial motion of objects, failing to capture the spatial-temporal dynamics of specific motion frames essential for detecting audible actions. In contrast, the problem of timing of vocalization is more sophisticated. It requires algorithms to capture specific moments within the action cycle, such as the exact moment when the foot contacts the ball in a kicking motion. This high-precision timing capability ensures the quality and accuracy of automated dubbing, making video content more vivid and realistic. It also helps other post-production tasks and applications that require high-precision timing.

[0004] In summary, this invention aims to fill an important gap in existing research and technology by providing an efficient and reliable solution to achieve high-precision timing positioning of vocal actions, thereby supporting automatic dubbing of video clips. Summary of the Invention

[0005] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a method and system for locating sound-generating actions based on inflection point flow. Through deep learning technology, it achieves frame-level collision sound-generating action frame localization, which greatly reduces the reliance on manual operation and can be applied to video content of various action types, thereby improving work efficiency.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] On one hand, the present invention provides a method for locating the timing of vocalization actions based on inflection point flow, comprising the following steps:

[0008] Read silent video containing vocalizations;

[0009] The silent video is treated as a displacement graph of an object that changes over time. Based on the laws of kinematics, inflection point flow analysis is performed on the silent video to obtain bidirectional motion flow and bidirectional inflection point flow.

[0010] Feature extraction was performed on image frames, bidirectional motion streams, and bidirectional inflection point streams in silent videos to obtain image context features, bidirectional motion features, and bidirectional velocity inflection point features.

[0011] Based on image context features, bidirectional motion features, and bidirectional velocity inflection point features, cross-kinematic features from image to motion and from image to inflection point are calculated and concatenated to obtain cross-kinematic aggregate features.

[0012] A discriminant map is obtained from cross-kinematic aggregation features. Cross-video contrastive learning is performed on the discriminant map for moving and non-moving regions, and intra-video smoothing constraints are applied to obtain activated moving region features. The discriminant map includes both moving and non-moving regions.

[0013] Cross-kinematic aggregation features and activated motion region features are encoded separately, then spatiotemporal feature fusion is performed, and frame-by-frame classification and prediction are conducted to identify whether each image frame is a frame containing sound. As a preferred technical solution, the step of performing inflection point flow analysis on silent video using an optical flow estimation algorithm based on kinematic laws specifically involves:

[0014] Silent video is treated as a time-varying object displacement graph, with optical flow used to represent velocity and optical flow difference used to represent acceleration.

[0015] The forward optical flow of an image frame and its next adjacent image frame is calculated using an optical flow estimation algorithm as its corresponding forward motion flow; the reverse optical flow of an image frame and its previous adjacent image frame is calculated as its corresponding reverse motion flow.

[0016] The forward motion flow of an image frame is differentially processed with the forward motion flow of its next adjacent image frame to obtain its corresponding forward inflection point flow. The reverse motion flow of an image frame is differentially processed with the reverse motion flow of its previous adjacent image frame to obtain its corresponding reverse inflection point flow.

[0017] The bidirectional motion flow is obtained by splicing the forward and reverse motion flows of an image frame, and the bidirectional inflection point flow is obtained by splicing the forward and reverse inflection point flows of an image frame.

[0018] As a preferred technical solution, the feature extraction uses three independent ResNet encoder networks to extract the context features of each image frame and its bidirectional motion flow and bidirectional inflection point flow, respectively. After feature concatenation, the image context features, bidirectional motion features and bidirectional velocity inflection point features of each image frame are obtained.

[0019] As a preferred technical solution, the splicing obtains cross-kinematic aggregation features, specifically as follows:

[0020] The image context features, bidirectional motion features, and bidirectional velocity inflection point features of each image frame are projected onto a linear layer to calculate the image-to-motion attention matrix and the image-to-inflection point attention matrix.

[0021] After passing through the Softmax activation layer, cross-kinematic features from image to motion and from image to inflection point are obtained respectively;

[0022] The cross-kinematic features of each image frame are obtained by concatenating the image context features, the cross-kinematic features from the image to the motion, and the cross-kinematic features from the image to the inflection point.

[0023] As a preferred technical solution, the cross-kinematic aggregation feature is processed by a 3×3 convolutional layer, a batch normalization layer, and a sigmoid operation to obtain a corresponding discriminant map. In the discriminant map, high-probability regions are motion regions, and low-probability regions are non-motion regions.

[0024] As a preferred technical solution, the step of performing cross-video comparative learning on the discriminant map of moving and non-moving regions and applying intra-video smoothing constraints specifically involves:

[0025] The motion and non-motion regions in the discriminant map are used to mask the cross-kinematic aggregation features to obtain motion region features and non-motion region features.

[0026] Construct positive and negative contrast target pairs based on features of moving and non-moving regions;

[0027] Based on positive and negative contrast target pairs, a cross-video contrast learning loss function is constructed and optimized for cross-kinematic features using a self-supervised learning approach to obtain activated motion region features;

[0028] The cross-video contrastive learning loss function is expressed as follows:

[0029]

[0030] in, The negative contrastive objective loss function is used. For positive contrast target pairs, the loss function is defined as the positive contrast target feature of the moving region. The positive contrast target loss function is used for positive contrast target pairs that are features of non-moving regions.

[0031] The video internal smoothing constraint is optimized based on the discriminant map of image frames in the same silent video, constructing a video temporal regularization loss function, expressed as:

[0032]

[0033] Among them, D p+2 D is the discriminant image of the (p+2)th image frame. p+1 This is the discriminant image for the (p+1)th image frame.

[0034] As a preferred technical solution, the negative contrast target loss function is constructed based on the motion region features and non-motion region features of image frames sampled from multiple silent videos:

[0035]

[0036] Where k is the number of image frames sampled from multiple silent videos. The inner product is represented as:

[0037]

[0038] For the motion region features of the p-th image frame, The non-motion region features of the q-th image frame;

[0039] The positive contrast target loss function is constructed based on motion region features or non-motion region features of image frames sampled from multiple silent videos:

[0040]

[0041] in, For positive contrast target pairs, the loss function is defined as either positive contrast target features or non-moving region features. For the motion region features or non-motion region features of the p-th image frame, For the motion region features or non-motion region features of the q-th image frame; It is 1 when p = q, otherwise it is 0; w p,q The penalty factor is represented as:

[0042]

[0043] α is a hyperparameter used for smoothness control. Inner product The ranking of cosine similarity pairs of motion region features or non-motion region features across all image frames.

[0044] As a preferred technical solution, 3D convolution is used to encode cross-kinematic aggregation features and activation motion region features respectively, and then the encoded output is input into a Transformer network with a self-attention module for spatiotemporal feature fusion to obtain spatiotemporal fused features;

[0045] Then, a fully connected layer is used to classify and predict each image frame one by one to identify whether each image frame is an action frame and obtain the recognition result.

[0046] As a preferred technical solution, when performing classification prediction frame by frame through fully connected layers, cross-entropy loss and focal loss are used for supervision:

[0047]

[0048] Where, λ ce and λ focal Cross-entropy loss and focal loss The weight parameters are: the cross-entropy loss is calculated using Gaussian-enhanced soft labeling.

[0049] On the other hand, a speech action temporal localization system based on inflection point flow is provided, which is applied to the above-mentioned speech action temporal localization method based on inflection point flow, including a video reading module, an inflection point flow analysis module, a feature extraction module, a feature aggregation module, a self-supervised optimization module, and a classification prediction module.

[0050] The video reading module is used to read silent videos that contain sound-producing actions;

[0051] The inflection point flow analysis module is used to treat silent video as a time-varying object displacement map, and to perform inflection point flow analysis on silent video according to the laws of kinematics to obtain bidirectional motion flow and bidirectional inflection point flow.

[0052] The feature extraction module is used to extract features from image frames, bidirectional motion streams and bidirectional inflection point streams in silent videos, respectively, to obtain image context features, bidirectional motion features and bidirectional velocity inflection point features.

[0053] The feature aggregation module is used to calculate cross-kinematic features from image to motion and from image to inflection point based on image context features, bidirectional motion features and bidirectional velocity inflection point features, and then concatenate them to obtain cross-kinematic aggregated features.

[0054] The self-supervised optimization module is used to obtain a discriminant map from cross-kinematic features, perform cross-video contrastive learning on the discriminant map for moving and non-moving regions, and apply intra-video smoothing constraints to obtain activated moving region features; the discriminant map contains both moving and non-moving regions.

[0055] The classification and prediction module is used to encode cross-kinematic features and activated motion region features respectively, then perform spatiotemporal feature fusion and classify and predict frame by frame to identify whether each image frame is a sound-generating action frame.

[0056] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0057] 1. Based on the laws of kinematics, this invention uses the second derivative based on position-time images combined with an optical flow estimation algorithm to perform inflection point flow analysis. The aim is to enrich the kinematic data by capturing the details of object state changes, thereby simulating collision sound-generating actions.

[0058] 2. This invention proposes an auxiliary optimization method based on self-supervised spatial localization features. By performing cross-video comparative learning on moving and non-moving regions in the discrimination image, and constructing a temporal regularization loss function within the video to perform smoothing constraints within the video, the obtained spatial information is used to enhance the network's representation ability, and the spatial location of the collision action is additionally identified as a secondary output.

[0059] 3. This invention achieves frame-level collision sound action frame localization, which can accurately identify the video frame corresponding to the collision sound action, ensuring the consistency and authenticity of audio-visual elements, and providing technical support for subsequent dubbing.

[0060] 4. This invention, through deep learning technology, significantly reduces reliance on manual operation, is applicable to video content with various action types, and improves work efficiency. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is an overall flowchart of the speech action timing localization method based on inflection point flow in an embodiment of the present invention.

[0063] Figure 2 This is a flowchart of the speech action timing localization method based on inflection point flow in an embodiment of the present invention.

[0064] Figure 3 This is a schematic diagram of the structure of the speech action timing localization system based on inflection point flow in an embodiment of the present invention. Detailed Implementation

[0065] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0066] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0067] The objective of this invention is to determine the moment when a sound-producing action occurs, with a target precision down to the frame level. This involves determining whether each frame contains an action capable of generating sound, thus contributing to audio-visual consistency. Sound-producing actions, especially collisions, are typically caused by sudden changes in force acting on an object, manifesting as inflection points in velocity. Furthermore, these collisions are a primary source of sound. Based on this assumption, this invention delves into the velocity inflection points and motion characteristics from both temporal and spatial perspectives. Previous action recognition methods have not fully explored and represented the potential of velocity inflection points within motion. Therefore, this invention proposes introducing inflection point flow to supplement traditional kinematic analysis (i.e., only including motion flow). This invention proposes inflection point flow to represent sudden changes in object velocity, enhancing traditional kinematic analysis as prior motion information. Simultaneously, it combines feature extraction and feature aggregation to extract motion-guided contextual features. Furthermore, this application introduces a self-supervised assisted optimization strategy, constraining the prediction of sound-producing actions by combining internal and external video perspectives. The specific steps and methods of this invention are as follows: Figure 1 , Figure 2 As shown, it includes the following steps:

[0068] S1. Read the silent video containing vocalizations I = {I t ∈R H×W×3 |t=1,2,...,T}, where T is the number of image frames, and H and W are the height and width of the image frame.

[0069] S2. Inflection Point Flow Analysis: The silent video is treated as a displacement graph of an object that changes over time. Inflection point flow analysis is performed on the silent video according to the laws of kinematics to obtain bidirectional motion flow and bidirectional inflection point flow.

[0070] The inflection point flow proposed in this application is used to represent sudden changes in the velocity of an object, enhancing traditional kinematic analysis by serving as prior information about motion. The principle is as follows:

[0071] Viewing a silent video as a time-varying object displacement graph, i.e., a two-dimensional displacement-time graph (i.e., an xt graph, where x is a two-dimensional displacement vector), and according to the laws of kinematics, by performing a first-order derivative operation on the time series of the object's displacement, the corresponding velocity-time graph (vt graph) can be obtained:

[0072]

[0073] Furthermore, by performing second derivative calculations on the displacement-time plot, an acceleration-time plot (at plot) can be obtained:

[0074]

[0075] According to Newton's first law, a sudden application of an external force will cause a sudden change in acceleration, thereby altering the object's state of motion. The acceleration function a(t) provides an important clue for identifying vocalization; therefore, the inflection point of velocity represented by acceleration a(t) is called the "inflection point flow." Next, the inflection point flow a... t := a(t) and motion flow v t The combination of :=v(t) and kinematic priors serves as a novel approach. Therefore, in practical applications, inflection point flow analysis of silent video is performed based on kinematic laws, specifically as follows:

[0076] S201. Treat the silent video as a time-varying object displacement graph, use optical flow to represent velocity, and use optical flow difference to represent acceleration.

[0077] S202. Use the optical flow estimation algorithm to calculate the forward optical flow of an image frame and its next adjacent image frame as its corresponding forward motion flow; calculate the reverse optical flow of an image frame and its previous adjacent image frame as its corresponding reverse motion flow.

[0078] S203. Perform a difference operation on the forward motion flow of the image frame and the forward motion flow of the next adjacent image frame to obtain its corresponding forward inflection point flow, and perform a difference operation on the reverse motion flow of the image frame and the reverse motion flow of the previous adjacent image frame to obtain its corresponding reverse inflection point flow.

[0079] S204. The forward motion flow and reverse motion flow of the image frame are spliced ​​together to obtain its bidirectional motion flow, and the forward inflection point flow and reverse inflection point flow of the image frame are spliced ​​together to obtain its bidirectional inflection point flow.

[0080] For example, for three adjacent image frames I in a silent video i-1 I i I i+1 Calculate image frame Ii The next adjacent image frame I i+1 Forward optical flow as image frame I i positive motion flow Calculate I i The image frame I adjacent to its predecessor i-1 The reverse optical flow is its reverse motion flow Then image frame I i positive motion flow The next adjacent image frame I i+1 positive motion flow Perform a difference operation to obtain image frame I. i Positive inflection point flow Similarly, image frame I i Reverse flow The image frame I that is adjacent to the previous one i-1 Reverse flow Perform a difference operation to obtain image frame I. i Reverse inflection point flow

[0081] S3. Feature Extraction: Feature extraction is performed on the image frames, bidirectional motion stream, and bidirectional inflection point stream in the silent video to obtain image context features, bidirectional motion features, and bidirectional velocity inflection point features.

[0082] Specifically, this invention employs three independent ResNet encoder networks (E... X E M and E C Extract the context features of each image frame and its bidirectional motion flow and bidirectional inflection point flow, respectively. Concatenate the features to obtain the image context features f of each image frame. i Bidirectional motion characteristics m i and bidirectional velocity inflection point characteristics c i .

[0083] S4. Cross-kinematic aggregation: Based on image context features, bidirectional motion features, and bidirectional velocity inflection point features, calculate cross-kinematic features from image to motion and from image to inflection point, and concatenate them to obtain cross-kinematic aggregation features.

[0084] To better utilize the guidance of kinematic priors, this invention introduces a cross-kinematic aggregation strategy, aiming to extract motion and velocity inflection point information from image context features; specifically:

[0085] S401. First, by projecting the image context features, bidirectional motion features, and bidirectional velocity inflection point features of each image frame onto a linear layer, the attention matrices from image to motion and from image to inflection point are calculated using the following formula:

[0086]

[0087] Among them, A mf(i) For the image-to-motion attention matrix, Q m(i) K is a bidirectional motion feature query vector. f(i) Let A be the image context feature key vector, T be the transpose operation, and A be the transpose operation. cf(i) Let Q be the attention matrix from the image to the inflection point. c(i) d represents the bidirectional velocity inflection point feature query vector, where d is the dimension of the query vector and the key vector.

[0088] S402, after passing through the Softmax activation layer, the cross-kinematic features from image to motion and from image to inflection point are obtained respectively:

[0089] h m(i) =V f(i) softmax(A mf(i) ), h c(i) =V f(i) softmax(A cf(i) ),

[0090] Where h m(i) and h c(i) These represent the cross-kinematic features from image to motion and from image to inflection point, respectively, while V f(i) By analyzing image context features f i Obtained by applying a linear layer.

[0091] S403. Finally, the image context features, image-to-motion cross-kinematic features, and image-to-inflection point cross-kinematic features of each image frame are concatenated to obtain the cross-kinematic aggregate feature F of each image frame. i .

[0092] S5. Self-supervised spatial-assisted optimization: Obtain a discriminant map from cross-kinematic features, perform cross-video comparative learning on the discriminant map for moving and non-moving regions, and apply smoothing constraints within the video to obtain activated moving region features; wherein, the discriminant map contains both moving and non-moving regions.

[0093] Previous temporal action localization methods did not explicitly consider the spatial motion of the learning object. Therefore, this invention introduces spatial motion localization as an auxiliary optimization task. This strategy can effectively improve the model's ability to identify the most discriminative regions of object motion, thereby better locating potential vocal action frames. First, a discriminant map needs to be obtained. This invention uses cross-kinematic aggregated features through a 3×3 convolutional layer, a batch normalization layer, and a sigmoid operation to obtain the corresponding discriminant map. High-probability regions in the discriminant map are considered motion regions, and low-probability regions are considered non-motion regions.

[0094] Next, cross-video contrastive learning and intra-video smoothing constraints are performed. Specifically, cross-video contrastive learning is conducted on the discriminant map for moving regions (high-probability regions) and non-moving regions (low-probability regions), and intra-video smoothing constraints are applied.

[0095] S501. Mask the cross-kinematic aggregation features using the moving and non-moving regions in the discriminant map to obtain the moving region features and non-moving region features, represented as follows:

[0096]

[0097] in, D represents the Hadamard product. p F is the discriminant image of the p-th image frame. p For the cross-kinematic aggregation features of the p-th image frame, For the motion region features of the p-th image frame, This represents the non-motion region features of the p-th image frame.

[0098] S502. Construct positive contrast target pairs and negative contrast target pairs based on motion region features and non-motion region features. The negative contrast target pairs are constructed based on motion region features and non-motion region features of image frames sampled from multiple silent videos, while the positive contrast target pairs are constructed based on either motion region features or non-motion region features of image frames sampled from multiple silent videos.

[0099] S503. Based on positive and negative contrast target pairs, a cross-video contrastive learning loss function is constructed and optimized using self-supervised learning to obtain activated motion region features. The cross-video contrastive learning loss function is expressed as follows:

[0100]

[0101] in, The negative contrastive objective loss function is used. For positive contrast target pairs, the loss function is defined as the positive contrast target feature of the moving region. The positive contrast target loss function is used for the positive contrast target pair, which is a feature of the non-moving region.

[0102] Specifically, the negative contrast target loss function is constructed based on negative contrast target pairs:

[0103]

[0104] Where k is the number of image frames sampled from the silent video. The inner product is represented as:

[0105]

[0106] For the motion region features of the p-th image frame, The non-motion region features of the q-th image frame;

[0107] The positive contrast target loss function is constructed based on positive contrast target pairs:

[0108]

[0109] in, For positive contrast target pairs, the loss function is defined as either positive contrast target features or non-moving region features. For the motion region features or non-motion region features of the p-th image frame, For the motion region features or non-motion region features of the q-th image frame; It is 1 when p = q, otherwise it is 0; w p,q This is a penalty factor used to penalize positive contrast pairs with low similarity in the ranking, denoted as:

[0110]

[0111] α is a hyperparameter used for smoothness control. Inner product The ranking of cosine similarity pairs of motion region features or non-motion region features across all image frames.

[0112] S504. Since actions in video are usually continuous, intra-video smoothing constraints are applied to ensure that the localization region does not change significantly between adjacent frames, thereby ensuring localization accuracy. Specifically, based on the discriminant map of image frames in the same silent video, an intra-video temporal regularization loss function is constructed for optimization, expressed as:

[0113]

[0114] Among them, D p+2 D is the discriminant image of the (p+2)th image frame. p+1 This is the discriminant image for the (p+1)th image frame.

[0115] Through the above self-supervised auxiliary optimization strategy, the spatial localization results are fed back to the vocal action prediction part and trained in a self-supervised learning manner. It can assist in the detection of vocal action frames by recognizing action regions.

[0116] S6. Spatiotemporal feature fusion and prediction: The cross-kinematic aggregation features and activated motion region features are encoded respectively, and then spatiotemporal feature fusion is performed and frame-by-frame classification and prediction are performed to identify whether each image frame is a sound-generating action frame.

[0117] After obtaining cross-kinematic convergence features and activated motion region features, 3D convolution is used to encode temporal and spatial motion information, respectively encoding the cross-kinematic convergence features and activated motion region features. In order to extract more detailed information about vocalization actions, the output of the 3D convolution is input into a Transformer network with a self-attention module for spatiotemporal feature fusion to obtain spatiotemporal fused features, which effectively locates potential instances of vocalization actions in the video.

[0118] Finally, a fully connected layer is used to classify and predict each image frame frame by frame, identifying whether each image frame is an action frame, and obtaining the recognition result. During classification and prediction, cross-entropy loss and focal loss are used for supervision.

[0119]

[0120] Where, λ ce and λ focal Cross-entropy loss and focal loss The weight parameters are determined; meanwhile, the cross-entropy loss is calculated using a Gaussian-enhanced soft-labeling technique. In this embodiment, λ... ce =1,λ focal =0.1.

[0121] Therefore, the total loss function of this method is obtained by weighted summation of the above three loss functions:

[0122]

[0123] Where, λ action , λ cont and λ temp These are the weight parameters for the loss term.

[0124] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0125] Based on the same idea as the inflection point flow-based vocal action timing localization method in the above embodiments, the present invention also provides an inflection point flow-based vocal action timing localization system, which can be used to execute the above-described inflection point flow-based vocal action timing localization method. For ease of explanation, the structural schematic diagram of the embodiment of the inflection point flow-based vocal action timing localization system only shows the parts related to the embodiments of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0126] like Figure 3 As shown, another embodiment of the present invention provides a speech action temporal localization system based on inflection point flow, including a video reading module, an inflection point flow analysis module, a feature extraction module, a feature aggregation module, a self-supervised optimization module, and a classification prediction module;

[0127] The video reading module is used to read silent videos that contain sound-producing actions;

[0128] The inflection point flow analysis module is used to treat silent video as a time-varying object displacement map. Based on the laws of kinematics, it performs inflection point flow analysis on silent video to obtain bidirectional motion flow and bidirectional inflection point flow.

[0129] The feature extraction module is used to extract features from image frames, bidirectional motion streams, and bidirectional inflection point streams in silent videos, respectively, to obtain image context features, bidirectional motion features, and bidirectional velocity inflection point features.

[0130] The feature aggregation module is used to calculate cross-kinematic features from image to motion and from image to inflection point based on image context features, bidirectional motion features, and bidirectional velocity inflection point features, and then concatenate them to obtain cross-kinematic aggregated features;

[0131] The self-supervised optimization module is used to obtain a discriminant map from cross-kinematic features. On the discriminant map, cross-video contrastive learning is performed on moving and non-moving regions, and intra-video smoothing constraints are applied to obtain features that activate moving regions. The discriminant map contains both moving and non-moving regions.

[0132] The classification and prediction module is used to encode cross-kinematic features and activated motion region features respectively, then perform spatiotemporal feature fusion and classify and predict frame by frame to identify whether each image frame is a sound-generating action frame.

[0133] It should be noted that the speech action timing localization system based on inflection point flow of the present invention corresponds one-to-one with the speech action timing localization method based on inflection point flow of the present invention. The technical features and beneficial effects described in the embodiments of the speech action timing localization method based on inflection point flow are applicable to the embodiments of the speech action timing localization system based on inflection point flow. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.

[0134] Furthermore, in the above embodiments of the speech action timing localization system based on inflection point flow, the logical division of each program module is only an example. In actual applications, the above functions can be assigned to different program modules as needed, for example, for the configuration requirements of the corresponding hardware or for the convenience of software implementation. That is, the internal structure of the speech action timing localization system based on inflection point flow is divided into different program modules to complete all or part of the functions described above.

[0135] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0136] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for locating vocalization actions based on inflection point flow, characterized in that, Includes the following steps: Read silent video containing vocalizations; The silent video is treated as a displacement graph of an object that changes over time. Based on the laws of kinematics, inflection point flow analysis is performed on the silent video to obtain bidirectional motion flow and bidirectional inflection point flow. Feature extraction was performed on image frames, bidirectional motion streams, and bidirectional inflection point streams in silent videos to obtain image context features, bidirectional motion features, and bidirectional velocity inflection point features. Based on image context features, bidirectional motion features, and bidirectional velocity inflection point features, cross-kinematic features from image to motion and from image to inflection point are calculated and concatenated to obtain cross-kinematic aggregate features. A discriminant map is obtained from cross-kinematic aggregation features. Cross-video contrastive learning is performed on the discriminant map for moving and non-moving regions, and intra-video smoothing constraints are applied to obtain activated moving region features. The discriminant map includes both moving and non-moving regions. The cross-kinematic aggregation features and activated motion region features are encoded separately, and then spatiotemporal feature fusion is performed and frame-by-frame classification and prediction are conducted to identify whether each image frame is a sound-generating action frame.

2. The method for locating vocalization actions based on inflection point flow according to claim 1, characterized in that, The inflection point flow analysis of silent video based on the laws of kinematics is as follows: Silent video is treated as a time-varying object displacement graph, with velocity represented by optical flow and acceleration represented by optical flow difference. The forward optical flow of an image frame and its next adjacent image frame is calculated using an optical flow estimation algorithm as its corresponding forward motion flow; the reverse optical flow of an image frame and its previous adjacent image frame is calculated as its corresponding reverse motion flow. The forward motion flow of an image frame is differentially processed with the forward motion flow of its next adjacent image frame to obtain its corresponding forward inflection point flow. The reverse motion flow of an image frame is differentially processed with the reverse motion flow of its previous adjacent image frame to obtain its corresponding reverse inflection point flow. The bidirectional motion flow is obtained by splicing the forward and reverse motion flows of an image frame, and the bidirectional inflection point flow is obtained by splicing the forward and reverse inflection point flows of an image frame.

3. The method for locating vocalization actions based on inflection point flow according to claim 1, characterized in that, The feature extraction employs three independent ResNet encoder networks to extract the context features of each image frame and its bidirectional motion flow and bidirectional inflection point flow, respectively. The features are then concatenated to obtain the image context features, bidirectional motion features, and bidirectional velocity inflection point features of each image frame.

4. The method for locating vocalization actions based on inflection point flow according to claim 1, characterized in that, The concatenation yields cross-kinematic aggregation features, specifically: The image context features, bidirectional motion features, and bidirectional velocity inflection point features of each image frame are projected onto a linear layer to calculate the image-to-motion attention matrix and the image-to-inflection point attention matrix. After passing through the Softmax activation layer, cross-kinematic features from image to motion and from image to inflection point are obtained respectively; The cross-kinematic features of each image frame are obtained by concatenating the image context features, the cross-kinematic features from the image to the motion, and the cross-kinematic features from the image to the inflection point.

5. The method for locating vocalization actions based on inflection point flow according to claim 1, characterized in that, The cross-kinematic aggregation features are processed through a 3×3 convolutional layer, a batch normalization layer, and a sigmoid operation to obtain the corresponding discriminant map. In the discriminant map, high-probability regions are motion regions, and low-probability regions are non-motion regions.

6. The method for locating vocalization actions based on inflection point flow according to claim 1, characterized in that, The step of performing cross-video comparative learning on the discriminant map for moving and non-moving regions and applying intra-video smoothing constraints specifically involves: The motion and non-motion regions in the discriminant map are used to mask the cross-kinematic aggregation features to obtain motion region features and non-motion region features. Construct positive and negative contrast target pairs based on features of moving and non-moving regions; Based on positive and negative contrast target pairs, a cross-video contrast learning loss function is constructed and optimized for cross-kinematic features using a self-supervised learning approach to obtain activated motion region features; The cross-video contrastive learning loss function is expressed as follows: ， in, The negative contrastive objective loss function is used. For positive contrast target pairs, the loss function is defined as the positive contrast target feature of the moving region. The positive contrast target loss function is used for positive contrast target pairs that are features of non-moving regions. The video internal smoothing constraint is optimized based on the discriminant map of image frames in the same silent video, constructing a video temporal regularization loss function, expressed as: ， in, D pt+2 For the first pt +2 image frames of discrimination maps, D pt+1 For the first pt The discrimination map of +1 image frame.

7. The method for locating vocalization actions based on inflection point flow according to claim 6, characterized in that, The negative contrast target loss function is constructed based on the motion region features and non-motion region features of image frames sampled from multiple silent videos: ， in, k The number of image frames sampled from multiple silent videos. The inner product is represented as: ， For the first p Motion region features of each image frame For the first q Non-motion region features of each image frame; The positive contrast target loss function is constructed based on motion region features or non-motion region features of image frames sampled from multiple silent videos: in, For positive contrast target pairs, the loss function is defined as either positive contrast target features or non-moving region features. For the first p Motion region features or non-motion region features of an image frame For the first q Motion region features or non-motion region features of an image frame; exist p = q The value is 1 if it is true, and 0 otherwise. w p,q The penalty factor is represented as: ， α These are hyperparameters used for smoothness control. Inner product Ranking of cosine similarity between motion region features and non-motion region features across all image frames.

8. The method for locating vocalization actions based on inflection point flow according to claim 1, characterized in that, 3D convolution is used to encode cross-kinematic aggregation features and activation motion region features respectively. Then, the encoded output is input into a Transformer network with a self-attention module for spatiotemporal feature fusion to obtain spatiotemporal fused features. Then, a fully connected layer is used to classify and predict each image frame one by one to identify whether each image frame is a frame of sound action and obtain the recognition result.

9. The method for locating vocalization actions based on inflection point flow according to claim 8, characterized in that, When performing classification prediction frame-by-frame using fully connected layers, cross-entropy loss and focal Monitoring losses: ， in, λ ce and λ focal Cross-entropy loss and focal loss The weight parameters are: the cross-entropy loss is calculated using Gaussian-enhanced soft labeling.

10. A vocalization action timing localization system based on inflection point flow, characterized in that, The method for locating vocal actions based on inflection point flow as described in any one of claims 1-9 includes a video reading module, an inflection point flow analysis module, a feature extraction module, a feature aggregation module, a self-supervised optimization module, and a classification prediction module. The video reading module is used to read silent videos that contain sound-producing actions; The inflection point flow analysis module is used to treat silent video as a time-varying object displacement map, and to perform inflection point flow analysis on silent video according to the laws of kinematics to obtain bidirectional motion flow and bidirectional inflection point flow. The feature extraction module is used to extract features from image frames, bidirectional motion streams, and bidirectional inflection point streams in silent videos, respectively, to obtain image context features, bidirectional motion features, and bidirectional velocity inflection point features. The feature aggregation module is used to calculate cross-kinematic features from image to motion and from image to inflection point based on image context features, bidirectional motion features and bidirectional velocity inflection point features, and then concatenate them to obtain cross-kinematic aggregated features. The self-supervised optimization module is used to obtain a discriminant map from cross-kinematic features, perform cross-video contrastive learning on the discriminant map for moving and non-moving regions, and apply intra-video smoothing constraints to obtain activated moving region features; the discriminant map contains both moving and non-moving regions. The classification prediction module is used to encode cross-kinematic features and activated motion region features respectively, then perform spatiotemporal feature fusion and classify and predict frame by frame to identify whether each image frame is a sound-emitting action frame.

Images