Information determination method, device and electronic equipment

Abstract

The present application provides an information determination method, device and electronic device, the method comprising: obtaining a target video; inputting the target video into a spatiotemporal linkage feature learning network to obtain a temporal heat map of the target video; determining a target key moment of the target video based on the temporal heat map; wherein the spatiotemporal linkage feature learning network is used to extract a spatiotemporal linkage feature map of the video in the temporal and spatial dimensions, and obtaining the temporal heat map based on the spatiotemporal linkage feature map. The present application can improve the recognition accuracy of the video.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an information determination method, device and electronic device. Background Art

[0002] With the rapid development and widespread popularity of smart phones and mobile Internet technologies, various short, medium and long video applications have been integrated into all aspects of people's work and life. In the process of video editing and processing, it is often necessary to find some key moments (temporal key-points) of events that occurred in the video, and then perform subsequent operations such as clip editing, track alignment, and special effects addition. For example, when the user dances, special effects can be automatically triggered by identifying certain actions that are completed in a coherent manner, or in multiple videos of the same dance with different costumes, automatic costume changes can be achieved by identifying key nodes.

[0003] Current video editing and processing usually involves breaking down the video into a series of independent frames, and then applying image understanding algorithms to each frame, resulting in low recognition accuracy. Summary of the invention

[0004] The present application provides an information determination method, device and electronic device to solve the problem of low video recognition accuracy.

[0005] In a first aspect, an embodiment of the present application provides an information determination method, including:

[0006] Get the target video;

[0007] Inputting the target video into a spatiotemporal linkage feature learning network to obtain a time domain heat map of the target video;

[0008] Based on the time domain heat map, determining a target key moment point of the target video;

[0009] Among them, the spatiotemporal linkage feature learning network is used to extract the spatiotemporal linkage feature map of the video in the time domain dimension and the spatial domain dimension, and obtain the time domain heat map based on the spatiotemporal linkage feature map.

[0010] In a second aspect, an embodiment of the present application further provides an information determination device, characterized in that it includes:

[0011] An acquisition module, used to acquire a target video;

[0012] An input module, used to input the target video into a spatiotemporal linkage feature learning network to obtain a time domain heat map of the target video;

[0013] A determination module, used to determine a target key moment point of the target video based on the time domain heat map;

[0014] Among them, the spatiotemporal linkage feature learning network is used to extract the spatiotemporal linkage feature map of the video in the time domain dimension and the spatial domain dimension, and obtain the time domain heat map based on the spatiotemporal linkage feature map.

[0015] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a transceiver, a memory, a processor, and a program stored in the memory and executable on the processor; the processor is used to read the program in the memory to implement the steps in the method described in the first aspect of the embodiment of the present application.

[0016] In a fourth aspect, an embodiment of the present application further provides a readable storage medium, on which a program is stored, and when the program is executed by a processor, the steps in the method described in the first aspect of the embodiment of the present application are implemented.

[0017] In an embodiment of the present application, the target video is input into the spatiotemporal linkage feature learning network, the spatiotemporal linkage feature map of the target video in the time domain and spatial domain dimensions is extracted, and the time domain heat map of the target video is obtained based on the spatiotemporal linkage feature map. Compared with the existing recognition through single-frame images, the present application obtains a time domain heat map by learning the before-and-after temporal relationship of continuously changing pictures in the video through the spatiotemporal linkage feature learning network, and then determines the target key moment points of the target video based on the time domain heat map, realizes the recognition of key moment points, and can improve the accuracy of recognition of key moment points in the target video. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 It is a flowchart of an information determination method provided in an embodiment of the present application;

[0020] Figure 2 It is a flowchart of a key moment point positioning method provided in an embodiment of the present application;

[0021] Figure 3 It is a schematic diagram of the construction and training of a spatiotemporal linkage convolutional network provided in an embodiment of the present application;

[0022] Figure 4 It is a schematic diagram of a spatiotemporal linkage convolutional layer provided in an embodiment of the present application;

[0023] Figure 5 It is a schematic diagram of a spatiotemporal linkage convolutional network provided in an embodiment of the present application;

[0024] Figure 6 It is a schematic diagram of a key moment point positioning system provided in an embodiment of the present application.

[0025] Figure 7 is a structural diagram of an information determination device provided in an embodiment of the present application;

[0026] Figure 8 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] The terms "first", "second" etc. in the embodiments of the present application are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. In addition, the terms "include" and "have" and any variation thereof are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units need not be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment. In addition, "and / or" is used in the present application to represent at least one of connected objects, such as A and / or B and / or C, representing the inclusion of separate A, separate B, separate C, and A and B all exist, B and C all exist, A and C all exist, and 7 situations that A, B and C all exist.

[0029] See also Figure 1 , Figure 1 is a flow chart of an information determination method provided in an embodiment of the present application, such as Figure 1 As shown, the following steps are included:

[0030] Step 101: Obtain target video.

[0031] It is understood that the target video may record one or more actions continuously through video frames that are continuous in time sequence, and the target video may be a video to be processed input by a user or captured in real time by a video capture device, for example: videos of sports events such as table tennis and badminton, dance videos uploaded by users, etc.

[0032] Step 102: input the target video into a spatiotemporal linkage feature learning network to obtain a time domain heat map of the target video.

[0033] Step 103: determining a target key moment point of the target video based on the time domain heat map;

[0034] Among them, the spatiotemporal linkage feature learning network is used to extract the spatiotemporal linkage feature map of the video in the time domain dimension and the spatial domain dimension, and obtain the time domain heat map based on the spatiotemporal linkage feature map.

[0035] For a coherent action in a video, we can find the key moments in the action and perform subsequent operations such as clip editing, audio track alignment, and special effects addition at the key moments. For example, we can add special effects at the moment when an athlete swings the racket; add special effects when the user completes certain actions in a coherent manner during the dance; or automatically identify key moments in multiple videos of the same dance with different costumes to achieve automatic clothing change.

[0036] It is understandable that the key moments in the above actions often need to be combined with the sequential information of the entire action in the video, so as to confirm the moment when the key moments are reached in the continuous action. In the embodiment of the present application, the above spatiotemporal linkage feature learning network learns the spatiotemporal linkage features of the video in the temporal and spatial dimensions by extracting spatiotemporal linkage feature maps in the temporal and spatial dimensions.

[0037] In the above-mentioned time-domain heat map, each pixel point can correspond to a moment in the above-mentioned target video, and the moment corresponding to the pixel point with greater heat is more likely to be a key moment point.

[0038] It should be noted that the key moment of an action in the above target video is the moment when the action reaches a certain posture, and the time period of the above target video can be divided into multiple moments according to the minimum moment unit, and the above target video is divided into multiple video frames according to the video frame rate. The above minimum moment unit and the above video frame rate do not necessarily match, that is, in the above target video, the above key moment can correspond to one or more video frames. Taking the above minimum moment unit as 0.1 second as an example, in the athlete's swing action, when the above target video is displayed at 30 frames / second, the moment of hitting the ball can be represented by 3 consecutive video frames, and the key moment can correspond to the 3 video frames; when the above target video is displayed at 10 frames / second, the moment of hitting the ball only corresponds to 1 video frame, and the key moment corresponds to the video frame.

[0039] Correspondingly, the target video is input into the spatiotemporal linkage feature learning network in the form of multiple continuous video frames to obtain the time domain heat map of the target video. The heat value of each pixel can represent a frame of image corresponding to each moment, or multiple frames of image corresponding to each moment.

[0040] In an embodiment of the present application, the target video is input into the spatiotemporal linkage feature learning network, the spatiotemporal linkage feature map of the target video in the time domain and spatial domain dimensions is extracted, and the time domain heat map of the target video is obtained based on the spatiotemporal linkage feature map. Compared with the existing recognition through single-frame images, the present application obtains a time domain heat map by learning the before-and-after temporal relationship of continuously changing pictures in the video through the spatiotemporal linkage feature learning network, and then determines the target key moment points of the target video based on the time domain heat map, realizes the recognition of key moment points, and can improve the accuracy of recognition of key moment points in the target video.

[0041] Optionally, the time domain heat map includes the heat of multiple time points in the target video;

[0042] Determining the target key time point of the target video based on the time domain heat map in step 103 includes:

[0043] Based on the heat of multiple time points in the target video, obtaining at least one peak and the heat of at least one peak of the time domain heat map;

[0044] Based on the heat of the at least one peak, the target key moment point is determined, the target key moment point is a peak among the at least one peak, and the heat of the peak corresponding to the target key moment point is greater than a preset threshold.

[0045] In the above-mentioned time domain heat map, multiple time points in the above-mentioned target video correspond to different pixel points respectively, and the time points corresponding to adjacent pixel points are also adjacent in the time domain. In this way, by representing the heat of the above-mentioned multiple time points in the above-mentioned time domain heat map, for a coherent action, its corresponding heat change in the time domain is also coherent. The actions that are coherent in the time domain in the above-mentioned target video are represented as the above-mentioned time domain heat map, and the heat change can be understood as the waveform change. The time point of the peak may become the key time point identified by the embodiment of the present application.

[0046] In an embodiment of the present application, based on the heat of multiple time points in the target video, at least one peak of the time domain heat map and the heat of at least one peak are obtained; based on the heat of the at least one peak, the target key moment point is determined, the target key moment point is a peak among the at least one peak, and the heat of the peak corresponding to the target key moment point is greater than a preset threshold, thereby realizing the identification of the key moment point.

[0047] Optionally, determining the target key moment point based on the heat of the at least one peak includes:

[0048] Based on the heat of the at least one peak, determining a candidate key moment point, the candidate key moment point being a peak among the at least one peak, and the heat of the peak corresponding to the candidate key moment point being greater than the preset threshold;

[0049] If there are at least two candidate key moments in the preset time window, determining the target key moment from the at least two candidate key moments based on a non-maximum suppression algorithm;

[0050] If there is one candidate key moment point within the preset time window, the one candidate key moment point is determined as the target key moment point.

[0051] It can be understood that, first, each peak in the above-mentioned time domain heat map can be obtained, and the peaks with heat greater than the above-mentioned preset threshold can be screened as candidate key moments. In the above-mentioned target video, if the video time is long, there may be multiple key moments, but for one of the coherent actions, there is usually only one key moment. For example, in sports events, taking the example of completing the coherent action of swinging the racket in 1 second, that is, the moment of swinging the racket and hitting the ball only exists once within 1 second. If the interval between the moments corresponding to the two identified peaks is less than 1 second, it can be considered that there is only one key moment in the two peaks, and the moment corresponding to the peak with greater heat can be selected as the key moment.

[0052] In this embodiment, the length of the above-mentioned preset time window can be preset according to experience or application scenarios to meet the actual situation. In a preset time window, if there are at least two candidate key moments, the candidate key moment with the highest heat can be selected as the above-mentioned target key moment through the non-maximum suppression algorithm; in a preset time window, if there is only one candidate key moment, it can be understood that the one candidate key moment is the above-mentioned target key moment.

[0053] In this implementation, if there are at least two candidate key moments in the preset time window, the target key moment is determined from the at least two candidate key moments based on the non-maximum suppression algorithm; if there is one candidate key moment in the preset time window, the one candidate key moment is determined as the target key moment. Thus, the recognition accuracy of the target key moment can be further improved.

[0054] Optionally, the spatiotemporal linkage feature learning network includes a first convolutional layer, a second convolutional layer and a third convolutional layer;

[0055] In step 102, the target video is input into a spatiotemporal linkage feature learning network to obtain a time domain heat map of the target video, including:

[0056] Inputting the target video into the first convolutional layer for feature extraction to obtain a spatiotemporal linkage feature map of the target video in the time domain and the spatial domain;

[0057] Inputting the spatiotemporal linkage feature map into the second convolutional layer to perform feature dimension reduction of the spatial domain dimension to obtain a temporal domain feature map;

[0058] The time domain feature map is input into the third convolutional layer for heat extraction to obtain a time domain heat map.

[0059] It can be understood that the first convolution layer is used to extract the spatiotemporal linkage feature map of the video in the time domain and spatial dimensions, the second convolution layer is used to perform feature dimension reduction of the spatiotemporal linkage feature map in the spatial dimension to obtain the time domain feature map, the third convolution layer is used to perform heat extraction on the time domain feature map to obtain the time domain feature map, and the first convolution layer, the second convolution layer and the third convolution layer can all be composed of a plurality of stacked convolution sub-layers.

[0060] Among them, the input of the above-mentioned first convolutional layer is the input of the above-mentioned spatiotemporal linkage feature learning network. Therefore, the number of input channels calculated by the above-mentioned first convolutional layer can correspond to the number of channels of the input video. If the input video is in color, the number of channels is 3 (corresponding to RGB three channels), and if the input video is in black and white, the number of channels is 1, thereby ensuring channel matching.

[0061] In this implementation, the input target video is processed in sequence through the first convolutional layer, the second convolutional layer and the third convolutional layer of the spatiotemporal linkage feature learning network to achieve feature extraction in the spatial and temporal dimensions, as well as feature dimensionality reduction in the spatial dimension, and obtain the temporal heat map.

[0062] Optionally, inputting the spatiotemporal linkage feature map into the second convolutional layer to perform feature dimension reduction of the spatial dimension to obtain the temporal feature map includes:

[0063] Transposing the spatiotemporal linkage feature map in time domain dimension to obtain a multi-channel time domain feature map;

[0064] The multi-channel time domain feature map is input into the second convolutional layer to perform feature dimension reduction of the spatial dimension to obtain the time domain feature map.

[0065] It can be understood that in this embodiment, it is necessary to obtain the time domain heat map of the target video, so as to determine the target key time points based on the time domain heat map. After the target video is input into the first convolutional layer for feature extraction, the obtained spatiotemporal linkage feature map can characterize the spatiotemporal joint features of the target video in the time domain and spatial domain dimensions, so the spatiotemporal linkage feature map can be reduced in the spatial domain dimension to obtain a time domain feature map that can generate a time domain heat map.

[0066] Specifically, the above-mentioned spatiotemporal linkage feature map can be understood as a multidimensional matrix in the time domain dimension and the spatial domain dimension, and can be transposed in the time domain dimension to transpose the spatiotemporal linkage feature map into a one-dimensional multi-channel time domain feature map. For example: the spatiotemporal linkage feature map can be expressed as T×W×H×C, and the transposed one-dimensional multi-channel time domain feature map can be expressed as T×1×C′, where T represents the number of time points of the video (divided according to the minimum time point unit), W represents the width of the video frame, H represents the length of the video frame, C represents the number of channels of the spatiotemporal linkage feature map, C′ represents the number of channels of the one-dimensional multi-channel time domain feature map, and C′=W×H×C. The above-mentioned second convolution layer can include a 1×1×c spatial bottleneck convolution (c represents the number of channels of the spatial bottleneck convolution) to perform feature dimensionality reduction on the one-dimensional multi-channel time domain feature map T×1×C′ to obtain a time domain feature map T×1×C″, wherein C″ is the number of channels of the time domain feature map. By adjusting c so that C″ is much smaller than C′, feature dimensionality reduction of the spatial dimension is achieved.

[0067] Optionally, the first convolution layer includes a spatiotemporal convolution kernel for extracting a spatiotemporal linkage feature map of the video;

[0068] The spatiotemporal convolution kernel is obtained based on the convolution kernel of the time domain dimension and the convolution kernel of the spatial domain dimension.

[0069] Among them, the above-mentioned spatiotemporal convolution kernel can be composed of the convolution kernel of the above-mentioned time domain dimension and the convolution kernel of the above-mentioned spatial dimension in series, or can be merged after convolution based on dual-track processing of two serial modes (the serial mode of "convolution kernel of time domain dimension + convolution kernel of spatial dimension" and the serial mode of "convolution kernel of spatial dimension + convolution kernel of time domain dimension"). It can also be a three-dimensional convolution kernel obtained by combining the one-dimensional convolution kernel of the above-mentioned time domain dimension and the two-dimensional convolution kernel of the above-mentioned spatial dimension to realize joint feature extraction in the time domain dimension and the spatial domain dimension.

[0070] Optionally, during the convolution process, the input of the first convolution layer may be added to the output of the spatiotemporal convolution kernel to implement residual learning.

[0071] In this embodiment, the target video is convolved by the spatiotemporal convolution kernel obtained by the convolution kernel based on the time domain dimension and the convolution kernel in the spatial domain dimension, thereby extracting the spatiotemporal linkage feature map of the video in the time domain dimension and the spatial domain dimension, and effectively learning the before-and-after correlation information of the action in the target video.

[0072] Optionally, the first convolution layer includes multiple sub-convolution layers, each sub-convolution layer includes a spatiotemporal convolution kernel, and the step size of the spatiotemporal convolution kernel is a positive integer greater than 1.

[0073] Specifically, if the first convolution layer includes m sub-convolution layers and the step size of the spatiotemporal convolution kernel is n, then in the convolution process, each time a sub-convolution layer is passed, an n-fold downsampling can be achieved. Through the first convolution layer, m×n-fold downsampling can be achieved, where m and n are both positive integers greater than 1.

[0074] In this embodiment, the first convolution layer includes multiple sub-convolution layers. During the convolution process, by performing convolution with a step size greater than 1 on the spatiotemporal convolution kernel in each sub-convolution layer, downsampling processing in the spatial dimension is achieved, which can reduce computational complexity and memory overhead.

[0075] Optionally, the first convolution layer includes a sub-convolution layer and a pooling layer that are arranged at intervals, and the input of the pooling layer is a spatiotemporal linkage feature map output by the sub-convolution layer connected to the pooling layer.

[0076] It should be noted that the first convolution layer may include multiple sub-convolution layers, and two pooling layers may be separated by several sub-convolution layers. The input of the pooling layer is the spatiotemporal linkage feature map output by the sub-convolution layer connected to the pooling layer, and the spatiotemporal linkage feature map is downsampled in the spatial dimension.

[0077] In this embodiment, through the sub-convolutional layers and pooling layers arranged at intervals in the first convolutional layer, the input of the pooling layer is the spatiotemporal linkage feature map output by the sub-convolutional layer connected to the pooling layer, and downsampling processing in the spatial dimension is also implemented, which can reduce computational complexity and memory overhead.

[0078] Optionally, the method further comprises the following steps:

[0079] Iteratively train the network to be trained using multiple videos with pre-annotated time points;

[0080] Using the loss function to calculate the result output by the network to be trained, to obtain a loss value;

[0081] Based on the loss value, the gradient of the network to be trained is returned to obtain the trained spatiotemporal linkage feature learning network.

[0082] It can be understood that the spatiotemporal linkage feature learning network used in step 102 can be a network obtained by training with multiple training sample data, and after the target key moment point is obtained in step 103, the spatiotemporal linkage feature learning network can be trained again. In this embodiment, the performance of the spatiotemporal linkage feature learning network is optimized by training the spatiotemporal linkage feature learning network.

[0083] Among them, the above-mentioned loss function can use loss functions such as Mean Squared Error (MSE), wing loss or adaptive wing loss, and this application does not impose any restrictions on this.

[0084] In this embodiment, the network to be trained is iteratively trained by using multiple videos with pre-marked time points; the result output by the network to be trained is calculated using a loss function to obtain a loss value; and the network to be trained is gradient-backed based on the loss value to obtain the trained spatiotemporal linkage feature learning network, thereby realizing the training of the spatiotemporal linkage feature learning network.

[0085] Optionally, the iterative training of the network to be trained using a plurality of videos with pre-marked time points may specifically include:

[0086] Obtaining time point annotations of the multiple videos;

[0087] Based on the time point annotations, generating heat maps of the multiple videos, and annotating the multiple videos with the heat maps;

[0088] The network to be trained is iteratively trained using multiple videos with heat map annotation.

[0089] Specifically, the heat map of the above video can be obtained by transforming the time point annotations of each video based on the Gaussian kernel function. It can be understood that the actions in the video are continuous, so the heat changes between the key moments of the action and the moments before and after them are also continuous. By converting the labeled data into a heat map, it is coordinated with the time domain heat map output by the above-mentioned spatiotemporal linkage feature learning network.

[0090] In this embodiment, based on the time point marking, a heat map of the multiple videos is generated, and the multiple videos are annotated with the heat map; thereby, the network to be trained is iteratively trained using the multiple videos annotated with the heat map, which can improve the efficiency and accuracy of the training process.

[0091] For ease of understanding, the specific examples are as follows:

[0092] The present application embodiment provides a key moment point positioning method, such as Figure 2 As shown, the following steps are included:

[0093] Step S21: obtaining video key point annotation data, specifically, obtaining training video samples, the training video samples including multiple video segments, each video segment containing key moment point annotation information, constituting annotation data;

[0094] Step S22: construct and train a spatio-temporal coordinated convolutional network based on the mixed dimensions of time domain and space domain:

[0095] For better understanding, the present application embodiment also provides a method for constructing and training a spatiotemporal convolutional network. Figure 3 As shown, the specific steps are as follows:

[0096] Step S31: construct a spatio-temporal coordinated convolution layer based on two-dimensional spatial convolution and one-dimensional temporal convolution. The basic component of the network is the spatio-temporal coordinated convolution layer. Figure 4 As shown, it includes a two-dimensional spatial convolution kernel (spatial convolution kernel) of size 1×s×s and a one-dimensional temporal convolution kernel (temporal convolution kernel) of size t×1×1. The two tracks of "spatial convolution + temporal convolution" and "temporal convolution + spatial convolution" are processed separately and then merged to form the basic unit of spatiotemporal linkage convolution feature learning. Optionally, the input and output can be added to achieve residual learning and reduce the difficulty of feature learning. At this time, the number of output channels of "spatial convolution + temporal convolution" and "temporal convolution + spatial convolution" are half of the number of channels of the original input feature map, ensuring that the number of output channels after the feature map is merged matches the original input. Among them, Figure 3 The dotted line indicates that residual learning is an optional step, that is, whether to perform residual learning is not an inevitable characteristic of spatiotemporal linkage learning. The spatiotemporal linkage convolution layer calculation can embed spatiotemporal feature extraction into the smallest unit - spatial adjacent pixels and temporal adjacent pixels, and extract spatiotemporal linkage features by learning spatiotemporal and spatial linkage features. Among them, s represents the width and height of the two-dimensional spatial convolution kernel, t represents the length of the one-dimensional temporal convolution kernel, s and t are adjustable parameters. In this embodiment, s = 3, t = 5;

[0097] Step S32: Construct a spatiotemporal linkage convolutional network and map the input video to a high-order spatiotemporal linkage semantic space. The spatiotemporal linkage convolutional layer can achieve spatiotemporal linkage feature learning with the smallest granularity. Based on the neural network (backbone network) composed of many spatiotemporal linkage convolutional layers, the input video can be mapped to a high-order spatiotemporal linkage semantic space to achieve feature extraction. Figure 5 A schematic diagram of a spatiotemporal linkage convolutional network provided in an embodiment of the present application is shown in FIG. Figure 5 As shown in the figure, the original size of the input video is T×W×H×C1, where T is the number of video frames (i.e., when the minimum time point unit matches the video frame rate), W and H are the width and height of the video frame, respectively, and C1 is the number of channels of the video frame (when the video is color, C1=3, corresponding to the RGB three channels, when the video is black and white, C1=1, and the number of input channels calculated in the first layer is also C1 to ensure channel matching). At the same time, in order to reduce computational complexity and memory overhead, the data is downsampled 2× based on the spatial dimension after each layer of calculation. Downsampling methods include but are not limited to 2× spatial pooling or spatial convolution with a stride of 2. The output of the backbone network is a spatiotemporal linkage semantic feature map mapped to a high-order feature space, and its size is T×W / r×H / r×C2, where r is the downsampling multiple, and C2 is the number of channels of the spatiotemporal linkage semantic feature map, which can be adjusted as needed. Common settings include 128, 256, 512 and other channel dimensions.

[0098] Step S33: Convert the spatiotemporal linkage semantic feature map into a time domain feature map, and calculate the time domain heat map. The spatiotemporal linkage semantic feature map obtained in the above step S32 contains information of mixed dimensions of time and space, and multiple rounds of downsampling have been performed in the space domain dimension, while the time series dimension still maintains the original resolution. At this time, the spatiotemporal linkage semantic feature map T×W / r×H / r×C2 can be transposed into a one-dimensional multi-channel time domain feature map of T×1×C3 based on the time domain dimension, where C3=W / r·H / r·C2, indicating the number of channels of the one-dimensional multi-channel time domain feature map; then use a 1×1 spatial bottleneck convolution to perform feature dimensionality reduction, and output the time domain feature map T×1×C4, where C4 is the number of channels of the time domain feature map, and C4 is an adjustable parameter, C4 is much smaller than C3, and in this embodiment, C4=256; finally, use a t×1 one-dimensional time domain convolution to generate a time domain heat map T×1.

[0099] Step S34: Generate a Gaussian heat map based on the labeled data, and implement model training in combination with the loss function and gradient return. The labeled data consists of a pair of training video samples and key moment point annotation information. The key moment point annotations are usually time coordinates and cannot be directly used as supervisory signals. At this time, they can be converted into Gaussian heat maps based on the Gaussian kernel function, and combined with gradient return and MSE as loss functions to construct supervisory signals and train the spatiotemporal linkage convolutional network. In addition, other related loss functions such as wing loss and adaptive wing loss are also applicable.

[0100] Step S23: Use the trained spatiotemporal linkage convolutional network to infer the target video and output a time domain heat map;

[0101] Specifically, the above-mentioned spatiotemporal linkage convolutional network that has completed training is obtained through the above steps. The spatiotemporal linkage convolutional network is a special network structure proposed in the embodiment of the present application. The spatiotemporal linkage convolutional network is constructed and trained based on the mixed dimensions of time domain and space domain, and is used to extract the spatiotemporal linkage features of the video in the spatial domain dimension and the time domain dimension.

[0102] Step S24: Calculate candidate key time points, and select the best time points based on time-domain non-maximum heat suppression.

[0103] In the embodiment of the present application, the key moment point is abstracted as a time-series key point in the time domain dimension, and the spatiotemporal linkage semantic feature map is converted into a temporal heatmap to realize key moment point detection. The position and heat of the peak are calculated in the temporal heatmap, and the candidate key moment point is obtained in combination with the preset heat threshold.

[0104] Specifically, when there are multiple candidate key moments within a short time range, what is usually detected is actually the same target key moment. At this time, non-maximum suppression can be performed based on the temporal window: the candidate points are sorted based on heat and other candidate points with lower heat than the candidate point within the time window are eliminated in turn. This method can be called temporal non-maximum heat suppression. The final result outputted by step S23 is the key moment location result of the target video.

[0105] like Figure 6 As shown, the embodiment of the present application also provides a key moment point positioning system, including:

[0106] The spatiotemporal linkage semantic feature extraction module M1 is used to execute the relevant algorithm logic in step S31, step S41 and step S23 in the above-mentioned key moment point positioning method embodiment, construct a spatiotemporal linkage convolutional network, map the original video to a high-order feature space and extract the spatiotemporal linkage semantic feature map;

[0107] Key moment point reasoning module M2: used to execute the relevant algorithm logic in step S42 and step S23 in the above key moment point positioning method embodiment, generate a time domain heat map based on the spatiotemporal linkage semantic feature map, and generate the final key moment point reasoning result through step S24;

[0108] Neural network training module M3: used to execute the data processing logic in steps S21 to S43 in the above network construction and training method embodiment, realize the collection and preservation of original video data and key moment point annotations, and the generation of time domain heat map, and train the neural network composed of M1 and M2 modules based on the "video + time domain heat map" training data and loss function, gradient return mechanism.

[0109] In this embodiment, the above-mentioned key moment point positioning method can be used to identify the moment when athletes swing their rackets to hit the ball in table tennis, badminton and other events to trigger phantom effects and particle tailing effects, thereby achieving cool creative videos; and after identifying the key moment points, perform operations such as clip editing, audio track alignment, and special effects addition, thereby reducing labor costs and improving video processing efficiency.

[0110] In the embodiment of the present application, by inputting the target video into the spatiotemporal linkage convolutional network, integrating the contextual temporal and spatial feature information in the video, obtaining a temporal heat map, and then detecting key moments, capturing the key moments of continuous actions, it is possible to effectively identify key moments triggered by the previous and subsequent associated information of continuous actions. That is, abandoning the use of image artificial intelligence (AI) to solve video problems, and using video AI to solve video problems is more in line with the essence of the business.

[0111] See also Figure 7 , Figure 7 Schematic diagram of the structure of an information determination device provided in an embodiment of the present application. Figure 7 As shown, the information determination device 700 includes:

[0112] An acquisition module 701 is used to acquire a target video;

[0113] An input module 702, used to input the target video into a spatiotemporal linkage feature learning network to obtain a time domain heat map of the target video;

[0114] A determination module 703, configured to determine a target key moment of the target video based on the time domain heat map;

[0115] Among them, the spatiotemporal linkage feature learning network is used to extract the spatiotemporal linkage feature map of the video in the time domain dimension and the spatial domain dimension, and obtain the time domain heat map based on the spatiotemporal linkage feature map.

[0116] Optionally, the time domain heat map includes the heat of multiple time points in the target video;

[0117] The determining module 703 may specifically include:

[0118] A first acquisition unit, configured to acquire at least one peak and the heat of at least one peak of the time domain heat map based on the heat of multiple time points in the target video;

[0119] The first determination unit is used to determine the target key moment point based on the heat of the at least one peak, the target key moment point is a peak among the at least one peak, and the heat of the peak corresponding to the target key moment point is greater than a preset threshold.

[0120] Optionally, the determining module 703 may specifically include:

[0121] A second determining unit is configured to determine a candidate key moment point based on the heat of the at least one peak, wherein the candidate key moment point is a peak among the at least one peak, and the heat of the peak corresponding to the candidate key moment point is greater than the preset threshold;

[0122] A third determining unit is configured to determine the target key moment point from the at least two candidate key moment points based on a non-maximum suppression algorithm if there are at least two candidate key moment points within a preset time window;

[0123] The fourth determining unit is configured to determine, if there is one candidate key moment point within the preset time window, the one candidate key moment point as the target key moment point.

[0124] Optionally, the spatiotemporal linkage feature learning network includes a first convolutional layer, a second convolutional layer and a third convolutional layer;

[0125] The input module 702 may specifically include:

[0126] A first input unit, used to input the target video into the first convolutional layer for feature extraction, so as to obtain a spatiotemporal linkage feature map of the target video in the time domain and the spatial domain;

[0127] A second input unit is used to input the spatiotemporal linkage feature map into the second convolutional layer to perform feature dimension reduction of the spatial domain dimension to obtain a temporal domain feature map;

[0128] The third input unit is used to input the time domain feature map into the third convolutional layer for heat extraction to obtain a time domain heat map.

[0129] Optionally, the second input unit may specifically include:

[0130] A transposition subunit, used for transposing the spatiotemporal linkage feature map in a time domain dimension to obtain a multi-channel time domain feature map;

[0131] An input subunit is used to input the multi-channel time domain feature map into the second convolutional layer to perform feature dimensionality reduction of the spatial dimension to obtain the time domain feature map.

[0132] Optionally, the first convolution layer includes a spatiotemporal convolution kernel for extracting a spatiotemporal linkage feature map of the video;

[0133] The spatiotemporal convolution kernel is obtained based on the convolution kernel of the time domain dimension and the convolution kernel of the spatial domain dimension.

[0134] Optionally, the first convolution layer includes multiple sub-convolution layers, each sub-convolution layer includes a spatiotemporal convolution kernel, and the step size of the spatiotemporal convolution kernel is a positive integer greater than 1.

[0135] Optionally, the first convolution layer includes a sub-convolution layer and a pooling layer that are arranged at intervals, and the input of the pooling layer is a spatiotemporal linkage feature map output by the sub-convolution layer connected to the pooling layer.

[0136] Optionally, the information determining device 700 may further include:

[0137] A training module, used for iteratively training a network to be trained using a plurality of videos with pre-annotated time points;

[0138] A calculation module, used to calculate the result output by the network to be trained using a loss function to obtain a loss value;

[0139] A feedback module is used to perform gradient feedback on the network to be trained based on the loss value to obtain the trained spatiotemporal linkage feature learning network

[0140] Optionally, the training module may specifically include:

[0141] A second acquisition unit, used to acquire time point annotations of the multiple videos;

[0142] A generating unit, configured to generate a heat map of the plurality of videos based on the time point annotations, and annotate the heat map of the plurality of videos;

[0143] A training unit is used to iteratively train the network to be trained using multiple videos with heat map annotation.

[0144] The information determination device 700 can implement the embodiment of the present application Figure 1 The various processes of the method embodiment and the same beneficial effects are achieved, and will not be described again here to avoid repetition.

[0145] The present application also provides an electronic device. Figure 1 The information determination method shown is similar, so the implementation of the electronic device can refer to the implementation of the method, and the repeated parts will not be repeated. Figure 8 As shown, the electronic device of the embodiment of the present application includes a memory 820, a transceiver 810, and a processor 800;

[0146] The memory 820 is used to store computer programs; the transceiver 810 is used to send and receive data under the control of the processor 800; the processor 800 is used to read the computer program in the memory 820 and perform the following operations:

[0147] Get the target video;

[0148] Inputting the target video into a spatiotemporal linkage feature learning network to obtain a time domain heat map of the target video;

[0149] Based on the time domain heat map, determining a target key moment point of the target video;

[0150] Among them, the spatiotemporal linkage feature learning network is used to extract the spatiotemporal linkage feature map of the video in the time domain dimension and the spatial domain dimension, and obtain the time domain heat map based on the spatiotemporal linkage feature map.

[0151] Among them, Figure 8 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking together various circuits of one or more processors represented by processor 800 and memory represented by memory 820. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 810 may be a plurality of components, namely, a transmitter and a transceiver, providing a unit for communicating with various other devices on a transmission medium. The processor 800 is responsible for managing the bus architecture and general processing, and the memory 820 may store data used by the processor 800 when performing operations.

[0152] The processor 800 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or a complex programmable logic device (CPLD). The processor may also adopt a multi-core architecture.

[0153] Optionally, the time domain heat map includes the heat of multiple time points in the target video;

[0154] Determining the target key moment of the target video based on the time domain heat map includes:

[0155] Based on the heat of multiple time points in the target video, obtaining at least one peak and the heat of at least one peak of the time domain heat map;

[0156] Based on the heat of the at least one peak, the target key moment point is determined, the target key moment point is a peak among the at least one peak, and the heat of the peak corresponding to the target key moment point is greater than a preset threshold.

[0157] Optionally, determining the target key moment point based on the heat of the at least one peak includes:

[0158] Based on the heat of the at least one peak, determining a candidate key moment point, the candidate key moment point being a peak among the at least one peak, and the heat of the peak corresponding to the candidate key moment point being greater than the preset threshold;

[0159] If there are at least two candidate key moments in the preset time window, determining the target key moment from the at least two candidate key moments based on a non-maximum suppression algorithm;

[0160] If there is one candidate key moment point within the preset time window, the one candidate key moment point is determined as the target key moment point.

[0161] Optionally, the spatiotemporal linkage feature learning network includes a first convolutional layer, a second convolutional layer and a third convolutional layer;

[0162] The step of inputting the target video into a spatiotemporal linkage feature learning network to obtain a time domain heat map of the target video includes:

[0163] Inputting the target video into the first convolutional layer for feature extraction to obtain a spatiotemporal linkage feature map of the target video in the time domain and the spatial domain;

[0164] Inputting the spatiotemporal linkage feature map into the second convolutional layer to perform feature dimension reduction of the spatial domain dimension to obtain a temporal domain feature map;

[0165] The time domain feature map is input into the third convolutional layer for heat extraction to obtain a time domain heat map.

[0166] Optionally, inputting the spatiotemporal linkage feature map into the second convolutional layer to perform feature dimension reduction of the spatial dimension to obtain the temporal feature map includes:

[0167] Transposing the spatiotemporal linkage feature map in time domain dimension to obtain a multi-channel time domain feature map;

[0168] The multi-channel time domain feature map is input into the second convolutional layer to perform feature dimension reduction of the spatial dimension to obtain the time domain feature map.

[0169] Optionally, the first convolution layer includes a spatiotemporal convolution kernel for extracting a spatiotemporal linkage feature map of the video;

[0170] The spatiotemporal convolution kernel is obtained based on the convolution kernel of the time domain dimension and the convolution kernel of the spatial domain dimension.

[0171] Optionally, the first convolution layer includes multiple sub-convolution layers, each sub-convolution layer includes a spatiotemporal convolution kernel, and the step size of the spatiotemporal convolution kernel is a positive integer greater than 1.

[0172] Optionally, the first convolution layer includes a sub-convolution layer and a pooling layer that are arranged at intervals, and the input of the pooling layer is a spatiotemporal linkage feature map output by the sub-convolution layer connected to the pooling layer.

[0173] Optionally, the processor 800 is further configured to read the computer program in the memory 820 and perform the following operations:

[0174] Iteratively train the network to be trained using multiple videos with pre-annotated time points;

[0175] Using the loss function to calculate the result output by the network to be trained, to obtain a loss value;

[0176] Based on the loss value, the gradient of the network to be trained is returned to obtain the trained spatiotemporal linkage feature learning network.

[0177] Optionally, the iterative training of the network to be trained using a plurality of videos with pre-marked time points includes:

[0178] Obtaining time point annotations of the multiple videos;

[0179] Based on the time point annotations, generating heat maps of the multiple videos, and annotating the multiple videos with the heat maps;

[0180] The network to be trained is iteratively trained using multiple videos with heat map annotation.

[0181] The electronic device provided in the embodiment of the present application can perform the above Figure 1 The implementation principle and technical effect of the method embodiment shown are similar, and this embodiment will not be repeated here.

[0182] The embodiment of the present application also provides a readable storage medium on which a program or instruction is stored. When the program or instruction is executed by a processor, the above Figure 1 The various processes of the method embodiment can achieve the same technical effect, and to avoid repetition, they will not be described again here.

[0183] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0184] The present application also provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the above Figure 1 The various processes of the method embodiment can achieve the same technical effect, and to avoid repetition, they will not be described again here.

[0185] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0186] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0187] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0188] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for determining information, characterized in that: include: Get the target video; Inputting the target video into a spatiotemporal linkage feature learning network to obtain a time domain heat map of the target video; Based on the time domain heat map, determining a target key moment point of the target video; The spatiotemporal linkage feature learning network is used to extract the spatiotemporal linkage feature map of the video in the time domain and the spatial domain, and obtain the time domain heat map based on the spatiotemporal linkage feature map; The spatiotemporal linkage feature learning network includes a first convolutional layer, a second convolutional layer and a third convolutional layer; The step of inputting the target video into a spatiotemporal linkage feature learning network to obtain a time domain heat map of the target video includes: Inputting the target video into the first convolutional layer for feature extraction to obtain a spatiotemporal linkage feature map of the target video in the time domain and the spatial domain; Inputting the spatiotemporal linkage feature map into the second convolutional layer to perform feature dimension reduction of the spatial domain dimension to obtain a temporal domain feature map; The time domain feature map is input into the third convolutional layer for heat extraction to obtain a time domain heat map.

2. The method according to claim 1, characterized in that The temporal heat map includes the heat of multiple time points in the target video; Determining the target key moment of the target video based on the time domain heat map includes: Based on the heat of multiple time points in the target video, obtaining at least one peak and the heat of at least one peak of the time domain heat map; Based on the heat of the at least one peak, the target key moment point is determined, the target key moment point is a peak among the at least one peak, and the heat of the peak corresponding to the target key moment point is greater than a preset threshold.

3. The method according to claim 2, characterized in that The determining the target key time point based on the heat of the at least one peak includes: Based on the heat of the at least one peak, determining a candidate key moment point, the candidate key moment point being a peak among the at least one peak, and the heat of the peak corresponding to the candidate key moment point being greater than the preset threshold; If there are at least two candidate key moments in the preset time window, determining the target key moment from the at least two candidate key moments based on a non-maximum suppression algorithm; If there is one candidate key moment point within the preset time window, the one candidate key moment point is determined as the target key moment point.

4. The method according to claim 1, characterized in that The step of inputting the spatiotemporal linkage feature map into the second convolutional layer to perform feature dimension reduction of the spatial domain dimension to obtain the temporal domain feature map includes: Transposing the spatiotemporal linkage feature map in time domain dimension to obtain a multi-channel time domain feature map; The multi-channel time domain feature map is input into the second convolutional layer to perform feature dimension reduction of the spatial dimension to obtain the time domain feature map.

5. The method according to claim 1, characterized in that The first convolution layer includes a spatiotemporal convolution kernel for extracting a spatiotemporal linkage feature map of the video; The spatiotemporal convolution kernel is obtained based on the convolution kernel of the time domain dimension and the convolution kernel of the spatial domain dimension.

6. The method according to claim 5, characterized in that The first convolution layer includes multiple sub-convolution layers, each sub-convolution layer includes a spatiotemporal convolution kernel, and the step size of the spatiotemporal convolution kernel is a positive integer greater than 1.

7. The method according to claim 5, characterized in that The first convolution layer includes a sub-convolution layer and a pooling layer that are arranged at intervals, and the input of the pooling layer is a spatiotemporal linkage feature map output by the sub-convolution layer connected to the pooling layer.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: Iteratively train the network to be trained using multiple videos with pre-annotated time points; Using the loss function to calculate the result output by the network to be trained, to obtain a loss value; Based on the loss value, the gradient of the network to be trained is returned to obtain the trained spatiotemporal linkage feature learning network.

9. The method according to claim 8, characterized in that The iterative training of the network to be trained using a plurality of videos with pre-marked time points includes: Obtaining time point annotations of the multiple videos; Based on the time point annotations, generating heat maps of the multiple videos, and annotating the multiple videos with the heat maps; The network to be trained is iteratively trained using multiple videos with heat map annotation.

10. An information determination device, characterized in that: include: An acquisition module, used to acquire a target video; An input module, used to input the target video into a spatiotemporal linkage feature learning network to obtain a time domain heat map of the target video; A determination module, used to determine a target key moment point of the target video based on the time domain heat map; The spatiotemporal linkage feature learning network is used to extract the spatiotemporal linkage feature map of the video in the time domain and the spatial domain, and obtain the time domain heat map based on the spatiotemporal linkage feature map; The spatiotemporal linkage feature learning network includes a first convolutional layer, a second convolutional layer and a third convolutional layer; The input module specifically includes: A first input unit, used to input the target video into the first convolutional layer for feature extraction, so as to obtain a spatiotemporal linkage feature map of the target video in the time domain and the spatial domain; A second input unit is used to input the spatiotemporal linkage feature map into the second convolutional layer to perform feature dimension reduction of the spatial domain dimension to obtain a temporal domain feature map; The third input unit is used to input the time domain feature map into the third convolutional layer for heat extraction to obtain a time domain heat map.

11. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the information determination method according to any one of claims 1 to 9 are implemented.

12. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by the processor, the steps of the information determination method according to any one of claims 1 to 9 are implemented.

Images