Video analysis method, apparatus, processor and storage medium

Abstract

The application discloses a video analysis method and device, a processor and a storage medium. The method comprises the following steps: obtaining a video to be analyzed; inputting the video into a feature processing model to obtain a global context relationship feature of the video and a local context relationship feature of the video; and determining a target class segment in the video based on the global context relationship feature of the video and the local context relationship feature of the video. The application solves the technical problem that the accuracy of determining a video segment of interest of a user is low when only relying on the local context relationship feature of the video.

Description

Technical Field

[0001] This invention relates to the technical field of video analysis and processing, and more specifically, to a video analysis method, apparatus, processor, and storage medium. Background Technology

[0002] In existing technologies, methods commonly used to determine video segments of interest to users include temporal sliding windows and probabilistic prediction of the segment of interest segment by segment. These methods can effectively locate video segments of interest to users in simple and ideal scenarios. However, in reality, video segments of interest often contain significant amounts of noise and irrelevant background frame noise, which may arise from the behavior itself or the shot composition during filming. These issues are ignored by current technologies, thus affecting the accuracy of determining the video segments of interest to users.

[0003] For example, a video might contain a cricket clip that the user needs to locate. Within this clip, there's a scene of cheers after a goal is scored. Current technology doesn't account for this interference, and relying solely on local context isn't sufficient to understand that the cheering scene should be part of the action requiring location. Therefore, it affects the accuracy of identifying the video clip the user is interested in.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides a video analysis method, apparatus, processor, and storage medium to at least solve the technical problem that relying solely on local contextual features of a video to determine the video segment of interest to a user results in low accuracy in identifying the segment.

[0006] According to one aspect of the present invention, a video analysis method is provided, comprising: acquiring a video to be analyzed; inputting the video into a feature processing model for processing to obtain features of the global context relationship of the video and features of the local context relationship of the video; and determining target class segments in the video based on the features of the global context relationship of the video and the features of the local context relationship of the video.

[0007] Further, processing the video input feature processing model to obtain the global context relationship features and the local context relationship features of the video includes: dividing the video into multiple video segments; encoding the multiple video segments to obtain a video feature matrix; and inputting the video feature matrix into the feature processing model for processing to obtain the global context relationship features and the local context relationship features of the video.

[0008] Furthermore, determining the target class segment in the video based on the features of the global context relationship and the features of the local context relationship of the video includes: determining the probability values ​​of multiple candidate video segments in the video as target class segments based on the features of the global context relationship and the features of the local context relationship of the video; and selecting the target class segment from the multiple candidate video segments according to the probability values ​​of the multiple candidate video segments as target class segments.

[0009] Furthermore, the method further includes: receiving a request instruction from the target object to obtain the target type segment; responding to the request instruction and pushing the target type segment to the target object; or, when it is detected that the target object is playing the video for the first time, pushing the target type segment to the target object.

[0010] Furthermore, after pushing the target class fragment to the target object, the method further includes: if it is detected that the target object has modified the target class fragment, obtaining the modification result; and correcting the feature processing model based on the modification result to update the feature processing model.

[0011] Furthermore, the feature processing model includes a global feature extraction module and a local feature extraction module. Processing the video into the feature processing model to obtain the global contextual relationship features and the local contextual relationship features of the video includes: inputting the video feature matrix into the global feature extraction module to extract global contextual relationship features; and inputting the video feature matrix into the local feature extraction module to extract local contextual relationship features.

[0012] Furthermore, the feature processing model includes a global feature extraction module and a local feature extraction module. The video is input into the feature processing model for processing to obtain the global contextual relationship features and the local contextual relationship features of the video. This includes: reducing the dimensionality of the video feature matrix to obtain a dimensionality-reduced feature matrix; inputting the dimensionality-reduced feature matrix into the global feature extraction module to extract global contextual relationship features; and inputting the dimensionality-reduced feature matrix into the local feature extraction module to extract local contextual relationship features.

[0013] Further, the reduced-dimensional feature matrix is ​​input into the global feature extraction module to extract global contextual features, thereby obtaining the global contextual features of the video. This includes: determining that the reduced-dimensional feature matrix is ​​subjected to three different linear projections to obtain linear projection Q, linear projection K, and linear projection V; obtaining features carrying global contextual relationships through linear projection Q, linear projection K, and linear projection V; and inputting the features carrying global contextual relationships into a feedforward neural network for processing to obtain the global contextual features of the video.

[0014] Furthermore, before processing the video input feature processing model, the method further includes: obtaining a video feature sample vector; inputting the video feature sample vector into the global feature extraction module for learning and training, while simultaneously supervising the global feature extraction module using a behavior classification loss function, so as to drive the global feature extraction module to capture features of global contextual relationships.

[0015] Furthermore, the local feature extraction module includes two graph convolutional layers, namely: a first graph convolutional layer and a second graph convolutional layer. The local feature extraction module processes the dimensionality-reduced feature matrix to obtain the local contextual features of the video. This process includes: passing the dimensionality-reduced feature matrix through the first graph convolutional layer to obtain the features of the first local contextual relationship; and inputting the features of the first local contextual relationship and the dimensionality-reduced feature matrix into the second graph convolutional layer, obtaining the local contextual features of the video through the output of the second graph convolutional layer.

[0016] Further, determining the probability values ​​of multiple candidate video segments as target class segments based on the features of the global context relationship and the features of the local context relationship of the video includes: fusing the features of the global context relationship and the features of the local context relationship of the video to obtain fused features; processing the fused features through a boundary point prediction module to obtain a first probability value of multiple candidate video segments as target class segments; processing the fused features through a confidence prediction module to obtain a second probability value of multiple candidate video segments as target class segments; and determining the probability values ​​of multiple candidate video segments as target class segments based on the first probability value and the second probability value.

[0017] According to one aspect of the present invention, a video analysis method is provided, comprising: receiving a service call request sent by a client, wherein the service call request carries a video to be analyzed; responding to the service call request, processing the video in a server by inputting the video into a feature processing model to obtain features of the global context relationship of the video and features of the local context relationship of the video; determining a target class segment in the video based on the features of the global context relationship of the video and the features of the local context relationship of the video; and outputting the target class segment.

[0018] According to one aspect of the present invention, a video analysis method is provided, comprising: acquiring a live video; inputting the live video into a feature processing model for processing to obtain features of the global context relationship of the live video and features of the local context relationship of the live video; and determining a target class segment in the live video based on the features of the global context relationship of the live video and the features of the local context relationship of the live video.

[0019] Furthermore, after determining the target class segment in the live video, the method further includes: editing the target class segment from the live video; and publishing the target class segment on the target application to promote the target object described in the target class segment.

[0020] According to one aspect of the present invention, a video analysis apparatus is provided, comprising: a first acquisition unit for acquiring a video to be analyzed; a first processing unit for processing the video into a feature processing model to obtain features of the global context relationship of the video and features of the local context relationship of the video; and a first determination unit for determining a target class segment in the video based on the features of the global context relationship of the video and the features of the local context relationship of the video.

[0021] According to one aspect of the present invention, a video analysis apparatus is provided, comprising: a receiving unit, configured to receive a service call request sent by a client, wherein the service call request carries a video to be analyzed; a second processing unit, configured to respond to the service call request, process the video input feature processing model in a server to obtain features of the global context relationship of the video and features of the local context relationship of the video; determine a target class segment in the video based on the features of the global context relationship of the video and the features of the local context relationship of the video; and an output unit, configured to output the target class segment.

[0022] According to one aspect of the present invention, a video analysis apparatus is provided, comprising: a second acquisition unit for acquiring a live video; a third processing unit for processing the live video into a feature processing model to obtain features of the global context relationship of the live video and features of the local context relationship of the live video; and a second determination unit for determining a target class segment in the live video based on the features of the global context relationship of the live video and the features of the local context relationship of the live video.

[0023] According to one aspect of the present invention, a storage medium is provided, the storage medium including a stored program, wherein, when the program is executed, it controls the device where the storage medium is located to perform the method described in any of the above-described embodiments.

[0024] According to one aspect of the present invention, a processor is provided for running a program, wherein the program executes the method described in any one of the above embodiments.

[0025] In this embodiment of the invention, a method is adopted to determine target class segments in a video based on the features of global context and local context. This involves acquiring the video to be analyzed, processing the video into a feature processing model to obtain the features of the video's global context and local context; and determining the target class segments in the video based on these features. This strengthens the representation of features, ensuring the accuracy of the determined target class segments, thereby improving the accuracy of determining target class segments in the video. This solves the technical problem that relying solely on the local context features of the video to determine the video segments of interest to the user results in low accuracy in identifying those segments. Attached Figure Description

[0026] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0027] Figure 1 This is a hardware structure block diagram of a computer terminal according to an embodiment of the present invention;

[0028] Figure 2 This is a flowchart of a video analysis method provided according to Embodiment 1 of the present invention;

[0029] Figure 3 This is a flowchart of a video analysis method provided according to Embodiment 2 of the present invention;

[0030] Figure 4 This is a flowchart of a video analysis method provided according to Embodiment 3 of the present invention;

[0031] Figure 5 This is a schematic diagram of a video analysis device provided according to Embodiment 4 of the present invention;

[0032] Figure 6 This is a schematic diagram of a video analysis device provided according to Embodiment 5 of the present invention;

[0033] Figure 7 This is a schematic diagram of a video analysis device provided according to Embodiment Six of the present invention;

[0034] Figure 8 This is a structural block diagram of an optional computer terminal according to an embodiment of the present invention. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] Example 1

[0038] According to an embodiment of the present invention, a video analysis method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0039] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a video analysis method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0040] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0041] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the video analysis method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the video analysis method of the aforementioned application. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0042] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0043] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0044] Under the aforementioned operating environment, this application provides the following: Figure 2 The video analysis method shown. Figure 2 This is a flowchart of a video analysis method according to Embodiment 1 of the present invention.

[0045] Step S201: Obtain the video to be analyzed.

[0046] For example, if the video to be analyzed is a recorded course video, to improve learning efficiency, it is necessary to locate the video segments that students need to focus on from the longer course video. Similarly, if the video to be analyzed is a recorded meeting video, to improve the efficiency of obtaining effective information, it is necessary to locate the meeting segments that users need to focus on from the longer meeting video.

[0047] Step S202: Process the video input feature processing model to obtain the features of the global context relationship of the video and the features of the local context relationship of the video.

[0048] Step S203: Based on the features of the global context relationship of the video and the features of the local context relationship of the video, determine the target class segment in the video.

[0049] In the above steps, the video input feature processing model is processed to obtain the features of the global context relationship of the video and the features of the local context relationship of the video. This can avoid the problem of accurately determining the target class segment in the video based on the features of the global context relationship of the video and the features of the local context relationship of the video, even if there is a lot of noise and irrelevant background frames in the video.

[0050] For course videos, the video analysis method provided in this application can quickly identify the video segments that students need to focus on based on the features of the global and local context relationships of the course video. For meeting videos, the same method can quickly identify the video segments that users need to focus on based on the features of the global and local context relationships of the meeting video.

[0051] By employing steps S201 to S203 to jointly determine target-type segments in the video based on features of both global and local contextual relationships, the representation of features is strengthened to ensure the accuracy of the determined target-type segments in the video. This achieves the technical effect of improving the accuracy of determining target-type segments in the video, thereby solving the technical problem that the accuracy of determining video segments of interest to users is low due to relying solely on the local contextual relationship features of the video.

[0052] Optionally, in the video analysis method provided in Embodiment 1 of this application, processing the video into a feature processing model to obtain the features of the global context relationship and the features of the local context relationship of the video includes: dividing the video into multiple video segments; encoding the multiple video segments to obtain a video feature matrix; and inputting the video feature matrix into the feature processing model to obtain the features of the global context relationship and the features of the local context relationship of the video.

[0053] In the above scheme, the video is divided into multiple video segments. This can be done by sequentially dividing the video into segments based on a predetermined number of image frames. For example, if the video contains 20,000 frames, frames 0-100 could be designated as video segment one, frames 101-200 as video segment two, and so on. Each video segment is then encoded using an encoder to obtain the video feature matrix. This encoder can be constructed from a pre-trained two-stream network, which extracts features F∈R for each video segment. C×T Where C is the number of channels, a video feature matrix is ​​obtained. The obtained video feature matrix is ​​then input into the feature processing model for processing to obtain the global contextual features and the local contextual features of the video.

[0054] Optionally, in the video analysis method provided in Embodiment 1 of this application, determining the target class segment in the video based on the features of the global context relationship and the features of the local context relationship of the video includes: determining the probability values ​​of multiple candidate video segments as target class segments based on the features of the global context relationship and the features of the local context relationship of the video; and selecting the target class segment from the multiple candidate video segments according to the probability values ​​of the multiple candidate video segments as target class segments.

[0055] In the above scheme, based on the characteristics of the global context relationship and the local context relationship of the video, the probability of multiple candidate video segments being the target segment is quantified. Based on the quantized probability value, the target segment can be selected from multiple candidate video segments, which can ensure the accuracy of the determined target segment in the video.

[0056] Optionally, in the video analysis method provided in Embodiment 1 of this application, the method further includes: receiving a request instruction from the target object to obtain a target type segment; responding to the request instruction and pushing the target type segment to the target object; or, when it is detected that the target object is playing the video for the first time, pushing the target type segment to the target object.

[0057] The target audience mentioned above can be those who are interested in target-type segments. For example, the target-type segment could be a segment in a course video that students need to pay attention to; when a student requests to access that segment, the target-type segment is pushed to the student. As another example, the target-type segment could be a segment in a meeting video that attendees need to pay attention to; when an attendee requests to access that segment, the target-type segment is pushed to the attendee. Additionally, if the target audience is watching the video for the first time, the target-type segment can be proactively pushed to attract their attention.

[0058] The above approach can respond to a request from the target object by pushing the located target fragment to the target object, or it can proactively push the fragment, thereby improving the efficiency of the target object in obtaining the target fragment and enhancing the user experience.

[0059] Optionally, in the video analysis method provided in Embodiment 1 of this application, after pushing the target segment to the target object, the method further includes: if it is detected that the target object has modified the target segment, obtaining the modification result; and correcting the feature processing model based on the modification result to update the feature processing model.

[0060] After locating the target segment, if the target user wishes to modify it, they can do so. The modified segment is then used as the result. Based on this result, the feature processing model can be trained and optimized to ensure that the optimized model can accurately locate video segments of interest to the user in the future.

[0061] Optionally, in the video analysis method provided in Embodiment 1 of this application, determining the probability values ​​of multiple candidate video segments as target class segments based on the features of the global context relationship and the features of the local context relationship of the video includes: fusing the features of the global context relationship and the features of the local context relationship of the video to obtain fused features; processing the fused features through a boundary point prediction module to obtain a first probability value of multiple candidate video segments as target class segments; processing the fused features through a confidence prediction module to obtain a second probability value of multiple candidate video segments as target class segments; and determining the probability values ​​of multiple candidate video segments as target class segments based on the first probability value and the second probability value.

[0062] It should be noted that the above-mentioned method of fusing the features of the global context and the local context of the video can be either early fusion or late fusion. Early fusion involves directly fusing the features of the global and local contexts of the video, and then inputting the fused features into the output modules (including the boundary point prediction module and the confidence prediction module). Late fusion involves inputting the features of the two branches (the features of the global context and the features of the local context) into the output modules separately, and then concatenating and fusing the features of the two branches before each classifier in the boundary point prediction module and the confidence prediction module in the output modules to obtain the fused features.

[0063] The boundary point prediction module described above is used to predict the probability that each candidate video segment in the video is the starting point. The probability of the endpoint The confidence prediction module iterates through all candidate video segments and scores their confidence. This module can connect two classifiers, for example, trained using a binary classification loss function and a regression loss function respectively. The predicted confidence scores for the two classifiers are denoted as follows: and The probability value p of the final candidate video segment being selected as the target segment is determined by the starting point probability, ending point probability, and confidence score of each pre-selected candidate video segment, and the formula is as follows:

[0064]

[0065] In other words, the probability value of each candidate video segment being the target segment can be calculated using the above formula.

[0066] It should be noted that, since all possibilities are traversed, the generated candidate video segments are dense and redundant. Soft-NMS (Soft Non-maximum suppression) can be used to remove redundant candidate video segments. After Soft-NMS processing, the remaining candidate video segments and their probability values ​​can be used as output.

[0067] For example, if the probability of video clip 1 being a segment of interest to the user is 0.8, video clip 2 is 0.9, video clip 3 is 0.81, and so on, and the preset threshold is 0.85, then by comparing the probability of each video clip being a segment of interest to the user with the preset threshold, video clip 2 is determined to be the target segment.

[0068] Optionally, in the video analysis method provided in Embodiment 1 of this application, the feature processing model includes a global feature extraction module and a local feature extraction module. The video is input into the feature processing model for processing to obtain the features of the global context relationship and the features of the local context relationship of the video. This includes: inputting the video feature matrix into the global feature extraction module to extract the features of the global context relationship, thereby obtaining the features of the global context relationship of the video; and inputting the video feature matrix into the local feature extraction module to extract the features of the local context relationship, thereby obtaining the features of the local context relationship of the video.

[0069] Since the feature processing model includes a global branch and a local branch, specifically, the global branch is a global feature extraction module, which can optionally be a transformer model, and the local branch is a local feature extraction module, which can optionally be a graph convolutional neural network model. The global branch uses the global feature extraction module to capture global contextual relationships, and the local branch uses the local feature extraction module to capture local contextual relationships, thereby obtaining the features of the global contextual relationships and the features of the local contextual relationships of the video.

[0070] To improve computational efficiency, the video analysis method provided in Embodiment 1 of this application performs dimensionality reduction on the video feature matrix to obtain a dimensionality-reduced feature matrix. For example, if the video feature matrix is ​​a 1024*1024 matrix, dimensionality reduction yields a 256*256 feature matrix. The dimensionality-reduced feature matrix is ​​then input into a Transformer model for global context feature extraction, yielding the global context features of the video. The dimensionality-reduced feature matrix is ​​also input into a graph convolutional network for local context feature extraction, yielding the local context features of the video. In other words, by reducing the dimensionality of the video feature matrix and then extracting the local and global context features based on the dimensionality-reduced feature matrix, computational efficiency can be further improved.

[0071] In the video analysis method provided in Embodiment 1 of this application, the dimensionality-reduced feature matrix is ​​input into a Transformer model for global context feature extraction. The resulting global context features include: determining the dimensionality-reduced feature matrix through three different linear projections to obtain linear projection Q, linear projection K, and linear projection V; and then applying the linear projections Q, K, and V using the following formula to obtain features carrying global context, where the formula is... in, It is a feature that carries global contextual relationships, F g d is the feature matrix after dimensionality reduction; d is the dimension of Q and K, and T is the number of video segments; the features carrying global context are input into the feedforward neural network for processing to obtain the global context features of the video.

[0072] The aforementioned feedforward neural network is called a Feed Forward Network, or FFN for short.

[0073] It should be noted that linear projection involves performing a linear matrix operation. For example, if the feature matrix after dimensionality reduction is Fg, then through linear projection Q, Q = Fg. g *w, where w is a parameter matrix. Using the above method, we can obtain the linear projection K and the linear projection V. (Using the above formula...) By performing calculations, features carrying global contextual relationships can be obtained. These features are then input into a feedforward neural network. Through linear projections from two linear layers in the feedforward neural network, the feature representation is further enhanced. Each linear layer can be followed by a non-linear activation function and a residual connection. Corresponding output and normalization layers are also added after each linear layer. The purpose is to guide the Transformer model to truly capture the necessary information, that is, to capture the features of the video's global contextual relationships.

[0074] Optionally, in the video analysis method provided in Embodiment 1 of this application, before inputting the video feature matrix into the dual-branch module for processing, the method further includes: obtaining a video feature sample vector; inputting the video feature sample vector into the global feature extraction module for learning and training, and simultaneously supervising the global feature extraction module by combining the behavior classification loss function, so as to drive the global feature extraction module to capture the features of global contextual relationships.

[0075] In the video analysis method provided in Embodiment 1 of this application, a behavior classification loss function is designed as an auxiliary loss function. For example, by classifying actions or backgrounds segment by segment, the Transformer model is driven to capture the correct contextual relationships. The behavior classification loss function L a The design is as follows:

[0076]

[0077] Where T is the number of video segments into which the video is divided. It is the predicted value of the behavior score for the i-th video segment. This is the true value of the action score for the i-th video segment. This segment classification supervision can directly force the Transformer model to learn how to properly capture global contextual relationships and generate high-quality features. It should be noted that the prediction results of each video segment only assist the Transformer model's learning during training, and the entire action prediction network can be removed during the testing phase.

[0078] Optionally, in the video analysis method provided in Embodiment 1 of this application, the local feature extraction module includes two graph convolutional layers, wherein the two graph convolutional layers include: a first graph convolutional layer and a second graph convolutional layer. The dimensionality-reduced feature matrix is ​​input into the graph convolutional network for processing to obtain the features of the local context of the video. This includes: obtaining the features of the first local context using the following formula in the first graph convolutional layer from the dimensionality-reduced feature matrix, wherein the formula is: Where W is the parameter matrix, F l Let A be the feature matrix after dimensionality reduction, ⊙ represent dot product, and M be the mask, which is used to limit the scope of the connection matrix. The features of the first local context relationship and the feature matrix after dimensionality reduction are input into the second graph convolutional layer. The local context relationship features of the video are obtained through the output of the second graph convolutional layer.

[0079] In order to more effectively capture local contextual relationships within local branches, a completely new graph is set up in this application. In the constructed graph, ν and ε represent nodes and edges, respectively. Nodes are each segment in the video, and edges are the dependencies between two nodes. In the local context relation model, edges are constructed based on the temporal distance between nodes, and the set of edges is defined as:

[0080] ε={(v_i, v_{i+k})|i∈{1,…,T};k∈{0,±1,…,±δ}},

[0081] Here, T represents the number of nodes, and 6 represents the maximum allowed temporal distance between connections. Specifically, the expression for graph convolutional layer operations is as follows: as follows:

[0082]

[0083] Where W is a trainable parameter matrix, F l Let A be the feature matrix input to the local branch, ⊙ represent the dot product, and M be a mask used to limit the scope of the connection matrix so that it only affects local features. The connection matrices of the graph convolutional layers in the above graph convolutional network are entirely obtained through training and have no constraints on A; therefore, they are not manually pre-defined as in related techniques. Two such graph convolutional layers are concatenated; that is, the first and second graph convolutional layers are concatenated. The input of the second graph convolutional layer is obtained by adding the input and output of the first convolutional layer, thereby acquiring features carrying global and local contextual relationships.

[0084] Optionally, the video to be analyzed is encoded using a two-stream network to obtain a video feature matrix. The video feature matrix is ​​then dimensionality-reduced using a one-dimensional convolutional network (Convld). This dimensionality-reduced feature matrix is ​​input into the global and local branches. In the global branch, output features carrying global context are calculated using linear mappings Q, K, and V as inputs to multi-head attention. These output features are then passed through a feedforward neural network to obtain the global context features of the video. In the local branch, two cascaded graph convolutional layers (GCN) are used to obtain the local context features of the video. The global and local context features are then fused and input into the output module (including a boundary point prediction module and a confidence prediction module). The boundary point prediction module is used to predict the probability that each candidate video segment in the video is a starting point. The probability of the endpoint The confidence prediction module iterates through all candidate video segments and scores their confidence. This module can connect two classifiers, for example, trained using a binary classification loss function and a regression loss function respectively. The predicted confidence scores for the two classifiers are denoted as follows: and The probability value p of the final candidate video segment being the target class segment. f The probability of the starting point, the probability of the ending point, and the confidence score corresponding to each pre-selected candidate video segment are jointly determined by the following formula:

[0085]

[0086] In other words, the probability value of each candidate video segment being the target segment can be calculated using the above formula. This ensures the accuracy of the identified target segments in the video, thereby improving the technical effect of determining the accuracy of the target segments in the video. This solves the technical problem that relying solely on the local contextual features of the video to determine the video segments of interest to the user results in relatively low accuracy in identifying the video segments of interest to the user.

[0087] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0088] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0089] Example 2

[0090] In the operating environment provided in Embodiment 1 above, this application provides the following: Figure 3 The video analysis method shown. Figure 3 This is a flowchart of a video analysis method according to Embodiment 2 of the present invention.

[0091] Step S301: Receive a service call request sent by the client, wherein the service call request carries the video to be analyzed.

[0092] Step S302: In response to the service call request, the video input feature processing model is processed in the server to obtain the global context relationship features and the local context relationship features of the video; based on the global context relationship features and the local context relationship features of the video, the target class segment in the video is determined.

[0093] In the above steps, the server responds to the client's service call request and processes the video input feature processing model to obtain the global context features and local context features of the video. This allows for accurate identification of target segments in the video, even when there is significant noise and irrelevant background frames, based on these global and local context features. For example, for course videos, the server can quickly identify the video segments students need to focus on based on the global and local context features. Similarly, for conference videos, the server can quickly identify the video segments users need to focus on based on the global and local context features.

[0094] Step S303: Output the target class fragment.

[0095] Through steps S301 to S303, the client triggers a service call request, and the server analyzes and processes the video, quickly and accurately identifying and outputting the target segment within the video. The solution in this application improves both the accuracy and efficiency of locating target segments in the video.

[0096] It should be noted that the specific technical solution for determining the target segment in the video on the server is the same as the specific solution in Embodiment 1, and will not be repeated here.

[0097] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0098] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0099] Example 3

[0100] In the operating environment provided in Embodiment 1 above, this application provides the following: Figure 4 The video analysis method shown. Figure 4 This is a flowchart of a video analysis method according to Embodiment 3 of the present invention.

[0101] Step S401: Obtain the live video.

[0102] Step S402: Input the live video into the feature processing model for processing to obtain the global context relationship features and the local context relationship features of the live video.

[0103] Step S403: Based on the features of the global context relationship of the live video and the features of the local context relationship of the live video, determine the target class segment in the live video.

[0104] By processing the live video into a feature processing model, the global contextual relationship features and the local contextual relationship features of the live video are obtained. This allows for accurate identification of target class segments in the live video even when there is a lot of noise and irrelevant background frames.

[0105] The above solution applies video analysis methods to live stream videos, which can be recorded during e-commerce live streams or other types of live streams. This solution allows for the quick and accurate identification of video segments from longer live streams that are of interest to the streamer or users.

[0106] It should be noted that the specific technical solution for determining the target segment in the live video application scenario is the same as the specific solution in Embodiment 1, and will not be repeated here.

[0107] Optionally, in the video analysis method provided in Embodiment 3 of this application, after determining the target class segment in the live video, the method further includes: editing the target class segment from the live video; and publishing the target class segment on the target application to promote the target object described in the target class segment.

[0108] For example, a live video may include introductions and explanations of various products by the host. A video segment explaining a particular lipstick may be identified in the live video. In the video analysis method provided in Embodiment 3 of this application, the video segment can be edited out of the live video and published on an application to promote the lipstick.

[0109] The above method can automatically cut out target-type segments after accurately identifying them in a video, and then use these segments to promote the target objects described within them, thereby improving the promotion effect.

[0110] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0111] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0112] Example 4

[0113] According to embodiments of the present invention, an apparatus for implementing the video analysis method in Embodiment 1 described above is also provided, such as... Figure 5 As shown, the device includes: a first acquisition unit 501, a first processing unit 502, and a first determination unit 503.

[0114] Specifically, the first acquisition unit 501 is used to acquire the video to be analyzed;

[0115] The first processing unit 502 is used to process the video input feature processing model to obtain the features of the global context relationship of the video and the features of the local context relationship of the video.

[0116] The first determining unit 503 is used to determine the target class segment in the video based on the features of the global context relationship of the video and the features of the local context relationship of the video.

[0117] In summary, in the video analysis apparatus provided in Embodiment 4 of this application, the first acquisition unit 501 acquires the video to be analyzed; the first processing unit 502 processes the video input feature processing model to obtain the global context relationship features and the local context relationship features of the video; the first determination unit 503 determines the target class segments in the video based on the global context relationship features and the local context relationship features of the video, thereby strengthening the representation of features and ensuring the accuracy of the determined target class segments in the video. This achieves the technical effect of improving the accuracy of determining the target class segments in the video, and solves the technical problem that the accuracy of determining the video segments of interest to the user is relatively low due to relying solely on the local context relationship features of the video.

[0118] It should be noted that the first acquisition unit 501, the first processing unit 502, and the first determination unit 503 mentioned above correspond to steps S201 to S203 in Embodiment 1. The three units and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above-mentioned unit modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0119] Example 5

[0120] According to embodiments of the present invention, an apparatus for implementing the video analysis method in Embodiment 2 described above is also provided, such as... Figure 6 As shown, the device includes a receiving unit 601, a second processing unit 602, and an output unit 603.

[0121] Specifically, the receiving unit 601 is used to receive a service call request sent by the client, wherein the service call request carries a video to be analyzed;

[0122] The second processing unit 602 is used to respond to the service call request, process the video input feature processing model in the server to obtain the global context relationship features and the local context relationship features of the video; and determine the target class segment in the video based on the global context relationship features and the local context relationship features of the video.

[0123] Output unit 603 is used to output the target class fragment.

[0124] In summary, in the video analysis apparatus provided in Embodiment 5 of this application, the receiving unit 601 receives a service call request sent by a client, wherein the service call request carries the video to be analyzed; the second processing unit 602 responds to the service call request, processes the video input feature processing model in the server to obtain the features of the global context relationship of the video and the features of the local context relationship of the video; based on the features of the global context relationship of the video and the features of the local context relationship of the video, the target class segment in the video is determined; the output unit 603 outputs the target class segment, which can quickly and accurately determine the target class segment in the video and output it. Through the solution in this application, the accuracy of locating target class segments in the video can be improved while simultaneously increasing the efficiency of locating target class segments.

[0125] It should be noted that the specific technical solution for determining the target segment in the video on the server is the same as the specific solution in Embodiment 1, and will not be repeated here.

[0126] It should be noted that the receiving unit 601, the second processing unit 602, and the output unit 603 mentioned above correspond to steps S301 to S302 in Embodiment 1. The three units and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 2. It should also be noted that the above-mentioned unit modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0127] Example 6

[0128] According to embodiments of the present invention, an apparatus for implementing the video analysis method in Embodiment 3 described above is also provided, such as... Figure 7 As shown, the device includes: a second acquisition unit 701, a third processing unit 702, and a second determination unit 703.

[0129] Specifically, the second acquisition unit 701 is used to acquire live video;

[0130] The third processing unit 702 is used to input the live video into the feature processing model for processing to obtain the global context relationship features and the local context relationship features of the live video.

[0131] The second determining unit 703 is used to determine the target class segment in the live video based on the features of the global context relationship of the live video and the features of the local context relationship of the live video.

[0132] In summary, in the video analysis apparatus provided in Embodiment Six of this application, the second acquisition unit 701 acquires the live video; the third processing unit 702 processes the live video into a feature processing model to obtain the global contextual relationship features and the local contextual relationship features of the live video; and the second determination unit 703 determines the target segment in the live video based on the global contextual relationship features and the local contextual relationship features of the live video. That is, by processing the live video into the feature processing model to obtain the global contextual relationship features and the local contextual relationship features of the live video, it is possible to accurately determine the target segment in the live video even if there is a large amount of noise and irrelevant background frames, based on the global contextual relationship features and the local contextual relationship features of the live video.

[0133] In the above solution, applied to live streaming videos, the live streaming video can be a video recorded during a live stream in an e-commerce scenario, or a video recorded during other types of live streams. This solution allows for the quick and accurate identification of video segments that the streamer or users are interested in from longer live streaming videos.

[0134] It should be noted that the specific technical solution for determining the target segment in the live video application scenario is the same as the specific solution in Embodiment 1, and will not be repeated here.

[0135] Optionally, in the video analysis apparatus provided in Embodiment Six of this application, the apparatus further includes: an editing unit, configured to edit out the target class segment from the live video after determining the target class segment in the live video; and a publishing unit, configured to publish the target class segment on a target application to promote the target object described in the target class segment.

[0136] For example, a live video may include the host's introduction and explanation of various products. A video segment explaining a particular lipstick may be selected from the live video. This video segment can be edited out of the live video and published on an application to promote the lipstick.

[0137] The above method can automatically cut out target-type segments after accurately identifying them in a video, and then use these segments to promote the target objects described within them, thereby improving the promotion effect.

[0138] It should be noted that the second acquisition unit 701, the third processing unit 702, and the second determination unit 703 mentioned above correspond to steps S401 to S403 in Embodiment 1. The three units and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 3. It should also be noted that the above-mentioned unit modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0139] Example 7

[0140] Embodiments of the present invention can provide a computer terminal, which can be any computer terminal device in a group of computer terminals. Optionally, in this embodiment, the computer terminal can also be replaced by a mobile terminal or other terminal device.

[0141] Optionally, in this embodiment, the computer terminal may be located in at least one of a plurality of network devices in a computer network.

[0142] In this embodiment, the computer terminal described above can execute the program code for the following steps in the video analysis method of the application: acquiring the video to be analyzed; inputting the video into a feature processing model for processing to obtain the features of the global context relationship of the video and the features of the local context relationship of the video; and determining the target class segment in the video based on the features of the global context relationship of the video and the features of the local context relationship of the video.

[0143] The aforementioned computer terminal can also execute program code for the following steps in the video analysis method of the application: inputting the video into a feature processing model for processing to obtain the features of the global context relationship and the features of the local context relationship of the video, including: dividing the video into multiple video segments; encoding the multiple video segments to obtain a video feature matrix; inputting the video feature matrix into the feature processing model for processing to obtain the features of the global context relationship and the features of the local context relationship of the video.

[0144] The aforementioned computer terminal can also execute program code for the following steps in the video analysis method of the application: determining the target class segment in the video based on the features of the global context relationship of the video and the features of the local context relationship of the video includes: determining the probability values ​​of multiple candidate video segments as target class segments based on the features of the global context relationship of the video and the features of the local context relationship of the video; and filtering the target class segment from the multiple candidate video segments according to the probability values ​​of the multiple candidate video segments as target class segments.

[0145] The aforementioned computer terminal can also execute program code for the following steps in the video analysis method of the application: the method further includes: receiving a request instruction from the target object to obtain the target type segment; responding to the request instruction and pushing the target type segment to the target object; or, when it is detected that the target object is playing the video for the first time, pushing the target type segment to the target object.

[0146] The aforementioned computer terminal can also execute program code for the following steps in the video analysis method of the application: after pushing the target class segment to the target object, the method further includes: if it is detected that the target object has modified the target class segment, obtaining the modification result; and correcting the feature processing model based on the modification result to update the feature processing model.

[0147] The aforementioned computer terminal can also execute program code for the following steps in the video analysis method of the application: The feature processing model includes a global feature extraction module and a local feature extraction module. The video is input into the feature processing model for processing to obtain the global context relationship features and the local context relationship features of the video, including: inputting the video feature matrix into the global feature extraction module to extract global context relationship features; and inputting the video feature matrix into the local feature extraction module to extract local context relationship features.

[0148] The aforementioned computer terminal can also execute program code for the following steps in the video analysis method of the application: The feature processing model includes a global feature extraction module and a local feature extraction module. Inputting the video into the feature processing model for processing to obtain the global contextual relationship features and the local contextual relationship features of the video includes: reducing the dimensionality of the video feature matrix to obtain a dimensionality-reduced feature matrix; inputting the dimensionality-reduced feature matrix into the global feature extraction module for global contextual relationship feature extraction to obtain the global contextual relationship features of the video; and inputting the dimensionality-reduced feature matrix into the local feature extraction module for local contextual relationship feature extraction to obtain the local contextual relationship features of the video.

[0149] The aforementioned computer terminal can also execute program code for the following steps in the video analysis method of the application: inputting the dimensionality-reduced feature matrix into the global feature extraction module to extract global contextual features, and obtaining the global contextual features of the video includes: determining that the dimensionality-reduced feature matrix is ​​subjected to three different linear projections to obtain linear projection Q, linear projection K, and linear projection V; obtaining features carrying global contextual relationships through linear projection Q, linear projection K, and linear projection V; and inputting the features carrying global contextual relationships into a feedforward neural network for processing to obtain the global contextual features of the video.

[0150] The aforementioned computer terminal can also execute program code for the following steps in the video analysis method of the application: before the video is input into the feature processing model for processing, the method further includes: obtaining a video feature sample vector; inputting the video feature sample vector into the global feature extraction module for learning and training, while supervising the global feature extraction module in combination with a behavior classification loss function, so as to drive the global feature extraction module to capture features of global contextual relationships.

[0151] The aforementioned computer terminal can also execute program code for the following steps in the video analysis method of the application: The local feature extraction module includes two graph convolutional layers, wherein the two graph convolutional layers include: a first graph convolutional layer and a second graph convolutional layer. The local feature extraction module processes the dimensionality-reduced feature matrix to obtain the features of the local context relationship of the video, including: passing the dimensionality-reduced feature matrix through the first graph convolutional layer to obtain the features of the first local context relationship; inputting the features of the first local context relationship and the dimensionality-reduced feature matrix into the second graph convolutional layer, and obtaining the features of the local context relationship of the video through the output of the second graph convolutional layer.

[0152] The aforementioned computer terminal can also execute program code for the following steps in the video analysis method of the application: determining the probability values ​​of multiple candidate video segments as target class segments based on the features of the global context relationship of the video and the features of the local context relationship of the video includes: fusing the features of the global context relationship of the video and the features of the local context relationship of the video to obtain fused features; processing the fused features through a boundary point prediction module to obtain a first probability value of multiple candidate video segments as target class segments; processing the fused features through a confidence prediction module to obtain a second probability value of multiple candidate video segments as target class segments; and determining the probability value of multiple candidate video segments as target class segments based on the first probability value and the second probability value.

[0153] The aforementioned computer terminal can also execute program code for the following steps in the video analysis method of the application: receiving a service call request sent by a client, wherein the service call request carries the video to be analyzed; responding to the service call request, processing the video in a feature processing model on the server to obtain the features of the global context relationship of the video and the features of the local context relationship of the video; determining the target class segment in the video based on the features of the global context relationship of the video and the features of the local context relationship of the video; and outputting the target class segment.

[0154] The aforementioned computer terminal can also execute program code for the following steps in the video analysis method of the application: acquiring live video; inputting the live video into a feature processing model for processing to obtain the global context relationship features and the local context relationship features of the live video; and determining the target class segment in the live video based on the global context relationship features and the local context relationship features of the live video.

[0155] Furthermore, after determining the target class segment in the live video, the method further includes: editing the target class segment from the live video; and publishing the target class segment on the target application to promote the target object described in the target class segment.

[0156] Optionally, Figure 8 This is a structural block diagram of a computer terminal according to an embodiment of the present invention. Figure 8 As shown, the computer terminal may include: one or more ( Figure 8 (Only one is shown in the image) processor and memory.

[0157] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the video analysis method and apparatus in this embodiment of the invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned video analysis method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0158] The processor can access information and applications stored in the memory via a transmission device to perform the following steps: acquiring the video to be analyzed; inputting the video into a feature processing model for processing to obtain features of the global context relationship and features of the local context relationship of the video; and determining the target class segment in the video based on the features of the global context relationship and the features of the local context relationship of the video.

[0159] The processor can also invoke information and applications stored in the memory via a transmission device to perform the following steps: processing the video input feature processing model to obtain the global context relationship features and the local context relationship features of the video, including: dividing the video into multiple video segments; encoding the multiple video segments to obtain a video feature matrix; and inputting the video feature matrix into the feature processing model for processing to obtain the global context relationship features and the local context relationship features of the video.

[0160] The processor can also invoke information and applications stored in the memory via a transmission device to perform the following steps: determining a target class segment in the video based on the features of the global context relationship and the features of the local context relationship of the video includes: determining the probability values ​​of multiple candidate video segments in the video as target class segments based on the features of the global context relationship and the features of the local context relationship of the video; and selecting the target class segment from the multiple candidate video segments according to the probability values ​​of the multiple candidate video segments as target class segments.

[0161] The processor can also invoke information and applications stored in the memory via a transmission device to perform the following steps: the method further includes: receiving a request instruction from the target object to obtain the target type fragment; responding to the request instruction and pushing the target type fragment to the target object; or, when it is detected that the target object is playing the video for the first time, pushing the target type fragment to the target object.

[0162] The processor can also invoke information and applications stored in the memory via a transmission device to perform the following steps: after pushing the target class fragment to the target object, the method further includes: if it is detected that the target object has modified the target class fragment, obtaining the modification result; and correcting the feature processing model based on the modification result to update the feature processing model.

[0163] The processor can also invoke information and applications stored in the memory via a transmission device to perform the following steps: The feature processing model includes a global feature extraction module and a local feature extraction module. Processing the video into the feature processing model to obtain the global context relationship features and the local context relationship features of the video includes: inputting the video feature matrix into the global feature extraction module to extract global context relationship features; and inputting the video feature matrix into the local feature extraction module to extract local context relationship features.

[0164] The processor can also invoke information and applications stored in the memory via a transmission device to perform the following steps: The feature processing model includes a global feature extraction module and a local feature extraction module. Processing the video into the feature processing model to obtain the global contextual relationship features and the local contextual relationship features of the video includes: reducing the dimensionality of the video feature matrix to obtain a dimensionality-reduced feature matrix; inputting the dimensionality-reduced feature matrix into the global feature extraction module to extract global contextual relationship features to obtain the global contextual relationship features of the video; and inputting the dimensionality-reduced feature matrix into the local feature extraction module to extract local contextual relationship features to obtain the local contextual relationship features of the video.

[0165] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: inputting the dimensionality-reduced feature matrix into the global feature extraction module to extract global contextual features, and obtaining the global contextual features of the video includes: determining that the dimensionality-reduced feature matrix is ​​passed through three different linear projections to obtain linear projection Q, linear projection K, and linear projection V; obtaining features carrying global contextual relationships through linear projection Q, linear projection K, and linear projection V; inputting the features carrying global contextual relationships into a feedforward neural network for processing to obtain the global contextual features of the video.

[0166] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: before processing the video input feature processing model, the method further includes: obtaining a video feature sample vector; inputting the video feature sample vector into the global feature extraction module for learning and training, and supervising the global feature extraction module in combination with the behavior classification loss function to drive the global feature extraction module to capture features of global contextual relationships.

[0167] The processor can also invoke information and applications stored in the memory via a transmission device to perform the following steps: The local feature extraction module includes two graph convolutional layers, wherein the two graph convolutional layers include a first graph convolutional layer and a second graph convolutional layer. The local feature extraction module processes the dimensionality-reduced feature matrix to obtain the local contextual features of the video, including: passing the dimensionality-reduced feature matrix through the first graph convolutional layer to obtain the features of the first local contextual relationship; inputting the features of the first local contextual relationship and the dimensionality-reduced feature matrix into the second graph convolutional layer, and obtaining the features of the local contextual relationship of the video through the output of the second graph convolutional layer.

[0168] The processor can also invoke information and applications stored in the memory via a transmission device to perform the following steps: determining the probability values ​​of multiple candidate video segments as target class segments based on the features of the global context relationship of the video and the features of the local context relationship of the video includes: fusing the features of the global context relationship of the video and the features of the local context relationship of the video to obtain fused features; processing the fused features through a boundary point prediction module to obtain a first probability value of multiple candidate video segments as target class segments; processing the fused features through a confidence prediction module to obtain a second probability value of multiple candidate video segments as target class segments; and determining the probability values ​​of multiple candidate video segments as target class segments based on the first probability value and the second probability value.

[0169] The processor can also invoke information and applications stored in the memory via a transmission device to perform the following steps: receiving a service call request sent by a client, wherein the service call request carries a video to be analyzed; responding to the service call request, processing the video input feature processing model in the server to obtain the global context relationship features and the local context relationship features of the video; determining the target class segment in the video based on the global context relationship features and the local context relationship features of the video; and outputting the target class segment.

[0170] The processor can also invoke information and applications stored in the memory via a transmission device to perform the following steps: acquiring live video; inputting the live video into a feature processing model for processing to obtain features of the global context relationship of the live video and features of the local context relationship of the live video; and determining the target class segment in the live video based on the features of the global context relationship of the live video and the features of the local context relationship of the live video.

[0171] The processor can also invoke information and applications stored in the memory via a transmission device to perform the following steps: after determining the target class segment in the live video, the method further includes: editing the target class segment from the live video; and publishing the target class segment on the target application to promote the target object described in the target class segment.

[0172] This invention provides a video analysis scheme. It employs a method to determine target-type segments in a video based on both global and local contextual features. This involves acquiring the video to be analyzed, inputting the video into a feature processing model to obtain global and local contextual features, and then determining target-type segments based on these features. This strengthens feature representation and ensures the accuracy of the determined target-type segments, thereby improving the accuracy of target-type segment identification. This solves the technical problem of low accuracy in identifying user-interested video segments relying solely on local contextual features.

[0173] Those skilled in the art will understand that Figure 8The structure shown is for illustrative purposes only. The computer terminal can also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a mobile internet device (MID), a PAD, and other terminal devices. Figure 8 This does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include components that are more advanced than those described above. Figure 8 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 8 The different configurations shown.

[0174] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0175] Example 4

[0176] Embodiments of the present invention also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the video analysis method provided in Embodiment 1.

[0177] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0178] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: acquiring the video to be analyzed; inputting the video into a feature processing model for processing to obtain features of the global context relationship of the video and features of the local context relationship of the video; and determining the target class segment in the video based on the features of the global context relationship of the video and the features of the local context relationship of the video.

[0179] The storage medium is also configured to store program code for performing the following steps: inputting the video into a feature processing model for processing to obtain features of the global context relationship and features of the local context relationship of the video, including: dividing the video into multiple video segments; encoding the multiple video segments to obtain a video feature matrix; inputting the video feature matrix into the feature processing model for processing to obtain features of the global context relationship and features of the local context relationship of the video.

[0180] The storage medium is also configured to store program code for performing the following steps: determining a target class segment in the video based on the features of the global context relationship of the video and the features of the local context relationship of the video includes: determining the probability values ​​of multiple candidate video segments in the video as target class segments based on the features of the global context relationship of the video and the features of the local context relationship of the video; and selecting the target class segment from the multiple candidate video segments according to the probability values ​​of the multiple candidate video segments as target class segments.

[0181] The storage medium is also configured to store program code for performing the following steps: the method further includes: receiving a request instruction from a target object to obtain the target class fragment; responding to the request instruction to push the target class fragment to the target object; or, when it is detected that the target object is playing the video for the first time, pushing the target class fragment to the target object.

[0182] The storage medium is also configured to store program code for performing the following steps: after pushing the target class fragment to the target object, the method further includes: if it is detected that the target object has modified the target class fragment, obtaining the modification result; and correcting the feature processing model based on the modification result to update the feature processing model.

[0183] The storage medium is also configured to store program code for performing the following steps: The feature processing model includes a global feature extraction module and a local feature extraction module. Processing the video into the feature processing model to obtain global contextual features and local contextual features of the video includes: inputting the video feature matrix into the global feature extraction module to extract global contextual features; and inputting the video feature matrix into the local feature extraction module to extract local contextual features.

[0184] The storage medium is also configured to store program code for performing the following steps: The feature processing model includes a global feature extraction module and a local feature extraction module. The process of inputting the video into the feature processing model to obtain global contextual features and local contextual features of the video includes: reducing the dimensionality of the video feature matrix to obtain a dimensionality-reduced feature matrix; inputting the dimensionality-reduced feature matrix into the global feature extraction module to extract global contextual features; and inputting the dimensionality-reduced feature matrix into the local feature extraction module to extract local contextual features.

[0185] The storage medium is also configured to store program code for performing the following steps: inputting the dimensionality-reduced feature matrix into the global feature extraction module to extract global contextual features, thereby obtaining the global contextual features of the video, including: determining that the dimensionality-reduced feature matrix is ​​subjected to three different linear projections to obtain linear projection Q, linear projection K, and linear projection V; obtaining features carrying global contextual relationships through linear projection Q, linear projection K, and linear projection V; and inputting the features carrying global contextual relationships into a feedforward neural network for processing to obtain the global contextual features of the video.

[0186] The storage medium is also configured to store program code for performing the following steps: before the video is input into the feature processing model for processing, the method further includes: obtaining a video feature sample vector; inputting the video feature sample vector into the global feature extraction module for learning and training, while supervising the global feature extraction module in conjunction with a behavior classification loss function, so as to drive the global feature extraction module to capture features of global contextual relationships.

[0187] The storage medium is also configured to store program code for performing the following steps: the local feature extraction module includes two graph convolutional layers, wherein the two graph convolutional layers include a first graph convolutional layer and a second graph convolutional layer; the local feature extraction module processes the dimensionality-reduced feature matrix to obtain the local contextual features of the video, including: passing the dimensionality-reduced feature matrix through the first graph convolutional layer to obtain the features of the first local contextual relationship; and inputting the features of the first local contextual relationship and the dimensionality-reduced feature matrix into the second graph convolutional layer, and obtaining the features of the local contextual relationship of the video through the output of the second graph convolutional layer.

[0188] The storage medium is also configured to store program code for performing the following steps: determining the probability values ​​of multiple candidate video segments in the video as target class segments based on the features of the global context relationship of the video and the features of the local context relationship of the video includes: fusing the features of the global context relationship of the video and the features of the local context relationship of the video to obtain fused features; processing the fused features through a boundary point prediction module to obtain a first probability value of multiple candidate video segments in the video as the target class segment; processing the fused features through a confidence prediction module to obtain a second probability value of multiple candidate video segments in the video as the target class segment; and determining the probability value of multiple candidate video segments in the video as the target class segment based on the first probability value and the second probability value.

[0189] The storage medium is also configured to store program code for performing the following steps: receiving a service call request sent by a client, wherein the service call request carries a video to be analyzed; responding to the service call request, processing the video in a feature processing model on a server to obtain features of the global context relationship and features of the local context relationship of the video; determining a target class segment in the video based on the features of the global context relationship and the features of the local context relationship of the video; and outputting the target class segment.

[0190] The storage medium is also configured to store program code for performing the following steps: acquiring live video; inputting the live video into a feature processing model for processing to obtain features of the global context relationship of the live video and features of the local context relationship of the live video; and determining the target class segment in the live video based on the features of the global context relationship of the live video and the features of the local context relationship of the live video.

[0191] The storage medium is also configured to store program code for performing the following steps: after determining the target class segment in the live video, the method further includes: editing the target class segment from the live video; and publishing the target class segment on a target application to promote the target object described in the target class segment.

[0192] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0193] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0194] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0195] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0196] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0197] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0198] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A video analysis method, characterized in that, include: Obtain the video to be analyzed; The video input feature processing model is used to process the video to obtain the features of the global context relationship and the features of the local context relationship of the video. Based on the features of the global context relationship and the features of the local context relationship of the video, target class segments in the video are determined. The target class segments are selected from multiple candidate video segments as probability values. The probability value of any candidate video segment as a target class segment is determined by the product of the start point probability value, the end point probability value, the first confidence score, and the second confidence score corresponding to the candidate video segment. The start point probability value represents the probability that the candidate video segment is a start point, and the end point probability value represents the probability that the candidate video segment is an end point. The first confidence score and the second confidence score are obtained by different classifiers scoring the confidence of the candidate video segment. The start point probability value, the end point probability value, the first confidence score, and the second confidence score are determined by the features of the global context relationship and the features of the local context relationship of the candidate video segment. The multiple candidate video segments are obtained by dividing the video into segments of a preset length.

2. The method according to claim 1, characterized in that, The video input feature processing model is used to process the video to obtain the features of the global context relationship and the features of the local context relationship of the video, including: The video is divided into multiple video segments; The multiple video segments are encoded to obtain a video feature matrix; The video feature matrix is ​​input into the feature processing model for processing to obtain the global context relationship features and the local context relationship features of the video.

3. The method according to claim 1, characterized in that, Based on the features of the global context relationship and the features of the local context relationship of the video, the target class segments in the video are determined as follows: Based on the features of the global context relationship of the video and the features of the local context relationship of the video, the probability values ​​of multiple candidate video segments in the video as target class segments are determined. The target segment is selected from the multiple candidate video segments based on their probability values.

4. The method according to claim 3, characterized in that, The method further includes: Receive a request instruction from the target object to obtain the target class fragment; In response to the request instruction, the target class fragment is pushed to the target object; or, When it is detected that the target object is playing the video for the first time, the target segment is pushed to the target object.

5. The method according to claim 4, characterized in that, After pushing the target class fragment to the target object, the method further includes: If it is detected that the target object has modified the target class fragment, obtain the modification result; The feature processing model is then corrected based on the modification results to update the feature processing model.

6. The method according to claim 2, characterized in that, The feature processing model includes a global feature extraction module and a local feature extraction module. The video is input into the feature processing model for processing to obtain the global contextual relationship features and the local contextual relationship features of the video, including: The video feature matrix is ​​input into the global feature extraction module to extract global contextual relationship features, thereby obtaining the global contextual relationship features of the video. The video feature matrix is ​​input into the local feature extraction module to extract features of local contextual relationships, thereby obtaining the features of local contextual relationships of the video.

7. The method according to claim 2, characterized in that, The feature processing model includes a global feature extraction module and a local feature extraction module. The video is input into the feature processing model for processing to obtain the global contextual relationship features and the local contextual relationship features of the video, including: The video feature matrix is ​​reduced in dimensionality to obtain the reduced feature matrix. The reduced feature matrix is ​​input into the global feature extraction module to extract global contextual features, thereby obtaining the global contextual features of the video. The reduced feature matrix is ​​input into the local feature extraction module to extract local contextual features, thereby obtaining the local contextual features of the video.

8. The method according to claim 7, characterized in that, The reduced feature matrix is ​​input into the global feature extraction module to extract global contextual features, resulting in the following global contextual features of the video: The reduced feature matrix is ​​determined to pass through three different linear projections to obtain linear projection Q, linear projection K, and linear projection V. By using linear projection Q, linear projection K, and linear projection V, features carrying global contextual relationships are obtained; The features carrying global context relationships are input into a feedforward neural network for processing to obtain the global context relationship features of the video.

9. The method according to claim 7 or 8, characterized in that, Before processing the video input feature processing model, the method further includes: Obtain video feature sample vectors; The video feature sample vector is input into the global feature extraction module for learning and training. At the same time, the global feature extraction module is supervised by the behavior classification loss function to drive the global feature extraction module to capture features of global contextual relationships.

10. The method according to claim 7, characterized in that, The local feature extraction module includes two graph convolutional layers, namely a first graph convolutional layer and a second graph convolutional layer. The reduced-dimensional feature matrix is ​​input into the local feature extraction module for processing to obtain the local contextual features of the video, including: The reduced feature matrix is ​​passed through the first graph convolutional layer to obtain the features of the first local context relationship; The features of the first local context relationship and the reduced feature matrix are input into the second graph convolutional layer, and the features of the local context relationship of the video are obtained through the output of the second graph convolutional layer.

11. The method according to claim 3, characterized in that, Based on the features of the global context relationship and the features of the local context relationship of the video, the probability values ​​of multiple candidate video segments in the video as target class segments are determined as follows: The features of the global context relationship of the video and the features of the local context relationship of the video are fused to obtain the fused features; The fused features are processed by the boundary point prediction module to obtain multiple candidate video segments in the video as the first probability value of the target class segment; The fused features are processed by a confidence prediction module to obtain multiple candidate video segments in the video as the second probability value of the target class segment; Based on the first probability value and the second probability value, the probability values ​​of multiple candidate video segments in the video as target class segments are determined.

12. A video analysis method, characterized in that, include: Receive a service call request sent by a client, wherein the service call request carries a video to be analyzed; In response to the service call request, the video input feature processing model is processed in the server to obtain the global context relationship features and the local context relationship features of the video. Based on the features of the global context relationship and the features of the local context relationship of the video, target class segments in the video are determined. The target class segments are selected from multiple candidate video segments as probability values. The probability value of any candidate video segment as a target class segment is determined by the product of the start point probability value, the end point probability value, the first confidence score, and the second confidence score corresponding to the candidate video segment. The start point probability value represents the probability that the candidate video segment is a start point, and the end point probability value represents the probability that the candidate video segment is an end point. The first confidence score and the second confidence score are obtained by different classifiers scoring the confidence of the candidate video segment. The start point probability value, the end point probability value, the first confidence score, and the second confidence score are determined by the features of the global context relationship and the features of the local context relationship of the candidate video segment. The multiple candidate video segments are obtained by dividing the video into segments of a preset length. Output the target class fragment.

13. A video analysis method, characterized in that, include: Get live video; The live video is input into a feature processing model for processing to obtain the global contextual relationship features and the local contextual relationship features of the live video. Based on the features of the global context relationship and the features of the local context relationship of the live video, target class segments in the live video are determined. The target class segments are selected from multiple candidate video segments included in the live video, with each candidate video segment's probability value determined by the product of its start point probability value, end point probability value, first confidence score, and second confidence score. The start point probability value represents the probability that the candidate video segment is a start point, and the end point probability value represents the probability that the candidate video segment is an end point. The first confidence score and the second confidence score are obtained by different classifiers scoring the confidence of the candidate video segment. The start point probability value, the end point probability value, the first confidence score, and the second confidence score are determined by the features of the global context relationship and the features of the local context relationship of the candidate video segments. The multiple candidate video segments are obtained by dividing the live video into segments of a preset length.

14. The method according to claim 13, characterized in that, After determining the target segment in the live video, the method further includes: The target type segment is edited from the live video; Publish the target class fragment on the target application to promote the target object described in the target class fragment.

15. A video analysis device, characterized in that, include: The first acquisition unit is used to acquire the video to be analyzed. The first processing unit is used to process the video input feature processing model to obtain the features of the global context relationship of the video and the features of the local context relationship of the video. The first determining unit is configured to determine target class segments in the video based on the features of the global context relationship and the features of the local context relationship of the video. The target class segments are selected from multiple candidate video segments as probability values. The probability value of any candidate video segment as a target class segment is determined by the product of the start point probability value, the end point probability value, the first confidence score, and the second confidence score corresponding to the candidate video segment. The start point probability value represents the probability that the candidate video segment is a start point, and the end point probability value represents the probability that the candidate video segment is an end point. The first confidence score and the second confidence score are obtained by different classifiers scoring the confidence of the candidate video segment. The start point probability value, the end point probability value, the first confidence score, and the second confidence score are determined by the features of the global context relationship and the features of the local context relationship of the candidate video segment. The multiple candidate video segments are obtained by dividing the video into segments of a preset length.

16. A video analysis device, characterized in that, include: A receiving unit is configured to receive a service call request sent by a client, wherein the service call request carries a video to be analyzed. The second processing unit is used to respond to the service call request, process the video input feature processing model in the server to obtain the global context relationship features and the local context relationship features of the video; and determine the target class segment in the video based on the global context relationship features and the local context relationship features of the video. The output unit is used to output the target class segment, wherein the target class segment is obtained by filtering from multiple candidate video segments contained in the video as probability values ​​of the target class segment. The probability value of any candidate video segment as the target class segment is determined by the product of the start point probability value, the end point probability value, the first confidence score, and the second confidence score corresponding to the candidate video segment. The start point probability value is used to characterize the probability that the candidate video segment is a start point, and the end point probability value is used to characterize the probability that the candidate video segment is an end point. The first confidence score and the second confidence score are obtained by different classifiers scoring the confidence of the candidate video segment. The start point probability value, the end point probability value, the first confidence score, and the second confidence score are determined by the global context relationship features and the local context relationship features of the candidate video segment. The multiple candidate video segments are obtained by dividing the video into segments of a preset length.

17. A video analysis device, characterized in that, include: The second acquisition unit is used to acquire live video; The third processing unit is used to input the live video into the feature processing model for processing to obtain the global context relationship features and the local context relationship features of the live video. The second determining unit is used to determine target class segments in the live video based on the features of the global context relationship and the features of the local context relationship of the live video. The target class segments are selected from multiple candidate video segments as probability values. The probability value of any candidate video segment as the target class segment is determined by the product of the start point probability value, the end point probability value, the first confidence score, and the second confidence score corresponding to the candidate video segment. The start point probability value represents the probability that the candidate video segment is a start point, and the end point probability value represents the probability that the candidate video segment is an end point. The first confidence score and the second confidence score are obtained by different classifiers scoring the confidence of the candidate video segment. The start point probability value, the end point probability value, the first confidence score, and the second confidence score are determined by the features of the global context relationship and the features of the local context relationship of the candidate video segment. The multiple candidate video segments are obtained by dividing the video into segments of a preset length.

18. A storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the storage medium is located to perform the method described in any one of claims 1 to 14.

19. A processor, characterized in that, The processor is used to run a program, wherein the program, when running, performs the method according to any one of claims 1 to 14.

Images