Nomination acquisition method, network training method, device, storage medium and equipment

Through the acquisition method of timing action nomination and the training method of refined networks, the problem of inaccurate video clip positioning is solved, the precise positioning of the start and end positions of the video clips is achieved, and the accuracy and efficiency of video clip recognition is improved.

CN114445757BActive Publication Date: 2024-11-19TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210173989.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-24
Publication Date
2024-11-19
Estimated Expiration
2042-02-24

AI Technical Summary

Technical Problem

Existing video highlight clip recognition technology is difficult to accurately locate the start and end positions of video clips, and can only determine the rough position.

Method used

Through the acquisition method of timing action nomination and the training method of refined network, the initial nomination sequence and feature sequence are used for pooling, the refined network is input to obtain the nomination position deviation, and the initial nomination sequence is corrected through the nomination position deviation, and the start and Nomination for accurate timing action of end position.

Benefits of technology

The precise positioning of the start and end positions of the video clips is achieved, and the accuracy and efficiency of video highlight clip recognition is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114445757B_ABST
    Figure CN114445757B_ABST
Patent Text Reader

Abstract

The present application discloses a method for obtaining a temporal action nomination, a method for training a refined network, a device for obtaining a temporal action nomination, a training device for a refined network, a computer-readable storage medium, and a computer device, which can be applied to scenarios such as social network services and video analysis. The method for obtaining a temporal action nomination includes: collecting an initial nomination sequence and an initial feature sequence from a target video, the initial nomination sequence is used to characterize a video clip containing a target object, and the initial feature sequence is a collection of abstract information extracted from the initial nomination sequence; performing pooling processing on the initial feature sequence to obtain an interesting feature sequence, and the interesting feature sequence is used to characterize multiple initial features of the same dimension; inputting the interesting feature sequence into a refined network to obtain a nomination position deviation, and the refined network meets a preset supervision condition; and obtaining a temporal action nomination according to the nomination position deviation and the initial nomination sequence.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of machine learning, and specifically to a method for obtaining time-series action nominations, a method for training a refined network, an apparatus for obtaining time-series action nominations, a training apparatus for a refined network, a computer-readable storage medium, and a computer device. Background Art

[0002] In the current video highlight segment recognition project, it is necessary to accurately locate the position of the corresponding segment. The current segment recognition technology can often only determine the rough position of the corresponding segment in the video, and it is difficult to accurately locate the start and end positions of the segment. Summary of the invention

[0003] The embodiments of the present application provide a method for obtaining timed action nominations, a method for training a refined network, an apparatus for obtaining intelligent timed action nominations, an apparatus for training a refined network, a computer-readable storage medium, and a computer device, which can obtain timed action nominations with accurate start and end positions.

[0004] On the one hand, a method for obtaining temporal action nominations is provided, the method comprising: collecting an initial nomination sequence and an initial feature sequence from a target video, the initial nomination sequence being used to characterize a video clip containing a target object, the initial feature sequence being a collection of abstract information extracted based on the initial nomination sequence; performing pooling processing on the initial feature sequence to obtain an interesting feature sequence, the interesting feature sequence being used to characterize multiple initial features of the same dimension; inputting the interesting feature sequence into a refined network to obtain a nomination position deviation, the refined network satisfying a preset supervision condition; and obtaining temporal action nominations based on the nomination position deviation and the initial nomination sequence.

[0005] On the other hand, a training method for a refinement network is provided, the method comprising: collecting a sample nomination sequence and a sample feature sequence from a sample video, the sample nomination sequence being used to characterize a video clip containing a sample object, the sample feature sequence being a collection of abstract information extracted from the sample nomination sequence, the sample video containing a real nomination, the real nomination comprising a known real start position and a real end position; sampling the sample nomination sequence to obtain a sampling nomination sequence, the sampling nomination sequence comprising a first preset number of positive samples and a second preset number of negative samples; obtaining a sampling feature sequence corresponding to the sampling nomination sequence according to the sample feature sequence, the sampling feature sequence being a collection of abstract information of the sampling nomination sequence; performing pooling processing on the sampling feature sequence to obtain a training feature sequence, the training feature sequence being used to characterize multiple sampling features of the same dimension; inputting the training feature sequence into the refinement network to determine a loss value of a supervision loss function; and completing the training of the refinement network when the loss value satisfies a preset supervision condition.

[0006] On the other hand, a device for acquiring temporal action nominations is provided, the device comprising: a first acquisition module, used to acquire an initial nomination sequence and an initial feature sequence from a target video, the initial nomination sequence being used to characterize a video clip containing a target object, the initial feature sequence being a collection of abstract information extracted based on the initial nomination sequence; a first pooling module, used to perform pooling processing on the initial feature sequence to acquire an interesting feature sequence, the interesting feature sequence being used to characterize multiple initial features of the same dimension; a refinement module, used to input the interesting feature sequence into a refinement network to acquire a nomination position deviation, the refinement network satisfying a preset supervision condition; and an acquisition module, used to acquire temporal action nominations based on the nomination position deviation and the initial nomination sequence.

[0007] On the other hand, a training device for a refined network is provided, characterized in that the device includes: a second acquisition module, used to collect a sample nomination sequence and a sample feature sequence from a sample video, the sample nomination sequence is used to characterize a video clip containing a sample object, and the sample feature sequence is a collection of abstract information extracted from the sample nomination sequence; a sampling module, used to sample the sample nomination sequence to obtain a sampling nomination sequence, the sampling nomination sequence includes a first preset number of positive samples and a second preset number of negative samples; a feature extraction module, used to obtain a sampling feature sequence corresponding to the sampling nomination sequence according to the sample feature sequence, and the sampling feature sequence is a collection of abstract information of the sampling nomination sequence; a second pooling module, used to perform pooling processing on the sampling feature sequence to obtain a training feature sequence, and the training feature sequence is used to characterize multiple sampling features of the same dimension; a supervision module, used to input the training feature sequence into the refined network to determine the loss value of the supervision loss function; and a termination module, used to complete the training of the refined network when the loss value meets a preset supervision condition.

[0008] On the other hand, a computer-readable storage medium is provided, which stores a computer program, and the computer program is suitable for being loaded by a processor to execute the steps in the method for obtaining timed action nominations as described in any of the above embodiments or the method for training a refined network as described in any of the above embodiments.

[0009] On the other hand, a computer device is provided, comprising a processor and a memory, wherein a computer program is stored in the memory, and wherein the processor calls the computer program stored in the memory to execute the steps in the method for obtaining nominations for sequential actions as described in any of the above embodiments or the method for training a refined network as described in any of the above embodiments.

[0010] On the other hand, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the steps in the method for obtaining temporal action nominations as described in any of the above embodiments or the method for training a refined network as described in any of the above embodiments.

[0011] The embodiment of the present application obtains the nomination position deviation through the refinement network, and corrects the start and end positions of the initial nomination sequence extracted from the target video through the nomination position deviation to obtain the timing action nomination with accurate start and end positions. The sample nominations extracted from the sample video are sampled to use the sample nominations with high richness and balance for training; the multi-scale features of the time dimension are extracted from the sample features to enrich the diversity of features in the training samples; the parameters of the refinement network are optimized through the supervised loss function to further improve the accuracy of the nomination position deviation generated by the refinement network. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1 A schematic diagram of an application scenario of a method for obtaining a sequential action nomination provided in an application embodiment.

[0014] Figure 2 A flowchart of a method for obtaining a time sequence action nomination provided in an embodiment of the present application.

[0015] Figure 3 A schematic diagram of an application scenario of the method for obtaining time-sequence action nominations provided in an embodiment of the present application.

[0016] Figure 4 A flowchart of a method for obtaining a time sequence action nomination provided in an embodiment of the present application.

[0017] Figure 5 A flowchart of a method for obtaining a time sequence action nomination provided in an embodiment of the present application.

[0018] Figure 6 A flowchart of a method for training a refined network provided in an embodiment of the present application.

[0019] Figure 7 A flowchart of a method for training a refined network provided in an embodiment of the present application.

[0020] Figure 8 A flowchart of a method for training a refined network provided in an embodiment of the present application.

[0021] Fig. 9 A schematic diagram of positive samples and negative samples provided in an embodiment of the present application.

[0022] Fig.10A schematic diagram of a sampling scenario for a refined network training method provided in an embodiment of the present application.

[0023] Fig.11 A flowchart of a method for training a refined network provided in an embodiment of the present application.

[0024] Fig.12 A schematic diagram of a feature fusion scenario of a refined network training method provided in an embodiment of the present application.

[0025] Fig.13 A schematic diagram of an application scenario of the method for obtaining time-sequence action nominations provided in an embodiment of the present application.

[0026] Fig.14 A flowchart of a method for training a refined network provided in an embodiment of the present application.

[0027] Fig.15 A flowchart of a method for training a refined network provided in an embodiment of the present application.

[0028] Fig.16 A schematic diagram of the structure of a device for obtaining sequential action nominations provided in an embodiment of the present application.

[0029] Fig.17 A schematic diagram of the structure of a training device for a refined network provided in an embodiment of the present application.

[0030] Fig.18 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0032] The embodiments of the present application provide a method for obtaining nominations for sequential actions, a model training method, an apparatus, a computer device, and a storage medium. Specifically, the method for obtaining nominations for sequential actions and the model training method of the embodiments of the present application can be executed by a computer device, wherein the computer device can be a terminal or a server or other device. The terminal can be a smart phone, a tablet computer, a laptop computer, a smart TV, a smart speaker, a wearable smart device, a smart car terminal or other devices. The terminal can also include a client, which can be a video client, a browser client, or an instant messaging client, etc. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms.

[0033] The embodiments of the present application can be applied to various scenarios such as instant messaging (such as WeChat, QQ, robot chat, etc.), natural language processing, machine learning, deep learning, cloud technology, artificial intelligence, smart transportation, etc.

[0034] First, some nouns or terms that appear in the description of the embodiments of the present application are explained as follows:

[0035] Artificial Intelligence (AI): It is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0036] Machine Learning (ML): It is a multi-disciplinary interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0037] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0038] Deep Learning (DL): It is a branch of machine learning and an algorithm that attempts to use multiple processing layers containing complex structures or composed of multiple nonlinear transformations to perform high-level abstraction of data. Deep learning is to learn the inherent laws and representation levels of training sample data. The information obtained in the learning process is of great help in the interpretation of data such as text, images and sounds. The ultimate goal of deep learning is to enable machines to have analytical learning capabilities like humans and to be able to recognize data such as text, images and sounds. Deep learning is a complex machine learning algorithm that has achieved results in speech and image recognition that far exceed previous related technologies.

[0039] Smart transportation fully utilizes new generation information technologies such as the Internet of Things, spatial perception, cloud computing, and mobile Internet in the entire transportation field, and comprehensively applies theories and tools such as transportation science, system methods, artificial intelligence, and knowledge mining. It aims to achieve comprehensive perception, deep integration, proactive service, and scientific decision-making. By building a real-time dynamic information service system, it deeply mines transportation-related data, forms a problem analysis model, and achieves improvements in the industry's resource allocation optimization capabilities, public decision-making capabilities, industry management capabilities, and public service capabilities. It promotes safer, more efficient, more convenient, more economical, more environmentally friendly, and more comfortable operation and development of transportation, and drives the transformation and upgrading of transportation-related industries.

[0040] Cloud technology: refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or local area network to achieve data computing, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool that can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of technical network systems require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, in the future, each item may have its own identification mark, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data require strong system backing support, which can only be achieved through cloud computing.

[0041] Blockchain system: It can be a distributed system formed by connecting clients and multiple nodes (any form of computing devices connected to the network, such as servers and user terminals) through network communication. The nodes form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on the Transmission Control Protocol (TCP). In a distributed system, any machine such as a server or terminal can join and become a node. The node includes the hardware layer, the middle layer, the operating system layer and the application layer.

[0042] Social Networking Services (SNS): includes social software and social networking sites, and also refers to information carriers where social networking has matured and become popular, such as SMS services. A social networking service is a platform that establishes social networks or social relationship connections between people. For example, shared interests, activities, backgrounds, or real-life connections. A social networking service includes social connections that represent each user (usually a profile) and various additional services. Most social networking services are online community services based on the web, and provide users with means of interaction on the Internet, such as email and instant messaging. Sometimes considered a social networking service, but in a broader sense, social networking services usually refer to individual-centered services and are centered on online community service groups. Social networking sites allow users to share their ideas, pictures, articles, activities, and events in their networks.

[0043] In the current video highlight segment recognition project, it is necessary to accurately locate the position of the corresponding segment. The current segment recognition technology can often only determine the rough position of the corresponding segment in the video, and it is difficult to accurately locate the starting position and the ending position of the segment. In order to accurately locate the starting position and the ending position of the segment, the present invention proposes a method for obtaining a temporal action nomination, which obtains the nomination position deviation through a refined network to obtain the temporal action nomination according to the nomination position deviation and the initial nomination sequence. Among them, the initial nomination sequence corresponds to a video segment with a known rough starting position and an ending position, and the temporal action nomination finally obtained is a video segment with a refined starting position and an ending position, so as to accurately locate the starting position and the ending position of the segment. The present invention also proposes a training method for a refined network, which supervises the refined network through a supervised loss function to optimize the refined network to obtain an accurate nomination position deviation.

[0044] Among them, the embodiments of the present application can be implemented in combination with cloud technology or blockchain network technology. For example, the method for obtaining the nomination of sequential actions and the method for training the refined network disclosed in the embodiments of the present application, wherein these data can be stored on the blockchain. For example, the trained refined network, the device for obtaining the nomination of sequential actions, and the training device for refined bowls and baskets can all be stored on the blockchain.

[0045] In order to facilitate the storage and query of the trained refined network, optionally, the method for obtaining the nomination of the sequential action also includes: sending the trained refined network to the blockchain network so that the nodes of the blockchain network fill the trained refined network into the new block, and when the consensus is reached on the new block, the new block is appended to the end of the blockchain. The embodiment of the present application can store the trained refined network on the chain to realize the backup of the record. When it is necessary to use the trained refined network to obtain the nominated position deviation, for example, when a wonderful clip is identified for a network video, the corresponding trained refined network can be directly and quickly obtained from the blockchain to obtain the starting position and ending position of the wonderful clip using the refined network, without the need for the training device of the refined network to perform a series of processing before the refined network can be obtained, thereby improving the efficiency of video analysis processing; similarly, when it is necessary to use the trained refined network to obtain the nominated position deviation, the corresponding trained refined network can be directly and quickly obtained from the blockchain, without the need for the training device of the refined network to perform a series of processing before the nominated position deviation can be obtained, thereby improving the efficiency of video analysis processing.

[0046] See also Figure 1 , Figure 1Schematic diagram of application scenarios of the method for obtaining the nomination of sequential actions provided in the embodiment of the present application. Take the method for obtaining the nomination of sequential actions executed by a computer device as an example, wherein the computer device can be a terminal or a server or other device. During the training phase of the refined network, the computer device obtains training sample data, and each training sample in the training sample data includes: collecting a sample nomination sequence and a sample feature sequence from a sample video, wherein the sample nomination sequence is used to characterize a video clip containing a sample object, and the sample feature sequence is a collection of abstract information extracted according to the sample nomination sequence; sampling the sample nomination sequence to obtain a sampling nomination sequence, wherein the sampling nomination sequence includes a first preset number of positive samples and a second preset number of negative samples; obtaining a sampling feature sequence corresponding to the sampling nomination sequence according to the sample feature sequence, wherein the sampling feature sequence is a collection of abstract information of the sampling nomination sequence; performing pooling processing on the sampling feature sequence to obtain a training feature sequence, wherein the training feature sequence is used to characterize multiple sampling features of the same dimension; inputting the training feature sequence into the refinement to determine the loss value of the supervision loss function; and completing the training of the refined network when the loss value satisfies the preset supervision condition. During the acquisition phase of temporal action nominations, a user can upload a target video through a client, a browser client or an instant messaging client installed in a computer device, and after the computer device acquires the uploaded target video; an initial nomination sequence and an initial feature sequence are collected from the target video, wherein the initial nomination sequence is used to characterize a video clip containing a target object, and the initial feature sequence is a collection of abstract information extracted according to the initial nomination sequence; pooling is performed on the initial feature sequence to obtain an interesting feature sequence, wherein the interesting feature sequence is used to characterize multiple initial features of the same dimension; the interesting feature sequence is input into a refined network to obtain a nomination position deviation, wherein the refined network satisfies a preset supervision condition; and temporal action nominations are acquired according to the nomination position deviation and the initial nomination sequence.

[0047] It should be noted that the process of obtaining the temporal action nomination can be completed in the server or in the terminal. When the method for obtaining the temporal action nomination and the training method for the refined network are both completed in the server middle end, when the trained refined network needs to be used, the target video or the feature sequence of interest can be input into the server. After the server generates the corresponding temporal action nomination, the obtained temporal action nomination is sent to the terminal for display. When the method for obtaining the temporal action nomination and the training method for the refined network are both completed in the terminal, when the trained refined network needs to be used, the target video or the feature sequence of interest can be input into the terminal. After the terminal generates the corresponding temporal action nomination, the terminal displays the generated temporal action nomination. When the training method for the refined network is completed in the server and the method for obtaining the temporal action nomination is completed in the terminal, when the trained refined network needs to be used, the target video or the feature sequence of interest can be input into the terminal. After the terminal generates the corresponding temporal action nomination, the terminal displays the temporal action nomination. Optionally, the trained model file (model file) in the server can be transplanted to the terminal. If it is necessary to obtain a temporal action nomination, the target video or the feature sequence of interest is input into the trained model file (model file), and the corresponding temporal action nomination can be obtained by calculation.

[0048] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.

[0049] An embodiment of the present application provides a method for obtaining a time sequence action nomination, which can be executed by a terminal or a server, or by both the terminal and the server. The embodiment of the present application takes the method for obtaining a time sequence action nomination executed by a server as an example for explanation.

[0050] See also Figure 2 , Figure 2 A flowchart of a method for obtaining a time sequence action nomination provided in an embodiment of the present application, the method comprising:

[0051] Step 210: Collect an initial nominated sequence and an initial feature sequence from the target video. The initial nominated sequence is used to characterize the video clip containing the target object, and the initial feature sequence is a collection of abstract information extracted from the initial nominated sequence.

[0052] Optionally, the target video is a video containing a target object. In one embodiment, the target video is a video of a basketball game, and the target object is a specific action, such as a "shooting" action, then a wonderful video clip containing the "shooting" action can be extracted based on the target video. Correspondingly, the initial nomination is a video clip containing the "shooting" action. Since there may be multiple "shooting" actions in a video of a basketball game, the number of initial nominations may be multiple, and multiple initial nominations constitute an initial nomination sequence. The initial feature sequence is a sequence composed of features extracted from a convolutional neural network. In one embodiment, the features obtained after abstract processing of the spatial information of the environment, objects, etc. and the dynamic features of the action in the initial nomination sequence using a convolutional neural network are the initial features corresponding to the initial nominations, and the sequence composed of the initial features corresponding to each initial nomination is the initial feature sequence.

[0053] Step 220: pooling the initial feature sequence to obtain a feature sequence of interest, where the feature sequence of interest is used to characterize multiple initial features of the same dimension.

[0054] Optionally, pooling is a concept in convolutional neural networks, which is a downsampling processing method. Pooling can adjust multiple initial features of different dimensions to the same dimension, so as to input features of the same dimension to the refinement network in subsequent processing, thereby reducing the complexity of the refinement network in processing data. In one embodiment, the dimension is 2D, and in other embodiments, the dimension can also be 3D, 4D, 5D, etc., which is not limited here.

[0055] Step 230: Input the feature sequence of interest into the refinement network to obtain the nominated position deviation, and the refinement network satisfies the preset supervision condition.

[0056] Optionally, the nominated position deviation is the deviation between the start position of the initial nominated predicted by the refinement network and the start position of the corresponding target segment in the target video, and the deviation between the end position of the initial nominated and the end position of the corresponding target segment in the target video.

[0057] See also Figure 3 For example, the target object is the "shooting" action, and the starting position of a "shooting" action segment in the target video is 3 minutes and 16 seconds into the target video, and the ending position is 3 minutes and 27 seconds into the target video. Among the initial nominations extracted for this target video, the initial nomination closest to the position of this "shooting" action segment has a starting position of 3 minutes and 14 seconds into the target video, and an ending position of 3 minutes and 29 seconds into the target video, which is two seconds later than the ending position of the corresponding "shooting" segment. The position deviation includes the starting position deviation and the ending position deviation. Figure 3In the illustrated embodiment, the start position deviation is "-2s", indicating that the start position of the initial nomination is 2 seconds earlier than the start position of the corresponding target segment; the end position deviation is "+2s", indicating that the end position of the initial nomination is 2 seconds later than the end position of the corresponding target segment.

[0058] Step 240: Obtaining a time sequence action nomination according to the nomination position deviation and the initial nomination sequence.

[0059] Optional, see Figure 3 ,exist Figure 3 In the illustrated embodiment, based on the initial nomination with the starting position being 3 minutes and 14 seconds and the ending position being 3 minutes and 29 seconds, the starting position of the initial nomination is delayed by 2 seconds according to the starting position deviation "-2s", and the ending position of the initial nomination is advanced by 2 seconds according to the ending position deviation "+2s", so that a timed action nomination with accurate starting and ending positions can be obtained, the starting position of the timed action nomination is consistent with the starting position of the corresponding target segment, and the ending position of the timed action nomination is consistent with the ending position of the corresponding target segment.

[0060] The more accurate the nominated position deviation obtained by the refined network is, the more accurate the start and end positions of the sequential action nominated according to the nominated position deviation and the initial nominated sequence are. The embodiment of the present application optimizes the refined network through preset supervision conditions to improve the accuracy of the nominated position deviation output by the refined network.

[0061] Optionally, by selecting different target objects, the method for obtaining the nomination of sequential actions can be adapted to different application scenarios. For example, if a specific wonderful action such as "shooting" or "shooting" is set as the target object, the nomination of the sequential action extracted from the target video is a segment where a specific wonderful action occurs. In the scene of playing sports events, these nominations of sequential actions can be played back as wonderful shots, reducing the waste of manpower in manually editing videos. For another example, in the application scenario of reviewing videos, the target scene in the video can be set as the target object, and the nomination of the sequential action extracted from the target video is a segment of the target scene, so that the segment can be directly deleted or processed by adding mosaics, etc. In this way, the processing of the target scene segment can be completed without the auditor watching the target scene segment, so as to protect the auditor's mental health. For another example, when looking for a specific person or object in the target video, the specific person or object can be set as the target object, so as to accurately locate the position where the specific person or object appears or disappears in the target video through the nomination of sequential actions, so that the viewer can quickly find the specific person or object.

[0062] Optional, see Figure 4, step 210: collecting an initial nomination sequence and an initial feature sequence from the target video, including:

[0063] Step 410: Collect a plurality of initial segments from the target video.

[0064] Optionally, the multiple initial segments are multiple video segments formed by randomly dividing the target video.

[0065] Step 420: Input the multiple initial segments into a feature extraction network to obtain multiple encoding features.

[0066] Optionally, the feature extraction network includes a dual-stream network consisting of a temporal information network and a spatial information network. The temporal information network is used to extract dynamic features of actions in the initial segment; the spatial information network is used to extract spatial information features of the initial segment including the environment, objects in the video, etc. Finally, the feature extraction network fuses the dynamic features and spatial information features to generate multiple encoding features, each encoding feature corresponding to an initial object, representing the abstracted action and spatial information of the corresponding initial object.

[0067] Step 430: Obtain an initial feature sequence based on multiple coding features.

[0068] Optionally, multiple coding features are input into a neural network including two layers of temporal convolutional layers to fix the temporal length of the multiple coding features and generate an initial feature sequence. The generated initial feature sequence has a fixed temporal length to facilitate the calculation of the boundary probability sequence and the confidence score matrix in subsequent processing.

[0069] Step 440: Obtain a boundary probability sequence and a confidence score matrix based on the initial feature sequence.

[0070] Optionally, the boundary probability sequence can be generated using the TEM (Temporal Evaluation Module, TEM) module. The TEM module uses a three-layer temporal convolutional layer to perform probability evaluation on each position in the input initial feature sequence, including the probability evaluation of the start of the action, the probability evaluation of the end of the action, and the probability evaluation of the ongoing action, and generates a boundary probability sequence. The boundary probability sequence is used to characterize the probability that each temporal position in the initial feature sequence is the start and end of the action. Any position representing the start of the action and a position representing the end of the action and the time sequence between the two can constitute a candidate nomination. Therefore, multiple candidate nominations can be obtained based on the boundary probability sequence.

[0071] Optionally, the confidence score matrix can be generated using a Proposal Evaluation Module (PEM). The PEM module is a multi-layer perceptron model with one hidden layer, which is used to perform confidence evaluation on each candidate nomination in the initial feature sequence to generate a confidence score matrix. The confidence score matrix of the PEM module is used to characterize the confidence of the candidate nomination. The higher the confidence, the higher the probability that the candidate nomination is determined as the initial nomination.

[0072] Step 450: Obtain an initial nomination sequence according to the boundary probability sequence and the confidence score matrix.

[0073] Optionally, multiple candidate nominations can be obtained according to the boundary probability sequence, the confidence of each candidate nomination can be obtained according to the confidence score matrix, and multiple initial nominations can be determined from the multiple candidate nominations according to the confidence of each candidate nomination to generate an initial nomination sequence.

[0074] Optional, see Figure 5 Step 240: Obtaining a time sequence action nomination according to the nomination position deviation and the initial nomination sequence, including:

[0075] Step 510: correcting the initial start position and the initial end position of each initial nomination in the initial nomination sequence according to the nomination position deviation, so as to obtain a refined start position and a refined end position corresponding to each initial nomination.

[0076] Optional, please combine Figure 3 , the initial nomination is the predicted video segment containing the target object. There may be deviations between the start and end positions of the initial nomination and the actual start and end positions of the video segment containing the target object, such as Figure 3 In the illustrated embodiment, the initial nominated start position is 2 seconds earlier than the corresponding target segment start position, and the initial nominated end position is 2 seconds later than the corresponding target segment end position. The nominated position deviation obtained according to the refined network is the deviation between the initial nominated start and end positions predicted by the refined network and the actual start and end positions of the video segment containing the target object. The more accurate the nominated position deviation is, the more accurate the refined start and refined end positions obtained after correcting the initial nominated start and end positions using the nominated position deviation are, and the closer they are to the corresponding actual start and end positions of the video segment containing the target object.

[0077] Step 520: Generate a time sequence action nomination according to the initial nomination and the refined start position and refined end position corresponding to the initial nomination.

[0078] Optional, please combine Figure 3, the starting position of the initial nomination is corrected to the corresponding refined starting position, and the ending position of the initial nomination is corrected to the corresponding refined ending position, so as to generate a temporal action nomination.

[0079] See also Figure 6 The embodiment of the present application also provides a training method for a refined network, which is used to optimize the refined network and improve the accuracy of the nominated position deviation output by the refined network. The training method for the refined network includes:

[0080] Step 610: Collect a sample nomination sequence and a sample feature sequence from the sample video. The sample nomination sequence is used to characterize the video clip containing the sample object. The sample feature sequence is a collection of abstract information extracted based on the sample nomination sequence. The sample video contains real nominations, and the real nominations include a known real start position and a real end position.

[0081] Optional, please combine Figure 2 The method of collecting sample nomination sequence and sample feature sequence from sample video is similar to the method of step 210, except that the sample video is a video with the real start and end positions of the real nomination (i.e., the video clip containing the target object) marked, and the target video is an arbitrary video; the sample nomination sequence and the sample feature sequence correspond to the sample video, and the initial nomination sequence and the initial feature sequence correspond to the initial video. The method of collecting sample nomination sequence is similar to the method of collecting initial nomination sequence, and the method of collecting sample feature sequence is similar to the method of collecting initial feature sequence.

[0082] Optional, see Figure 7 , step 610: collecting a sample nomination sequence and a sample feature sequence from a sample video, including:

[0083] Step 710: Collect multiple sample segments from a sample video.

[0084] Optional, please combine Figure 4 The method of step 710 is similar to the method of step 410, except that the initial segment is a segment extracted from the initial video, and the sample segment is a segment extracted from the sample video.

[0085] Step 720: Input the plurality of sample segments into a feature extraction network to obtain a plurality of first features.

[0086] Optional, please combine Figure 4 The method of step 720 is similar to the method of step 420, except that the first feature is a feature corresponding to the sample segment, and the encoding feature is a feature corresponding to the initial segment.

[0087] Optionally, the feature extraction network in step 720 includes a dual-stream network consisting of a temporal information network and a spatial information network, which is consistent with the feature extraction network in step 420 and will not be repeated here.

[0088] Step 730: Obtain a sample feature sequence according to the multiple first features.

[0089] Optional, please combine Figure 4 The method of step 730 is similar to the method of step 430, except that the sample feature sequence corresponds to the first feature and the initial feature sequence corresponds to the coding feature.

[0090] Step 740: Obtain a sample probability sequence and a sample score matrix according to the sample feature sequence.

[0091] Optional, please combine Figure 4 The method of step 740 is similar to the method of step 440. The boundary probability sequence in step 740 can be generated using the TEM module, and the confidence score matrix in step 740 can be generated using the PEM module, which will not be repeated here.

[0092] Step 750: Obtain a sample nomination sequence according to the sample probability sequence and the sample score matrix.

[0093] Optional, please combine Figure 4 The method of step 750 is similar to the method of step 450, the difference is that the initial nomination sequence in step 450 is obtained according to the sample probability sequence and sample score matrix corresponding to the initial feature sequence, while the sample nomination sequence in step 750 is obtained according to the sample probability sequence and sample score matrix corresponding to the sample probability sequence, which will not be repeated here.

[0094] See also Figure 6 , the training method of the refined network also includes:

[0095] Step 620: Sampling the sample nomination sequence to obtain a sample nomination sequence, where the sample nomination sequence includes a first preset number of positive samples and a second preset number of negative samples.

[0096] Please combine Figure 8 , Optionally, step 620: sampling the sample nomination sequence to obtain the sample nomination sequence, including:

[0097] Step 810: Obtain the first intersection-over-union ratio between each sample nomination in the sample nomination sequence and the true nomination.

[0098] Optionally, the first intersection-over-union ratio between the sample nomination and the real nomination can represent the proportion of the time sequence overlap between the sample nomination and the real nomination, so as to judge the closeness between the sample nomination obtained in step 610 and the real nomination. The larger the value of the first intersection-over-union ratio, the higher the proportion of the time sequence overlap between the sample nomination and the real nomination, and the closer the time sequence of the sample nomination is to the time sequence of the real nomination; the smaller the value of the first intersection-over-union ratio, the lower the proportion of the time sequence overlap between the sample nomination and the real nomination, and the greater the deviation between the time sequence of the sample nomination and the time sequence of the real nomination.

[0099] Step 820: Among multiple first intersection-and-union ratios, samples corresponding to the first intersection-and-union ratio greater than the first preset ratio are nominated as positive samples, and samples corresponding to the first intersection-and-union ratio less than the second preset ratio are nominated as negative samples, so as to obtain a first preset number of positive samples and a second preset number of negative samples.

[0100] Optionally, the value range of the intersection-and-union ratio is [0, 1], the first preset ratio is set to 0.7, and the second preset ratio is 0.3. That is, when the value of a first intersection-and-union ratio is greater than 0.7, the sample nomination corresponding to the first intersection-and-union ratio is determined as a positive sample; when the value of a first intersection-and-union ratio is less than 0.3, the sample nomination corresponding to the first intersection-and-union ratio is determined as a negative sample; when the value of a first intersection-and-union ratio is greater than or equal to 0.3 and less than or equal to 0.7, the sample nomination corresponding to the first intersection-and-union ratio is discarded. In other embodiments, under the condition that the first preset ratio is greater than the second preset ratio, the first preset ratio can also be set to other values, for example, the first preset ratio is set to 0.6, 0.5, 0.8, 0.9, etc., which are not listed here one by one; the second preset ratio can also be set to other values, for example, the second preset ratio is set to 0.1, 0.2, 0.4, etc., which are not listed here one by one.

[0101] Step 830: Generate a sampling nomination sequence according to a first preset number of positive samples and a second preset number of negative samples.

[0102] Optionally, positive samples and negative samples are sampled from the sample nomination sequence to generate a sample nomination sequence, so that the sample nomination sequence has high richness and balance, and representative samples of different types are sampled. Among them, positive samples are sample nominations whose timing is obviously close to the timing of the real nomination, and negative samples are sample nominations whose timing is obviously not close to the timing of the real nomination. Positive samples and negative samples are representative. The discarded sample nominations that are neither positive samples nor negative samples have timings that are neither obviously close to the timing of the real nomination nor obviously not close to the timing of the real nomination, and are not representative, so they are discarded in the sampling process.

[0103] Optionally, the first preset number is 10 times the second preset number, that is, the sampling process retains the majority of positive samples to generate the sampling nomination sequence. Fig. 9 ,Optionally, in some embodiments, the number of sample nominations is consistent with the number of real nominations, and each real nomination corresponds to a unique sample nomination. In some embodiments, for example Fig. 9 In the illustrated implementation, each real nomination corresponds to one or more sample nominations. Fig. 9 In the illustrated embodiment, sample nominations Sy1, Sy2, and Sy3 all correspond to the real nomination Sz1, wherein Sy1 is a positive sample, Sy2 is a negative sample, and Sy3 is neither a positive sample nor a negative sample. When determining the start and end positions of the real nomination Sz1 based on the sample nominations and the nomination position deviations of the sample nominations, we would most like to determine it based on the positive sample Sy1 in combination with the nomination position deviation of the positive sample Sy1, because the timing of the positive sample Sy1 is closest to the timing of the real nomination Sz1, the nomination position deviation of the positive sample Sy1 is small, and it is easier to be accurately obtained; we would least like to determine it based on the negative sample Sy2 in combination with the nomination position deviation of the negative sample Sy2, because the timing of the negative sample Sy2 is least close to the timing of the real nomination Sz1, the nomination position deviation of the negative sample Sy2 is large, and it is not easy to be accurately obtained. The sampling nomination sequence with positive and negative samples enables the refined network to have the ability to distinguish positive and negative samples. The number of positive samples in the sampling nomination sequence accounts for the majority, so as to mainly optimize the ability of the refined network to obtain the corresponding nomination position deviation according to the positive samples, and improve the operating efficiency of the refined network. Optionally, in other embodiments, the first preset number is not limited to 10 times the second preset number, and can also be any value greater than the second preset number, for example, the first preset number is 5 times, 3 times, 15 times, 20 times, etc., of the second preset number, which are not listed here one by one.

[0104] See also Figure 6 , the training method of the refined network also includes:

[0105] Step 630: Obtain a sampling feature sequence corresponding to the sampling nomination sequence according to the sample feature sequence. The sampling feature sequence is a collection of abstract information of the sampling nomination sequence.

[0106] Please combine Fig.10 Optionally, the sampling nomination sequence is a sequence of a part of sample nominations obtained after sampling the sample nominations in the sample nomination sequence, and each sample nomination has a corresponding sample feature in the sample feature sequence. After this part of the sample nominations is sampled to become sampling nominations, they still have a corresponding relationship with the sample features in the sample feature sequence. For example, Fig.10In the illustrated embodiment, the sample nomination Sy1 corresponds to the sample feature Fy1, and after the sample nomination Sy1 is sampled to become the sample nomination Sc1, the sample nomination Sc1 still corresponds to the sample feature Fy1. The function of step 630 is to find the sample feature corresponding to each sample nomination in the sampled sample nomination sequence, and generate a sample feature sequence based on these sample features.

[0107] Please combine Fig.11 Optionally, obtaining a sampling feature sequence corresponding to the sampling nomination sequence according to the sample feature sequence includes:

[0108] Step 1110: Input the sample feature sequence into at least two layers of time series convolution for processing to obtain a fused feature sequence, which represents the sample feature sequence after abstraction on a time scale.

[0109] Optionally, the purpose of obtaining a fused feature sequence is to extract multi-scale features of sample features in the time dimension to enrich the diversity of features. Different sample features may correspond to time series of different sizes, and there may be large time series gaps between some sample features. For example, for sample features corresponding to the same target object "running action", some "running action" may have a time series that lasts only a few seconds, while some "running action" may last for hundreds of seconds. Therefore, the sample feature sequence can be input into at least two layers of temporal convolution for processing to extract multi-scale features of sample features in the time dimension, so that the obtained fused feature sequence can represent video clips of different lengths.

[0110] Step 1120: Determine the fusion feature corresponding to each sampling nomination in the fusion feature sequence.

[0111] Optional, please combine Fig.10 and Fig.12 , each sampling nomination has a corresponding sample feature, and the fusion feature is the feature of the sample feature mapping, so the mapping relationship between each sampling nomination and the fusion feature can be determined. For example, in Fig.12 In the illustrated embodiment, the sampling nomination Sc1 corresponds to the sample feature Fy1, and the fusion feature Fr1 is obtained by mapping the sample feature Fy1. Therefore, it can be determined that the sampling nomination Sc1 corresponds to the fusion feature Fr1.

[0112] Step 1130: Generate a sampling feature sequence according to multiple fusion features corresponding to the sampling nomination.

[0113] Optionally, a sampling feature sequence generated by multiple fused features can represent video clips of different lengths and has high feature diversity.

[0114] See also Figure 6 , the training method of the refined network also includes:

[0115] Step 640: Perform pooling processing on the sampled feature sequence to obtain a training feature sequence, where the training feature sequence is used to represent multiple sampled features with the same dimension.

[0116] Optionally, the method of step 640 is similar to the method of step 220, the difference being that the pooled feature sequences are respectively an initial feature sequence and a sampling feature sequence, and the corresponding acquired feature sequences are respectively a feature sequence of interest and a training feature sequence, and both have the function of adjusting multiple features of different dimensions to the same dimension, which will not be repeated here.

[0117] Step 650: Input the training feature sequence into the refinement network to determine the loss value of the supervised loss function.

[0118] Optional, please combine Fig.13 , the refinement network includes a regression network, a foreground classification network and an intersection-over-union network, and the supervision loss function includes a regression loss function for supervising the regression network, a foreground classification loss function for supervising the foreground classification network, and an intersection-over-union loss function for supervising the intersection-over-union network. The regression network is used to generate a predicted start position and a predicted end position according to the input features, and the regression function used by the regression network is not limited. The regression loss function is used to determine the deviation between the predicted start position and the actual start position, and the deviation between the predicted end position and the actual end position, so as to optimize the parameters of the regression network and improve the prediction accuracy of the regression network. The foreground classification network is used to predict the probability that the nomination corresponding to the feature belongs to the foreground or background according to the input features. The foreground classification loss is used to determine the accuracy of the probability predicted by the foreground classification network, so as to optimize the parameters of the foreground classification network and improve the prediction accuracy of the foreground classification network. The intersection-over-union network is used to generate a predicted intersection-over-union ratio between the nomination corresponding to the feature and the actual nomination according to the input features. The intersection-over-union loss is used to determine the deviation between the predicted intersection-over-union ratio and the actual intersection-over-union ratio between the nomination corresponding to the feature and the actual nomination.

[0119] Optionally, in other embodiments, the supervision loss function may also include other loss functions, which are not limited here.

[0120] Optional, please combine Fig.14 , step 650: input the training feature sequence into the refinement network to determine the loss value of the supervision loss function, including:

[0121] Step 1410: Input the training feature sequence into the regression network to obtain the predicted position deviation of the sample nomination.

[0122] Optionally, the predicted position deviation includes a predicted starting position deviation between a start position nominated by the sample and a true start position, and a predicted ending position deviation between an end position nominated by the sample and a true end position.

[0123] Step 1420: Obtain the predicted nomination according to the predicted position deviation and the actual nomination.

[0124] Optionally, the start position and end position of the corresponding sample nomination are corrected according to the predicted position deviation to obtain the predicted nomination. The start position of the predicted nomination is the predicted start position, and the end position of the predicted nomination is the predicted end position.

[0125] Step 1430: Determine the regression loss based on the predicted position deviation, the predicted nomination, and the maximum second intersection-over-union ratio between the predicted nomination and the true nomination.

[0126] Optionally, the predicted nomination corresponding to the actual nomination may include multiple nominations, the intersection-and-union ratio between the predicted nomination and the actual nomination may be multiple, and the maximum second intersection-and-union ratio is the maximum value of the multiple second intersection-and-union ratios.

[0127] Optionally, the regression loss function is as shown in Formula 1.

[0128] Formula 1:

[0129] Where N is the number of sequences,

[0130] t is ,t ie They respectively represent the real start and end positions of the real nomination, which can be determined based on the real nomination. Respectively represent the prediction start and prediction end positions of the prediction nomination, which can be determined according to the prediction position deviation and the prediction nomination. iou is the maximum second intersection-over-union ratio between the predicted nomination and the true nomination. L reg is the regression loss value, used to determine t is and The deviation between ie and The deviation between .

[0131] Step 1440: Input the training feature sequence into the foreground-background classification network to obtain the foreground probability, which is used to characterize the probability that the sample nomination corresponding to the training feature belongs to the foreground.

[0132] Step 1450: Determine the foreground classification loss based on the foreground probability and the maximum second intersection-over-union ratio.

[0133] Optionally, the foreground classification loss function is as described in Formula 2.

[0134] Formula 2:

[0135] Among them, p iG represents the probability that the sample nomination corresponding to the i-th training feature belongs to the foreground. iou is the maximum second intersection-over-union ratio between the predicted nomination and the true nomination. L cls is the foreground classification loss. Optionally, nominations with an IoU greater than 0.7 are classified as foreground, and nominations with an IoU less than 0.3 are classified as background. After the optimization is completed, the foreground-background classification network can identify the probability that the initial nomination corresponding to the feature of interest input to the refinement network belongs to the foreground, so as to obtain the temporal action nomination based on the initial nomination belonging to the foreground, thereby improving the accuracy of the obtained temporal action nomination.

[0136] Step 1460: Input the training feature sequence into the IoU network to obtain the predicted IoU.

[0137] Step 1470: Obtain the true intersection-over-union ratio according to the predicted nomination and the true nomination.

[0138] Step 1480: Determine the IoU loss based on the predicted IoU and the actual IoU.

[0139] Optionally, the intersection-over-union loss function is as described in Formula 3.

[0140] Formula 3:

[0141] Among them, I i represents the true intersection-over-union ratio of the i-th predicted nomination to the real one, L represents the predicted-predicted intersection-over-union ratio corresponding to the i-th predicted nomination. iou Optionally, the first IoU in step 810 and the second IoU in step 1430 can be obtained by IoU network calculation. Optimizing the IoU network using the IoU loss function can make the IoU generated by the IoU network more accurate.

[0142] See also Figure 6 , the training method of the refined network also includes:

[0143] Step 660: When the loss value satisfies the preset supervision condition, the training of the refined network is completed.

[0144] Optional, see Fig.15 , step 660: completing the training of the refined network when the loss value satisfies the preset supervision condition, further comprising:

[0145] Step 1510: When the regression loss is less than a preset regression loss threshold, and the foreground classification loss is less than a preset foreground classification loss threshold, the training of the refined network is completed; or

[0146] Step 1520: When the regression loss is less than a preset regression loss threshold, the foreground classification loss is less than a preset foreground classification loss threshold, and the intersection-over-union loss is less than a preset intersection-over-union loss threshold, the training of the refined network is completed.

[0147] Optionally, in the method of step 1510, the training of the refined network can be completed only when the regression loss is less than a preset regression loss threshold, and the foreground classification loss is less than a preset foreground classification loss threshold. In this way, the training process of the refined network can be simplified and the training efficiency can be improved. In the method of step 1520, the training of the refined network can be completed only when the regression loss is less than a preset regression loss threshold, the foreground classification loss is less than a preset foreground classification loss threshold, and the intersection-over-union loss is less than a preset intersection-over-union loss threshold, so as to improve the accuracy of the nominated position deviation generated by the refined network.

[0148] To better illustrate the training method of the refined network provided in the embodiment of the present application, please refer to Figure 6 The process of the training method of the refined network provided in the embodiment of the present application can be summarized into the following steps:

[0149] Step 610: Sampling the sample nomination sequence to obtain a sample nomination sequence, where the sample nomination sequence includes a first preset number of positive samples and a second preset number of negative samples;

[0150] Step 620: Obtain a sampling feature sequence corresponding to the sampling nomination sequence according to the sample feature sequence, where the sampling feature sequence is a collection of abstract information of the sampling nomination sequence;

[0151] Step 630: Perform pooling processing on the sampled feature sequence to obtain a training feature sequence, where the training feature sequence is used to represent multiple sampled features with the same dimension;

[0152] Step 640: Input the training feature sequence into the refinement network to determine the loss value of the supervision loss function; and

[0153] Step 650: When the loss value satisfies the preset supervision condition, the training of the refined network is completed.

[0154] All of the above technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described in detail here.

[0155] The method for obtaining the temporal action nomination in the embodiment of the present application obtains the nomination position deviation through a refinement network, so as to correct the start position and end position of the initial nomination sequence extracted from the target video through the nomination position deviation, so as to obtain the temporal action nomination with accurate start and end positions.

[0156] The training method of the refined network of the embodiment of the present application performs sampling processing on the sample nominations extracted from the sample video to utilize the sample nominations with high richness and balance for training; extracts multi-scale features of the time dimension from the sample features to enrich the diversity of features in the training samples; and optimizes the parameters of the refined network through the supervised loss function to further improve the accuracy of the nomination position deviation generated by the refined network.

[0157] In order to better implement the method for obtaining the time sequence action nomination of the embodiment of the present application, the embodiment of the present application also provides a device 600 for obtaining the time sequence action nomination. Fig.16 , Fig.16 A schematic diagram of the structure of a device 600 for obtaining a time sequence action nomination provided in an embodiment of the present application. The device 600 for obtaining a time sequence action nomination may include:

[0158] The first acquisition module 601 is used to acquire an initial nominated sequence and an initial feature sequence from a target video, wherein the initial nominated sequence is used to characterize a video clip containing a target object, and the initial feature sequence is a collection of abstract information extracted from the initial nominated sequence;

[0159] A first pooling module 602 is used to perform pooling processing on the initial feature sequence to obtain an interesting feature sequence, where the interesting feature sequence is used to represent multiple initial features with the same dimension;

[0160] A refinement module 603, used to input the interest feature sequence into a refinement network to obtain a nominated position deviation, and the refinement network satisfies a preset supervision condition; and

[0161] The acquisition module 604 is used to acquire the time sequence action nomination according to the nomination position deviation and the initial nomination sequence.

[0162] Optionally, the first acquisition module 601 may also be used for:

[0163] Collecting multiple initial segments from the target video;

[0164] Inputting the multiple initial fragments into a feature extraction network to obtain multiple encoding features;

[0165] Obtaining an initial feature sequence according to a plurality of encoding features;

[0166] Obtaining a boundary probability sequence and a confidence score matrix according to the initial feature sequence; and

[0167] The initial nomination sequence is obtained according to the boundary probability sequence and the confidence score matrix.

[0168] The Get module can also be used to:

[0169] Correcting the initial start position and the initial end position of each initial nomination in the initial nomination sequence according to the nomination position deviation to obtain a refined start position and a refined end position corresponding to each initial nomination; and

[0170] A temporal action nomination is generated according to the initial nomination and the refined start position and the refined end position corresponding to the initial nomination.

[0171] Optionally, the present embodiment also provides a training device 700 for a refined network. Fig.17 As shown, Fig.17 A structural diagram of a refined network training device 700 provided in an embodiment of the present application. The refined network training device includes:

[0172] The second acquisition module 701 is used to acquire a sample nomination sequence and a sample feature sequence from a sample video, wherein the sample nomination sequence is used to characterize a video segment containing a sample object, and the sample feature sequence is a collection of abstract information extracted from the sample nomination sequence;

[0173] A sampling module 702 is used to sample the sample nomination sequence to obtain a sample nomination sequence, where the sample nomination sequence includes a first preset number of positive samples and a second preset number of negative samples;

[0174] A feature extraction module 703 is used to obtain a sampling feature sequence corresponding to the sampling nomination sequence according to the sample feature sequence, wherein the sampling feature sequence is a collection of abstract information of the sampling nomination sequence;

[0175] A second pooling module 704 is used to perform pooling processing on the sampled feature sequence to obtain a training feature sequence, where the training feature sequence is used to represent multiple sampled features with the same dimension;

[0176] A supervision module 705, configured to input the training feature sequence into the refinement network to determine a loss value of a supervision loss function; and

[0177] The termination module 706 is used to complete the training of the refined network when the loss value meets the preset supervision condition.

[0178] Optionally, the second acquisition module 701 may also be used for:

[0179] Collecting multiple sample clips from a sample video;

[0180] Inputting a plurality of sample segments into a feature extraction network to obtain a plurality of first features;

[0181] Acquire a sample feature sequence according to a plurality of first features;

[0182] Obtaining a sample probability sequence and a sample score matrix according to the sample feature sequence; and

[0183] According to the sample probability sequence and sample score matrix, the sample nomination sequence is obtained.

[0184] The sampling module 702 may also be used to:

[0185] Obtain the first intersection-over-union ratio between each sample nomination in the sample nomination sequence and the true nomination;

[0186] Nominating samples corresponding to the first intersection-and-union ratios greater than the first preset ratio among the plurality of first intersection-and-union ratios as positive samples, and nominating samples corresponding to the first intersection-and-union ratios less than the second preset ratio as negative samples, so as to obtain a first preset number of positive samples and a second preset number of negative samples; and

[0187] A sampling nomination sequence is generated according to a first preset number of positive samples and a second preset number of negative samples.

[0188] The feature extraction module 703 may also be used to:

[0189] The sample feature sequence is input into at least two layers of time series convolution for processing to obtain a fused feature sequence, which represents the sample feature sequence after time scale abstraction;

[0190] Determining the fusion feature corresponding to each sample nomination in the fusion feature sequence; and

[0191] A sampling feature sequence is generated according to a plurality of fused features corresponding to the sampling nominations.

[0192] The supervision module 705 may also be used to:

[0193] Input the training feature sequence into the regression network to obtain the predicted position deviation of the sample nomination;

[0194] Get predicted nominations based on predicted position deviation and real nominations;

[0195] Determine the regression loss based on the predicted position deviation, the predicted nomination, and the maximum second intersection-over-union ratio between the predicted nomination and the true nomination;

[0196] Input the training feature sequence into the foreground-background classification network to obtain the foreground probability, which is used to represent the probability that the sample nomination corresponding to the training feature belongs to the foreground; and

[0197] The foreground classification loss is determined based on the foreground probability and the maximum second intersection-over-union ratio.

[0198] The supervision module 705 may also be used to:

[0199] Input the training feature sequence into the INR network to obtain the predicted INR;

[0200] Obtain the actual intersection-over-combination ratio based on the predicted nominations and the actual nominations;

[0201] Determine the intersection-over-merge loss based on the predicted intersection-over-merge and the actual intersection-over-merge.

[0202] The termination module 706 may also be used to:

[0203] When the regression loss is less than a preset regression loss threshold and the foreground classification loss is less than a preset foreground classification loss threshold, the training of the refinement network is completed; or

[0204] When the regression loss is less than the preset regression loss threshold, the foreground classification loss is less than the preset foreground classification loss threshold, and the intersection-over-union loss is less than the preset intersection-over-union loss threshold, the training of the refined network is completed.

[0205] It should be noted that the functions of each module in the device 600 for acquiring sequential action nominations and the device 700 for training a refined network in the embodiment of the present application can correspond to the specific implementation methods of any of the above-mentioned method embodiments, which will not be repeated here.

[0206] Each unit in the above device can be implemented in whole or in part by software, hardware or a combination thereof. Each unit can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each unit.

[0207] For example, the device 600 for acquiring time-series action nominations and the device 700 for training a refined network can be integrated in a terminal or server that has a storage device and a processor and has computing capabilities, or the device 600 for acquiring time-series action nominations and the device 700 for training a refined network are the terminal or server.

[0208] Optionally, the present application further provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0209] Fig.18 A schematic structural diagram of a computer device provided in an embodiment of the present application, such as Fig.18As shown, the computer device 800 may include: a communication interface 801, a memory 802, a processor 803 and a communication bus 804. The communication interface 801, the memory 802, and the processor 803 communicate with each other through the communication bus 804. The communication interface 801 is used for the device 800 to communicate data with external devices. The memory 802 can be used to store software programs and modules, and the processor 803 runs the software programs and modules stored in the memory 802, such as the software programs of the corresponding operations in the aforementioned method embodiments.

[0210] Optionally, the processor 803 can call the software program and module stored in the memory 802 to perform the following operations: collect an initial nomination sequence and an initial feature sequence from the target video, the initial nomination sequence is used to characterize the video clip containing the target object, and the initial feature sequence is a collection of abstract information extracted according to the initial nomination sequence; perform pooling processing on the initial feature sequence to obtain an interesting feature sequence, and the interesting feature sequence is used to characterize multiple initial features of the same dimension; input the interesting feature sequence into the refinement network to obtain the nomination position deviation, and the refinement network meets the preset supervision conditions; and obtain the temporal action nomination according to the nomination position deviation and the initial nomination sequence.

[0211] Optionally, the processor 803 can call the software program and module stored in the memory 802 to perform the following operations: collect a sample nomination sequence and a sample feature sequence from a sample video, the sample nomination sequence is used to characterize a video clip containing a sample object, the sample feature sequence is a collection of abstract information extracted according to the sample nomination sequence, the sample video contains a real nomination, and the real nomination includes a known real start position and a real end position; sample the sample nomination sequence to obtain a sampling nomination sequence, the sampling nomination sequence includes a first preset number of positive samples and a second preset number of negative samples; obtain a sampling feature sequence corresponding to the sampling nomination sequence according to the sample feature sequence, the sampling feature sequence is a collection of abstract information of the sampling nomination sequence; perform pooling processing on the sampling feature sequence to obtain a training feature sequence, the training feature sequence is used to characterize multiple sampling features of the same dimension; input the training feature sequence into the refinement network to determine the loss value of the supervision loss function; and complete the training of the refinement network when the loss value meets the preset supervision condition.

[0212] Optionally, the computer device 800 may be integrated in a terminal or server that has a storage device and a processor and has computing capabilities, or the computer device 800 is the terminal or server.

[0213] The present application also provides a computer-readable storage medium for storing a computer program. The computer-readable storage medium can be applied to a computer device, and the computer program enables the computer device to execute the corresponding processes in the above-mentioned methods in the embodiments of the present application, which will not be described in detail for the sake of brevity.

[0214] The present application also provides a computer program product, which includes computer instructions, which are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the corresponding processes in the above-mentioned methods in the embodiments of the present application, which will not be described here for the sake of brevity.

[0215] The present application also provides a computer program, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the corresponding processes in the above-mentioned methods in the embodiments of the present application, which will not be described here for the sake of brevity.

[0216] It should be understood that the processor of the embodiment of the present application may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor are combined and performed. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, and other mature storage media in the art. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0217] It can be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0218] It should be understood that the above-mentioned memory is exemplary but not restrictive. For example, the memory in the embodiments of the present application may also be static random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synch link DRAM, SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DR RAM), etc. That is to say, the memory in the embodiments of the present application is intended to include but not limited to these and any other suitable types of memory.

[0219] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0220] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0221] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0222] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0223] In addition, each functional unit in the embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0224] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer or a server) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0225] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for obtaining a time sequence action nomination, characterized in that: include: Collecting an initial nominated sequence and an initial feature sequence from a target video, wherein the initial nominated sequence is used to characterize a video clip containing a target object, and the initial feature sequence is a collection of abstract information extracted from the initial nominated sequence; Performing pooling processing on the initial feature sequence to obtain an interesting feature sequence, where the interesting feature sequence is used to represent multiple initial features with the same dimension; Inputting the feature sequence of interest into a refinement network to obtain a nominated position deviation, wherein the refinement network satisfies a preset supervision condition; and Acquire a time sequence action nomination according to the nominated position deviation and the initial nominated sequence; The step of training the refined network includes: Collecting a sample nomination sequence and a sample feature sequence from a sample video, wherein the sample nomination sequence is used to characterize a video clip containing a sample object, and the sample feature sequence is a collection of abstract information extracted from the sample nomination sequence, and the sample video contains a real nomination, and the real nomination includes a known real start position and a real end position; Sampling the sample nomination sequence to obtain a sample nomination sequence, wherein the sample nomination sequence includes a first preset number of positive samples and a second preset number of negative samples; Acquire a sampling feature sequence corresponding to the sampling nomination sequence according to the sample feature sequence, wherein the sampling feature sequence is a collection of abstract information of the sampling nomination sequence; Performing pooling processing on the sampling feature sequence to obtain a training feature sequence, wherein the training feature sequence is used to represent multiple sampling features with the same dimension; Inputting the training feature sequence into the refinement network to determine a loss value of a supervised loss function; and The training of the refined network is completed when the loss value meets the preset supervision condition.

2. The method for obtaining a time sequence action nomination according to claim 1, characterized in that: The collecting of the initial nominated sequence from the target video comprises: Collecting a plurality of initial segments from the target video; Inputting a plurality of the initial segments into a feature extraction network to obtain a plurality of encoding features; Acquire the initial feature sequence according to the plurality of encoding features; Obtaining a boundary probability sequence and a confidence score matrix according to the initial feature sequence; and The initial nomination sequence is obtained according to the boundary probability sequence and the confidence score matrix.

3. The method for obtaining a time sequence action nomination according to claim 2, characterized in that: The feature extraction network includes a dual-stream network consisting of a temporal information network and a spatial information network.

4. The method for obtaining a time sequence action nomination according to claim 1, characterized in that: The step of obtaining a time sequence action nomination according to the nominated position deviation and the initial nominated sequence includes: Correcting the initial start position and the initial end position of each initial nomination in the initial nomination sequence according to the nomination position deviation to obtain a refined start position and a refined end position corresponding to each initial nomination; and The time sequence action nomination is generated according to the initial nomination and the refined start position and the refined end position corresponding to the initial nomination.

5. A training method for a refined network, characterized in that: include: Collecting a sample nomination sequence and a sample feature sequence from a sample video, wherein the sample nomination sequence is used to characterize a video clip containing a sample object, and the sample feature sequence is a collection of abstract information extracted from the sample nomination sequence, and the sample video contains a real nomination, and the real nomination includes a known real start position and a real end position; Sampling the sample nomination sequence to obtain a sample nomination sequence, wherein the sample nomination sequence includes a first preset number of positive samples and a second preset number of negative samples; Acquire a sampling feature sequence corresponding to the sampling nomination sequence according to the sample feature sequence, wherein the sampling feature sequence is a collection of abstract information of the sampling nomination sequence; Performing pooling processing on the sampling feature sequence to obtain a training feature sequence, wherein the training feature sequence is used to represent multiple sampling features with the same dimension; Inputting the training feature sequence into the refinement network to determine a loss value of a supervised loss function; and When the loss value satisfies a preset supervision condition, the training of the refined network is completed; The loss value includes regression loss and foreground classification loss, and the inputting the training feature sequence into the refinement network to determine the loss value of the supervision loss function includes: Inputting the training feature sequence into a regression network to obtain the predicted position deviation of the sample nomination; Obtaining a predicted nomination according to the predicted position deviation and the actual nomination; Determining a regression loss according to the predicted position deviation, the predicted nomination, and a maximum second intersection-over-union ratio between the predicted nomination and the true nomination; Inputting the training feature sequence into a foreground-background classification network to obtain a foreground probability, wherein the foreground probability is used to characterize the probability that the sample nomination corresponding to the training feature belongs to the foreground; and A foreground classification loss is determined according to the foreground probability and the second maximum intersection-over-union ratio.

6. The training method according to claim 5, characterized in that: The collecting of the sample nomination sequence and the sample feature sequence from the sample video includes: Collecting a plurality of sample clips from the sample video; Inputting the plurality of sample segments into a feature extraction network to obtain a plurality of first features; Acquire the sample feature sequence according to the plurality of the first features; Acquire a sample probability sequence and a sample score matrix according to the sample feature sequence; and The sample nomination sequence is obtained according to the sample probability sequence and the sample score matrix.

7. The training method according to claim 5, characterized in that: The step of sampling the sample nomination sequence to obtain a sample nomination sequence comprises: Obtaining a first intersection-over-union ratio between each sample nomination in the sample nomination sequence and the true nomination; Nominate samples corresponding to the first intersection-and-union ratios greater than a first preset ratio among the plurality of first intersection-and-union ratios as the positive samples, and nominate samples corresponding to the first intersection-and-union ratios less than a second preset ratio as the negative samples, so as to obtain a first preset number of the positive samples and a second preset number of the negative samples; and The sampling nomination sequence is generated according to a first preset number of the positive samples and a second preset number of the negative samples.

8. The training method according to claim 5, characterized in that: The acquiring, according to the sample feature sequence, a sampling feature sequence corresponding to the sampling nomination sequence comprises: Input the sample feature sequence into at least two layers of temporal convolution for processing to obtain a fused feature sequence, wherein the fused feature sequence represents the sample feature sequence after time scale abstraction; Determining the fusion feature corresponding to each of the sample nominations in the fusion feature sequence; and The sampling feature sequence is generated according to the plurality of fused features corresponding to the sampling nomination.

9. The training method according to claim 8, characterized in that: The loss function further includes an intersection-over-union loss, and the step of inputting the training feature sequence into the refinement network to determine the loss value of the supervision loss function further includes: Inputting the training feature sequence into an IoU network to obtain a predicted IoU; Obtaining a true intersection-over-union ratio according to the predicted nomination and the true nomination; and An IoU loss is determined according to the predicted IoU and the true IoU.

10. The training method according to claim 9, characterized in that: The loss value includes regression loss, foreground classification loss and intersection-over-union loss, and the training of the refined network is completed when the loss value satisfies a preset supervision condition, including: When the regression loss is less than a preset regression loss threshold, and the foreground classification loss is less than a preset foreground classification loss threshold, completing the training of the refinement network; or When the regression loss is less than a preset regression loss threshold, the foreground classification loss is less than a preset foreground classification loss threshold, and the intersection-over-union loss is less than a preset intersection-over-union loss threshold, the training of the refined network is completed.

11. A device for obtaining a time sequence action nomination, characterized in that: The device for obtaining the time sequence action nomination includes: A first acquisition module is used to acquire an initial nominated sequence and an initial feature sequence from a target video, wherein the initial nominated sequence is used to characterize a video clip containing a target object, and the initial feature sequence is a collection of abstract information extracted from the initial nominated sequence; A first pooling module is used to perform pooling processing on the initial feature sequence to obtain an interesting feature sequence, where the interesting feature sequence is used to represent multiple initial features with the same dimension; A refinement module, configured to input the feature sequence of interest into a refinement network to obtain a nominated position deviation, wherein the refinement network satisfies a preset supervision condition; and An acquisition module, configured to acquire a time sequence action nomination according to the nominated position deviation and the initial nominated sequence; A second acquisition module is used to collect a sample nomination sequence and a sample feature sequence from a sample video, wherein the sample nomination sequence is used to characterize a video clip containing a sample object, and the sample feature sequence is a collection of abstract information extracted from the sample nomination sequence, and the sample video contains a real nomination, and the real nomination includes a known real start position and a real end position; A sampling module, configured to sample the sample nomination sequence to obtain a sample nomination sequence, wherein the sample nomination sequence includes a first preset number of positive samples and a second preset number of negative samples; A feature extraction module, used to obtain a sampling feature sequence corresponding to the sampling nomination sequence according to the sample feature sequence, wherein the sampling feature sequence is a collection of abstract information of the sampling nomination sequence; A second pooling module is used to perform pooling processing on the sampled feature sequence to obtain a training feature sequence, where the training feature sequence is used to represent multiple sampled features with the same dimension; A supervision module, configured to input the training feature sequence into the refinement network to determine a loss value of a supervision loss function; and The termination module is used to complete the training of the refined network when the loss value meets the preset supervision condition.

12. A training device for a refined network, characterized in that: The training device comprises: A second acquisition module is used to collect a sample nomination sequence and a sample feature sequence from a sample video, wherein the sample nomination sequence is used to characterize a video clip containing a sample object, and the sample feature sequence is a collection of abstract information extracted from the sample nomination sequence, and the sample video contains a real nomination, and the real nomination includes a known real start position and a real end position; A sampling module, configured to sample the sample nomination sequence to obtain a sample nomination sequence, wherein the sample nomination sequence includes a first preset number of positive samples and a second preset number of negative samples; A feature extraction module, used to obtain a sampling feature sequence corresponding to the sampling nomination sequence according to the sample feature sequence, wherein the sampling feature sequence is a collection of abstract information of the sampling nomination sequence; A second pooling module is used to perform pooling processing on the sampled feature sequence to obtain a training feature sequence, where the training feature sequence is used to represent multiple sampled features with the same dimension; A supervision module, configured to input the training feature sequence into the refinement network to determine a loss value of a supervision loss function; and A termination module, used to complete the training of the refined network when the loss value satisfies a preset supervision condition; The loss value includes regression loss and foreground classification loss, and the supervision module is specifically used for: Inputting the training feature sequence into a regression network to obtain the predicted position deviation of the sample nomination; Obtaining a predicted nomination according to the predicted position deviation and the actual nomination; Determining a regression loss according to the predicted position deviation, the predicted nomination, and a maximum second intersection-over-union ratio between the predicted nomination and the true nomination; Inputting the training feature sequence into a foreground-background classification network to obtain a foreground probability, wherein the foreground probability is used to characterize the probability that the sample nomination corresponding to the training feature belongs to the foreground; and A foreground classification loss is determined according to the foreground probability and the second maximum intersection-over-union ratio.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is suitable for being loaded by a processor to execute the steps in the method for obtaining temporal action nominations as described in any one of claims 1 to 4, or the steps in the method for training a refined network as described in any one of claims 5 to 10.

14. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the processor is used to execute the steps in the method for obtaining temporal action nominations as described in any one of claims 1 to 4, or the steps in the method for training a refined network as described in any one of claims 5 to 10 by calling the computer program stored in the memory.

15. A computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the method for obtaining temporal action nominations as described in any one of claims 1 to 4, or the steps of the method for training a refined network as described in any one of claims 5 to 10.