Knowledge-enhanced video scene recognition method and system

Abstract

The present disclosure provides a knowledge enhancement-based video scene recognition method, comprising: receiving a to-be-recognized video comprising a plurality of video frames and video-level text; extracting a keyword from the video-level text, and obtaining frame-level global temporal features, frame-level local temporal features, and frame-level local non-temporal features of the plurality of video frames; constructing video-level temporal features based on the frame-level global temporal features to determine first scene label information of the to-be-recognized video; constructing video-level knowledge enhancement features based on the obtained frame-level local temporal features, frame-level local non-temporal features, and keyword to determine second scene label information of the to-be-recognized video; and fusing the first scene label information and the second scene label information to determine a scene of the to-be-recognized video.

Description

TECHNICAL FIELD

[0001] The present disclosure mainly relates to image processing, and particularly relates to scene recognition. BACKGROUND

[0002] With the development of Internet technology, the mainstream form of content has experienced gradual development from the pure text era to the graphic era, and then to the current video and live era. Compared with pure text and graphic content forms, video content is more rich and attractive to users.

[0003] Video understanding aims to automatically identify and analyze the content in the video through intelligent analysis technology. Scene recognition is one of the angles of video understanding, and there are currently scene recognition methods based on time sequence features, non-time sequence features or video descriptions. However, due to the richness and complexity of video content, the scene recognition efficiency of the three types of scene recognition methods is not high.

[0004] Therefore, there is a need for an efficient video scene recognition scheme in the art. SUMMARY

[0005] To solve the above technical problems, the present disclosure provides a knowledge-enhanced video scene recognition scheme, which simultaneously incorporates the analysis of the time sequence features, non-time sequence features and video descriptions of the video, learns a robust video representation through the construction of a dual-flow architecture of time sequence and non-time sequence, and introduces external knowledge to assist feature fusion, to achieve high-performance video scene recognition.

[0006] In an embodiment of the present disclosure, a knowledge-enhanced video scene recognition method is provided, comprising: receiving a to-be-recognized video comprising a plurality of video frames and video-level text; extracting keywords from the video-level text, and obtaining frame-level global time sequence features and frame-level local time sequence features of the plurality of video frames, and frame-level local non-time sequence features; constructing video-level time sequence features based on the frame-level global time sequence features to determine first scene label information of the to-be-recognized video; constructing video-level knowledge-enhanced features based on the obtained frame-level local time sequence features, frame-level local non-time sequence features and keywords to determine second scene label information of the to-be-recognized video; and fusing the first scene label information and the second scene label information to determine the scene of the to-be-recognized video.

[0007] In another embodiment of the present disclosure, the plurality of video frames are key frames of the to-be-recognized video.

[0008] In yet another embodiment of the present disclosure, the frame-level global time sequence features and the frame-level local time sequence features of the plurality of video frames are obtained by fusing multi-modal features of the video frames.

[0009] In another embodiment of the present disclosure, the multi-modal features include video, image, text, audio and the like.

[0010] In yet another embodiment of the present disclosure, the acquiring the frame-level local non-temporal feature of the plurality of video frames comprises: detecting a local region containing an entity in the video frame and extracting a feature of the detected local region; and performing intra-frame inference using the feature of the detected local region in the video frame to acquire the frame-level local non-temporal feature.

[0011] In another embodiment of the present disclosure, the constructing the video-level knowledge enhanced feature based on the acquired frame-level local temporal feature, the frame-level local non-temporal feature and the keyword comprises: performing frame-level feature fusion on the acquired frame-level local temporal feature and the frame-level local non-temporal feature using an attention mechanism to acquire a frame-level local feature of each video frame; and fusing the frame-level local features of the plurality of video frames using a feature of the keyword to acquire the video-level knowledge enhanced feature.

[0012] In yet another embodiment of the present disclosure, the determining the first or second scene label information of the video to be recognized is performed using a hierarchical multi-label prediction model.

[0013] In another embodiment of the present disclosure, the fusing the first scene label information and the second scene label information is achieved through knowledge distillation.

[0014] In an embodiment of the present disclosure, a video scene recognition system based on knowledge enhancement is provided, comprising: a receiving module configured to receive a video to be recognized comprising a plurality of video frames and a video-level text; an extracting module configured to extract a keyword from the video-level text and acquire frame-level global temporal features and frame-level local temporal features, and frame-level local non-temporal features of the plurality of video frames; a temporal feature learning module configured to construct a video-level temporal feature based on the frame-level global temporal features to determine first scene label information of the video to be recognized; a non-temporal feature learning module configured to construct a video-level knowledge enhanced feature based on the acquired frame-level local temporal features, the frame-level local non-temporal features and the keyword to determine second scene label information of the video to be recognized; and a scene recognition module configured to fuse the first scene label information and the second scene label information to determine a scene of the video to be recognized.

[0015] In yet another embodiment of the present disclosure, in the model training phase, both the temporal feature learning module and the non-temporal feature learning module participate in model training.

[0016] In another embodiment of the present disclosure, in the inference test phase, only the temporal feature learning module participates in inference test.

[0017] In an embodiment of the present disclosure, a computer readable storage medium storing instructions is provided, which when executed cause a machine to perform the method as described above.

[0018] This summary is provided to introduce some concepts of the following detailed description in a simplified form. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF DRAWINGS

[0019] The above summary of the present disclosure, and the following detailed description of the present disclosure will be better understood when read in conjunction with the appended drawings. It should be understood that the figures are merely schematic and are not drawn to scale. It should be understood that the present application can take many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the application to those skilled in the art. In the drawings, like reference numerals refer to like elements throughout.

[0020] Figure 1 is a flow chart illustrating a knowledge-enhanced video scene recognition method according to an embodiment of the present disclosure;

[0021] Figure 2 is a schematic diagram illustrating a temporal feature learning framework according to an embodiment of the present disclosure;

[0022] Figure 3 is a schematic diagram illustrating a non-temporal feature learning framework according to an embodiment of the present disclosure;

[0023] Figure 4 is a schematic diagram illustrating a knowledge-enhanced video scene recognition framework according to an embodiment of the present disclosure;

[0024] Figure 5 is a block diagram illustrating a knowledge-enhanced video scene recognition system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] In order to make the above objectives, features and advantages of the present disclosure more obvious and comprehensible, the specific embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0026] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that the present disclosure can be practiced without the specific details presented herein. In other instances, well-known methods have not been described in detail in order to avoid obscuring the present disclosure.

[0027] In recent years, the explosive growth of video data has made the analysis and inference of video content increasingly important. One major application of artificial intelligence on video is video understanding, which aims to solve the problem of "semantic gap". Video structural analysis is an important branch of video understanding, which is to identify scene labels based on video content.

[0028] Video understanding itself is complex due to the diversity and redundancy of video content and the inherent "gap" between the multi-modal information (video, image, text, behavior, etc.) of the video. Therefore, it is necessary to effectively and comprehensively understand the content of the video. Based on the insight into the features of the video content, the multi-modal information contained in the video content includes global / local information, time / non-time information, visual / text information, etc. However, due to the diversity and discreteness of the video content, it is difficult to distinguish useful and useless information.

[0029] Current video understanding techniques are based on modeling of spatial-temporal relationship, and attention is mainly focused on the time information of the video, while the non-time information of the video is often ignored. The introduction of external knowledge helps to build and measure the relationship between different modal information of the video content, but since the knowledge graph usually has a huge amount of information, it is difficult to fuse the multi-modal information of the video with external knowledge while maintaining the high cost-effectiveness of the video scene recognition architecture.

[0030] Therefore, there is a need in the art for a solution that can fuse the multi-modal information of the video while quickly extracting useful information for video scene recognition, and achieve efficient video scene recognition.

[0031] The present application provides a knowledge-enhanced video scene recognition scheme, which simultaneously incorporates the analysis of the time sequence features, non-time sequence features and video description of the video, and achieves high-performance video scene recognition through robust video expression and the introduction of external knowledge to assist feature fusion.

[0032] Figure 1 FIG. 1 is a flowchart illustrating a knowledge-enhanced video scene recognition method 100 according to an embodiment of the present disclosure.

[0033] The knowledge-enhanced video scene recognition scheme of the present disclosure is based on the understanding that both the time sequence information and the non-time sequence information of the video are valuable and can complement each other, taking into account the time sequence features and non-time sequence features of the video frames, introducing knowledge enhancement based on the text modal and strengthening the fusion with other modalities, thereby achieving efficient video scene recognition.

[0034] At 102, a video to be recognized including a plurality of video frames and video-level text is received.

[0035] The received video to be recognized contains video-level text and a plurality of video frames. The video-level text describes the video-level text features, which are embodied by keywords. The video frames, as continuous frames, include the time sequence information contained between the continuous frames, and the meaningful non-time sequence information of a single frame. Scene recognition is to identify the appropriate scene label based on the video frame sequence and the associated description text.

[0036] The plurality of video frames can be key frames of the video to be recognized. The video frames usually include multi-modal features, such as but not limited to, video, image, text, audio, and the like.

[0037] At 106, keywords are extracted from the video-level text, and frame-level global temporal features and frame-level local temporal features, and frame-level local non-temporal features of the plurality of video frames are obtained.

[0038] Step 106 prepares for subsequent temporal feature learning and non-temporal feature learning.

[0039] The frame-level global temporal features and the frame-level local temporal features of the plurality of video frames are obtained by fusing the multi-modal features of the video frames. Specifically, two-dimensional features and three-dimensional features are extracted from the plurality of video frames, text features are extracted from the video-level text, and the frame-level global temporal features and the frame-level local temporal features are obtained after fusion. Further, local regions (referred to as “candidate local regions”) containing entities in each frame are detected from the plurality of video frames, features thereof are extracted, and inference within the frame is performed using the features of the detected local regions in the video frames to obtain the frame-level local non-temporal features.

[0040] The process of obtaining the frame-level features will be described in detail below with reference to Figure 2 and Figure 3

[0041] Keywords are extracted from the video-level text to integrate node and edge information in the knowledge graph into scene recognition based on the keywords.

[0042] At 108, video-level temporal features are constructed based on the frame-level global temporal features to determine first scene label information of the video to be recognized.

[0043] To aggregate global information, video-level temporal features are constructed based on the frame-level global temporal features. In an embodiment of the present disclosure, an encoder (for example, a transformer encoder) is used to construct the frame-level global temporal features into video-level temporal features together with position embedding. The process of constructing the video-level temporal features will be described in detail below with reference to Figure 2

[0044] After obtaining the video-level temporal features, multi-level multi-label prediction can be used to determine the first scene label information of the video to be recognized. In an embodiment of the present disclosure, a multi-layer perception (MLP) is used to obtain scores of two layers of scene label prediction, which are added together as the first scene label information of the video to be recognized.

[0045] At 110, video-level knowledge-enhanced features are constructed based on the obtained frame-level local temporal features, frame-level local non-temporal features, and keywords to determine second scene label information of the video to be recognized.​​

[0046] The acquired frame-level local timing features and frame-level local non-timing features are fused by using an attention mechanism to obtain frame-level local features of each video frame, and the frame-level local features of multiple video frames are fused using the keyword to obtain video-level knowledge enhanced features.

[0047] In an embodiment of the present disclosure, for the frame-level local timing features and frame-level local non-timing features obtained in step 106, fused frame-level local features can be constructed. By using the fused frame-level local features and the keyword, the similarity between different fused frame-level local features and the keyword semantics can be determined, thereby obtaining the weights of the fused frame-level local features.

[0048] In this embodiment, further, the fused frame-level local features can be fused to different degrees based on specific keyword semantics to construct video-level knowledge enhanced features. The process of constructing the video-level knowledge enhanced features will be described in detail below with reference to Figure 3 Further detailed description.

[0049] Similarly, after obtaining the video-level knowledge enhanced features, multi-level multi-label prediction can be used to determine the second scene label information of the video to be recognized, similar to step 108.

[0050] In 112, the first scene label information and the second scene label information are fused to determine the scene of the video to be recognized.

[0051] In an embodiment of the present disclosure, the first scene label information determined by the timing feature learning and the second scene label information determined by the non-timing feature learning can be fused by means of knowledge distillation (for example, self-distillation).

[0052] In an embodiment of the present disclosure, in the model training phase, both the timing feature learning and the non-timing feature learning participate in model training; while in the inference test phase, only the lighter timing feature learning participates, thereby balancing the performance and efficiency of the video scene recognition model.

[0053] Thus, the knowledge-enhanced video scene recognition method of the present disclosure simultaneously incorporates the analysis of the timing features, non-timing features and video descriptions of the video, achieves high-performance and strong scalability of video scene recognition by robust video representation and the introduction of external knowledge to assist feature fusion.

[0054] Figure 2 is a schematic diagram showing a timing feature learning framework according to an embodiment of the present disclosure.

[0055] In an embodiment of the present disclosure, a dual-flow framework of timing feature learning framework and non-timing feature learning framework is adopted for the scene recognition task. As shown in Figure 2The illustrated time sequence feature learning framework includes frame-level time sequence feature acquisition, video-level time sequence feature acquisition, and hierarchical multi-label prediction, etc., which takes a multi-modal bidirectional Transformer as the backbone and uses a Transformer as a feature encoder.

[0056] As shown in the time sequence feature learning framework, the video content is modeled from the overall and time sequence of the video. Those skilled in the art can understand that other backbone architectures and feature encoders (including but not limited to RNN, LSTM, GRU, etc.) can be used to implement the time sequence feature learning framework, and even as technology advances, other more advanced neural networks and encoders for feature learning can be included in the technical solutions of the present disclosure, which will not be repeated here. Figure 2

[0057] For each video v, N f frames are sampled as key frames, and frame-level two-dimensional 2D features of each key frame are extracted Then, N c consecutive frames centered on each sampled key frame are collected, and frame-level three-dimensional 3D features are extracted For the text description associated with the video v, video-level text features f text are extracted.

[0058] Based on the extracted above features, the features are spliced and refined to obtain frame-level global time sequence features and frame-level local time sequence features

[0059]

[0060]

[0061]

[0062] τ∈{gframe,lframe},

[0063] wherein denotes splicing, denotes splicing features, and denotes refining features, and δ(·) denotes an activation function. W1, W2, b1, b2 denote weight matrices and bias vectors in a fully connected layer. W3 denotes a residual transformation matrix, and Norm denotes a layer normalization operation.

[0064] In the embodiments of the present disclosure, further, a specific classification label [CLS] and frame-level global time sequence features ​The position embedding is input to the self-attention layer of the Transformer encoder. After encoding, the Transformer encoder generates a series of outputs, and the output corresponding to a specific classification label [CLS] is the video-level temporal feature e. t_video :

[0065]

[0066] e t_video =o0,

[0067] Where O refers to the output sequence, pos j (j∈[1,N) f The number of keyframes sampled is 1.

[0068] After obtaining video-level temporal features e t_video Then, hierarchical multi-label prediction is adopted, that is, multilayer perceptron (MLP) is used to obtain the label score of the scene recognition label in order to determine the first scene label information of the video to be recognized.

[0069] Figure 3 This is a schematic diagram illustrating a non-temporal feature learning framework according to an embodiment of the present disclosure.

[0070] In one embodiment of this disclosure, a dual-stream framework, combining a temporal feature learning framework and a non-temporal feature learning framework, is employed for scene recognition tasks. For example... Figure 3 The non-temporal feature learning framework shown includes frame-level local feature fusion, video-level feature fusion, and hierarchical multi-label prediction. In this embodiment, the Transformer is also used as the feature encoder.

[0071] like Figure 3 The non-temporal feature learning framework shown models video content from the perspective of local regions frame by frame and from a non-temporal angle. Those skilled in the art will understand that other backbone architectures and feature encoders can be used to implement the temporal feature learning framework. Furthermore, with technological advancements, more advanced neural networks and encoders for other feature learning can be incorporated into the technical solution of this disclosure, which will not be elaborated upon here.

[0072] First, a detection model (e.g., CNN, Faster-RCNN, Mask-R-CNN, YOLO, SSD, etc.) is used to detect candidate local regions and extract their 2D features. The 2D features of the m-th candidate region in the j-th keyframe are represented as follows:

[0073] Then the candidate region features of each key frame are input into the self-attention layer of the Transformer encoder, and intra-frame reasoning is performed on the detected candidate regions of the same key frame to obtain enhanced features of each candidate region

[0074]

[0075] where N r denotes the number of candidate regions extracted in the same key frame, and is the obtained frame-level local non-temporal feature.

[0076] Then, the frame-level local temporal feature obtained as Figure 2 and the frame-level local non-temporal feature obtained above are fused. Specifically, the frame-level local non-temporal feature obtained as Figure 2 is taken as an element Query in a given Target, and the frame-level local non-temporal feature obtained above is taken as a constituent element in Source.

[0077] The frame-level local non-temporal feature obtained above is imagined to be composed of a series of <Key, Value> data pairs. At this time, by calculating the similarity or correlation between Query and each Key, the weight coefficient of the Value corresponding to each Key is obtained, and then the Value is weighted and summed to obtain the final Attention value:

[0078]

[0079]

[0080]

[0081]

[0082]

[0083] where denotes the weight matrix corresponding to the query, key and value of the self-attention module, and dKey represents the dimension of the key vector, is the obtained frame-level local feature.

[0084] After obtaining the fused frame-level local feature, the entity embedding G θ from the knowledge graph G (including but not limited to ConceptNet, DBpedia, etc.) is used as external knowledge to extract N kA keyword. The entity embedding corresponding to the keyword in the pre-trained knowledge graph is used as the feature of the keyword, and the local features of different frames of the same video are fused to obtain the knowledge-enhanced keyword-level non-temporal feature corresponding to each keyword. After mean pooling and residual calculation, the video-level knowledge-enhanced feature is obtained.

[0085] After obtaining the video-level knowledge-enhanced feature, a hierarchical multi-label prediction is performed, i.e., a multi-layer perception (MLP) is used to obtain the label score of the scene recognition label to determine the second scene label information of the video to be recognized.

[0086] Specifically, the entity abstract embedding features corresponding to each scene label in the pre-trained knowledge graph are used as the features of each video scene, and the process of calculating the knowledge-enhanced video-level feature to calculate the scene label score is converted into the process of calculating the similarity between the knowledge-enhanced video-level feature and each scene label. After fusing the multi-layer label scores, the refined scene label score is obtained, i.e., the scene label information determined in the non-temporal feature learning framework.

[0087] Figure 4 FIG. 1 is a schematic diagram illustrating a knowledge-enhanced video scene recognition framework according to an embodiment of the present disclosure.

[0088] The first scene label information determined as shown in Figure 2 and the second scene label information determined as shown in Figure 3 are fused in the knowledge-enhanced video scene recognition framework as shown in Figure 4 to make a final video scene judgment.

[0089] The knowledge-enhanced video scene recognition framework as shown in Figure 4 fuses the first scene label information of the video to be recognized determined by the temporal feature learning and the second scene label information (i.e., the score of the label) of the video to be recognized determined by the non-temporal feature learning through the method of knowledge distillation (such as but not limited to self-distillation).

[0090] In particular, in the model training stage, both the temporal feature learning framework and the non-temporal feature learning framework participate in the model training; in the inference test stage, only the lighter temporal feature learning framework participates.

[0091] For a given video, the topK labels are output according to the score of the label, or all labels with a score greater than a certain threshold, so as to balance the performance and efficiency of the model.

[0092] Thus, the knowledge-enhanced video scene recognition framework of the present disclosure utilizes a dual-stream framework design based on the timing and non-timing angles: the timing information contained between the continuous frames of the video and the valuable information in the single frame independent of the timing can be utilized; the keywords extracted from the text description and their abstract expression in the pre-trained knowledge graph can be used to guide the non-timing fusion of multi-modal features and assist the label score calculation, so as to improve the performance of the model; the information learned from the timing and non-timing angles can be fused by using the self-distillation module, and only the light timing feature module is used during inference to balance the performance and efficiency of the model.

[0093] Figure 5 FIG. 5 is a block diagram illustrating a knowledge-enhanced video scene recognition system 500 according to an embodiment of the present disclosure.

[0094] The knowledge-enhanced video scene recognition system 500 according to an embodiment of the present disclosure includes a receiving module 502, an extracting module 506, a timing feature learning module 508, a non-timing feature learning module 510, and a scene recognition module 512.

[0095] The receiving module 502 receives a video to be recognized including a plurality of video frames and video-level text.

[0096] The video to be recognized received by the receiving module 502 contains video-level text and a plurality of video frames. The video-level text describes the video-level text features, which are embodied by keywords. The video frames, as continuous frames, include the timing information contained between the continuous frames and the meaningful non-timing information of a single frame. Scene recognition is to identify the appropriate scene label based on the video frame sequence and the associated description text.

[0097] The plurality of video frames can be key frames of the video to be recognized. The video frames usually include multi-modal features, such as but not limited to, video, image, text, audio, etc.

[0098] The extracting module 506 extracts keywords from the video-level text, and obtains frame-level global timing features and frame-level local timing features of the plurality of video frames, and frame-level local non-timing features.

[0099] The extracting module 506 prepares for the subsequent timing feature learning and non-timing feature learning.

[0100] The extraction module 506 obtains the frame-level global temporal features and the frame-level local temporal features of the plurality of video frames by fusing the multi-modal features of the video frames. Specifically, the extraction module 506 extracts two-dimensional features and three-dimensional features from the plurality of video frames, extracts text features from the video-level text, and obtains the frame-level global temporal features and the frame-level local temporal features after fusion. Further, the extraction module 506 detects local regions (referred to as “candidate local regions”) containing entities in each frame from the plurality of video frames, extracts features thereof, and performs intra-frame inference using the features of the detected local regions in the video frames to obtain frame-level local non-temporal features.

[0101] The extraction module 506 also extracts keywords from the video-level text to incorporate node and edge information in the knowledge graph into scene recognition based on the keywords.

[0102] The temporal feature learning module 508 constructs video-level temporal features based on the frame-level global temporal features to determine the first scene label information of the video to be recognized.

[0103] To aggregate global information, the temporal feature learning module 508 constructs video-level temporal features based on the frame-level global temporal features. In an embodiment of the present disclosure, an encoder (for example, a transformer encoder) is used to construct the frame-level global temporal features into video-level temporal features together with position embedding.

[0104] After obtaining the video-level temporal features, the temporal feature learning module 508 can use multi-level multi-label prediction to determine the first scene label information of the video to be recognized. In an embodiment of the present disclosure, a multi-layer perception (MLP) is used to obtain scores of two layers of scene label prediction, which are added together as the first scene label information of the video to be recognized.

[0105] The non-temporal feature learning module 510 constructs video-level knowledge-enhanced features based on the obtained frame-level local temporal features, frame-level local non-temporal features, and keywords to determine the second scene label information of the video to be recognized.

[0106] The non-temporal feature learning module 510 uses an attention mechanism to perform frame-level feature fusion on the obtained frame-level local temporal features and frame-level local non-temporal features to obtain frame-level local features of each video frame, and uses the keywords to fuse frame-level local features of a plurality of video frames to obtain video-level knowledge-enhanced features.

[0107] In an embodiment of the present disclosure, the non-temporal feature learning module 510 can construct fused frame-level local features for the frame-level local temporal features and the frame-level local non-temporal features obtained in step 106. Using the fused frame-level local features and the keywords, the non-temporal feature learning module 510 can determine the similarity between different fused frame-level local features and keyword semantics, thereby obtaining the weights of the fused frame-level local features.

[0108] In this embodiment, further, the non-temporal feature learning module 510 fuses the fused frame-level local features to different degrees based on specific keyword semantics, thereby constructing video-level knowledge-enhanced features.

[0109] Similarly, after obtaining the video-level knowledge-enhanced features, the non-temporal feature learning module 510 can determine the second scene label information of the video to be recognized using multi-level multi-label prediction.

[0110] The scene recognition module 512 fuses the first scene label information and the second scene label information to determine the scene of the video to be recognized.

[0111] In an embodiment of the present disclosure, the scene recognition module 512 fuses the first scene label information determined by the temporal feature learning and the second scene label information determined by the non-temporal feature learning through knowledge distillation (e.g., self-distillation).

[0112] Thus, the knowledge-enhanced video scene recognition system of the present disclosure simultaneously incorporates the analysis of the temporal features, non-temporal features, and video descriptions of a video, achieves robust video representation, and introduces external knowledge to assist feature fusion, thereby achieving high-performance and highly expandable video scene recognition.

[0113] The various steps and modules of the knowledge-enhanced video scene recognition method and system described above can be implemented in hardware, software, or a combination thereof. If implemented in hardware, the various illustrative steps, modules, and circuits described in connection with the present disclosure can be implemented or performed with a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic component, hardware component, or any combination thereof. The general purpose processor can be a microprocessor, a controller, a controller with a state machine, etc. If implemented in software, the various illustrative steps, modules, and circuits described in connection with the present disclosure can be stored as one or more instructions or code on a computer-readable medium and executed by a processing unit. A computer-readable medium can be any available medium or media including a volatile or non-volatile medium, an electromagnetic medium such as a computer diskette, a floppy disk, a RAM, a flash memory, a ROM, an EPROM, an EEPROM, a register, a hard disk, a removable disk, a CD-ROM, a cloud storage, etc. The computer-readable medium can be coupled to the processing unit such that the processing unit can read information from, and write information to, the computer-readable medium. In addition, the software-based embodiment can be uploaded, downloaded, or remotely accessed through a suitable communication means. Such suitable communication means include, for example, the Internet, the World Wide Web, an intranet, software applications, cable (including fiber optic cable), magnetic communications, electromagnetic communications (including RF, microwave, and infrared communications), electronic communications, or other such communication means.

[0114] It should also be noted that the embodiments can be described as a process which is depicted as a flowchart, a flow diagram, a structure diagram, or a block diagram. Although a flowchart can describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations can be re-arranged.

[0115] The disclosed systems, devices, and systems should not be limited in any way by the above description. Rather, the present disclosure encompasses all novel and non-obvious features and aspects of the various disclosed embodiments, alone and in various combinations and sub-combinations with one another. The disclosed systems, devices, and systems are not limited to any particular aspect or feature or combination of them, nor do any of the disclosed embodiments require the presence of any particular advantage or solve a particular or all technical problems.

[0116] The embodiments of the present application described above with reference to the drawings are merely exemplary and are not intended to limit the present application to the specific embodiments described. The above-described embodiments are merely illustrative, and are not intended to limit the present application to the specific embodiments described. Many modifications can be made by those skilled in the art without departing from the spirit and scope of the present application, and these modifications are also within the scope of the present application.

Claims

1.A method of video scene recognition based on knowledge enhancement, comprising: receiving a video to be recognized including a plurality of video frames and video-level text; extracting keywords from the video-level text, and obtaining frame-level global temporal features and frame-level local temporal features of the plurality of video frames, and frame-level local non-temporal features; constructing video-level temporal features based on the frame-level global temporal features to determine first scene label information of the video to be recognized; constructing video-level knowledge enhancement features based on the obtained frame-level local temporal features, frame-level local non-temporal features and the keywords to determine second scene label information of the video to be recognized; and fusing the first scene label information and the second scene label information to determine a scene of the video to be recognized. 2.The method of claim 1, wherein the plurality of video frames are key frames of the video to be recognized. 3.The method of claim 1, wherein the obtaining frame-level global temporal features and frame-level local temporal features of the plurality of video frames is performed by fusing multi-modal features of the video frames. 4.The method of claim 3, wherein the multi-modal features include video, image, text, and audio features. 5.The method of claim 1, wherein the obtaining frame-level local non-temporal features of the plurality of video frames comprises: detecting local regions containing entities in the video frames and extracting features of the detected local regions; and performing intra-frame inference using the features of the detected local regions in the video frames to obtain frame-level local non-temporal features. 6.The method of claim 1, wherein the constructing video-level knowledge enhancement features based on the obtained frame-level local temporal features, frame-level local non-temporal features and the keywords comprises: performing frame-level feature fusion of the obtained frame-level local temporal features and frame-level local non-temporal features using an attention mechanism to obtain frame-level local features of each video frame; and fusing the frame-level local features of the plurality of video frames using features of the keywords to obtain video-level knowledge enhancement features. 7.The method of claim 1, wherein the determining first or second scene label information of the video to be recognized is performed using a hierarchical multi-label prediction model. 8.The method of claim 1, wherein the fusing the first scene label information and the second scene label information is achieved by knowledge distillation. 9.A system of video scene recognition based on knowledge enhancement, comprising: a receiving module that receives a video to be recognized including a plurality of video frames and video-level text; an extracting module that extracts keywords from the video-level text, and obtains frame-level global temporal features and frame-level local temporal features of the plurality of video frames, and frame-level local non-temporal features; a temporal feature learning module that constructs video-level temporal features based on the frame-level global temporal features to determine first scene label information of the video to be recognized; a non-temporal feature learning module that constructs video-level knowledge enhancement features based on the obtained frame-level local temporal features, frame-level local non-temporal features and the keywords to determine second scene label information of the video to be recognized; and a fusing module that fuses the first scene label information and the second scene label information to determine a scene of the video to be recognized. ​ ​ ​ A scene recognition module fuses the first scene label information and the second scene label information to determine a scene of the video to be recognized. 10.The system of claim 9, wherein in a model training phase, both the time-series feature learning module and the non-time-series feature learning module participate in model training. 11.The system of claim 9, wherein in an inference testing phase, only the time-series feature learning module participates in inference testing. 12.A computer-readable storage medium storing instructions that, when executed, cause a machine to perform the method of any one of claims 1-8.

Images