Video content detection methods

Abstract

This invention provides a video content detection method, comprising: in response to receiving target text input from an object, generating a text scene graph based on the target text, the text scene graph including the action to be detected and the target object related to the detection intent; acquiring a video scene graph of the video to be detected; wherein the video scene graph is used to represent the temporal information of the action of a first object in the video to be detected; and processing the text scene graph and the video scene graph using a video content detection model to generate a target detection result. The target detection result indicates whether the video to be detected includes the action to be detected of the target object. This video content detection method can accurately locate target objects and their related actions in complex multi-object video scenes, effectively avoiding false detections and missed detections.

Description

Technical Field

[0001] This invention relates to the field of video content recognition, and more specifically, to a video content detection method. Background Technology

[0002] In today's era of rapid digital information dissemination, video data has become a crucial carrier for information dissemination and sharing. With the increasing prevalence of social media and video platforms, vast amounts of videos are uploaded, distributed, and shared. However, effectively detecting video content and preventing the leakage of critical information during this distribution and sharing process has become a critical issue in the information society. Existing video content detection methods, however, struggle to dynamically identify content based on users' evolving detection needs. Summary of the Invention

[0003] In view of this, the present invention provides a video content detection method, comprising: in response to receiving target text input by an object, generating a text scene graph based on the target text, the text scene graph including a detectable action and a target object related to the detection intent; acquiring a video scene graph of a video to be detected; wherein the video scene graph is used to represent the action sequence information of a first object in the video to be detected; processing the text scene graph and the video scene graph using a video content detection model to generate a target detection result; wherein the target detection result indicates whether the video to be detected includes the detectable action of the target object.

[0004] According to an embodiment of the present invention, the video content detection model includes a video extraction module, a text extraction module, and a detection module; the video content detection model processes the text scene graph and the video scene graph to generate target detection results, including: processing the video scene graph and the video to be detected using the video extraction module to generate multimodal features for describing the action changes of a first object; processing the text scene graph using the text extraction module to generate semantic features for describing the action to be detected; and generating target detection results based on the similarity between the multimodal features and the semantic features using the detection module.

[0005] According to an embodiment of the present invention, the multimodal features include frame fusion features and action fusion features; the video extraction module includes a visual extraction submodule, a semantic extraction submodule, and a convolution submodule; the video extraction module processes the video scene graph and the video to be detected to generate multimodal features describing the action changes of the first object, including: using the semantic extraction submodule to semantically process the video scene graph to obtain the category features of the first object and the category features of the action performed by the first object in each video frame of the video to be detected; using the visual extraction submodule to process the video scene graph and the video to be detected to obtain the visual features of the first object and the visual features of the action performed by the first object in each video frame of the video to be detected; for each video frame, concatenating the category features and visual features of the first object to obtain object fusion features; using the convolution submodule to perform a spatiotemporal graph convolution operation on the object fusion features to obtain frame fusion features; for each video frame, concatenating the category features and visual features of the action performed by the first object to obtain action fusion features.

[0006] According to an embodiment of the present invention, a visual extraction submodule processes a first object and a video to be detected in a video scene graph to obtain visual features of the first object and visual features of the actions performed by the first object. This includes: using the visual extraction submodule to extract features from video frames in the video to be detected that include the first object within the region where the first object is located, thereby obtaining visual features of the first object and visual features of the actions performed by the first object. According to an embodiment of the present invention, semantic features include category features of the target object and category features of the actions to be detected; a text extraction module processes a text scene graph to generate semantic features describing the actions to be detected, including: using the text extraction module to semantically process the text scene graph to obtain category features of the target object and category features of the actions to be detected.

[0007] According to an embodiment of the present invention, a target detection result is generated by utilizing the detection module based on the similarity between multimodal features and semantic features, including: using the detection module to obtain node matching score, frame matching score and video matching score based on multimodal features and semantic features respectively; comparing the average value of node matching score, frame matching score and video matching score with a first threshold, and determining the target detection result based on the comparison result.

[0008] According to an embodiment of the present invention, the detection module includes a node matching submodule, a frame matching submodule, and a video matching submodule. The node matching score is determined as follows: For a single video frame, the node matching submodule calculates a first initial similarity between the category features of each target object and the frame fusion features, and determines the maximum value of the first initial similarity as the object similarity corresponding to each target object; the object similarities of multiple target objects are summed and averaged to obtain the first similarity of the video frame; For a single video frame, the node matching submodule calculates a second initial similarity between the category features of each action to be detected and the action fusion features of each first object on the video frame, and determines the maximum value of the second initial similarity as the action similarity; the action similarities of multiple actions to be detected are summed and averaged to obtain the second similarity of the video frame; the first similarity and the second similarity of the same video frame are added together to obtain the node-level similarity of the corresponding video frame, and the largest node-level similarity among the node-level similarities of multiple video frames is determined as the node matching score.

[0009] According to an embodiment of the present invention, the frame fusion sub-features corresponding to each first object in a single video frame are weighted and aggregated to obtain the first graph-level features of the single video frame; the action fusion features corresponding to each action in a single video frame are weighted and aggregated to obtain the second graph-level features of the single video frame; the first graph-level features and the second graph-level features of the same video frame are concatenated to obtain the object graph-level features; the category features of multiple target objects are weighted and fused to obtain the third graph-level features; the category features of multiple actions to be detected are weighted and fused to obtain the fourth graph-level features; the third graph-level features and the fourth graph-level features are concatenated to obtain the text graph-level features; the similarity scores between the text graph-level features and the multiple object graph-level features are determined respectively to obtain the frame matching similarity scores of the multiple video frames, and the largest frame matching similarity score is taken as the frame matching score.

[0010] According to an embodiment of the present invention, the video matching submodule performs average pooling on the object graph-level features of multiple video frames in the video to be detected to obtain the global features of the object; the similarity between the global features of the object and the text graph-level features is calculated to obtain the video matching score.

[0011] According to an embodiment of the present invention, when the target detection result indicates that the video to be detected includes the target object's action, the frame matching score is used to enhance the object graph-level features of the video frame to which the frame matching score belongs, thereby obtaining enhanced features; the global features of the object and the text graph-level features are concatenated to obtain an action vector; the action vector is concatenated with the enhanced features to obtain a video frame sequence; the video frame sequence is input into a recurrent neural network to output the start and end times of the target object performing the action to be detected in the video to be detected.

[0012] Limiting the actions of the first object to the target text allows users' search intent to cover more custom actions, enabling video content detection to go beyond the limited actions specified by the model. This invention's method detects whether a video contains the action to be detected through a model, reducing manual review of video content and improving video detection efficiency. In complex multi-object video scenarios, it can accurately locate the target object and its related actions, effectively avoiding false positives and false negatives. Simultaneously, by processing the video scene graph of the video to be detected and the text scene graph of the target text through the video content detection model, detection efficiency is greatly improved, enabling rapid screening of large numbers of videos in a short time, meeting the high requirements for real-time performance and accuracy in fields such as video review. Attached Figure Description

[0013] The above and other objects, features and advantages of the present invention will become more apparent from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0014] Figure 1 This illustration schematically shows an exemplary system architecture to which a video content detection method can be applied according to an embodiment of the present invention;

[0015] Figure 2 A flowchart illustrating a video content detection method according to an embodiment of the present invention is shown schematically.

[0016] Figure 3 The schematic diagram illustrates the principle of a video content detection method according to an embodiment of the present invention;

[0017] Figure 4 The schematic diagram illustrates the principle of a video extraction module according to an embodiment of the present invention;

[0018] Figure 5 The schematic diagram illustrates the principle of a convolutional submodule according to an embodiment of the present invention;

[0019] Figure 6 This diagram illustrates a data processing flowchart of a video content detection method according to an embodiment of the present invention.

[0020] Figure 7 A block diagram of a video content detection apparatus according to an embodiment of the present invention is schematically shown; and

[0021] Figure 8 A block diagram of an electronic device suitable for video content detection according to an embodiment of the present invention is shown schematically. Detailed Implementation

[0022] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0023] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0024] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0025] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0026] In the embodiments of this invention, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to maintain the security of user personal information and network security.

[0027] In the embodiments of the present invention, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0028] Currently, in the field of video content detection, the demand for content detection in videos is increasing. However, existing methods for video content detection typically predefine the objects to be detected before detection, and then simply determine whether the content to be detected exists in the video by detecting specific objects, such as faces or license plates. These methods have many shortcomings, for example:

[0029] While the methods described above can identify explicit content in videos to some extent, some information may not be directly presented through the visual features of a particular object, but rather through the relationships between objects or changes in their actions. Therefore, there is a lack of effective means to detect implicit and complex content. Implicit and complex content can include behaviors or social relationships. Moreover, current technologies generally rely on manual review for the detection of complex content, which is inefficient and prone to false positives and false negatives.

[0030] On the one hand, the above methods are generally only applicable to the detection of a single object, ignoring the content of interactions between objects in multiple dimensions that may be contained in the video. The spatial positional relationship of multiple objects and the video content of interaction changes in the temporal dimension may cause information leakage.

[0031] On the other hand, the above methods can generally only perform detection based on the detection requirements specified in the detection model. However, different users have different requirements for the content to be detected, which often makes it difficult to perform flexible and varied detection and meet the dynamic detection needs of users.

[0032] In view of this, embodiments of the present invention provide a video content detection method, comprising: in response to receiving target text input by an object, generating a text scene graph based on the target text, the text scene graph including a detectable action and a target object related to the detection intent; acquiring a video scene graph of a video to be detected; wherein the video scene graph is used to characterize the action sequence information of a first object in the video to be detected; processing the text scene graph and the video scene graph using a video content detection model to generate a target detection result; wherein the target detection result indicates whether the video to be detected includes the detectable action of the target object.

[0033] The method of this invention can limit the actions of the first object based on the target text, and can cover more custom actions according to actual conditions; thus meeting the dynamic detection needs of users. By using a model to detect whether a video contains the action to be detected, it reduces the chances of missed or false detections that can occur with manual detection, thereby improving video detection efficiency.

[0034] Figure 1 The illustration schematically depicts an exemplary system architecture to which a video content detection method can be applied according to embodiments of the present invention. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to embodiments of the present invention, in order to help those skilled in the art understand the technical content of the present invention, but do not mean that embodiments of the present invention cannot be used in other devices, systems, environments or scenarios.

[0035] like Figure 1As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0036] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social media platform software, etc. (for example only).

[0037] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0038] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0039] It should be noted that the video content detection method provided in this embodiment of the invention can generally be executed by server 105. Correspondingly, the video content detection system provided in this embodiment of the invention can generally be located in server 105. The video content detection method provided in this embodiment of the invention can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the video content detection device provided in this embodiment of the invention can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Alternatively, the video content detection method provided in this embodiment of the invention can also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or by other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the video content detection device provided in the embodiments of the present invention may also be disposed in the first terminal device 101, the second terminal device 102 or the third terminal device 103, or disposed in other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103.

[0040] For example, the video to be detected can be stored in any one of the first terminal device 101, the second terminal device 102, or the third terminal device 103 (e.g., the first terminal device 101, but not limited thereto), or it can be stored on an external storage device and imported into the first terminal device 101. Then, the first terminal device 101 can locally execute the video content detection method and location positioning method provided in the embodiments of the present invention, or send the video to be detected to other terminal devices, servers, or server clusters, and have the other terminal devices, servers, or server clusters that receive the video to be detected execute the video content detection method and location positioning method provided in the embodiments of the present invention.

[0041] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0042] Figure 2 A flowchart illustrating a video content detection method according to an embodiment of the present invention is shown.

[0043] like Figure 2 As shown, the video content detection method 200 includes operations S210 to S230.

[0044] In operation S210, in response to receiving target text as object input, a text scene graph is generated based on the target text. The text scene graph includes the action to be detected and the target object related to the detection intent.

[0045] According to an embodiment of the present invention, the aforementioned object is a user with video content detection needs. The target text is text information written by the user based on their detection intent for the video to be detected. In this embodiment, a text scene graph model can be used to transform the target text input by the user into a text scene graph, which can represent the action to be detected and the target object that the user wants to detect.

[0046] According to embodiments of the present invention, the user's detection intent may be: the user wants to detect whether an embrace occurs in the video to be detected, or the user wants to detect whether a man and a woman embrace in the video to be detected, or the user wants to detect whether a man and a woman gaze at each other in the video to be detected. Both the man and the woman can be considered as target objects, and the embrace and gaze can be considered as actions to be detected. That is, in this application, the detection intent can be both the action to be detected and the target object.

[0047] In operation S220, a video scene graph of the video to be detected is obtained; wherein, the video scene graph is used to represent the action timing information of the first object in the video to be detected.

[0048] According to an embodiment of the present invention, a video scene graph generation model can be used to convert the video to be detected into a video scene graph.

[0049] According to an embodiment of the present invention, the action timing information of the first object included in the above-described video scene diagram is the action change of the first object or the action change between two or more first objects. For example, the two first objects can be a man and a woman, and the action change between the two first objects can be that they change from looking at each other to dancing or from looking at each other to embracing.

[0050] According to embodiments of the present invention, before detection, video scene diagrams can be generated in advance for multiple videos to be detected, so that the video scene diagrams can be directly used for detection after the user inputs the target text. Alternatively, video scene diagrams can be temporarily generated based on the current videos to be detected, so as to perform video content detection based on the target text.

[0051] In operation S230, the text scene graph and video scene graph are processed using a video content detection model to generate target detection results; the target detection results indicate whether the video to be detected contains the target object's action.

[0052] According to embodiments of the present invention, the text scene graph obtained in operation S210 and the video scene graph generated in operation S220 can be input into the video content detection model for comparison, thereby obtaining a target detection result. The target detection result can indicate whether the video to be detected includes the action to be detected. For example, if the user inputs the action of embracing in the target text, and an image of embracing appears in the video to be detected, it will be detected. Furthermore, if the user inputs the action to be detected in the target text, it can also be identified in the video to be detected. For example, if the user inputs the action of embracing a man and a woman, the target detection result can indicate whether an image of a man and a woman embracing appears in the video to be detected.

[0053] According to embodiments of the present invention, the target detection result can also indicate whether the video to be detected includes a target object changing from a first detected action to a second detected action. If the user inputs a change in the detected action between the detected objects in the target text, it can also be identified in the video to be detected. For example, if the user inputs that a man and a woman first look at each other and then hug, the target detection result can indicate whether there is a scene in the video to be detected where a man and a woman change from looking at each other to hugging.

[0054] According to an embodiment of the present invention, the video content detection model is trained based on sample text and sample videos used to describe the action information of sample objects in sample videos. When preparing training data, videos from a public dataset can be used as video samples, and then sample text can be written based on the video content. The sample text can be divided into two types: positive samples that accurately describe the action changes of objects in the video samples, and negative samples that are completely unrelated to the action changes of objects in the video samples. The collected sample videos and sample texts are divided into training, validation, and test sets in an 8:1:1 ratio and stored for easy retrieval during subsequent training of the video content detection model.

[0055] During training, the model can be evaluated using a validation set in a timely manner, calculating metrics such as accuracy, recall, and F1 score. Based on the evaluation results, analyze any problems with the model, such as overfitting or underfitting. If overfitting occurs, add regularization terms or use Dropout techniques; if underfitting occurs, adjust the video content detection model architecture, increasing the number of network layers or adjusting hyperparameters. Once the performance of the video content detection model on the validation set has stabilized, use a test set to comprehensively evaluate the finally trained model, obtaining its generalization performance metrics to ensure that the model can accurately detect video content based on sample text and sample videos in practical applications.

[0056] According to embodiments of the present invention, defining the actions of the first object based on the target text allows the user's search intent to cover more custom actions, enabling video content detection to go beyond the limited actions specified by the model. The method of the present invention detects whether a video contains the action to be detected through a model, reducing manual review of video content and improving video detection efficiency. In complex multi-object video scenarios, it can accurately locate target objects and their related actions, effectively avoiding false detections and missed detections. Simultaneously, through automated model processing, detection efficiency is greatly improved, enabling rapid screening of large amounts of video in a short time, meeting the high requirements for real-time performance and accuracy in fields such as security monitoring and video review, providing strong technical support for related industries, and possessing broad application value and good social benefits.

[0057] According to an embodiment of the present invention, the video content detection model includes a video extraction module, a text extraction module, and a detection module; the video content detection model processes the text scene graph and the video scene graph to generate target detection results, including: processing the video scene graph and the video to be detected using the video extraction module to generate multimodal features for describing the action changes of a first object; processing the text scene graph using the text extraction module to generate semantic features for describing the action to be detected; and generating target detection results based on the similarity between the multimodal features and the semantic features using the detection module.

[0058] The following is for reference. Figure 3 In conjunction with specific embodiments, Figure 2 The method shown will be further explained.

[0059] Figure 3 The illustration shows a schematic diagram of the principle of a video content detection method according to an embodiment of the present invention.

[0060] According to embodiments of the present invention, the aforementioned multimodal features can be fused features that integrate visual information features and textual information features. Multimodal features can more comprehensively display information about the changes in the action of the first object. Semantic features can represent the specific information expressed in the target text output by the user. Converting the action to be detected into semantic features is to facilitate the calculation of the similarity between multimodal features and semantic features.

[0061] According to embodiments of the present invention, when generating a Text Scene Graph (TSG), the target object can be treated as a node, and the relationships between the nodes of the target object can be connected as relation edges. For example, a text scene graph generation model can be used to parse the target text input by the user and generate the corresponding text scene graph. .in, A collection of nodes representing the target object in the text. It can represent the node of the first target object in the target text. It can represent the node of the second target object in the target text. This can represent the total number of nodes representing the target object contained in the target text, as shown below. The meaning remains unchanged. This represents the set of relation edges between the nodes of the target object. It can represent the relation edges of the node of the first target object in the target text. It can represent the relation edge of the node of the second target object in the target text. This can represent the total number of relation edges between nodes of the target object included in the target text, as shown below. The meaning remains unchanged. The above... It can represent the target text identifier, appearing in the following other locations. The meaning remains unchanged. The nodes and relation edges used to represent the target object are obtained from the target text.

[0062] To more clearly describe all objects in the embodiments of the present invention, the target object / first object will be used instead of the node of the target object / first object below.

[0063] For example, in the target object middle, This represents the ID of the target object, which corresponds to a word in the target text. Relationship edges for each target object. Represented as a triplet. ,in, Can be the target object With the target object Relationship predicates. It can represent the number of the target object.

[0064] According to embodiments of the present invention, when generating a Video Scene Graph (VSG), static scene graphs for corresponding video frames can be generated first, and then the static scene graphs can be correlated along time to obtain the video scene graph. For example, a scene graph model can be used to generate a static scene graph for each video frame of the video to be detected. In static scene diagrams In the table, the node set of the first object is... , ;in, It can represent video frames. The first in The first object's node. This refers to the video frame number, which appears in the following other locations. The meaning remains unchanged. and All of these are the numbers of the first object. and They can represent nodes of different first objects. A node that can represent the first object Visual vectors, The node can be the first object Category tags, The node can be the first object The bounding box coordinates in the video frame. It can represent video frames. The first total number of objects in the following... The meaning remains unchanged.

[0065] In static scene image In the first object's set of relation edges , Among them, in video frames The relational predicate of the first object The node representing the first object and the node of the first object The interaction between them.

[0066] According to an embodiment of the present invention, after obtaining a static scene diagram, video frames in the video to be detected... The first object And another video frame The first object in Form the first object pair ( , The score is determined by the connection between the two first objects in the first object pair. To establish cross-frame co-reference relationships, the calculation method for the connection score between the first objects is based on formula (1):

[0067]

[0068] in, Cosine similarity is used to measure the visual similarity between two objects. Representing video frames respectively The first object and the first object The visual vector. Representing video frames respectively The first object and the first object The bounding box coordinates. The intersection-over-union ratio (IoU) represents the bounding boxes of two first objects. Spatial location weight parameters This is used to control the impact of the spatial location of the two first objects on the connection score, and the temporal distance between the two first objects is also considered when calculating the connection score. The influence of the spatial location of the two first objects is limited, meaning that the connection score of first objects that are far apart in time is mainly determined by the feature similarity of the visual vectors of the two first objects, rather than the degree of spatial overlap.

[0069] According to an embodiment of the present invention, based on obtaining multiple connection scores, for the first object Select connections with a connection score greater than the connection score threshold. Another first object Perform connections and form a set of time-copied edges by connecting all the first object connections across frames. .in, Represents the first object and the first object The relationship between the two sides.

[0070] According to an embodiment of the present invention, the temporal common edges of adjacent frames in the set of temporal common edges are combined with a static scene graph to obtain a video scene graph. .in, The total number of frames in the video to be tested; the following other locations appearing... The meaning remains unchanged. and These are static scene images from frames 1 and 2, respectively. This refers to the time common edge from frame 1 to frame 2. This refers to the common index edge between frames 2 and 3. In practical applications, the spatial location weight parameter... It can be 0.5, the connection score threshold. It can be 0.7.

[0071] like Figure 3As shown, a video scene graph generation model is used to generate a video scene graph 302 from the video to be detected 301. The video extraction module 310 processes the video to be detected 301 and the video scene graph 302 to generate multimodal features 303, which are used to describe the action changes of the first object. Since the video scene graph is in the form of text information, it is also necessary to combine it with the video frames in the video to be detected for feature extraction. For example, the video scene graph can show that the current frame includes a man and a woman hugging through its text features. The video extraction module can extract the visual features of the man and woman hugging in the corresponding video frame, and then combine the visual features with the text features displayed in the video scene graph to form the multimodal features 303. The information contained in the multimodal features 303 is richer than the simple text information in the video scene graph. For the cross-frame action information displayed in the text information contained in the video scene graph, the video extraction module can also extract the visual features.

[0072] According to an embodiment of the present invention, a text scene graph 305 is obtained by using a text scene graph generation model on the target text 304. The text extraction module 320 extracts the text scene graph 305 semantically to obtain semantic features 306. The semantic features 306 may include the user-inputted action to be detected, the target object, and the action to be detected on the target object.

[0073] According to an embodiment of the present invention, for the multimodal features 303 and semantic features 306 obtained above, the detection module 330 can perform detection through semantic similarity to obtain target detection result 307. For example, the cosine similarity algorithm is used to calculate the similarity between multimodal features 303 and semantic features 306. Generally, the value of cosine similarity is between -1 and 1, and the closer the value is to 1, the higher the similarity. The detection module 330 can also set a similarity threshold for judgment. For example, if the cosine similarity obtained above is higher than a preset threshold, the probability that the action to be detected appears in the corresponding video frame is higher. The detection module 330 can also optimize the detection results that are higher than the threshold. Redundant detection results are eliminated by non-maximum suppression algorithm. For multiple overlapping and highly similar detection results, only the one with the highest score is retained as the final result. At the same time, the confidence of the detection results can be calibrated, and the similarity score can be converted into a confidence score that is more in line with the actual application scenario, and finally an accurate target detection result is generated and output.

[0074] According to embodiments of the present invention, by comparing the multimodal features obtained from the video to be detected with the semantic features in the target text, continuous actions in the video to be detected can be detected, making the detection not limited to actions within a single frame image, resulting in more comprehensive detection results and avoiding missed detections of video content.

[0075] According to an embodiment of the present invention, the multimodal features include frame fusion features and action fusion features; the video extraction module includes a visual extraction submodule, a semantic extraction submodule, and a convolution submodule; the video extraction module processes the video scene graph and the video to be detected to generate multimodal features describing the action changes of the first object, including: using the semantic extraction submodule to semantically process the video scene graph to obtain the category features of the first object and the category features of the action performed by the first object in each video frame of the video to be detected; using the visual extraction submodule to process the video scene graph and the video to be detected to obtain the visual features of the first object and the visual features of the action performed by the first object in each video frame of the video to be detected; for each video frame, concatenating the category features and visual features of the first object to obtain object fusion features; using the convolution submodule to perform a spatiotemporal graph convolution operation on the object fusion features to obtain frame fusion features; for each video frame, concatenating the category features and visual features of the action performed by the first object to obtain action fusion features.

[0076] The following is for reference. Figure 4 The feature extraction operation performed by the video extraction module 310 will be further explained in conjunction with specific embodiments.

[0077] Figure 4 The schematic diagram illustrates the principle of a video extraction module according to an embodiment of the present invention.

[0078] According to embodiments of the present invention, for example, the category feature of the first object indicates that the first object is specifically a man, a woman, or a plant, etc.; the category feature of the action performed by the first object refers to the first object dancing, making eye contact, or hugging, etc. The aforementioned category features are semantic textual information. The visual features of the first object and the visual features of the action performed by the first object are visual information extracted from the video frame images. Therefore, after feature concatenation, for example, the object fusion feature for a man includes textual and visual information describing a man. The action fusion feature for dancing describes textual and visual information about dancing.

[0079] According to an embodiment of the present invention, the semantic extraction submodule 311 is used to semantically process the video scene graph 302 to obtain the category features 401 of the first object and the category features 402 of the actions performed by the first object contained in each video frame.

[0080] For example, for a video scene graph including video frames Static scene image Using word embedding matrix to divide video frames The first object in Category tags Transformed into the category features of the first object Using word embedding matrices to divide video frames The first object's relational predicate The transformation into the category features of the action performed by the first object. ;in, and video frames One-hot encoding of the first object and relational predicate, and This is the pre-trained video word embedding matrix.

[0081] According to an embodiment of the present invention, the visual extraction submodule 312 processes the video to be detected 301 and the video scene graph 302 to obtain the visual features 403 of the first object and the visual features 404 of the action performed by the first object contained in each video frame of the video to be detected.

[0082] According to an embodiment of the present invention, specifically, the visual extraction submodule 312 is used to extract features from video frames in the video to be detected that include the first object within the area where the first object is located, thereby obtaining the visual features 403 of the first object and the visual features 404 of the actions performed by the first object.

[0083] For example, the visual extraction submodule is used based on the first object bounding box Visual features are extracted from the corresponding video frames of the video to be detected to obtain the visual features of the first object. The visual extraction submodule is used to extract the joint region based on the relation edges of the first object. Extract visual features to obtain the visual features of the action performed by the first object. .in, This indicates the visual extraction submodule. This represents the union region of the bounding boxes of the two first objects. Represents video frames The first object The bounding box. In practical applications, the dimensions of visual features can be 2048 when extracting visual features.

[0084] For each video frame, the category feature 401 and the visual feature 403 of the first object are concatenated to obtain the object fusion feature 405; the object fusion feature 405 is subjected to spatiotemporal graph convolution operation using the convolution submodule 313 to obtain the frame fusion feature 406; for each video frame, the category feature 402 of the action performed by the first object and the visual feature 404 of the action performed by the first object are concatenated to obtain the action fusion feature 407.

[0085] For example, for video frames in the video to be detected The category features of the first object and the visual features of the first object By splicing the data, we obtain the object fusion features. . It can be a pre-trained concatenation parameter matrix. It is a non-linear activation function. Convolutional submodule 313 is used to fuse object features. Perform spatiotemporal graph convolution operations to obtain frame fusion features. Similar to the calculation method for object fusion features, for video frames in the video to be detected... The category features of the actions performed by the first object and visual features By splicing the data, motion fusion features are obtained. The non-linear activation function can be either the ReLU function or the Sigmoid function.

[0086] According to an embodiment of the present invention, the multimodal feature 303 may include the frame fusion feature obtained in the above steps. and motion fusion features .

[0087] The following is for reference. Figure 5 The spatiotemporal graph convolution operation performed by the convolution submodule 313 will be further explained in conjunction with specific embodiments.

[0088] Figure 5 The schematic diagram illustrates the principle of a convolutional submodule according to an embodiment of the present invention.

[0089] According to an embodiment of the present invention, for the above-mentioned spatiotemporal graph convolution operation, the object fusion feature 405 can be convolved by the spatial graph convolution unit 3131 and the temporal graph convolution unit 3132 respectively to obtain the spatial feature 501 and the temporal feature 502; then the spatial feature 501 and the temporal feature 502 are summed to obtain the frame fusion feature 406.

[0090] For example, when updating features in a video scene graph using a graph convolutional network, an adjacency matrix of spatial relationships can be defined to capture the visual relationships between the first objects within a single video frame. Identifiers that can represent spatial relationships. This indicates the video frame number. For video frames in the video to be detected... Static scene image If the first object is paired with Predicate relation of the first object If they are connected, then the corresponding elements in the adjacency matrix are... ,otherwise For the above video frames adjacency matrix Each row in the formula is normalized. The normalization process is described in formula (2).

[0091]

[0092] in, This represents the first object pair. The elements in the corresponding adjacency matrix.

[0093] For example, when using a graph convolutional network to update features in a video scene graph, to obtain the dynamic changes of the first object in adjacent frames over time, multiple adjacency matrices of the first object in the time dimension can be defined. , It can be an identifier that indicates a time relationship.

[0094] Based on the above set of time co-pointing edges Assign values ​​to the adjacency matrix in the time dimension, when video frames... The first object in and its adjacent next video frame The first object in When representing the same first object, the elements of the corresponding adjacency matrix are... Set to 1 otherwise to 0. The same applies to the adjacency matrix. Normalize each row in the formula, and refer to formula (3) for the normalization process:

[0095]

[0096] in, It can represent the first The total number of the first objects included in the frame. This indicates when the video frame The first object in and its adjacent next video frame The first object in The elements in the corresponding adjacency matrix.

[0097] For example, video frames can be... The feature matrix composed of the object fusion features of all first objects is denoted as ,in, For the transpose operator, appear in the following other positions. The meaning remains unchanged. For the feature dimension of the first object, Represents the first object Object fusion features. The process of performing spatial graph convolution operation is referenced in formula (4); the process of performing temporal graph convolution operation is referenced in formula (5):

[0098]

[0099]

[0100] in, Can be The pre-trained spatial convolutional graph parameter matrix of the layer, Can be The pre-trained temporal graph convolution parameter matrix of the layer is used in the temporal graph convolution operation in the form of video frames. The result of the previous time graph convolution is used as input for time graph convolution operation. It can be the number of the convolutional layers. Represents a non-linear activation function. For video frames The The feature matrix after convolution of the layer space graph. For video frames +1 of The feature matrix after convolution of the layer time sequence graph.

[0101] For example, in order to integrate feature representations of spatial and temporal relationships, video frames are... The The result of spatial graph convolution operation of layers and video frames The The result of the temporal graph convolution operation of the layer Summing the results, we can obtain the first... Mid-video frames after convolution Feature matrix :Graphic convolution feature representation The calculation reference formula (6) is as follows:

[0102]

[0103] According to an embodiment of the present invention, after After spatiotemporal graph convolution of the layers, video frames The set of feature representations of all first objects after spatiotemporal graph convolution is called frame fusion feature. ,in, Let be the feature dimension of the Lth layer graph convolutional network. For video frames... The first object in Object fusion features The features obtained after spatiotemporal graph convolution can be counted as frame fusion sub-features. In practical applications, the spatiotemporal feature update of video scene graphs employs a two-layer graph convolutional network, namely... Feature dimensions of spatial graph convolution and temporal graph convolution Both can be set to 512.

[0104] According to embodiments of the present invention, frame fusion features obtained using spatiotemporal graph convolution can capture the dynamic changes of the same first object between video frames, effectively characterizing complex and implicit actions in the video to be detected. When generating object fusion features and action fusion features, both semantic and visual aspects are used to describe the first object and the actions performed by the first object, enabling more accurate detection of video content based on target text.

[0105] According to an embodiment of the present invention, the semantic features include the category features of the target object and the category features of the action to be detected; the text scene graph is processed by the text extraction module to generate semantic features for describing the action to be detected, including: the text scene graph is semantically processed by the text extraction module to obtain the category features of the target object and the category features of the action to be detected.

[0106] According to an embodiment of the present invention, in order to achieve a unified cross-modal representation of the video to be detected and the text scene graph, a text extraction module is used to process the text scene graph. The target object and its relation edges are encoded with features, and a bidirectional GRU operation is performed on the resulting feature codes to obtain semantic features, which include the category features of the target object and the category features of the action to be detected.

[0107] For example, using a pre-trained text word embedding matrix For each target object Encode to obtain the initial embedding vector representation. ,in Represents the target object The one-hot encoded vectors of the corresponding words in the target text, and the text word embedding matrix. An initialization method consistent with the video word embedding matrix is ​​adopted.

[0108] For example, a neural network can be used to encode features of the sequence path of the target object formed by the original word order, obtaining the forward and backward hidden state representations of the target object. A Bi-GRU network can be used as the neural network. Target object The forward hidden state representation can be ; Represents the target object The hidden state forward is represented as, This indicates a forward GRU operation. Target object. The backward hidden state representation can be ; Represents the target object Backward hidden state representation, This represents a backward GRU operation. The target object is obtained by calculating the average of the hidden states in both the forward and backward directions. Category features .

[0109] For example, for the set of relation edges of the target object in the text scene graph Using the method described above, target objects are formed into triplet forms. Target object and relational predicates The corresponding word embedding features are respectively represented as The embedding vector sequence of triple nodes is used as input for feature encoding. Finally, the hidden state output by the Bi-GRU network is used, and the average value of the hidden state is taken to obtain the category feature of the action to be detected. Category features of the action to be detected The calculation method is based on formula (7):

[0110]

[0111] in, This can represent the sequential concatenation of word embedding features of the target object. In practical applications, the dimension of the hidden layers of the Bi-GRU network can be uniformly set to 512.

[0112] According to an embodiment of the present invention, the text extraction module decomposes the target text into target objects and actions to be detected, so that video content detection can be performed from two aspects: target object detection and target action detection, thereby improving the accuracy of detection.

[0113] According to an embodiment of the present invention, for a single video frame, a first initial similarity is calculated between the category features and frame fusion features of each target object using a node matching submodule, and the maximum value of the first initial similarity is determined as the object similarity corresponding to each target object; the object similarities of multiple target objects are summed and averaged to obtain the first similarity of the video frame; for a single video frame, a second initial similarity is calculated between the category features of each action to be detected and the action fusion features of each first object on the video frame using a node matching submodule, and the maximum value of the second initial similarity is determined as the action similarity; the action similarities of multiple actions to be detected are summed and averaged to obtain the second similarity of the video frame; the first similarity and the second similarity of the same video frame are added together to obtain the node-level similarity of the corresponding video frame, and the largest node-level similarity among the node-level similarities of multiple video frames is determined as the node matching score.

[0114] According to an embodiment of the present invention, for a single video frame, a first initial similarity can represent the similarity between each target object and multiple first objects, where the largest can be used as the object similarity, and the average of the object similarities of multiple target objects is taken to obtain the first similarity. A second initial similarity can represent the similarity between each action to be detected and the actions performed by multiple first objects, where the largest can be used as the action similarity, and the average of the action similarities of multiple actions to be detected is taken to obtain the second similarity. Since node-level similarity combines the first and second similarities, the node matching score obtained through node-level similarity can detect the entire video to be detected from both object and action perspectives. The method for obtaining the node matching score is described in detail below with reference to embodiments.

[0115] For example, the video frames of the video to be detected can be calculated according to the steps described above. Multimodal features, including frame fusion features. and motion fusion features Semantic features can be obtained from the target text, including the category features of the target object. and category features of the action to be detected Multimodal and semantic features are used for granular node matching to obtain node matching scores.

[0116] For example, for each target object, its relationship with the video frame is calculated. Frame fusion sub-features corresponding to all first objects in Frame fusion features The first initial similarity is calculated, and the maximum value of the first initial similarity is determined as the object similarity corresponding to each target object; the object similarities of multiple target objects are summed and averaged to obtain the video frame. First similarity First similarity The calculation reference formula (8) is as follows:

[0117]

[0118] in, This indicates the number of the target object in the target text. It can represent the total number of target objects in the target text.

[0119] For example, for each target object's relation edge The corresponding category features of the action to be detected With video frames Action fusion features of each first object The second initial similarity between the two actions is calculated, and the maximum value of the second initial similarity is determined as the action similarity. The action similarities of multiple actions to be detected are summed and averaged to obtain the video frame. Second similarity Second similarity The calculation reference formula (9) is as follows:

[0120]

[0121] in, It can represent video frames. The total number of relation edges of the first object in the array. It can represent the total number of relation edges of the nodes of the target object included in the target text.

[0122] For example, video frames First similarity Second similarity Add them together to get the video frames. Node-level similarity Finally, all videos in the testing video were... The node matching score is obtained by finding the maximum value among the node-level similarities of each video frame. = , Indicates the node matching identifier.

[0123] According to embodiments of the present invention, the first similarity score effectively focuses on the feature information in the video frame that best matches the target object, avoiding matching errors caused by confusion of features from multiple objects. For example, in a video frame containing various vehicles, such as cars, bicycles, and motorcycles, this method can accurately identify the part with the highest matching degree to the target vehicle category features, highlighting key object features, so that the first similarity score can accurately reflect the overall matching degree of the target object in the video frame. The second similarity score can accurately capture the features that best match the action to be detected from complex video action scenes, improving the accuracy of action matching. For example, in sports event videos, for the action to be detected, such as "shooting," it can accurately identify the moment with the highest matching degree to the "shooting" feature among various player action postures, so that the second similarity score can effectively measure the matching situation of actions in the video frame. Selecting the maximum node-level similarity score as the score ensures that the result selected is the most representative of the matching degree of the video frame, providing an accurate data foundation for subsequent in-depth analysis and decision-making based on video content.

[0124] According to an embodiment of the present invention, the frame fusion feature includes a frame fusion sub-feature corresponding to each first object, and the frame matching score is determined by: weighted aggregation of the frame fusion sub-features corresponding to each first object in a single video frame to obtain a first graph-level feature of the single video frame; weighted aggregation of the action fusion features corresponding to each action in a single video frame to obtain a second graph-level feature of the single video frame; concatenation of the first graph-level feature and the second graph-level feature of the same video frame to obtain an object graph-level feature; weighted fusion of the category features of multiple target objects to obtain a third graph-level feature; weighted fusion of the category features of multiple actions to be detected to obtain a fourth graph-level feature; concatenation of the third graph-level feature and the fourth graph-level feature to obtain a text graph-level feature; determination of the similarity score between the text graph-level feature and the multiple object graph-level features to obtain the frame matching similarity score of each of the multiple video frames, and taking the largest frame matching similarity score as the frame matching score.

[0125] According to an embodiment of the present invention, the first graph-level feature is the feature of each first object after weighted aggregation, which can represent the features of all first objects in a single video frame; similarly, the second graph-level feature can represent the features of the actions performed by all first objects in a single video frame. The object graph-level feature obtained by concatenating the first graph-level feature and the second graph-level feature can represent the overall features of the first object itself and its actions. Similarly, the text graph-level feature can represent the overall features of the target object and the action to be detected. Thus, the frame matching score can detect whether the content to be detected appears in the video to be detected as a whole. The method for obtaining the frame matching score is described in detail below with reference to the embodiments.

[0126] According to an embodiment of the present invention, the frame fusion sub-feature is the video frame. Corresponding frame fusion features The frame fusion sub-features corresponding to all first objects included in the data .

[0127] According to an embodiment of the present invention, multimodal features are input into the graph embedding layer to obtain object graph-level features. The semantic features are input into the graph embedding layer to obtain text graph-level features. , object graph-level features Text graph-level features Frame granularity matching is performed to obtain a frame matching score. The following detailed description is provided in conjunction with embodiments of the present invention.

[0128] According to an embodiment of the present invention, the frame fusion sub-features corresponding to each first object in a single video frame are weighted and aggregated to obtain the first graph-level features of the single video frame.

[0129] For example, to obtain video frames First-level features Multi-scale attention is used to fuse the sub-features of the first object in the frame. Aggregation is performed. First-level graph features. The calculation method is based on formula (10):

[0130]

[0131] in, It can represent video frames. The first in Attention weights for the first object, The calculation method can be found in formula (11):

[0132]

[0133] in, It can represent an exponential function. It can represent the first object. Frame fusion sub-features, Represents video frames The number of the first object in the middle, It can be a multi-scale attention mapping function.

[0134] According to an embodiment of the present invention, for video frames Action fusion features corresponding to each action Perform weighted aggregation to obtain video frames. Second-level features .

[0135] For example, with the first graph-level features Similarly, second-level features The calculation method can be found in formula (12):

[0136]

[0137] in, For video frames The total number of relation edges in the first object. It can represent the first The relation edge of the first object Attention weights Indicates the number of the relation edge of the first object. The calculation method can be found in formula (13):

[0138]

[0139] in, Represents video frames The first object pair in the middle ( , Action fusion features Indicates the number of the relation edge of the first object.

[0140] For example, video frames First-level features Second-level features By concatenating the features, we obtain the object graph-level features. = . Graph-level features of the computational objects mentioned above. Similarly, it is possible to analyze the category features of multiple target objects in the target text. After weighted fusion, the third-level features are obtained. ; Category features of multiple actions to be detected in the target text After weighted fusion, the fourth-level features are obtained. ; third-level features and fourth-level features By concatenating the features, we obtain the text graph-level features. .

[0141] According to embodiments of the present invention, text graph-level features are determined respectively. Object graph-level features corresponding to each video frame The similarity scores between the frames are used to obtain the frame matching similarity scores for each of the multiple video frames. Frame matching similarity score The calculation method is based on formula (14):

[0142]

[0143] in, Indicates the frame match identifier.

[0144] For example, the highest frame matching similarity score is finally used as the frame matching score. , Indicates the frame match identifier.

[0145] According to an embodiment of the present invention, the frame matching score is used to represent the semantic similarity between the text scene graph obtained from the target text and the first object and the overall actions performed by the first object in each frame of the video to be detected. Assigning weights based on the importance of different first objects in the video scene highlights key first object features and effectively avoids interference from secondary first objects, making the obtained first graph-level features more representative and accurately reflecting the core features of the first object in a single video frame. Similarly, the second graph-level features obtained by weighted aggregation of action fusion features can also focus on key action information and enhance the expressive power of action features. Text graph-level features and object graph-level features encode the video frame content and text semantics from different perspectives. By calculating the similarity score between them, the potential correlation between video frames and text descriptions can be deeply explored. Using the highest frame matching similarity score as the frame matching score ensures that the selected score best reflects the degree of matching between the two, effectively improving the accuracy of matching video frames and text descriptions. Compared to traditional frame matching methods, this approach can more accurately locate video frames that match the target text description in complex video scenes, significantly improving the accuracy and reliability of the target detection system in video analysis scenarios. It provides more accurate basic data for subsequent intelligent analysis and decision-making based on video content, enhancing the practicality and adaptability of video content detection methods.

[0146] According to an embodiment of the present invention, the video matching submodule performs average pooling on the object graph-level features of multiple video frames in the video to be detected to obtain the global features of the object; the similarity between the global features of the object and the text graph-level features is calculated to obtain the video matching score.

[0147] According to an embodiment of the present invention, the above-mentioned similarity calculation can be cosine similarity.

[0148] According to an embodiment of the present invention, the object graph-level features of multiple video frames in the video to be detected are utilized by the video matching submodule. Perform mean pooling to obtain the global features of the object. Global characteristics of the object Text graph-level features Perform video granularity matching between them to obtain a video matching score. , This is a video matching identifier. Video matching score. The calculation method is based on formula (15):

[0149]

[0150] in, This indicates the calculation of the Euclidean norm.

[0151] According to embodiments of the present invention, when performing average pooling on the object graph-level features of multiple video frames, this process can effectively integrate the local differences in the features of the first object within a single video frame, aggregate the first object information scattered across different frames, eliminate feature fluctuations caused by factors such as shooting angle and lighting changes, and thus obtain globally representative global object features. Compared to traditional video text matching methods, embodiments of the present invention globally integrate features through average pooling and calculate similarity based on the integrated features, significantly improving the accuracy and stability of video matching scores. This enables rapid and accurate filtering of video content matching text descriptions from massive amounts of video data, greatly improving the efficiency of video retrieval and analysis, providing more reliable technical support for applications such as intelligent monitoring and video recommendation, and enhancing the system's adaptability and practicality in complex real-world applications.

[0152] According to an embodiment of the present invention, a target detection result is generated by utilizing the detection module based on the similarity between multimodal features and semantic features, including: using the detection module to obtain node matching score, frame matching score and video matching score based on multimodal features and semantic features respectively; comparing the average value of node matching score, frame matching score and video matching score with a first threshold, and determining the target detection result based on the comparison result.

[0153] According to an embodiment of the present invention, when training the detection module, the range of the first threshold value can be determined. Generally, if the calculated average value is greater than the first threshold, it indicates that the video to be detected contains the content to be detected.

[0154] For example, calculate the node matching score mentioned above. Frame matching score Video matching score The average score is obtained by averaging the average values. . This is used to indicate the average score.

[0155] When detecting video content, if the average score Greater than the first threshold If the first threshold is met, it can be determined that the video contains the target object from the user-input target text; otherwise, it is determined that the video does not contain the target object. In practical applications, the first threshold... Based on the performance of the validation set, it can be set to 0.5.

[0156] According to embodiments of the present invention, averaging the three matching scores can comprehensively balance the influence of scores from different dimensions, reducing the risk of misjudgment due to deviations in scores from a single dimension. For example, when the node matching score is low, but the frame matching score and video matching score are high, averaging can prevent the overall matching situation from being overlooked due to excessive focus on deficiencies at the node level. This fusion method makes the detection results more robust and adaptable to the diversity and complexity of multimodal data.

[0157] According to an embodiment of the present invention, the video content detection method of the present invention includes: when the target detection result indicates that the video to be detected includes a target object and a target action, using frame matching score to analyze the object graph-level features of the video frame to which the frame matching score belongs. Feature enhancement is performed to obtain enhanced features; the global features of the object and the text graph-level features are concatenated to obtain the action vector. The action vector is concatenated with the enhanced features to obtain a video frame sequence. The video frame sequence is then input into a recurrent neural network, which outputs the start and end times of the target object performing the action to be detected in the video.

[0158] For example, the frame matching similarity score for each video frame can be used. As weights, the object graph-level features of each frame Perform feature enhancement on associated frames to obtain enhanced features. Enhanced features The calculation method is based on formula (16):

[0159]

[0160] in, , It can represent video frames The frame matching similarity score is mapped to The result after that, It can be used to characterize video frames. The degree of relevance to the target text. The function used for mapping can be... function.

[0161] For example, global characteristics of an object Text graph-level features By concatenating the vectors, we obtain the action vectors. ; to move the vector spliced ​​to enhanced features The front end obtains the video frame sequence. . The feature dimension representing a video frame sequence. The enhancement features of the first frame are shown in the table. This indicates the enhanced features of the second frame. The total number of frames in the video to be tested. Represents the space of real numbers.

[0162] Among them, action vector The index in the video frame sequence is 0, and the action vector It includes global relationship information between the video to be detected and the target text. If the content to be detected does not exist in the video to be detected, this node can serve as both a start and end marker to clearly indicate that no content segment to be detected exists.

[0163] According to an embodiment of the present invention, if the target detection result indicates that the video to be detected includes a target object and a target action, the above-mentioned video frame sequence is... Input two unidirectional recurrent neural networks (Long Short-Term Memory). For position prediction, a recurrent neural network is used to capture the temporal dependencies of the start and end boundaries of the segments of the action to be detected that contain the target object. The calculation method of the temporal dependencies is shown in formulas (17) and (18).

[0164]

[0165]

[0166] in, Video frames that can be represented The initial hidden state, Video frames that can be represented The initial hidden state, Video frames that can be represented End of hidden state, Video frames that can be represented End of hidden state, Indicates the starting identifier. Indicates the end of the process. Represents a video frame sequence Video frames in Its characteristics.

[0167] For example, the probability distribution of the start of content segments is predicted using two feed-forward neural networks (FFNs). Probability distribution of the ending position , The calculation method is based on formula (19). The calculation method is based on formula (20):

[0168]

[0169]

[0170] in, This represents the sequence of start and hidden states, consisting of the start and hidden states of all video frames. This represents the sequence of end-hidden states consisting of the end-hidden states of all video frames. This represents the concatenation operation of features. This represents the activation function.

[0171] For example, finally through and By obtaining the starting and ending positions of the content to be detected, the position of the video content can be predicted.

[0172] According to embodiments of the present invention, the frame matching score can reflect the degree of matching between a video frame and the target text. Based on this, feature enhancement can highlight key features related to the target object and the action to be detected, while suppressing irrelevant information. For example, when detecting a pedestrian running in the target text, features related to running, such as the pedestrian's posture and speed, can be enhanced, while irrelevant factors such as the color of the pedestrian's clothing can be weakened. This makes the enhanced features more targeted and representative, facilitating the prediction of the location of the content to be detected using neural networks.

[0173] According to embodiments of the present invention, the following references Figure 6 The data processing procedure of the present invention will be further explained.

[0174] Figure 6 This diagram illustrates a data processing flowchart of a video content detection method according to an embodiment of the present invention.

[0175] like Figure 6 As shown, the target text input by the user is: "A woman wearing a blue shirt is making eye contact with a man." In the generated text scene graph, the user's detection intent could be: the user wants to detect whether the video to be detected shows a woman wearing a blue shirt making eye contact with a man. In the text scene graph, the man, woman, and blue shirt can all be considered as target objects, while eye contact and clothing can be considered as actions to be detected.

[0176] According to an embodiment of the present invention, a video scene graph is generated based on a video to be detected. The action timing information of the first object included in the video scene graph can be the action change of the first object itself or the action change between two first objects. For example, Figure 6As shown: The two first objects can be a man and a woman, and the change in action between the two first objects can be from looking at each other to embracing. In the video scene diagram, the man, woman, and blue shirt can be the first objects, and embracing, looking at each other, and wearing clothes can be the actions performed by the first objects.

[0177] According to an embodiment of the present invention, the text extraction module is used to encode the target object and the action to be detected in the text scene graph, and the obtained feature codes are subjected to a bidirectional GRU operation to obtain semantic features. The convolution submodule is used to perform spatiotemporal graph convolution on the object fusion features of the first object to obtain frame fusion features, and the frame fusion features and the action fusion features of the action performed by the first object are used as multimodal features.

[0178] According to embodiments of the present invention, multimodal features and semantic features can be used for granular node matching to obtain node matching scores.

[0179] According to an embodiment of the present invention, multimodal features are input into the graph embedding layer to obtain object graph-level features. The semantic features are input into the graph embedding layer to obtain text graph-level features. , object graph-level features Text graph-level features Perform frame granularity matching to obtain frame matching scores.

[0180] According to an embodiment of the present invention, the object graph-level features of multiple video frames in the video to be detected are utilized by the video matching submodule. Perform mean pooling to obtain the global features of the object. Global characteristics of an object Text graph-level features Video granularity matching is performed between the two to obtain a video matching score.

[0181] According to an embodiment of the present invention, in the video content detection method, the frame matching similarity score of each video frame can be used as a weight to perform associated frame feature enhancement on the object graph-level features of each frame, thereby obtaining enhanced features and global object features. Text graph-level features By concatenating the vectors, we obtain the action vectors. . The action vector The data is concatenated to the front end of the enhanced features to obtain a video frame sequence. Finally, the video frames are input into a recurrent neural network for position prediction. Figure 6 It can be seen from the data that the start and end times of the scene when the man and woman are hugging were located in the video to be tested.

[0182] Generally, the supervision of model training is determined by a loss function. In the embodiments of this invention, the video content detection method can also use a location prediction model for prediction. The final loss function can be used when training the location prediction model. Supervision of training. Final loss function. You can refer to formula (21):

[0183]

[0184] in, For frame-level loss function, Let be the video granularity loss function. This is the cross-entropy loss function. As the first hyperparameter, This is the second hyperparameter. and This is used to weigh the three loss functions mentioned above.

[0185] According to an embodiment of the present invention, the frame granularity loss function This is used to improve the feature similarity between text scene graphs and positive sample video frames, while increasing the difference between them and negative sample video frames. To achieve this, a frame-granularity loss function with intervals can be constructed. , You can refer to formula (22):

[0186]

[0187] in, Text scene diagram With video frames Static scene diagram of positive samples Similarity score, For text scene diagram With video frames Static scene diagram of negative samples The similarity score can be calculated using cosine similarity. The interval hyperparameter for frame-level matching. By minimizing the similarity between the target text and positive sample frames in the video, while maximizing the similarity with negative sample frames, accurate detection of the action and target object is achieved. In the actual training process, the interval hyperparameter in the frame-granularity loss function... It can be set to 0.4.

[0188] According to an embodiment of the present invention, the video granularity loss function You can refer to formula (23):

[0189]

[0190] in, Global features of objects representing positive samples Text-level graph features Similarity score between them Global features of objects representing negative samples Text-level graph features The similarity score between them. The similarity score can be cosine similarity.

[0191] According to an embodiment of the present invention, a positive sample represents a video containing a target object and an action to be detected, and a negative sample represents a video that does not contain a target object and an action to be detected. By minimizing the similarity between the target text and positive sample frames in the video, while maximizing the similarity with negative sample frames, accurate detection of the action and target object is achieved. In the actual training process, the video granularity loss function interval parameter... It can be set to 0.3.

[0192] In an embodiment of the present invention, the cross-entropy loss function You can refer to formula (24):

[0193]

[0194] in, The content to be detected begins to appear in the video frame. The probability at that location. The content to be detected ends at the end of the video frame. The probability at that location. The content to be detected begins to appear in the video frame. One-hot vector labels at the location, The content to be detected ends at the end of the video frame. The unique hot vector label at the location.

[0195] In the actual training process, the first weight parameter It can be Second weighting parameter It can be The model was optimized using the Adam optimizer with an initial learning rate of 1e-4 and a batch size of 32. Training was performed for 150 epochs, and the model with the best performance on the validation set was selected for final testing.

[0196] Figure 7 A block diagram of a video content detection apparatus according to an embodiment of the present invention is shown schematically.

[0197] like Figure 7As shown, the video content detection device 700 includes a text scene graph generation module 710, a video scene graph acquisition module 720, and a target detection module 730.

[0198] The text scene graph generation module 710 is used to generate a text scene graph based on the target text received as object input in response to the target text received.

[0199] The video scene graph acquisition module 720 is used to acquire the video scene graph of the video to be detected;

[0200] The target detection module 730 is used to process the text scene graph and the video scene graph using a video content detection model to generate target detection results.

[0201] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present invention, or at least part of the functions of any one or more of them, can be implemented in a single module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present invention can be implemented by being divided into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present invention can be at least partially implemented as hardware circuits, such as field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems-on-a-chip, systems-on-a-substrate, systems-on-package, application-specific integrated circuits (ASICs), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuits, or implemented in software, hardware, and firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present invention can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0202] For example, any plurality of the text scene graph generation module 710, video scene graph acquisition module 720, and target detection module 730 can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits can be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of the present invention, at least one of the text scene graph generation module 710, video scene graph acquisition module 720, and target detection module 730 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the text scene graph generation module 710, the video scene graph acquisition module 720, and the object detection module 730 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0203] It should be noted that the video content detection device in the embodiments of the present invention corresponds to the video content detection method in the embodiments of the present invention. For a detailed description of the video content detection device, please refer to the video content detection method section, which will not be repeated here.

[0204] Figure 8 A block diagram of an electronic device suitable for implementing the methods described above, according to an embodiment of the present invention, is shown schematically. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0205] like Figure 8 As shown, an electronic device 800 according to an embodiment of the present invention includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 88 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0206] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0207] According to an embodiment of the present invention, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the input / output (I / O) interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output (I / O) interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0208] According to embodiments of the present invention, the method flow according to embodiments of the present invention can be implemented as a computer software program. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by processor 801, it performs the functions defined in the system of the embodiments of the present invention. According to embodiments of the present invention, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0209] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0210] According to embodiments of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0211] For example, according to embodiments of the present invention, a computer-readable storage medium may include the ROM 802 and / or RAM 803 described above and / or one or more memories other than ROM 802 and RAM 803.

[0212] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of the present invention. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the video content detection method provided in the embodiments of the present invention.

[0213] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this embodiment of the invention. According to embodiments of the invention, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0214] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0215] According to embodiments of the present invention, program code for executing the computer programs provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0216] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention can be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or pairings fall within the scope of this invention.

[0217] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. A video content detection method, characterized in that, The method includes: In response to receiving target text as object input, a text scene graph is generated based on the target text, the text scene graph including the action to be detected and the target object related to the detection intent; Obtain a video scene graph of the video to be detected; wherein, the video scene graph is used to represent the action timing information of the first object in the video to be detected; The video scene graph and the video to be detected are processed using the video extraction module to generate multimodal features that describe the changes in the action of the first object; The text scene graph is processed using a text extraction module to generate semantic features that describe the action to be detected; The detection module generates target detection results based on the similarity between the multimodal features and the semantic features; The target detection result indicates whether the video to be detected includes the action to be detected of the target object, and the multimodal features include frame fusion features and action fusion features; The frame fusion feature is obtained by performing a spatiotemporal graph convolution operation on the object fusion feature; for each video frame, the object fusion feature is obtained by concatenating the category feature and visual feature of the first object, and the action fusion feature is obtained by concatenating the category feature and visual feature of the action performed by the first object.

2. The method according to claim 1, characterized in that, The video extraction module includes a visual extraction submodule, a semantic extraction submodule, and a convolutional submodule; The step of using a video extraction module to process the video scene graph and the video to be detected to generate multimodal features describing the action changes of the first object includes: The semantic extraction submodule is used to semantically process the video scene graph to obtain the category features of the first object and the category features of the action performed by the first object in each video frame of the video to be detected. The visual extraction submodule is used to process the video scene map and the video to be detected to obtain the visual features of the first object and the visual features of the action performed by the first object in each video frame of the video to be detected. For each video frame, the category features and visual features of the first object are concatenated to obtain object fusion features; The convolutional submodule is used to perform a spatiotemporal graph convolution operation on the object fusion features to obtain frame fusion features; For each video frame, the category features and visual features of the action performed by the first object are concatenated to obtain action fusion features.

3. The method according to claim 2, characterized in that, The The visual extraction submodule processes the first object in the video scene graph and the video to be detected to obtain the visual features of the first object and the visual features of the actions performed by the first object, including: The visual extraction submodule is used to extract features from video frames in the video to be detected that include the first object within the area where the first object is located, thereby obtaining the visual features of the first object and the visual features of the actions performed by the first object.

4. The method according to claim 3, characterized in that, The semantic features include the category features of the target object and the category features of the action to be detected; the step of processing the text scene graph using the text extraction module to generate semantic features describing the action to be detected includes: The text extraction module is used to semantically process the text scene graph to obtain the category features of the target object and the category features of the action to be detected.

5. The method according to claim 4, characterized in that, The step of generating target detection results using the detection module based on the similarity between the multimodal features and the semantic features includes: The detection module uses the multimodal features and semantic features to obtain node matching scores, frame matching scores, and video matching scores, respectively. The average of the node matching score, the frame matching score, and the video matching score is compared with a first threshold, and the target detection result is determined based on the comparison result.

6. The method according to claim 5, characterized in that: The detection module includes a node matching submodule, a frame matching submodule, and a video matching submodule; the node matching score is determined in the following way: For a single video frame, the node matching submodule is used to calculate a first initial similarity between the category features of each target object and the frame fusion features, and the maximum value of the first initial similarity is determined as the object similarity corresponding to each target object; the object similarities of multiple target objects are summed and averaged to obtain the first similarity of the video frame; For a single video frame, the node matching submodule is used to calculate the second initial similarity between the category feature of each action to be detected and the action fusion feature of each first object on the video frame, and the maximum value of the second initial similarity is determined as the action similarity. The action similarities of multiple actions to be detected are summed and averaged to obtain the second similarity of the video frame. The first similarity and the second similarity of the same video frame are added together to obtain the node-level similarity of the corresponding video frame, and the largest node-level similarity among multiple video frames is determined as the node matching score.

7. The method according to claim 6, characterized in that: The frame fusion feature includes a frame fusion sub-feature corresponding to each of the first objects, and the frame matching score is determined in the following manner: Weighted aggregation is performed on the frame fusion sub-features corresponding to each of the first objects in a single video frame to obtain the first graph-level features of the single video frame; The motion fusion features corresponding to each action in a single video frame are weighted and aggregated to obtain the second graph-level features of the single video frame. The first and second graph-level features of the same video frame are concatenated to obtain the object graph-level features; The category features of multiple target objects are weighted and then fused to obtain the third-level graph features; The category features of multiple actions to be detected are weighted and then fused to obtain the fourth-level features; The third-level graph feature and the fourth-level graph feature are concatenated to obtain the text graph feature; The similarity scores between the text graph-level features and the multiple object graph-level features are determined respectively to obtain the frame matching similarity scores of multiple video frames, and the largest frame matching similarity score is taken as the frame matching score.

8. The method according to claim 7, characterized in that: The video matching score is determined in the following way: The video matching submodule is used to perform average pooling on the object graph-level features of multiple video frames in the video to be detected to obtain the global features of the object. The similarity between the global features of the object and the graph-level features of the text is calculated to obtain the video matching score.

9. The method according to claim 8, characterized in that, The method further includes: When the target detection result indicates that the target action is included in the video to be detected, the object graph-level features of the video frame to which the frame matching score belongs are enhanced using the frame matching score to obtain enhanced features; The action vector is obtained by concatenating the global features of the object and the graph-level features of the text. The motion vector is concatenated with the enhanced features to obtain a video frame sequence; The video frame sequence is input into a recurrent neural network, which outputs the start and end times of the target object performing the action to be detected in the video.

Images