Video processing method and device, electronic equipment and storage medium

Abstract

The present application relates to the technical field of computer vision, and discloses a video processing method and device, electronic equipment and storage medium, wherein the video processing method comprises: acquiring a target video segment, the target video segment comprising a plurality of continuous video images; performing target detection processing on the plurality of video images to identify a plurality of targets in the plurality of video images, and obtaining target detection results of the plurality of targets; constructing a space-time scene graph of the plurality of targets in the target video segment based on the target detection results; inputting the space-time scene graph and the target video segment into a pre-trained scene semantic reasoning model to perform semantic feature extraction, and obtaining a target semantic description text of the target video segment. The space dimension and the time dimension are combined to comprehensively analyze the changes of the plurality of targets, and then the target semantic description text of the target video segment is obtained, so that the staff can quickly understand the content of the target video segment through the target semantic description text.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a video processing method and device, electronic equipment and a storage medium. BACKGROUND

[0002] In related technologies, whether the video content collected by the front-end camera device is abnormal is mainly analyzed by using a manual method. However, when a large amount of video content collected by multiple camera devices is analyzed by using this method, a large amount of time cost is consumed, the analysis efficiency is low, and the analysis result is prone to deviation.

[0003] Therefore, there is an urgent need for a method capable of improving the video analysis efficiency. SUMMARY

[0004] Therefore, the present application provides a video processing method and device, electronic equipment and a storage medium to solve the problem of low video analysis efficiency.

[0005] In a first aspect, the present application provides a video processing method, which comprises:

[0006] obtaining a target video segment, the target video segment comprising a plurality of continuous video images;

[0007] performing target detection processing on the plurality of video images to identify a plurality of targets in the plurality of video images, and obtaining target detection results of the plurality of targets;

[0008] constructing a spatio-temporal scene graph of the plurality of targets in the target video segment based on the target detection results;

[0009] inputting the spatio-temporal scene graph and the target video segment into a pre-trained scene semantic reasoning model to perform semantic feature extraction, and obtaining a target semantic description text of the target video segment.

[0010] In this way, the changes of the plurality of targets can be comprehensively analyzed in combination with the spatial dimension and the temporal dimension, and then the pre-trained scene semantic reasoning model is used to perform semantic characterization to obtain the target semantic description text of the target video segment, so that the staff can quickly understand the content of the target video segment through the target semantic description text, thereby saving time cost and improving the video detection efficiency when subsequent targeted analysis is performed.

[0011] In an optional implementation, the target detection results comprise sub-target detection results of each target in different video images, and constructing the spatio-temporal scene graph of the plurality of targets in the target video segment based on the target detection results comprises:

[0012] Based on the sub-target detection results of multiple targets in different frame video images, the interaction relationship of multiple targets in each frame video image is identified, resulting in multiple spatial maps, with different spatial maps corresponding to different frame video images;

[0013] Based on the sub-target detection results of multiple targets in different frame video images and the temporal order of multiple frame video images in the target video segment, the positional changes of each target in the target time period are tracked to obtain multiple time series diagrams. Different time series diagrams correspond to different targets, and the target time period is the time period corresponding to the target video segment.

[0014] Based on multiple spatial maps and multiple temporal maps, a spatiotemporal scene map of multiple targets in a target video segment is constructed.

[0015] This approach can fully explore the spatial interaction relationships between targets within the same video frame, thereby enabling the resulting spatial map to effectively assist the scene semantic reasoning model in comprehensively perceiving the scene state corresponding to the target video segment and improving the accuracy of understanding the scene content corresponding to the target video segment.

[0016] In one optional implementation, based on the sub-target detection results of multiple targets in different frame video images, the interaction relationship of multiple targets in each frame video image is identified to obtain multiple spatial maps, including:

[0017] Based on the sub-target detection results of multiple targets in the current frame video image, determine the position information and local region information of each target in the current frame video image;

[0018] Based on the location information and local region information of each target, the strength of the interaction relationship between every two targets in the current frame video image is determined;

[0019] An adjacency matrix is ​​constructed based on the interaction strength between every two targets in the current frame video image, and a spatial graph corresponding to the current frame video image is constructed based on the adjacency matrix.

[0020] In one optional implementation, the multiple targets include a first target and a second target. Based on the location information and local region information of each target, the interaction strength between every two targets in the current frame video image is determined, including:

[0021] Based on the location information of the first target, extract the first visual features of the first target in the current frame video image;

[0022] Based on the location information of the second target, extract the second visual features of the second target in the current frame video image;

[0023] Based on the Euclidean distance between the first visual feature and the second visual feature, the semantic relevance between the first target and the second target is determined;

[0024] Based on the location information of the first target, the local region information of the first target, the location information of the second target, and the local region information of the second target, the spatial overlap of the first target and the second target in the current frame video image is determined, and the positional correlation between the first target and the second target is obtained.

[0025] The strength of the interaction relationship between the first and second objectives is obtained by using semantic relevance and positional relevance.

[0026] This method can fully combine the positional relationships between targets and local region information in the current frame video image, thereby effectively expressing the spatial interaction state of two targets in the current frame video image.

[0027] In one optional implementation, based on the sub-target detection results of multiple targets in different frames of video images and the temporal order of the multiple frames of video images in the target video segment, the positional changes of each target within the target time period are tracked to obtain multiple time-series graphs, including:

[0028] Based on the sub-target detection results of multiple targets in different frame video images, the region location of each target in each frame video image is determined respectively;

[0029] Based on the regional location of each target in each frame of video image and the temporal order of multiple video images in the target video segment, the motion trajectory of each target within the target time period is constructed, so as to construct the time sequence diagram of the corresponding target based on the motion trajectory.

[0030] In this approach, constructing a temporal graph corresponding to each target can integrate the temporal information of the spatial graph at different times, reflecting the motion continuity of the same target between frames. As a result, when performing semantic description on the target video segment, the target action can be fully understood, thereby helping to improve the accuracy of describing the event content included in the scene corresponding to the target video segment.

[0031] In one alternative implementation, the plurality of targets includes a first target, and the plurality of video images includes a first video image and a first frame video image;

[0032] Based on the sub-target detection results of multiple targets in different frames of video images, the region location of each target in each frame of video image is determined, including:

[0033] Based on the sub-target detection results of the first target in the first frame of the video image, determine the location of the first region of the first target in the first frame of the video image;

[0034] Based on the sub-target detection results of multiple targets in the first frame of the video image, the position of the second region of each target in the first frame of the video image is determined respectively;

[0035] The first region location is semantically matched with each second region location. The second region location with the highest semantic matching degree with the first region location is taken as the target region location of the first target in the first frame video image.

[0036] In this approach, visual features are used to identify whether targets in different regions are the same target, which reduces the occurrence of misidentification and helps improve the accuracy of identification. This helps ensure the accuracy of tracking when tracking the movement trajectory of each target within the target time period, and thus improves the accuracy of semantic description.

[0037] In one optional implementation, a spatiotemporal scene graph of multiple targets in a target video segment is constructed based on multiple spatial graphs and multiple temporal graphs, including:

[0038] By treating multiple targets as graph nodes in the spatiotemporal scene graph, multiple target nodes are obtained;

[0039] Multiple spatial graphs are used as spatial edges of the spatiotemporal scene graph, and arranged according to the temporal order of the corresponding video frames in the target video segment to obtain multiple target spatial edges.

[0040] Multiple time series graphs are used as time series edges of a spatiotemporal scene graph to obtain multiple target time series edges;

[0041] Each target node is connected to its corresponding spatial edge and temporal edge to obtain a spatiotemporal scene graph of multiple targets in the target video segment.

[0042] In one optional implementation, the spatiotemporal scene graph and the target video clip are input into a pre-trained scene semantic reasoning model for semantic feature extraction to obtain the target semantic description text of the target video clip, including:

[0043] The spatiotemporal scene map and the target video clip are input into a pre-trained scene semantic reasoning model. The target encoder built into the scene semantic reasoning model performs feature encoding on the spatiotemporal scene map and the target video clip to obtain encoded semantic features.

[0044] The target semantic description text of the target video segment is obtained by decoding the encoded semantic features through the target decoder built into the scene semantic reasoning model.

[0045] In one alternative implementation, the target encoder includes: a spatial encoder, a temporal encoder, and a global context encoder;

[0046] By using a target encoder embedded in the scene semantic reasoning model, feature encoding is performed on the spatiotemporal scene graph and the target video segment to obtain encoded semantic features, including:

[0047] By performing first feature encoding on each spatial graph in the spatiotemporal scene graph using a spatial encoder, multiple first feature codes are obtained. These multiple first feature codes are then combined with multiple spatial... Figure One One-to-one correspondence;

[0048] The temporal scene graph is processed by a temporal encoder to perform second feature encoding on each temporal graph, resulting in multiple second feature codes. These multiple second feature codes are then combined with multiple temporal graphs. Figure One One-to-one correspondence;

[0049] The target video segment is processed by a global context encoder to obtain the third feature code.

[0050] Multiple first feature codes, multiple second feature codes, and third feature codes are used as encoding semantic features.

[0051] This approach can fully explore the social interaction relationships between multiple targets in a target video clip and the motion state of each target within the target time period from both temporal and spatial dimensions, thereby enabling the obtained target semantic description text to better fit the scene content corresponding to the target video clip.

[0052] In one optional implementation, a spatial encoder is used to perform first feature encoding processing on each spatial map in the spatiotemporal scene map to obtain multiple first feature codes, including:

[0053] Within the spatial encoder, the first feature extraction process is performed on the current spatial graph to obtain the node features of multiple target nodes. The current spatial graph is any spatial graph in the spatiotemporal scene graph.

[0054] The node features of multiple target nodes are subjected to mean pooling to obtain the first feature code corresponding to the current spatial graph.

[0055] In one optional implementation, a second feature encoding process is performed on each temporal graph in the spatiotemporal scene graph using a temporal encoder to obtain multiple second feature codes, including:

[0056] Within the temporal encoder, a second feature extraction process is performed on the current temporal graph to obtain the image features of the target corresponding to the current temporal graph at different times. The current temporal graph is any temporal graph in the spatiotemporal scene graph.

[0057] Determine the feature cluster centers of the encoded semantic features;

[0058] Clustering encoding is performed on the size information, image features, and feature clustering centers of the target corresponding to the current time series graph at different times to obtain the second feature code corresponding to the current time series graph.

[0059] In one alternative implementation, the target decoder includes a first decoder, a second decoder, and a global vocabulary.

[0060] The encoded semantic features are decoded by the target decoder built into the scene semantic reasoning model to obtain the target semantic description text of the target video segment, including:

[0061] The first semantic description text of the target video segment is obtained by decoding multiple second and third feature codes using the first decoder and global vocabulary.

[0062] By using a second decoder and a global vocabulary, multiple first and third feature codes are decoded to obtain the second semantic description text of the target video segment.

[0063] The first semantic description text is multiplied by the first weight corresponding to the first decoder to obtain the first intermediate parameter;

[0064] The second semantic description text is multiplied by the first weight corresponding to the second decoder to obtain the second intermediate parameter;

[0065] By combining the first and second intermediate parameters, the target semantic description text of the target video segment is obtained.

[0066] In one alternative implementation, the method further includes:

[0067] Construct a quadruple sample, which consists of a normal video clip sample with the same video duration, a normal reference video clip sample, an abnormal video clip sample, and an abnormal reference video clip sample.

[0068] The quadruplet samples are input into the target encoder for feature encoding processing to obtain four sets of segment encoding semantic features. The four sets of segment encoding semantic features include the first segment encoding semantic features corresponding to normal video segment samples, the second segment encoding semantic features corresponding to normal reference video segment samples, the third segment encoding semantic features corresponding to abnormal video segment samples, and the fourth segment encoding semantic features corresponding to abnormal reference video segment samples.

[0069] Based on the semantic features encoded in the first segment and the semantic features encoded in the second segment, the first feature interval of the normal sample pair is determined.

[0070] Based on the semantic features encoded in the third and fourth segments, the second feature spacing of the abnormal sample pairs is determined.

[0071] Based on the second segment encoded semantic features and the fourth segment encoded semantic features, the third feature distance between normal sample pairs and abnormal sample pairs is determined.

[0072] Using the first feature spacing, the second feature spacing, and the third feature spacing, the parameters of the target encoder are adjusted through a preset contrastive loss function. If the obtained loss value is less than or equal to a specified loss threshold, the adjustment of the parameters of the target encoder is stopped, and the updated target encoder is obtained.

[0073] The scene semantic reasoning model is updated based on the updated target encoder to obtain the updated scene semantic reasoning model.

[0074] In this approach, under an open set setting, the updated scene semantic reasoning model can be used to generate target semantic description text for a target video segment while simultaneously detecting whether the target video segment is a normal segment. When an abnormal segment is detected, it can be accurately identified and described, thereby enhancing the anomaly detection capability.

[0075] In one alternative implementation, the method further includes:

[0076] Normal video samples are input into the updated target encoder for feature extraction to obtain the target feature code.

[0077] Clustering is performed on the target feature codes to determine the target metric centers of the target feature codes;

[0078] Determine the center distance between the feature cluster centers of the encoded semantic features and the target metric center;

[0079] If a center distance greater than the specified distance threshold exists, the target video segment is determined to be an abnormal video segment.

[0080] In one alternative implementation, the method further includes:

[0081] If there is no center distance greater than the specified distance threshold, the target video segment is determined to be a normal video segment.

[0082] In one optional implementation, acquiring the target video segment includes:

[0083] Acquire the target video;

[0084] Determine the segment duration for which the target video will be segmented;

[0085] The target video is divided into segments according to their duration, resulting in multiple video segments, each including the target video segment.

[0086] In one optional implementation, determining the segment duration for which the target video is to be segmented includes:

[0087] Determine the number of video segments based on the total duration of the target video;

[0088] The quotient of the total duration divided by the number of video segments is taken as the segment duration.

[0089] In a second aspect, the present invention provides a video processing apparatus, the apparatus comprising:

[0090] The acquisition module is used to acquire the target video segment, which includes multiple consecutive video images;

[0091] The detection module is used to perform target detection processing on multiple frames of video images to identify multiple targets in the multiple frames of video images and obtain the target detection results of multiple targets;

[0092] The first construction module is used to construct a spatiotemporal scene map of multiple targets in the target video segment based on the target detection results;

[0093] The first processing module is used to input the spatiotemporal scene graph and the target video clip into a pre-trained scene semantic reasoning model for semantic feature extraction, so as to obtain the target semantic description text of the target video clip.

[0094] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the video processing method described in the first aspect or any corresponding embodiment thereof.

[0095] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the video processing method described in the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0096] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0097] Figure 1 This is a flowchart illustrating a video processing method according to an embodiment of the present invention;

[0098] Figure 2This is a flowchart illustrating another video processing method according to an embodiment of the present invention;

[0099] Figure 3 This is a spatiotemporal scene diagram according to an embodiment of the present invention;

[0100] Figure 4 This is a flowchart illustrating a method for training a scene semantic reasoning model according to an embodiment of the present invention;

[0101] Figure 5 This is a flowchart illustrating another video processing method according to an embodiment of the present invention;

[0102] Figure 6 This is a flowchart illustrating another method for training a semantic reasoning model for a specific scenario according to an embodiment of the present invention.

[0103] Figure 7 This is a comparative schematic diagram of anomaly detection according to an embodiment of the present invention;

[0104] Figure 8 This is a flowchart illustrating another video processing method according to an embodiment of the present invention;

[0105] Figure 9 This is a schematic diagram of the network framework of an abnormal event detection system according to an embodiment of the present invention;

[0106] Figure 10 This is a structural block diagram of a video processing apparatus according to an embodiment of the present invention;

[0107] Figure 11 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0108] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0109] In related technologies, content analysis of video footage captured by front-end camera equipment is mainly performed manually. This involves continuously observing the video content to check for any abnormalities and then conducting targeted analysis. However, this method of video analysis requires a large amount of manpower, resulting in high operating costs. Furthermore, staff may experience decreased attention spans after prolonged observation, leading to low analysis efficiency.

[0110] In view of this, embodiments of the present invention provide a video processing method, comprising: acquiring a target video segment, the target video segment comprising a series of consecutive video frames; performing target detection processing on the multiple video frames to identify multiple targets in the multiple video frames, and obtaining target detection results for the multiple targets; constructing a spatiotemporal scene map of the multiple targets in the target video segment based on the target detection results; and inputting the spatiotemporal scene map and the target video segment into a pre-trained scene semantic reasoning model for semantic feature extraction to obtain target semantic description text of the target video segment. Through the video processing method provided by the present invention, by constructing a spatiotemporal scene map of the target video segment, the changes of multiple targets can be comprehensively analyzed by combining spatial and temporal dimensions. Furthermore, a pre-trained scene semantic reasoning model is used for semantic characterization to obtain target semantic description text of the target video segment. This allows staff to quickly understand the content of the target video segment through the target semantic description text, thereby saving time and improving video detection efficiency during subsequent targeted analysis.

[0111] According to an embodiment of the present invention, a video processing method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0112] This embodiment provides a video processing method that can be used in electronic devices, such as computers and servers. Figure 1 This is a flowchart of a video processing method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0113] Step S101: Obtain the target video segment.

[0114] In this embodiment of the invention, the target video segment includes a series of consecutive video frames. The target video segment can be understood as the segment to be detected. For example, the target video segment can be extracted from video captured by a front-end camera device in a public area, or it can be extracted from video stored locally or on a server; this is not limited in this invention.

[0115] Step S102: Perform target detection processing on multiple frames of video images to identify multiple targets in the multiple frames of video images and obtain target detection results for multiple targets.

[0116] In this embodiment of the invention, in order to facilitate a targeted description of the content in the target video segment, target detection processing is performed on the multi-frame video images to identify multiple targets in the multi-frame video images, thereby obtaining the target detection results of each target in the target video segment.

[0117] Since existing object detection technologies are relatively mature, any object detection algorithm can be used to process multiple frames of video images, and this invention is not limited to any particular algorithm. For example, the YOLO (You Only Look Once, a neural network model for object detection) algorithm or the FastRegion-CNN (RCNN) algorithm (a convolutional neural network model for object detection) algorithm can be used to process multiple frames of video images to obtain object detection results for multiple objects.

[0118] When performing target detection processing on multiple frames of video images, either asynchronous or synchronous detection methods can be used. If asynchronous detection is used, each frame can be detected sequentially according to its chronological order within the target video segment, thus helping to avoid missed detections. If synchronous detection is used, multi-channel detection can be employed, performing target detection processing on each frame simultaneously, thereby improving target detection efficiency.

[0119] Step S103: Based on the target detection results, construct a spatiotemporal scene map of multiple targets in the target video segment.

[0120] In this embodiment of the invention, based on the target detection results, multiple targets involved in each frame of video image can be determined. In order to make it easier to represent the changes of multiple targets in the target video segment in a concrete way, a spatiotemporal scene graph of multiple targets in the target video segment is constructed to describe the changes of multiple targets in space and time through the spatiotemporal scene graph, so that the target video segment can be given a targeted semantic description in the future.

[0121] Step S104: Input the spatiotemporal scene map and the target video clip into the pre-trained scene semantic reasoning model for semantic feature extraction to obtain the target semantic description text of the target video clip.

[0122] In this embodiment of the invention, the scene semantic reasoning model can be understood as a model specifically designed to perform scene semantic reasoning on the scene content of a video clip, and then use text for targeted semantic description. The spatiotemporal scene graph and the target video clip are input into the pre-trained scene semantic reasoning model. The model then combines the semantic features of the scene content of the spatiotemporal scene graph with each frame of the target video clip to perform targeted semantic characterization, thereby obtaining the target semantic description text for the target video clip. This allows staff to quickly understand the scene content corresponding to the target video clip through the target semantic description text.

[0123] The video processing method provided in this embodiment can comprehensively analyze the changes of multiple targets by combining spatial and temporal dimensions, and then use a pre-trained scene semantic reasoning model to perform semantic characterization to obtain the target semantic description text of the target video segment. This allows staff to quickly understand the content of the target video segment through the target semantic description text, thereby saving time and improving video detection efficiency when conducting targeted analysis.

[0124] This embodiment provides a video processing method that can be used in the aforementioned electronic devices, such as computers and servers. Figure 2 This is a flowchart of a video processing method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0125] Step S201: Obtain the target video segment. For details, please refer to [link to relevant documentation]. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0126] Step S202 involves performing target detection processing on multiple frames of video images to identify multiple targets within the images, thereby obtaining target detection results for each target. For details, please refer to [link to relevant documentation]. Figure 1 Step S102 of the illustrated embodiment will not be described again here.

[0127] Step S203: Based on the target detection results, construct a spatiotemporal scene map of multiple targets in the target video segment.

[0128] Specifically, the target detection result includes the sub-target detection result of each target in different frame video images, and the above step S203 includes:

[0129] Step S2031: Based on the sub-target detection results of multiple targets in different frame video images, identify the interaction relationship of multiple targets in each frame video image to obtain multiple spatial maps.

[0130] In this embodiment of the invention, to facilitate the determination of the social interaction relationships between different targets within each frame of a video image, the spatial positional relationships of multiple targets within each frame of a video image can be clarified based on the sub-target detection results of multiple targets in different frames of video images. This allows for the identification of the interaction relationships between multiple targets in each frame of a video image, thereby generating a spatial map corresponding to each frame of a video image. Different spatial maps correspond to different frames of video images.

[0131] In some optional implementations, step S2031 above includes:

[0132] Step a1: Based on the sub-target detection results of multiple targets in the current frame video image, determine the position information and local region information of each target in the current frame video image.

[0133] In this approach, the current frame video image is any one of multiple video frames. To construct the spatial map corresponding to the current frame video image, the position information and local region information of each target in the current frame video image are determined based on the sub-target detection results of multiple targets in the current frame video image. This position information and local region information are used to determine the spatial positional relationship between multiple targets, thereby clarifying the interaction relationship between targets in the current frame video image.

[0134] Step a2: Based on the location information and local region information of each target, determine the strength of the interaction relationship between every two targets in the current frame video image.

[0135] In this approach, the spatial overlap between any two targets can be determined using their location information, thus establishing the positional correlation between them. The interaction between any two targets can be determined using their local region information, thus establishing the semantic correlation between them. Therefore, to facilitate the description of the interaction relationships between targets in the current frame of the video image, the strength of the interaction relationship between any two targets in the current frame is determined based on the location and local region information of each target.

[0136] In some alternative implementations, the multiple objectives include a first objective and a second objective, and step a2 above includes:

[0137] Step a21: Extract the first visual features of the first target in the current frame video image based on the location information of the first target.

[0138] Step a22: Based on the location information of the second target, extract the second visual features of the second target in the current frame video image.

[0139] Step a23: Determine the semantic relevance between the first target and the second target based on the Euclidean distance between the first visual feature and the second visual feature.

[0140] Step a24: Based on the location information of the first target, the local region information of the first target, the location information of the second target, and the local region information of the second target, determine the spatial overlap between the first target and the second target in the current frame video image, and obtain the positional correlation between the first target and the second target.

[0141] Step a25: Obtain the interaction strength between the first target and the second target through semantic relevance and positional relevance.

[0142] Specifically, to facilitate determining whether there is an interaction between the first target and the second target, the first visual feature v of the first target in the current frame video image is extracted based on the position information of the first target. i Based on the location information of the second target, extract the second visual feature v of the second target in the current frame video image. j To pass through the first visual feature v i Second visual feature v j The Euclidean distance between them indicates whether they are semantically related. Among them, the first visual feature v i The semantic features of the first target can be extracted from the local region of the video image in the current frame using pre-trained convolutional neural networks (CNNs). Similarly, the second visual feature v j Alternatively, it can be used to extract the first visual feature v i The same technical means of extraction are not limited in this invention. The first visual feature and the second visual feature may include, but are not limited to, any one or more of the following features: color features, texture features, and shape features corresponding to the target.

[0143] Based on the first visual feature v i With second visual feature v j The Euclidean distance between the two objectives determines the semantic relevance between the first and second objectives, thereby clarifying the semantic correlation (IR) between them. sem (i,j). The relevant formulas are as follows:

[0144]

[0145] Among them, L2(v i ,v j ) represents the first visual feature v i With second visual feature v j The Euclidean distance between them, maxp,q (L2(v p ,v q The distance () represents the Euclidean distance among multiple Euclidean distances within the current frame of the video image, where the distance value is the largest. In other words, the semantic relevance between the first and second targets needs to be determined by combining the Euclidean distances between the visual features of each target in the current frame of the video image. This makes the obtained semantic relevance more closely aligned with the scene corresponding to the current frame of the video image, and thus more effective.

[0146] Based on the location information of the first target and the local area information of the first target, the first spatial location area(o) of the first target can be determined. i (t) Based on the location information of the second target and the local region information of the second target, the second spatial location area(o) of the second target can be determined. j (t) The spatial overlap between the first and second targets in the current frame video image is determined using the following intersection-union ratio (IUGR) formula, thereby obtaining the positional correlation (IR) between the first and second targets. iou (i,j):

[0147]

[0148] Where t represents the current time order of the current frame video image among multiple video frames.

[0149] Determining the interaction strength between the first and second targets using the above method can fully combine the positional relationships between targets and local region information in the current frame video image, thereby effectively expressing the spatial interaction state of the two targets in the current frame video image.

[0150] Step a3: Construct an adjacency matrix based on the interaction strength between every two targets in the current frame video image, and construct a spatial graph corresponding to the current frame video image based on the adjacency matrix.

[0151] In this approach, a higher interaction strength indicates a stronger interaction between the two corresponding targets; conversely, a lower interaction strength indicates a weaker interaction between the two corresponding targets. Therefore, to fully express the spatial relationships between targets in a given frame of video image, an adjacency matrix is ​​constructed based on the interaction strength between every two targets in the current frame of video image, and then a spatial graph corresponding to the current frame of video image is constructed based on the adjacency matrix.

[0152] By constructing a spatial graph in the above manner, the spatial interaction relationships between targets within the same video frame can be fully explored. This allows the resulting spatial graph to effectively assist the scene semantic reasoning model in comprehensively perceiving the scene state corresponding to the target video segment, thereby improving the accuracy of understanding the scene content corresponding to the target video segment.

[0153] In one example, if the spatial graph corresponding to each frame of video image uses an adjacency matrix of size N×N... If we represent the target video segment as a sequence, then the multiple spatial graphs corresponding to the target video segment can be represented as a sequence, and each spatial graph G in the spatial graph sequence G... t The corresponding frame images are arranged in chronological order within the target video segment. For example, if the target video segment contains T frames of video images, the expression for the spatial graph sequence G can be as follows: G = {G} t}, t=1,...,T.

[0154] Step S2032: Based on the sub-target detection results of multiple targets in different frame video images and the temporal order of multiple frame video images in the target video segment, track the positional changes of each target within the target time period to obtain multiple time series diagrams.

[0155] In this embodiment of the invention, different time sequence diagrams correspond to different targets, and the target time period is the time period corresponding to the target video segment. By using the sub-target detection results of multiple targets in different frame video images, the positional information of each target in different frame video images can be clearly identified. Then, by combining the time sequence of multiple frame video images in the target video segment for tracking, it can be determined whether the position of each target changes within the target time period, thereby expressing it continuously and obtaining the time sequence diagram corresponding to each target. This makes the motion trajectory of each target within the target video segment more concrete and helps to provide a targeted semantic description of the motion state of the target within the target video segment.

[0156] In some optional implementations, step S2032 above includes:

[0157] Step b1: Based on the sub-target detection results of multiple targets in different frame video images, determine the region location of each target in each frame video image.

[0158] In this method, based on the sub-target detection results of multiple targets in different frame video images, the regional positions of different targets in each frame video image can be clearly identified, and then integrated to clearly identify the regional position of each target in each frame video image.

[0159] In some optional implementations, the multiple targets include a first target, and the multiple video frames include a first video frame and a first video image. Step b1 above includes:

[0160] Step b11: Determine the location of the first target in the first region of the first target in the first frame of the video image based on the sub-target detection results of the first target in the first frame of the video image.

[0161] Step b12: Based on the sub-target detection results of multiple targets in the first frame video image, determine the second region position of each target in the first frame video image.

[0162] Step b13: Perform semantic matching processing on the first region position and each second region position respectively, and take the second region position with the highest semantic matching degree with the first region position as the target region position of the first target in the first frame video image.

[0163] Specifically, the first region position of the first target in the first frame of the video image is used as the starting position for tracking the positional changes of the first target. The first frame of the video image can be any non-first frame from multiple video images. Since multiple targets exist in each frame of the video image, to facilitate targeted tracking of the motion changes of the first target within the target time period, the second region position of each target in the first frame of the video image is determined based on the sub-target detection results of multiple targets in the first frame of the video image. Since the probability of the same target appearing in different video frames in consecutive multi-frame video images is low, the region position of the same target in different frame video images can be identified by semantic matching. Then, the first region position is semantically matched with each second region position, and the second region position with the highest semantic matching degree with the first region position is taken as the target region position of the first target in the first frame of the video image. The higher the semantic matching degree, the smaller the difference in visual features between the two, and the greater the probability that they are the same target. The lower the semantic matching degree, the greater the difference in visual features between the two, and the lower the probability that they are the same target. Therefore, the target corresponding to the second region with the highest semantic matching degree is determined as the first target. Thus, the movement path of the first target can be tracked from the time corresponding to the first frame of the video image to the time corresponding to the first frame of the video image based on the position change of the first target between the first frame of the video image and the first frame of the video image.

[0164] The semantic matching degree (IR) between the first region location and each second region location can be calculated using the following formula. sem (1,m):

[0165]

[0166] Where 1 represents the sorting position of the first video frame in the multi-frame video image, m represents the sorting position of the first video frame in the multi-frame video image, L2(v1,v m ) represents the third visual feature v1 and the fourth visual feature v m The Euclidean distance between them, the third visual feature v1 is the visual feature of the first target within the first region, and the fourth visual feature v m The maximum value is the visual feature of the target in the first frame of the video image corresponding to the current second region location. i,q (L2(v1,v q The distance ()) represents the largest Euclidean distance among the multiple fourth visual features. The third visual feature can be obtained by extracting semantic features from the local image corresponding to the first region location using a pre-trained visual feature extraction model. Similarly, the fourth visual features can be obtained by the same visual feature extraction model by extracting semantic features from the local image corresponding to the current second region location. This visual feature extraction model can be trained using a Convolutional Neural Network (CNN) framework. Since the training process for models used for visual feature extraction is relatively mature in existing technologies, it will not be elaborated further.

[0167] Identifying whether targets in different regions are the same target through visual features can reduce the occurrence of misidentification and help improve identification accuracy. This helps ensure the accuracy of tracking the movement trajectory of each target within the target time period, thereby improving the accuracy of subsequent semantic description.

[0168] In some alternative implementations, based on the sub-target detection results of the target in different frames of video images, the targets included in each frame of video image and the target region position of each target in that frame of video image can be identified separately. Using the target types marked in each sub-target detection result, the target type most similar to the first target in the first frame of video image is determined as a candidate target type. If the number of targets corresponding to a candidate target type is 1, the target corresponding to the candidate target type is directly determined as the first target, and the second position region of this candidate target type in the first frame of video image is taken as the target region position, thereby improving the efficiency of determining the target region position. If the number of targets corresponding to a candidate target type is greater than 1, the targets corresponding to the candidate target type are taken as candidate targets, and then the first region position of the first target in the first frame of video image is semantically matched with the second region positions of each candidate target in the first frame of video image. The second region position with the highest semantic matching degree is taken as the target region position of the first target in the first frame of video image.

[0169] Step b2: Based on the regional position of each target in each frame of video image and the temporal order of multiple frames of video images in the target video segment, construct the motion trajectory of each target within the target time period, so as to construct the time sequence diagram of the corresponding target based on the motion trajectory.

[0170] In this method, based on the temporal order of multiple video images in the target video segment and the regional position of the same target in each video image, the regional position change of the same target at different times within the target time period can be clearly identified, and targeted tracking can be carried out. By constructing the motion trajectory of each target within the target time period, the continuous state of the target's motion between frames can be concretely expressed, thereby obtaining the time sequence diagram corresponding to the target.

[0171] By constructing a temporal graph corresponding to each target using the above method, the temporal information of the spatial graph at different times can be integrated to reflect the motion continuity of the same target between frames. As a result, when performing semantic description on the target video segment, the target action can be fully understood, which helps to improve the accuracy of describing the event content included in the scene corresponding to the target video segment.

[0172] Step S2033: Based on multiple spatial maps and multiple temporal maps, construct a spatiotemporal scene map of multiple targets in the target video segment.

[0173] In this embodiment of the invention, different spatial maps correspond to different video frames. Each spatial map reflects the social interaction relationships between targets in the corresponding video frame, and each temporal map reflects the motion state of each target at different times within the target time period. By combining the two, a spatiotemporal scene map of multiple targets in the target video segment is constructed. This allows for a concrete representation of the social interaction relationships and motion states between targets, facilitating semantic feature extraction in a pre-trained scene semantic reasoning model without compromising the temporal information of the target video segment. This, in turn, improves the accuracy of semantic description of the target video segment.

[0174] In some optional implementations, step S2033 above includes:

[0175] Step c1: Use multiple targets as graph nodes in the spatiotemporal scene graph to obtain multiple target nodes.

[0176] Step c2 involves using multiple spatial graphs as spatial edges of a spatiotemporal scene graph, and arranging them according to the temporal order of the corresponding video frames in the target video segment to obtain multiple target spatial edges.

[0177] Step c3: Use multiple time sequence graphs as time sequence edges of the spatiotemporal scene graph to obtain multiple target time sequence edges.

[0178] Step c4: Connect each target node to its corresponding target spatial edge and its corresponding target temporal edge to obtain a spatiotemporal scene graph of multiple targets in the target video segment.

[0179] Specifically, since multiple spatial diagrams and multiple time series diagrams are formed using each target as a connection point, in order to construct multiple spatial diagrams and multiple time series diagrams together to form a structure like... Figure 3 The spatiotemporal scene graph corresponding to the target video segment is represented by multiple targets as graph nodes, resulting in multiple target nodes (solid circles). Multiple spatial graphs are used as spatial edges of the spatiotemporal scene graph, and arranged according to the temporal order of the corresponding video frames in the target video segment, resulting in multiple target spatial edges (solid lines). Multiple temporal graphs are used as temporal edges of the spatiotemporal scene graph, resulting in multiple target temporal edges (dashed lines). Each target node is connected to its corresponding target spatial edge and its corresponding target temporal edge, resulting in the spatiotemporal scene graph of multiple targets within the target video segment.

[0180] Step S204: Input the spatiotemporal scene map and the target video clip into the pre-trained scene semantic reasoning model for semantic feature extraction to obtain the target semantic description text of the target video clip.

[0181] Specifically, step S204 includes:

[0182] Step S2041: Input the spatiotemporal scene map and the target video segment into the pre-trained scene semantic reasoning model. Through the target encoder built into the scene semantic reasoning model, perform feature encoding processing on the spatiotemporal scene map and the target video segment to obtain encoded semantic features.

[0183] In this embodiment of the invention, in order to fully explore the semantic information of the scene content corresponding to the target video segment and facilitate targeted description in the form of text, after inputting the spatiotemporal scene map and the target video segment into the pre-trained scene semantic reasoning model, the target encoder built into the scene semantic reasoning model is first used to perform feature encoding processing on the spatiotemporal scene map and the target video segment, thereby obtaining encoded semantic features that can fully express the scene content corresponding to the spatiotemporal scene map, so that targeted semantic characterization can be performed subsequently.

[0184] In some optional implementations, the target encoder includes: a spatial encoder, a temporal encoder, and a global context encoder, and step S2041 above includes:

[0185] Step d1 involves performing first feature encoding on each spatial map in the spatiotemporal scene map using a spatial encoder to obtain multiple first feature codes.

[0186] In this approach, multiple first feature codes are coupled with multiple spatial codes. Figure One One-to-one correspondence. A spatial encoder can be understood as an encoder used to encode spatial graphs. To construct the interaction information between targets within a frame, the spatial encoder performs first feature encoding processing on each spatial graph in the spatiotemporal scene graph, so as to express the social interaction relationship of each target in the same spatial graph through data, and thus obtain the first feature encoding corresponding to each spatial graph.

[0187] In some optional implementations, step d1 above includes:

[0188] Step d11: Within the spatial encoder, perform the first feature extraction process on the current spatial map to obtain the node features of multiple target nodes.

[0189] Step d12: Perform mean pooling on the node features of multiple target nodes to obtain the first feature code corresponding to the current spatial graph.

[0190] Specifically, the spatial encoder can be trained using a Graph Convolutional Network (GCN) framework. Within the spatial encoder, the GCN can be used to calculate the activation values ​​of each target node based on the interaction relationships of spatial edge labels. Simultaneously, the target nodes are updated iteratively, and the first feature extraction process is performed on the current spatial graph to obtain the node features of multiple target nodes.

[0191] The graph convolutional layer can be represented by the following formula:

[0192]

[0193] in, For the current spatial graph G t Original target node features For the pre-trained weight parameter matrix, Let A be the node features of the target node in the final output, and let I be the adjacency matrix and I be the diagonal matrix.

[0194] The node features of multiple target nodes are subjected to mean pooling to obtain the first feature encoding corresponding to the current spatial graph.

[0195] In some examples, the first feature encoding corresponding to each spatial map can be expressed by the following expression: SF = {sf} t}, t=1,...,T.

[0196] Step d2 involves performing second feature encoding on each temporal sequence in the spatiotemporal scene graph using a temporal encoder to obtain multiple second feature codes.

[0197] In this approach, multiple second feature codes are coupled with multiple temporal sequences. Figure One One-to-one correspondence. A time sequence encoder can be understood as an encoder used to encode time sequence diagrams. To facilitate spatiotemporal statistics on the spatial position changes of the same target at different times, a time sequence encoder is used to perform second feature encoding processing on each time sequence diagram in the spatiotemporal scene diagram, so as to express the motion characteristics of each target in the target time period in an encoded form, thereby obtaining the second feature code corresponding to each time sequence diagram.

[0198] In some alternative implementations, step d2 above includes:

[0199] Step d21: Within the time encoder, perform second feature extraction processing on the current time sequence graph to obtain the image features of the target corresponding to the current time sequence graph at different times.

[0200] Step d22: Determine the feature cluster centers for encoding semantic features.

[0201] Step d23: Perform clustering encoding on the size information, image features and feature clustering centers of the target corresponding to the current time series graph at different times to obtain the second feature code corresponding to the current time series graph.

[0202] Specifically, to fully describe the spatial position changes of the same target at different times in a time series graph, a trainable Vector of Locally Aggregated Descriptors (VLAD) model constructed using a Convolutional-Gated Recurrent Unit (C-GRU) is trained using a magic-like method to obtain a time series encoder. Within the time series encoder, a second feature extraction is first performed on the current time series graph to obtain the image features of the target at different times. Feature cluster centers for each image feature are then determined. These cluster centers are combined with the size information of the target at different times, the image features, and the feature cluster centers to perform clustering encoding, thereby obtaining a second feature code that describes the motion state of the target in the current time series graph.

[0203] The timing encoder performs a second feature extraction process on the current timing graph to obtain the second feature code tf. it The formula used can be as follows:

[0204]

[0205] Wherein, H and W are the local feature maps r extracted from the target corresponding to the current time series map. it Maximum height and maximum width, D represents the pre-trained feature cluster centers, where D is r it The number of channels, a it These are pre-trained feature clustering weights.

[0206] Step d3 involves performing third feature encoding on the target video segment using a global context encoder to obtain the third feature code.

[0207] In this approach, the global context encoder is an encoder trained using a VLAD model as the framework. Within the global context encoder, a specified local convolutional kernel is used to perform sliding convolution on each frame of video image with a specified stride to obtain target local features. These target local features are then used as local image features of the corresponding frame of video image. This is combined with the local image features of each frame of video image for global context encoding, resulting in a third feature code. This allows for subsequent decoding of the first and second feature codes, enabling reasonable parsing based on the context of the target video segment and better matching the scene content corresponding to the target video segment. For example, if the current frame of video image is 40*40 pixels, a specified local convolutional kernel with a size of 3*3 is used to perform overlapping sliding convolution on the current frame of video image with a specified stride of 2*2, obtaining a target local feature of size 20*20 pixels. The feature value at each pixel of the target local feature corresponds to the feature of a 3*3 region in the original image, and thus this target local feature can be used as the local image feature of the corresponding frame of video image.

[0208] Step d4: Encode multiple first feature codes, multiple second feature codes, and third feature codes as semantic features.

[0209] Step S2042: The encoded semantic features are decoded by the target decoder built into the scene semantic reasoning model to obtain the target semantic description text of the target video segment.

[0210] In this embodiment of the invention, by decoding the encoded semantic features through a target decoder, words at each position of the statements used to describe the target video segment can be predicted in a targeted manner, thereby obtaining target semantic description text that can be used to describe the scene content corresponding to the target video segment.

[0211] In some optional implementations, the target decoder includes a first decoder, a second decoder, and a global vocabulary, and step S2042 above includes:

[0212] Step e1 involves decoding multiple second and third feature codes using a first decoder and a global vocabulary library to obtain the first semantic description text of the target video segment.

[0213] In this approach, multiple second feature encodings are first processed step-by-step using a hierarchical attention module that includes temporal attention and target attention to obtain the first feature, highlighting the target region while distinguishing the contributions of different targets. The third feature encoding is then processed using a temporal attention module to obtain the second feature, highlighting keyframe video images that can represent multiple video images. Within the first decoder, the processed first and second features are decoded, and then combined with a global vocabulary to predict the word prediction probability distribution at each position when semantically describing the target video segment, thus obtaining the first semantic description text. That is, in the first semantic description text, the words at each position are the words with the highest prediction probability from the first decoder.

[0214] Step e2 involves decoding multiple first and third feature codes using a second decoder and a global vocabulary library to obtain the second semantic description text of the target video segment.

[0215] In this approach, a temporal attention module is first used to process multiple first and third feature codes. Then, within the second decoder, the processed results are combined with the global vocabulary to analyze the word prediction probability distribution at each position when semantically describing the target video segment, thus obtaining the second semantic description text. That is, in the second semantic description text, the word at each position is the word with the highest prediction probability from the second decoder.

[0216] Step e3: Multiply the first semantic description text by the first weight corresponding to the first decoder to obtain the first intermediate parameter.

[0217] In this approach, since the first semantic description text is obtained based on the second and third feature encodings, it is more inclined to describe the motion state of each target in the target video segment within the target time period. The second semantic description text is more inclined to describe the social interaction relationships of multiple targets in the same frame of the target video segment. Therefore, to obtain the target semantic description text used to describe the scene content corresponding to the target video segment, the first semantic description text is first multiplied by the first weight corresponding to the first decoder to obtain the first intermediate parameter.

[0218] Step e4 involves multiplying the second semantic description text by the first weight corresponding to the second decoder to obtain the second intermediate parameter. The processing principle of this step is the same as that of step e3, and will not be repeated here.

[0219] Step e5: Combine the first intermediate parameter and the second intermediate parameter to obtain the target semantic description text of the target video segment.

[0220] In this method, the first semantic description text and the second semantic description text have the same length, but the words at different positions may differ. Since the first weight and the second weight are also different, in order to obtain a target semantic description text that can reasonably describe the scene content corresponding to the target video segment, the words at the same position in the first intermediate parameter and the second intermediate parameter are compared one by one, and the word with the highest probability value is retained, thus obtaining the target semantic description text of the target video segment.

[0221] For example, the first intermediate parameter is: Target A and target B are running on the road. The second intermediate parameter is: Target A and target B are running on the track. If the weight of "road" is 89.5% and the weight of "track" is 94.4%, then the final target semantic description text is: Target A and target B are running on the track.

[0222] The video processing method provided in this embodiment can fully explore the social interaction relationships between multiple targets in a target video segment and the motion state of each target within the target time period from the time and space dimensions, so that the obtained target semantic description text can better fit the scene content corresponding to the target video segment.

[0223] In one implementation scenario, the process of obtaining the target semantic description text through a pre-trained scene semantic reasoning model can be as follows: Figure 4 As shown.

[0224] A pre-built framework for the scene semantic reasoning model is constructed, including: a target encoder framework, a hierarchical attention module, a temporal attention module, and a target decoder framework. The target encoder framework includes a spatial encoder, a temporal encoder, and a global up-and-down encoder model. A GCN is used to construct the spatial encoder, a VLAD model containing C-GRU units is used to construct the temporal encoder, and a VLAD model is used to construct the global up-and-down encoder model. The target decoder framework includes a first decoder and a second decoder, both of which consist of multiple GRU units.

[0225] Video samples for model training are obtained from a large video database on the network, and a spatiotemporal scene graph of the video sample is constructed. The video samples and their spatiotemporal scene graphs are input into a pre-built framework for a scene semantic reasoning model, and the scene semantic reasoning model is trained using a cross-entropy loss function until the obtained loss function value is less than or equal to a specified threshold. The training is then complete, and the scene semantic reasoning model is obtained.

[0226] In some examples, during the training of the scene semantic reasoning model, the hidden layer a of the temporal encoder is targeted. it The update process is as follows:

[0227] z it =σ(W z *r it +U z *a i(t-1) )

[0228] h it =σ(W r *r it +U r *a i(t-1) )

[0229]

[0230]

[0231] Among them W z W r W a And U z U r U a denoted by 2D convolution kernel, * denotes convolution operation, ⊙ denotes element-wise multiplication operation, and σ() is the sigmoid non-linear activation function. When hidden layer a... it Once the update is confirmed, it can be used as the feature clustering weight for the second feature extraction process of the time series graph.

[0232] This embodiment provides a video processing method that can be used in the aforementioned electronic devices, such as computers and servers. Figure 5 This is a flowchart of a video processing method according to an embodiment of the present invention, such as... Figure 5 As shown, the process includes the following steps:

[0233] Step S501: Obtain the target video segment.

[0234] Step S502: Perform target detection processing on multiple frames of video images to identify multiple targets in the multiple frames of video images and obtain target detection results for multiple targets.

[0235] Step S503: Based on the target detection results, construct a spatiotemporal scene map of multiple targets in the target video segment.

[0236] Step S504: Input the spatiotemporal scene map and the target video clip into the pre-trained scene semantic reasoning model for semantic feature extraction to obtain the target semantic description text of the target video clip.

[0237] Step S505: Construct quadruple samples.

[0238] In this embodiment of the invention, to achieve abnormal event detection under an open set setting, a four-tuple sample is first constructed. This four-tuple sample is then used to train a scene semantic reasoning model to identify the semantic features of normal video segments. Subsequently, when semantically describing an input target video segment, it can simultaneously determine whether there are abnormal frame video images within the target video segment, thus determining whether the target video segment is an abnormal video segment. The four-tuple sample consists of normal video segment samples of the same duration, normal reference video segment samples, abnormal video segment samples, and abnormal reference video segment samples.

[0239] Step S506: Input the quadruple samples into the target encoder for feature encoding processing to obtain four sets of segment encoded semantic features.

[0240] In this embodiment of the invention, the four sets of segment coding semantic features include a first segment coding semantic feature corresponding to a normal video segment sample, a second segment coding semantic feature corresponding to a normal reference video segment sample, a third segment coding semantic feature corresponding to an abnormal video segment sample, and a fourth segment coding semantic feature corresponding to an abnormal reference video segment sample. Each segment coding semantic feature can be obtained by mapping the corresponding original video segment sample to a feature space. The original video segment samples include: normal video segment samples, normal reference video segment samples, abnormal video segment samples, or abnormal reference video segment samples.

[0241] Step S507: Determine the first feature spacing of normal sample pairs based on the first segment encoded semantic features and the second segment encoded semantic features.

[0242] In an embodiment of the present invention, the first feature distance between the first segment encoded semantic features and the second segment encoded semantic features is calculated using the following Euclidean distance formula:

[0243]

[0244] In this case, the semantic features encoded in the first segment can be substituted into f. * Substitute the semantic features encoded in the second segment into f · And thus obtain As the first feature spacing.

[0245] Step S508: Based on the semantic features encoded in the third and fourth segments, determine the second feature distance between the abnormal sample pairs. The principle used to calculate the second feature distance is the same as that used to calculate the first feature distance, and will not be elaborated further here.

[0246] Step S509: Based on the semantic features encoded in the second and fourth segments, determine the third feature distance between normal and abnormal sample pairs. The principle used to calculate the third feature distance is the same as that used to calculate the first feature distance, and will not be elaborated further here.

[0247] Step S510: Using the first feature spacing, the second feature spacing, and the third feature spacing, the parameters of the target encoder are adjusted through a preset contrast loss function. If the obtained loss value is less than or equal to the specified loss threshold, the adjustment of the parameters of the target encoder is stopped, and the updated target encoder is obtained.

[0248] In this embodiment of the invention, the preset contrast loss function is a contrast loss function specifically set for the semantic features of the quadruplet segment encoding. By performing boundary metric learning through this contrast loss function, the distance between features of normal video segments can be reduced, while the features of abnormal video segments can be separated from the features of normal video segments, thereby enabling accurate identification of whether the input target video segment is a normal video segment.

[0249] The formula for the contrastive loss function is as follows:

[0250]

[0251]

[0252] in, Let n be the first feature distance between normal video clip sample n and normal reference video clip sample an. The second feature distance between the abnormal video segment sample p and the abnormal reference video segment sample ap. The distance between the normal reference video segment sample *an* and the abnormal reference video segment sample *ap* is the third feature distance. Wherein, as... Figure 6 As shown, when When the distance between features of the same type of sample pair is... The value of is set to 1, which minimizes the loss function. As the number of samples gradually decreases, similar sample features continuously cluster in the feature space; conversely, when the number of samples increases... When the distance between features of heterogeneous sample pairs is... The value of is set to 0, which minimizes the loss function. As the number of samples gradually increases, the features of outlier samples move further and further away in the feature space. By minimizing the QC-Loss, we can simultaneously bring the distance between members within the normal feature distribution and the known outlier feature distribution (intra-class distance) closer together, and expand the distance between members of the two distributions (inter-class distance), forming a compact normal and outlier distribution that simultaneously creates feature boundaries between the two distributions.

[0253] Step S511: Update the scene semantic reasoning model according to the updated target encoder to obtain the updated scene semantic reasoning model.

[0254] In this embodiment of the invention, the scene semantic reasoning model is updated according to the updated target encoder, so that the updated scene semantic reasoning model can automatically detect whether the target video segment is a normal video segment during the semantic feature extraction process, thereby improving the practicality of video detection.

[0255] The video processing method provided in this embodiment can, under an open set setting, utilize an updated scene semantic reasoning model to generate target semantic description text for a target video segment while simultaneously detecting whether the target video segment is a normal segment. When an abnormal segment is detected, it can be accurately identified and described, thereby enhancing the anomaly detection capability.

[0256] In some optional implementations, the video processing method described above further includes the following steps:

[0257] Step S512: Input the normal video sample into the updated target encoder for feature extraction to obtain the target feature code.

[0258] Step S513: Perform clustering processing on the target feature encoding to determine the target metric center of the target feature encoding.

[0259] Step S514: Determine the center distance between the feature cluster centers that encode semantic features and the target metric center.

[0260] Step S515: If there is a center distance greater than the specified distance threshold, the target video segment is determined to be an abnormal video segment.

[0261] Specifically, the updated target encoder is used to extract semantic features from normal video samples to obtain target feature codes. In some optional implementations, Att-sConv and tConv-Sia networks can be used to extract spatiotemporal features from normal video samples to obtain target feature codes. Since there are multiple normal video samples, the target feature codes are equivalent to the set Ω of target feature codes for each normal video sample. By clustering the target feature codes, the target metric center of the target feature codes can be determined and used as the metric center for measuring whether the video segment to be detected is a normal segment. The center distance between the feature cluster center of the encoded semantic features of the target video segment and the target metric center is calculated using the Euclidean standard distance formula, and this center distance is used as the anomaly score, calculated as follows:

[0262]

[0263] in, The feature cluster centers, f, represent the encoded semantic features of the target video segment. c This represents the target metric center. If the video segment to be tested contains anomalous events (known or unknown types), given that normal features have already clustered into a compact distribution in the feature space, its semantic features will deviate from f. c This results in a larger s(t) value; conversely, semantic features will be closer to f. c This results in a small s(t) value. Therefore, to determine whether the target video segment contains abnormal events, a preset distance threshold θ is used. The preset distance threshold θ can be understood as the maximum distance required to determine if the video segment to be detected is a normal video segment; that is, it is considered to be the maximum score required to determine if the video segment to be detected is a normal video segment. If s(t) > θ, then the target video segment is determined to be an abnormal video segment.

[0264] In some alternative embodiments, the video processing method described above further includes the following steps:

[0265] Step S516: If there is no center distance greater than the specified distance threshold, the target video segment is determined to be a normal video segment.

[0266] In one implementation scenario, the above comparison process for determining whether a target video segment is a normal video segment can be utilized. Figure 7 Example: The target measurement center is located at the center point of the circle, the circle contains the set Ω of target feature codes for each normal video sample, and the measurement boundary is a specified distance threshold θ. If the calculated center distance s(t) > θ, the target video segment is considered an abnormal video segment. If the calculated center distance s(t) ≤ θ, the target video segment is considered a normal video segment.

[0267] In some optional implementation scenarios, the process from training the scene semantic reasoning model to applying the trained scene semantic reasoning model to perform video detection on the target video segment and obtain the target semantic description text can be as follows: Figure 8 As shown in the diagram. For clarity and ease of description, the semantic features of the first segment of a normal video clip sample are represented by normal features, the semantic features of the second segment of a normal reference video clip sample are represented by normal reference features, the semantic features of the third segment of an abnormal video clip sample are represented by abnormal features, and the semantic features of the fourth segment of an abnormal reference video clip sample are represented by abnormal reference features. Since the specific implementation process of each step has been described above, it will not be repeated here.

[0268] In some alternative implementations, the target video segment can be acquired by: acquiring the target video; determining the segment length to be segmented in the target video; dividing the target video according to the segment length to obtain multiple video segments, wherein the multiple video segments include the target video segment. Using segmentation for video detection can improve video detection efficiency, enabling rapid location of abnormal video segments when anomalies are found in the target video.

[0269] In some alternative implementations, the segment duration can be determined as follows: the number of video segments is determined based on the total duration of the target video; the quotient of the total duration divided by the number of video segments is used as the segment duration.

[0270] Based on the same inventive concept, this invention also provides an anomaly event detection system for video surveillance. The network framework of the anomaly event detection system can be as follows: Figure 9 As shown, this abnormal event detection system comprises four modules: a video data acquisition and preprocessing module 901, a video data storage module 902, a video abnormal event detection module 903, and a system management module 904. Data interaction between the different modules is achieved through network transmission.

[0271] The video data acquisition and preprocessing module 901 primarily undertakes two tasks: firstly, it receives video stream data transmitted back from the front-end camera equipment via the network and encodes and decodes the raw video stream to obtain MP4 video files; secondly, it performs distortion and blurring operations on the video data to optimize video image quality. The preprocessed video data is then saved to the server.

[0272] After reading video data, the video anomaly detection module 903, following its designed algorithm flow, detects anomalies in the video and provides semantic descriptions based on four sub-modules: object detection and spatiotemporal scene graph construction, semantic feature extraction, boundary metric learning, and anomaly event discrimination and description. Specifically, the object detection and spatiotemporal scene graph construction sub-module segments the original video (target video) into target video segments, regresses the scene target region, and constructs the spatiotemporal scene graph corresponding to the target video segment. The semantic feature extraction sub-module uses the spatiotemporal scene graph as input and employs a pre-trained video scene semantic description model to obtain high-level semantic features of the video segments. The boundary metric learning sub-module, based on the semantic features of the video segments, designs a four-tuple contrastive loss function for metric learning under an open-set setting, updating the encoder parameters of the pre-trained semantic description model. The anomaly event discrimination sub-module, based on the metric learning results, calculates the distance between the semantic features of the test video segment and the center of the normal feature distribution to achieve anomaly event detection, while simultaneously using the semantic description model to provide anomaly event descriptions. The front-end interface of the video anomaly detection module 903 can be implemented using the PyQt framework provided by Python.

[0273] The system management module 904 serves as the underlying support for the other three modules, and is responsible for providing organizational management services, such as process management, coordinating the priority of different modules, and avoiding process errors.

[0274] In the video processing method provided by this invention, firstly, the target video sequence is segmented into multiple segments of fixed time length, and target detection is performed on the video segments to regress the position coordinates of different targets in the scene. Based on this, a spatiotemporal scene graph is constructed using the detected targets as nodes, modeling the interaction relationships of different targets and the trajectory of the same target from both spatial and temporal dimensions of the video segments. Secondly, based on the spatiotemporal scene graph, a pre-trained semantic description model is introduced to extract high-level semantic features of the video segments. Thirdly, using the semantic features as input, a four-tuple contrastive loss function is designed to perform boundary metric learning under an open set setting, updating and fine-tuning the parameters of the semantic description model. This brings members within the same feature distribution closer together while expanding the distance between members of dissimilar feature distributions, constructing a compact normal video segment feature distribution and a known type of abnormal video segment feature distribution, and forming a metric boundary between the two distributions. Finally, using the metric learning results, the Euclidean distance between the corresponding semantic features of the video segment under test and the center of the normal feature distribution is calculated as an anomaly score, used to determine whether the segment contains an abnormal event. Simultaneously, the semantic description model outputs anomaly description text.

[0275] This embodiment also provides a video processing apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0276] This embodiment provides a video processing device, such as... Figure 10 As shown, it includes:

[0277] The acquisition module 1001 is used to acquire a target video segment, which includes a series of consecutive video images.

[0278] The detection module 1002 is used to perform target detection processing on multiple frames of video images to identify multiple targets in the multiple frames of video images and obtain target detection results for multiple targets;

[0279] The first construction module 1003 is used to construct a spatiotemporal scene map of multiple targets in the target video segment based on the target detection results;

[0280] The first processing module 1004 is used to input the spatiotemporal scene graph and the target video segment into a pre-trained scene semantic reasoning model for semantic feature extraction, so as to obtain the target semantic description text of the target video segment.

[0281] In some optional implementations, the target detection results include the sub-target detection results of each target in different frame video images. The first construction module 1003 includes: a first processing unit, used to identify the interaction relationship of multiple targets in each frame video image based on the sub-target detection results of multiple targets in different frame video images, and obtain multiple spatial maps, with different spatial maps corresponding to different frame video images; a second processing unit, used to track the positional changes of each target in the target time period based on the sub-target detection results of multiple targets in different frame video images and the temporal order of multiple frame video images in the target video segment, and obtain multiple temporal maps, with different temporal maps corresponding to different targets, and the target time period being the time period corresponding to the target video segment; and a construction unit, used to construct a spatiotemporal scene map of multiple targets in the target video segment based on the multiple spatial maps and the multiple temporal maps.

[0282] In some optional implementations, the first processing unit includes: a first execution unit, configured to determine the position information and local region information of each target in the current frame video image based on the sub-target detection results of multiple targets in the current frame video image; a second execution unit, configured to determine the interaction strength between every two targets in the current frame video image based on the position information and local region information of each target; and a first sub-construction unit, configured to construct an adjacency matrix based on the interaction strength between every two targets in the current frame video image, so as to construct a spatial graph corresponding to the current frame video image based on the adjacency matrix.

[0283] In some optional implementations, the multiple targets include a first target and a second target. The second execution unit includes: a first extraction unit, configured to extract a first visual feature of the first target in the current frame video image based on the location information of the first target; a second extraction unit, configured to extract a second visual feature of the second target in the current frame video image based on the location information of the second target; a first determination unit, configured to determine the semantic relevance between the first target and the second target based on the Euclidean distance between the first visual feature and the second visual feature; a second determination unit, configured to determine the spatial overlap between the first target and the second target in the current frame video image based on the location information of the first target, the local region information of the first target, the location information of the second target, and the local region information of the second target, thereby obtaining the positional relevance between the first target and the second target; and a second sub-execution unit, configured to obtain the interaction strength between the first target and the second target through the semantic relevance and the positional relevance.

[0284] In some optional implementations, the second processing unit includes: a third execution unit, configured to determine the region position of each target in each frame of video image based on the sub-target detection results of multiple targets in different frame video images; and a second sub-construction unit, configured to construct the motion trajectory of each target within the target time period based on the region position of each target in each frame of video image and the temporal order of multiple frame video images in the target video segment, so as to construct a time sequence map of the corresponding target based on the motion trajectory.

[0285] In some optional implementations, the multiple targets include a first target, and the multiple video images include a first video image and a first frame video image; the third execution unit includes: a third determining unit, configured to determine the first region position of the first target in the first frame video image based on the sub-target detection results of the first target in the first frame video image; a fourth determining unit, configured to determine the second region position of each target in the first frame video image based on the sub-target detection results of the multiple targets in the first frame video image; and a matching unit, configured to perform semantic matching processing on the first region position and each second region position respectively, and take the second region position with the highest semantic matching degree with the first region position as the target region position of the first target in the first frame video image.

[0286] In some optional implementations, the construction unit includes: a fourth execution unit, used to use multiple targets as graph nodes of a spatiotemporal scene graph to obtain multiple target nodes; a fifth execution unit, used to use multiple spatial graphs as spatial edges of a spatiotemporal scene graph, and arrange them according to the temporal order of the corresponding frame video images of each spatial graph in the target video segment to obtain multiple target spatial edges; a sixth execution unit, used to use multiple temporal graphs as temporal edges of a spatiotemporal scene graph to obtain multiple target temporal edges; and a seventh execution unit, used to connect each target node to its corresponding target spatial edge and its corresponding target temporal edge to obtain a spatiotemporal scene graph of multiple targets in the target video segment.

[0287] In some optional implementations, the first processing module 1004 includes: an encoding module, used to input the spatiotemporal scene graph and the target video segment into a pre-trained scene semantic reasoning model, and to perform feature encoding processing on the spatiotemporal scene graph and the target video segment through a target encoder built into the scene semantic reasoning model to obtain encoded semantic features; and a decoding module, used to decode the encoded semantic features through a target decoder built into the scene semantic reasoning model to obtain the target semantic description text of the target video segment.

[0288] In some optional implementations, the target encoder includes: a spatial encoder, a temporal encoder, and a global context encoder; the encoding module includes: a first encoding unit, used to perform first feature encoding processing on each spatial graph in the spatiotemporal scene graph through the spatial encoder to obtain multiple first feature codes, and the multiple first feature codes and multiple spatial... Figure One One-to-one correspondence; the second encoding unit is used to perform second feature encoding processing on each temporal graph in the spatiotemporal scene graph through a temporal encoder, resulting in multiple second feature codes, and the multiple second feature codes are associated with multiple temporal graphs. Figure OneOne-to-one correspondence; the third encoding unit is used to perform third feature encoding processing on the target video segment through the global context encoder to obtain the third feature encoding; the eighth execution unit is used to encode multiple first feature encodings, multiple second feature encodings and third feature encodings as semantic features.

[0289] In some optional implementations, the first encoding unit includes: a first feature extraction unit, configured to perform a first feature extraction process on the current spatial map within the spatial encoder to obtain node features of multiple target nodes, wherein the current spatial map is any spatial map in the spatiotemporal scene map; and a first processing unit, configured to perform mean pooling processing on the node features of the multiple target nodes to obtain the first feature encoding corresponding to the current spatial map.

[0290] In some optional implementations, the second encoding unit includes: a second feature extraction unit, used to perform second feature extraction processing on the current time sequence map within the time sequence encoder to obtain image features of the target corresponding to the current time sequence map at different times, wherein the current time sequence map is any time sequence map in the spatiotemporal scene map; a fifth determination unit, used to determine the feature clustering center of the encoded semantic features; and a second processing unit, used to perform clustering encoding processing on the size information, image features, and feature clustering center of the target corresponding to the current time sequence map at different times to obtain the second feature encoding corresponding to the current time sequence map.

[0291] In some optional implementations, the target decoder includes a first decoder, a second decoder, and a global vocabulary; the decoding module includes: a first decoding unit, used to decode multiple second feature codes and third feature codes using the first decoder and the global vocabulary to obtain a first semantic description text of the target video segment; a second decoding unit, used to decode multiple first feature codes and third feature codes using the second decoder and the global vocabulary to obtain a second semantic description text of the target video segment; a third processing unit, used to multiply the first semantic description text by a first weight corresponding to the first decoder to obtain a first intermediate parameter; a fourth processing unit, used to multiply the second semantic description text by a first weight corresponding to the second decoder to obtain a second intermediate parameter; and a fifth processing unit, used to combine the first intermediate parameter and the second intermediate parameter to obtain the target semantic description text of the target video segment.

[0292] In some optional implementations, the method further includes: a second construction module for constructing quadruple samples, wherein the quadruple samples consist of normal video segment samples, normal reference video segment samples, abnormal video segment samples, and abnormal reference video segment samples of the same video duration; a second processing module for inputting the quadruple samples into a target encoder for feature encoding processing to obtain four sets of segment encoded semantic features, wherein the four sets of segment encoded semantic features include a first segment encoded semantic feature corresponding to the normal video segment sample, a second segment encoded semantic feature corresponding to the normal reference video segment sample, a third segment encoded semantic feature corresponding to the abnormal video segment sample, and a fourth segment encoded semantic feature corresponding to the abnormal reference video segment sample; and a first execution module for performing feature encoding based on the first segment encoded semantic feature and the second segment encoded semantic feature. The system comprises the following modules: a first feature interval for normal sample pairs; a second execution module for determining the second feature interval for abnormal sample pairs based on the third and fourth segment encoded semantic features; a third execution module for determining the third feature interval between normal and abnormal sample pairs based on the second and fourth segment encoded semantic features; an adjustment module for adjusting the parameters of the target encoder using the first, second, and third feature intervals and a preset contrastive loss function; and an update module for updating the scene semantic inference model based on the updated target encoder.

[0293] In some optional implementations, the method further includes: a feature extraction module for inputting normal video samples into the updated target encoder for feature extraction to obtain target feature codes; a fourth execution module for performing clustering processing on the target feature codes to determine the target metric center of the target feature codes; a distance determination module for determining the center distance between the feature cluster centers of the encoded semantic features and the target metric center; and a first judgment module for determining that if there is a center distance greater than a specified distance threshold, the target video segment is an abnormal video segment.

[0294] In some optional implementations, the method further includes: a second determination module, used to determine that the target video segment is a normal video segment if there is no center distance greater than a specified distance threshold.

[0295] In some optional implementations, the acquisition module includes: an acquisition unit for acquiring the target video; a sixth determining unit for determining the segment duration of the target video to be segmented; and a segmentation unit for dividing the target video according to the segment duration to obtain multiple video segments, wherein the multiple video segments include the target video segment.

[0296] In some optional implementations, the sixth determining unit includes: a quantity determining unit, used to determine the number of video segments based on the total duration of the target video; and a sixth determining subunit, used to take the quotient of the total duration divided by the number of video segments as the segment duration.

[0297] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0298] In this embodiment, the video processing device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0299] This invention also provides an electronic device having the above-described features. Figure 10 The video processing device shown.

[0300] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of an electronic device provided in an optional embodiment of the present invention, such as... Figure 11 As shown, the electronic device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise as required. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 11 Take a processor 10 as an example.

[0301] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0302] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0303] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0304] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0305] The electronic device also includes an input device 30 and an output device 40. The processor 10, memory 20, input device 30, and output device 40 can be connected via a bus or other means. Figure 10 Taking the example of a connection between China and Israel via a bus.

[0306] Input device 30 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the electronic device, such as a touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 40 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The aforementioned display devices include, but are not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touch screen.

[0307] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0308] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method of video processing, the method comprising: The method comprises: obtaining a target video clip, the target video clip comprising a plurality of continuous video images; performing target detection processing on the plurality of video images to identify a plurality of targets in the plurality of video images, to obtain target detection results of the plurality of targets, wherein the target detection results comprise sub-target detection results of each target in different frames of video images; based on the sub-target detection results of the plurality of targets in different frames of video images, identifying the interaction relationship of the plurality of targets in each frame of video image, to obtain a plurality of spatial graphs, different spatial graphs corresponding to different frames of video images; based on the sub-target detection results of the plurality of targets in different frames of video images and the time sequence of the plurality of video images in the target video clip, tracking the position change of each target in a target period, to obtain a plurality of time sequence graphs, different time sequence graphs corresponding to different targets, the target period being a time period corresponding to the target video clip; based on the plurality of spatial graphs and the plurality of time sequence graphs, constructing a spatio-temporal scene graph of the plurality of targets in the target video clip; inputting the spatio-temporal scene graph and the target video clip into a pre-trained scene semantic reasoning model to perform semantic feature extraction, to obtain a target semantic description text of the target video clip.

2. The method of claim 1, wherein, The method comprises: based on the sub-target detection results of the plurality of targets in different frames of video images, identifying the interaction relationship of the plurality of targets in each frame of video image, to obtain a plurality of spatial graphs, comprising: determining the position information and the local area information of each target in the current frame of video image according to the sub-target detection results of the plurality of targets in the current frame of video image; based on the position information and the local area information of each target, respectively determining the interaction relationship strength between each two targets in the current frame of video image, wherein the interaction relationship strength is used to indicate the interaction strength between two targets; 3. The method of claim 2, wherein, constructing an adjacency matrix according to the interaction relationship strength between each two targets in the current frame of video image, to construct the spatial graph corresponding to the current frame of video image according to the adjacency matrix. The plurality of targets comprises a first target and a second target, and the method comprises: extracting a first visual feature of the first target in the current frame of video image according to the position information of the first target; extracting a second visual feature of the second target in the current frame of video image according to the position information of the second target; determining the semantic correlation degree between the first target and the second target based on the Euclidean distance between the first visual feature and the second visual feature; determining the position correlation degree between the first target and the second target in the current frame of video image according to the position information of the first target, the local area information of the first target, the position information of the second target and the local area information of the second target, to obtain the position correlation degree between the first target and the second target. An interaction strength between the first target and the second target is obtained through the semantic correlation and the location correlation.

4. The method of claim 1, wherein, The position changes of each target in the target period are tracked based on the sub-target detection results of the multiple targets in different frames of video images and the time sequence of the multiple frames of video images in the target video segment, and multiple time sequence graphs are obtained, including: The area positions of each target in each frame of video image are determined respectively according to the sub-target detection results of the multiple targets in different frames of video images; The motion trajectories of each target in the target period are constructed according to the area positions of each target in each frame of video image and the time sequence of the multiple frames of video images in the target video segment, so as to construct the time sequence graph of the corresponding target according to the motion trajectory.

5. The method of claim 4, wherein the multiple targets include a first target, and the multiple frames of video images include a first frame of video image and a first frame of video image, wherein the first frame of video image is any non-first frame of video image in the multiple frames of video image; the area positions of each target in each frame of video image are determined respectively according to the sub-target detection results of the multiple targets in different frames of video images, including: a first area position of the first target in the first frame of video image is determined according to the sub-target detection result of the first target in the first frame of video image; a second area position of each target in the first frame of video image is determined according to the sub-target detection results of the multiple targets in the first frame of video image; the first area position is matched with each second area position respectively to obtain a target area position of the first target in the first frame of video image, which has the highest semantic matching degree with the first area position.

6. The method of claim 1, wherein, the spatio-temporal scene graph of the multiple targets in the target video segment is constructed based on the multiple spatial graphs and the multiple time sequence graphs, including: the multiple targets are taken as graph nodes of the spatio-temporal scene graph to obtain multiple target nodes; the multiple spatial graphs are taken as spatial edges of the spatio-temporal scene graph, and the multiple target spatial edges are obtained by arranging the multiple spatial graphs according to the time sequence of the corresponding frames of video image in the target video segment; the multiple time sequence graphs are taken as time sequence edges of the spatio-temporal scene graph to obtain multiple target time sequence edges; each target node is connected with the corresponding target spatial edge and the corresponding target time sequence edge respectively to obtain the spatio-temporal scene graph of the multiple targets in the target video segment.

7. The method of claim 6, wherein, the spatio-temporal scene graph and the target video segment are input into a pre-trained scene semantic reasoning model to extract semantic features, and a target semantic description text of the target video segment is obtained, including: inputting the spatio-temporal scene graph and the target video segment into a pre-trained scene semantic inference model, performing feature coding processing on the spatio-temporal scene graph and the target video segment by a target encoder built in the scene semantic inference model, to obtain coded semantic features; performing decoding processing on the coded semantic features by a target decoder built in the scene semantic inference model, to obtain a target semantic description text of the target video segment.

8. The method of claim 7, wherein, the target encoder comprises a spatial encoder, a temporal encoder, and a global context encoder; the performing feature coding processing on the spatio-temporal scene graph and the target video segment by the target encoder built in the scene semantic inference model comprises: performing first feature coding processing on each spatial graph in the spatio-temporal scene graph by the spatial encoder, respectively, to obtain a plurality of first feature codes, the plurality of first feature codes corresponding to the plurality of spatial graphs one by one; performing second feature coding processing on each temporal graph in the spatio-temporal scene graph by the temporal encoder, respectively, to obtain a plurality of second feature codes, the plurality of second feature codes corresponding to the plurality of temporal graphs one by one; performing third feature coding processing on the target video segment by the global context encoder, to obtain a third feature code; the plurality of first feature codes, the plurality of second feature codes, and the third feature code are taken as the coded semantic features.

9. The method of claim 8, wherein, the performing first feature coding processing on each spatial graph in the spatio-temporal scene graph by the spatial encoder comprises: performing first feature extraction processing on a current spatial graph in the spatial encoder, to obtain node features of the plurality of target nodes, the current spatial graph being any one of the spatial graphs in the spatio-temporal scene graph; performing mean pooling processing on the node features of the plurality of target nodes, to obtain a first feature code corresponding to the current spatial graph.

10. The method of claim 8, wherein, the performing second feature coding processing on each temporal graph in the spatio-temporal scene graph by the temporal encoder comprises: performing second feature extraction processing on a current temporal graph in the temporal encoder, to obtain image features of a target at different time instants corresponding to the current temporal graph, the current temporal graph being any one of the temporal graphs in the spatio-temporal scene graph; determining a feature clustering center of the coded semantic features; performing clustering coding processing on size information, image features, and the feature clustering center of the target at different time instants corresponding to the current temporal graph, to obtain a second feature code corresponding to the current temporal graph.

11. The method of any one of claims 8-10, wherein, the target decoder comprises a first decoder, a second decoder, and a global word library; the performing decoding processing on the coded semantic features by the target decoder built in the scene semantic inference model comprises: The first semantic description text and the first weight corresponding to the first decoder are multiplied to obtain a first intermediate parameter. The second semantic description text and the first weight corresponding to the second decoder are multiplied to obtain a second intermediate parameter. The first intermediate parameter and the second intermediate parameter are combined to obtain a target semantic description text of the target video segment. The method further comprises: A four-tuple sample is constructed, and the four-tuple sample is composed of a normal video segment sample, a normal reference video segment sample, an abnormal video segment sample, and an abnormal reference video segment sample with the same video duration; 12. The method of claim 11, wherein, The four-tuple sample is input into the target encoder for feature encoding processing to obtain four groups of segment encoding semantic features, including a first segment encoding semantic feature corresponding to the normal video segment sample, a second segment encoding semantic feature corresponding to the normal reference video segment sample, a third segment encoding semantic feature corresponding to the abnormal video segment sample, and a fourth segment encoding semantic feature corresponding to the abnormal reference video segment sample; A first feature distance between a normal sample pair is determined based on the first segment encoding semantic feature and the second segment encoding semantic feature; A second feature distance between an abnormal sample pair is determined based on the third segment encoding semantic feature and the fourth segment encoding semantic feature; A third feature distance between the normal sample pair and the abnormal sample pair is determined based on the second segment encoding semantic feature and the fourth segment encoding semantic feature; The first feature distance, the second feature distance, and the third feature distance are used to adjust the parameters of the target encoder through a preset contrast loss function, and if a loss value obtained is less than or equal to a specified loss threshold, the adjustment of the parameters of the target encoder is stopped, and an updated target encoder is obtained. The scene semantic reasoning model is updated according to the updated target encoder to obtain an updated scene semantic reasoning model. The method further comprises: The normal video segment sample is input into the updated target encoder for feature extraction to obtain a target feature code; 13. The method of claim 12, wherein, The target feature code is clustered to determine a target metric center of the target feature code; A center distance between a feature clustering center of the encoding semantic feature and the target metric center is determined; If there is a center distance greater than a specified distance threshold, the target video segment is determined to be an abnormal video segment. The method further comprises: If there is no center distance greater than the specified distance threshold, the target video segment is determined to be a normal video segment.

14. The method of claim 13, wherein, The target video segment is obtained, comprising: A target video is obtained.

15. The method of claim 1, wherein, ​ ​ determine a segment duration to be used for segmenting the target video; divide the target video according to the segment duration to obtain a plurality of video segments, the plurality of video segments including the target video segment.

16. The method of claim 15, wherein, The determination of the segment duration to be used for segmenting the target video includes: determine a number of video segments according to a total duration of the target video; divide the total duration by the number of video segments to obtain a quotient as the segment duration.

17. A video processing apparatus, comprising: The apparatus includes: an acquisition module configured to acquire a target video segment, the target video segment including a plurality of continuous video images; a detection module configured to perform target detection on the plurality of video images to identify a plurality of targets in the plurality of video images and obtain target detection results of the plurality of targets, wherein the target detection results include sub-target detection results of each target in different video images; a first construction module configured to identify interaction relationships of the plurality of targets in each video image based on the sub-target detection results of the plurality of targets in different video images, obtain a plurality of spatial graphs, different spatial graphs corresponding to different video images, track position changes of each target in a target period based on the sub-target detection results of the plurality of targets in different video images and a time sequence of the plurality of video images in the target video segment, obtain a plurality of time sequence graphs, different time sequence graphs corresponding to different targets, the target period being a time period corresponding to the target video segment, and construct a spatio-temporal scene graph of the plurality of targets in the target video segment based on the plurality of spatial graphs and the plurality of time sequence graphs; a first processing module configured to input the spatio-temporal scene graph and the target video segment into a pre-trained scene semantic inference model to extract semantic features and obtain a target semantic description text of the target video segment.

18. An electronic device, comprising: include: a memory and a processor, which are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the video processing method in any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are used to make a computer execute the video processing method in any one of claims 1 to 16.

Images