Industrial surveillance video query and positioning method and device based on natural language

Abstract

The embodiments of the present disclosure disclose a natural language-based industrial surveillance video query and positioning method and device. A specific implementation of the method includes: in response to receiving a query text input by a user and performing text feature extraction to obtain query text features; extracting features from the surveillance video to obtain a video frame feature vector sequence; performing foreground and background separation on the video frame feature vector sequence to obtain a video frame foreground feature vector sequence; performing global information extraction on each video frame feature vector in the video frame feature vector sequence to obtain a video frame spatial feature vector sequence; performing foreground enhancement on each video frame spatial feature vector in the video frame spatial feature vector sequence to obtain an enhanced video frame feature vector sequence; performing video frame positioning on the enhanced video frame feature vector sequence to generate video positioning fragment information. This implementation can improve the efficiency and accuracy of surveillance video query and positioning tasks.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and in particular to a natural language-based industrial surveillance video query and positioning method and device. Background Art

[0002] During the production of mechanical parts and components or the import and export of products by customs, surveillance videos are often required for subsequent review. However, there is a large amount of redundant and sparsely distributed information in the video data, which poses a challenge to video retrieval and fine-grained positioning tasks. At present, when querying and positioning surveillance videos, the usual approach is to use natural language video positioning technology to solve video retrieval and positioning tasks. The goal of natural language video positioning technology is to accurately locate the corresponding video timestamp based on the query statement.

[0003] However, when the above method is used to query and locate the surveillance video, the following technical problems often occur:

[0004] Natural language video localization models may be negatively affected by background deception, paying more attention to the background and easily ignoring foreground entities, resulting in difficulty in learning robust foreground entity feature representations and weak anti-background interference capabilities, making it difficult to accurately locate the corresponding surveillance video timestamp based on the query statement, thereby increasing manual intervention and data processing time, and further reducing the efficiency and accuracy of surveillance video query localization tasks.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0006] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0007] Some embodiments of the present disclosure propose a natural language-based industrial surveillance video query and positioning method and device to solve one or more of the technical problems mentioned in the above background technology section.

[0008] In a first aspect, some embodiments of the present disclosure provide a natural language-based method for querying and locating surveillance videos in an industrial field, the method comprising: in response to receiving a query text input by a user for a surveillance video, performing text feature extraction on the query text to obtain query text features, wherein the query text is an action description of an entity in the surveillance video; performing feature extraction on video images of consecutive frames in the surveillance video to obtain a video frame feature vector sequence; performing foreground and background separation on each video frame feature vector in the video frame feature vector sequence based on the query text features to obtain a video frame foreground feature vector sequence; performing global information extraction on each video frame feature vector in the video frame feature vector sequence , obtaining a video frame spatial feature vector sequence; according to the above video frame foreground feature vector sequence, performing foreground enhancement on each video frame spatial feature vector in the above video frame spatial feature vector sequence to obtain an enhanced video frame feature vector sequence; according to the above query text feature, performing video frame positioning on each enhanced video frame feature vector in the above enhanced video frame feature vector sequence to generate video positioning segment information, and displaying and storing the video positioning segment corresponding to the video positioning segment information, wherein the above video positioning segment information includes the start timestamp and end timestamp of the corresponding query text feature appearing in the above surveillance video, and the above video positioning segment is the video segment between the start timestamp and the end timestamp.

[0009] In a second aspect, some embodiments of the present disclosure provide an industrial surveillance video query and positioning device based on natural language, the device comprising: a text feature extraction unit, configured to extract text features from the query text in response to receiving a query text input by a user for a surveillance video, so as to obtain a query text feature, wherein the query text is an action description of an entity in the surveillance video; a feature extraction unit, configured to extract features from video images of consecutive frames in the surveillance video, so as to obtain a video frame feature vector sequence; a foreground and background separation unit, configured to separate the foreground and background of each video frame feature vector in the video frame feature vector sequence according to the query text feature, so as to obtain a video frame foreground feature vector sequence; and a global information extraction unit, configured to extract features from each video frame in the video frame feature vector sequence. A frame feature vector is subjected to global information extraction to obtain a video frame spatial feature vector sequence; a foreground enhancement unit is configured to perform foreground enhancement on each video frame spatial feature vector in the video frame spatial feature vector sequence according to the video frame foreground feature vector sequence to obtain an enhanced video frame feature vector sequence; a video frame positioning unit is configured to perform video frame positioning on each enhanced video frame feature vector in the enhanced video frame feature vector sequence according to the query text feature to generate video positioning segment information, and display and store the video positioning segment corresponding to the video positioning segment information, wherein the video positioning segment information includes a start timestamp and an end timestamp of the corresponding query text feature appearing in the monitoring video, and the video positioning segment is a video segment between the start timestamp and the end timestamp.

[0010] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0011] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0012] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the natural language-based industrial surveillance video query and positioning method of some embodiments of the present disclosure, the efficiency and accuracy of the surveillance video query and positioning task can be improved. Specifically, the reason for the reduction in the efficiency and accuracy of the surveillance video query and positioning task is that the natural language video positioning model may be negatively affected by background deception, pay more attention to the background and easily ignore the foreground entity, resulting in difficulty in learning robust foreground entity feature representation, and weak anti-background interference ability, so that it is difficult to accurately locate the corresponding surveillance video timestamp according to the query statement, thereby increasing manual intervention and data processing time. Based on this, the natural language-based industrial surveillance video query and positioning method of some embodiments of the present disclosure, first, in response to receiving the query text input by the user for the surveillance video, the query text is subjected to text feature extraction to obtain the query text feature. Among them, the query text is a description of the action of the entity in the surveillance video. Through text feature extraction, the matching accuracy between the query statement and the video content can be improved. Secondly, the video image of the continuous frames in the surveillance video is subjected to feature extraction to obtain a video frame feature vector sequence. Thus, the feature understanding of each image frame of the video is improved. Again, according to the query text features, each video frame feature vector in the video frame feature vector sequence is separated from the foreground and background to obtain a video frame foreground feature vector sequence. In this way, the model can focus on the foreground information related to the query text, especially when processing surveillance videos with complex backgrounds, the influence of background interference information on the accuracy of the model can be reduced. Then, global information is extracted from each video frame feature vector in the video frame feature vector sequence to obtain a video frame space feature vector sequence. In this way, the overall spatial features can be obtained from each video frame image, and the global perception ability of the action in the video frame can be improved. Afterwards, according to the video frame foreground feature vector sequence, each video frame space feature vector in the video frame space feature vector sequence is foreground enhanced to obtain an enhanced video frame feature vector sequence. By enhancing the foreground features, it is helpful to avoid the occurrence of background deception problems, thereby improving the ability to locate the video segments related to the query text. Finally, according to the query text features, each enhanced video frame feature vector in the enhanced video frame feature vector sequence is video frame positioned to generate video positioning segment information, and the video positioning segment corresponding to the video positioning segment information is displayed and stored. The video positioning segment information includes the start timestamp and end timestamp of the corresponding query text feature appearing in the surveillance video, and the video positioning segment is the video segment between the start timestamp and the end timestamp. By applying the foreground enhanced spatial feature vector to the positioning of the video frame, the exact time period related to the query text can be accurately generated, reducing manual intervention and data processing time, thereby improving the efficiency and accuracy of the surveillance video query positioning task. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0014] Figure 1 is a flow chart of some embodiments of the natural language-based industrial surveillance video query and positioning method according to the present disclosure;

[0015] Figure 2 It is a structural schematic diagram of some embodiments of the natural language-based industrial surveillance video query and positioning device according to the present disclosure;

[0016] Figure 3 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0018] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0019] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0020] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0021] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0022] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0023] Figure 1 The process 100 of some embodiments of the natural language-based industrial surveillance video query and positioning method according to the present disclosure is shown. The natural language-based industrial surveillance video query and positioning method comprises the following steps:

[0024] Step 101 , in response to receiving a query text input by a user for a surveillance video, extracting text features from the query text to obtain query text features.

[0025] In some embodiments, the execution subject of the natural language-based industrial surveillance video query and positioning method can be hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or it can be implemented as a single server or a single terminal device. When the computing device is embodied as software, it can be installed in the hardware devices listed above. It can be implemented as multiple software or software modules for providing distributed services, or it can be implemented as a single software or software module. No specific limitation is made here.

[0026] In other embodiments, the execution subject may extract text features from the query text in response to receiving a query text input by a user for a surveillance video to obtain query text features, wherein the query text is a description of an action of an entity in the surveillance video.

[0027] For example, for surveillance videos of the production process of mechanical parts and components, the query text can be "parts falling from the conveyor belt." For surveillance videos of customs import and export products, the query text can be "loading and unloading of goods with special marks."

[0028] In some optional implementations of some embodiments, the execution subject extracts text features from the query text in response to receiving a query text input by a user for a surveillance video to obtain a query text feature, which may include the following steps:

[0029] The first step is to perform a word segmentation operation on the query text to obtain a query word segmentation set. The query text can be segmented by a preset word segmentation algorithm to obtain a query word segmentation set. Here, a query word segmentation is a word in the query text. One query text corresponds to at least one query word segmentation.

[0030] As an example, the word segmentation algorithm may include but is not limited to at least one of the following: NLTK (Natural Language Toolkit, natural language processing toolkit), Jieba word segmentation algorithm, etc.

[0031] The second step is to extract characters from the query text to obtain a query character set, wherein each word in the query text can be used as a character in the query character set to obtain the query character set.

[0032] The third step is to embed each query word in the query word set into a word vector to obtain a query word vector set. Among them, through a preset word embedding algorithm, each query word in the query word set can be embedded into a word vector to obtain a query word vector set. Secondly, the query word vector is an ordered sequence that represents the query word. Here, one query word corresponds to one query word vector.

[0033] As an example, a word embedding algorithm may include but is not limited to at least one of the following: Word2Vec (Word to Vector) algorithm, GLoVe (Global Vectors for Word Representation) algorithm, etc.

[0034] The fourth step is to perform a convolution operation on each query character in the query character set to obtain a query character feature vector set. In particular, a convolution operation can be performed on each query character in the query character set to obtain a query character feature vector set by using a preset character-level convolution algorithm.

[0035] As an example, the character-level convolution algorithm may include but is not limited to at least one of the following: Char-CNN (Char-Convolutional Neural Network), Char-Transformer (character-level transformer), etc.

[0036] The fifth step is to encode each query word segmentation vector in the query word segmentation vector set and each query character feature vector in the query character feature vector set to obtain query text features. In particular, each query word segmentation vector in the query word segmentation vector set and each query character feature vector in the query character feature vector set can be encoded by a preset text encoder to obtain query text features.

[0037] As an example, the text encoder may include but is not limited to at least one of the following: Mamba-790m (790m model of the selection strategy state space model), T5 (Text-to-Text Transfer Transformer, text-to-text converter model), etc.

[0038] In practice, we construct different types of dictionaries to record the query sequence number, video number, start timestamp, end timestamp, total video length, and query word set. In addition, we also construct a character dictionary to record the character number in each word.

[0039] Step 102: extract features from video images of consecutive frames in the surveillance video to obtain a video frame feature vector sequence.

[0040] In some embodiments, the execution subject may extract features from the video images of consecutive frames in the surveillance video to obtain a video frame feature vector sequence, wherein the video frame feature vector sequence is a sequence composed of feature vectors corresponding to each image frame in the video.

[0041] In some optional implementations of some embodiments, the execution subject extracts features from the video images of consecutive frames in the surveillance video to obtain a video frame feature vector sequence, which may include the following steps:

[0042] The first step is to perform frame processing on the surveillance video to obtain a video frame image sequence. Here, the surveillance video is converted into a video frame image sequence at a preset frame rate. The preset frame rate can be 8 frames per second or 24 frames per second.

[0043] The second step is to normalize each video frame image in the above video frame image sequence to obtain a standard video frame vector sequence. First, the video frame images of different sizes can be converted to a fixed size to obtain a standard video frame image sequence, such as 226×226 pixels. Secondly, the above standard video frame image is flattened into a one-dimensional vector, and the vector dimension can be 226×226 (51076). Among them, each standard video frame vector in the above standard video frame vector sequence has the same vector dimension.

[0044] The third step is to normalize each of the standard video frame vectors in the standard video frame vector sequence to obtain a standard video frame vector sequence. The standard video frame vector can be adjusted to a standard dimension of 224×224. Here, the standard dimension is the standard image input dimension of the deep learning model.

[0045] The fourth step is to normalize the vector values ​​of each standard video frame vector in the above standard video frame vector sequence to obtain a normalized video frame vector sequence. The values ​​in the above standard video frame vector are color values ​​corresponding to image pixels, usually between 0 and 255. Here, these color values ​​are scaled to between -1 and 1 through normalization.

[0046] The fifth step is to extract video features from each normalized video frame vector in the above normalized video frame vector sequence based on a preset time window to obtain a video frame feature vector sequence. The above time window is the number of video frames that need to be considered for each feature extraction. For example, if the time window size is 10 frames, 10 consecutive normalized video frame vectors will be analyzed each time during feature extraction. Here, a pre-trained video feature extraction algorithm can be used to extract video features from the above normalized video frame vectors to obtain a video frame feature vector. Secondly, video feature extraction can be performed on each frame, or it can be a comprehensive feature extraction of frames within the entire time window. Afterwards, these video frame feature vectors are spliced ​​into a complete video frame feature vector sequence.

[0047] As an example, the video feature extraction algorithm may include but is not limited to at least one of the following: an I3D (Inflated 3DConvNet, inflated three-dimensional convolutional network) algorithm, a C3D (Convolutional 3D, three-dimensional convolution) algorithm, and the like.

[0048] Step 103 , based on the query text feature, separate the foreground and background of each video frame feature vector in the video frame feature vector sequence to obtain a video frame foreground feature vector sequence.

[0049] In some embodiments, the execution subject may separate the foreground and background of each video frame feature vector in the video frame feature vector sequence according to the query text feature to obtain a video frame foreground feature vector sequence.

[0050] In some optional implementations of some embodiments, the execution subject performs foreground and background separation on each video frame feature vector in the video frame feature vector sequence according to the query text feature to obtain the video frame foreground feature vector sequence, which may include the following steps:

[0051] The first step is to determine the target text in the query text based on the query text features to generate a target text set. First, the query word segmentation vector representing the noun in the query text features is determined as the entity query word segmentation vector. After that, the query word segmentation corresponding to the above entity query word segmentation vector is returned, that is, the target text. Among them, the above target text represents the entity in the query text. For example, in the query text "Parts fell off the conveyor belt", "conveyor belt" and "parts" are target texts used to represent the two entities of conveyor belt and parts.

[0052] In the second step, for each video frame feature vector in the above video frame feature vector sequence, the following foreground and background separation steps are performed to generate a video frame foreground feature vector in the video frame foreground feature vector sequence:

[0053] Sub-step 1, semantically segment the above-mentioned video frame feature vector to obtain a semantic segmentation result map and a segmentation result label set. Wherein, by a preset semantic segmentation algorithm, the above-mentioned video frame feature vector can be semantically segmented to obtain a semantic segmentation result map and a segmentation result label set. Here, the above-mentioned semantic segmentation result map contains at least one segmentation area, and each segmentation area corresponds to a segmentation result label. Wherein, the above-mentioned segmentation area is a collection of pixels in the semantic segmentation result map.

[0054] As an example, the semantic segmentation algorithm may include, but is not limited to, at least one of the following: UperNet (Unified Perceptual Net) algorithm, Vmamba (Visual mamba visual selection strategy state space model) algorithm.

[0055] Sub-step 2: The segmentation result labels in the segmentation result label set that do not match any target text in the target text set are determined as background labels to obtain a background label set. The segmentation result labels in the segmentation result label set that match any target text in the target text set are determined as foreground labels, and the remaining labels are determined as background labels to obtain a background label set.

[0056] For example, the segmentation result label set includes the labels "conveyor belt", "parts", "ground", "cabinet", and "wall", and the target text set includes "conveyor belt" and "parts". Among them, "conveyor belt" and "parts" are foreground labels, and "ground", "cabinet", and "wall" are background labels.

[0057] Sub-step three, determining the segmented area corresponding to each background label in the background label set in the semantic segmentation result map, and setting the vector value corresponding to the segmented area in the video frame feature vector to zero, to obtain the video frame foreground feature vector. Here, the segmented area corresponding to each background label in the background label set is determined as the background area. Afterwards, the vector value of the video frame feature vector corresponding to the background area is set to 0, to obtain the video frame foreground feature vector.

[0058] In practice, it is found that when using technologies based on convolutional neural networks or vision transformers (ViT) to obtain global information of video frame images in surveillance videos, the following technical problems exist:

[0059] When extracting features, the convolutional neural network will lose a certain amount of receptive field. The computational complexity of the ViT model increases quadratically with the increase of feature dimensions. Secondly, the ViT model may have an implicit bias problem caused by the order of image block input determined by human reading habits. That is, due to the non-causal nature between image data, the strategy of directly dividing and tiling according to the human reading order will lead to limitations in the receptive field. Therefore, it is easy to cause spatial information loss, thereby reducing video comprehension ability and positioning accuracy, resulting in reduced efficiency in querying and positioning tasks for surveillance videos. Combining the visual processing method based on the state space model can not only reduce the amount of calculation, but also achieve results that are better than the current best model in multiple visual tasks such as image classification, object detection and semantic segmentation. It can be decided to adopt the following solution.

[0060] Step 104 , extracting global information from each video frame feature vector in the video frame feature vector sequence to obtain a video frame spatial feature vector sequence.

[0061] In some embodiments, the execution entity may perform global information extraction on each video frame feature vector in the video frame feature vector sequence to obtain a video frame spatial feature vector sequence.

[0062] In some optional implementations of some embodiments, the execution subject extracts global information from each video frame feature vector in the video frame feature vector sequence to obtain a video frame spatial feature vector sequence, which may include the following steps:

[0063] Each video frame feature vector in the above video frame feature vector sequence is input into a preset visual state space model, and the following steps are performed to generate a video frame spatial feature vector in the video frame spatial feature vector sequence:

[0064] Sub-step 1: performing image block segmentation on the above video frame feature vector to obtain a video frame image block set. First, the above video frame feature vector is converted into a corresponding feature map. Then, the feature map is segmented into video frame image blocks of the same size to obtain a video frame image block set.

[0065] Sub-step 2: linearly process each video frame image block in the video frame image block set to generate a one-dimensional image block vector set, wherein each video frame image block is flattened into a one-dimensional image block vector to obtain a one-dimensional image block vector set.

[0066] Sub-step three, sequentially transforming each one-dimensional image block vector in the one-dimensional image block vector set to obtain a corresponding transformed image block vector set. The order of each one-dimensional image block vector in the one-dimensional image block vector set can be changed by matrix transposition or flipping to obtain the corresponding transformed image block vector set.

[0067] In practice, four different orders of image block arrangements can be obtained by the following formula:

[0068] .

[0069] in, Represents a video frame image. It is the first video frame image block in the video frame image block set. is the one-dimensional image block vector corresponding to the image block of the first video frame. The function represents flattening the two-dimensional image block into a one-dimensional feature vector, that is, The dimension of can be changed from (H, W) to (H×W). express The transposed vector after transposition represents a sequence of image blocks in another order. Function represents the transpose operation. express A flipped version of express is a flipped version of . and Equivalent to the corresponding reverse feature input. Function represents the flip operation.

[0070] Sub-step 4, according to the serial number corresponding to each video frame image block in the above-mentioned video frame image block set, each transformed image block vector in the above-mentioned transformed image block vector set is positionally embedded to obtain a position embedding matrix. According to the serial number corresponding to each video frame image block in the above-mentioned video frame image block set, each transformed image block vector in the above-mentioned transformed image block vector set is sequentially placed in the position embedding matrix. Here, each column of the position embedding matrix corresponds to the transformed image block vector in the transformed image block vector set in sequence.

[0071] Sub-step 5: extract spatial features from the position embedding matrix to obtain a spatial feature vector of the video frame. Here, the spatial feature extraction of the position embedding matrix can be performed through a preset visual state space module to obtain a spatial feature vector of the video frame. The visual state space module adopts a setting similar to ViM (Vision Mamba, visual selection strategy state space model), and introduces a bidirectional propagation mechanism inside the Mamba (selection strategy state space model) structure.

[0072] The above step 104 and its related contents, as an inventive point of an embodiment of the present disclosure, solve the above technical problem "the video understanding ability and positioning accuracy are reduced, resulting in a reduction in the efficiency of querying and positioning tasks for surveillance videos." The factors that lead to the above technical problems are often as follows: the convolutional neural network will lose a certain receptive field when extracting features, and the computational complexity of the ViT model increases quadratically with the increase of feature dimensions. Secondly, the ViT model may have an implicit bias problem caused by the input order of image blocks determined by people's reading habits, which is easy to cause spatial information loss. If the above factors are solved, the model's ability to obtain global information of video frame images in surveillance videos can be improved, thereby improving the accuracy and efficiency of querying and positioning tasks for surveillance videos. In order to achieve this effect, first, a more diverse image block sequence generation method is used to avoid the problem of spatial information loss caused by a single image block input order, thereby obtaining a video feature representation containing richer contextual information, reducing the impact of implicit bias, and improving the model's video understanding ability and positioning accuracy. Secondly, a visual state space model is used for global information extraction. The visual state space model uses bidirectional sequence modeling to capture the contextual relationships and spatial information in visual data, which can not only reduce the amount of computation but also improve the ability to extract spatial features of the video, thereby improving the accuracy and efficiency of querying and positioning tasks on surveillance videos.

[0073] Step 105 , based on the video frame foreground feature vector sequence, foreground enhancement is performed on each video frame spatial feature vector in the video frame spatial feature vector sequence to obtain an enhanced video frame feature vector sequence.

[0074] In some embodiments, the execution subject may perform foreground enhancement on each video frame spatial feature vector in the video frame spatial feature vector sequence according to the video frame foreground feature vector sequence to obtain an enhanced video frame feature vector sequence. First, the video frame foreground feature vector and the video frame spatial feature vector may be vector dimensionally aligned to obtain an aligned foreground feature vector and an aligned spatial feature vector, respectively. Afterwards, the result of element-by-element addition of the aligned foreground feature vector and the aligned spatial feature vector may be determined as an enhanced video frame feature vector to obtain an enhanced video frame feature vector sequence.

[0075] Step 106 , based on the query text feature, perform video frame positioning on each enhanced video frame feature vector in the enhanced video frame feature vector sequence to generate video positioning segment information, and display and store the video positioning segment corresponding to the video positioning segment information.

[0076] In some embodiments, the execution subject may perform video frame positioning on each enhanced video frame feature vector in the enhanced video frame feature vector sequence according to the query text feature to generate video positioning segment information, and display and store the video positioning segment corresponding to the video positioning segment information. The video positioning segment information includes the start timestamp and end timestamp of the corresponding query text feature appearing in the surveillance video. Here, the video positioning segment is the video segment between the start timestamp and the end timestamp.

[0077] In some optional implementations of some embodiments, the execution subject performs video frame positioning on each enhanced video frame feature vector in the enhanced video frame feature vector sequence according to the query text feature to generate video positioning segment information, and displays and stores the video positioning segment corresponding to the video positioning segment information, which may include the following steps:

[0078] In the first step, for each enhanced video frame feature vector in the enhanced video frame feature vector sequence, the following steps are performed to generate a video segment start frame and a video segment end frame:

[0079] Sub-step one, input the enhanced video frame feature vector sequence and the query text feature into the state space module in the preset surveillance video positioning model to perform multimodal feature mapping to obtain the video frame modal feature sequence and the query text modal feature. Wherein, the surveillance video positioning model includes a state space module, a cross-modal interaction module, a feedforward network module, and a prediction head module. Here, the state space module includes a state space submodule and a multi-layer perceptron layer. Wherein, the state space submodule processes the enhanced video frame feature vector sequence and the query text feature respectively, and makes the feature vectors extracted from the features of the two different modalities closer in the common subspace by sharing parameters.

[0080] As an example, the state space module includes but is not limited to at least one of the following: SSM (State Space Model), Mamba (Selection Strategy State Space Model).

[0081] Sub-step two, input the above-mentioned video frame modal feature sequence and the query text modal feature into the above-mentioned cross-modal interaction module to obtain text-to-video attention features and video-to-text attention features. Among them, through the attention mechanism algorithm, the global relationship between the modalities of the above-mentioned video frame modal feature sequence and the above-mentioned query text modal feature can be extracted to obtain text-to-video attention features and video-to-text attention features. Here, the text-to-video attention feature represents the importance of text features to video features. At the same time, the video-to-text attention feature represents the importance of video features to text features. Among them, the above-mentioned cross-modal interaction module includes an attention mechanism module.

[0082] As an example, the attention mechanism algorithm includes but is not limited to at least one of the following: Transformer model, Switch Transformer model.

[0083] In sub-step three, the result of dot multiplication of the text-to-video attention feature and each video frame modal feature in the above video frame modal feature sequence is determined as the text-to-video attention feature to generate a text-to-video attention feature sequence.

[0084] In sub-step four, the result of dot multiplication of the video-to-text attention feature with each video frame modal feature in the above video frame modal feature sequence is determined as the video-to-text attention feature to generate a video-to-text attention feature sequence.

[0085] Sub-step five: input the above-mentioned video frame modal feature sequence, the above-mentioned video-to-text attention feature, the above-mentioned video-to-text attention feature sequence and the above-mentioned text-to-video attention feature sequence into the above-mentioned feedforward network module for forward feature extraction to obtain a fused video frame feature sequence.

[0086] As an example, the feedforward network includes but is not limited to at least one of the following: a convolutional neural network (CNN), a generalized regression neural network (GRNN).

[0087] Sub-step six, input each fused video frame feature in the above fused video frame feature sequence into the prediction head module to obtain the video clip start frame and the video clip end frame. Among them, the above prediction head module is composed of two unidirectional long short-term memory networks. First, input each fused video frame feature in the above fused video frame feature sequence into two unidirectional long short-term memory networks to obtain the probability value of the above fused video frame feature being the video clip start frame and the probability value of the video clip end frame respectively. Among them, the above video clip start frame and the video clip end frame are the first frame and the last frame in which the corresponding query text feature appears in the above surveillance video.

[0088] In practice, the output of two unidirectional long short-term memory networks fuses the video frame features into scores for the start frame and the end frame of the video clip, as shown in the following formula:

[0089] .

[0090] in, Represents a video frame in a video. Represents a fused video frame feature in the fused video frame feature sequence. Indicates the number of surveillance videos The fused video frame features corresponding to the frame image. Represents the memory feature vector extracted by the long short-term memory network. The starting memory feature vector indicating that the video frame is the starting frame. Indicates the end of memorizing feature vectors. and Respectively represent Frame The starting memory feature vector and the ending memory feature vector of the video frame. and Respectively represent Frame The starting memory feature vector and the ending memory feature vector of the video frame. Represents a long short-term memory network. and is the weight matrix used to predict the start frame and the end frame. and is the corresponding deviation. Indicates that the The fused video frame features of each video frame are concatenated with the starting memory feature vector. Represents the score of the LSTM network output. and Respectively represent the score of each video frame as the start frame and the end frame in the video. and Respectively represent Frames Video frames are scored for the start frame and the end frame.

[0091] Optionally, after obtaining the score of each video frame as the starting frame or the ending frame, the score of the video image frame can be converted into a probability distribution using the Softmax function. Specifically, by applying the Softmax function, the probabilities corresponding to the scores of the starting frame and the ending frame can be obtained respectively. and .here, , .in, represents the probability value. Secondly, , Here, we can use the joint probability distribution of the maximized boundary conditions to get the possible boundary combinations, the formula is:

[0092] .

[0093] in, For possible boundary combinations, is the starting frame, is the end frame corresponding to the above start frame. express and An ordered pair. is the joint probability distribution function that maximizes the boundary conditions.

[0094] In the second step, the start timestamp and the end timestamp corresponding to the start frame of the video segment and the end frame of the video segment are determined as the video positioning segment information.

[0095] Optionally, the surveillance video positioning model is trained and generated by the following steps:

[0096] The first step is to obtain sample videos, sample texts and corresponding sample labels. The sample videos, sample texts and corresponding sample labels can be samples in a public dataset. Here, the public dataset includes sample videos, semantic annotations of behaviors in the sample videos, and the start and end timestamps of the video clips.

[0097] As an example, the public dataset includes but is not limited to at least one of the following: Charades-STA dataset, ActivityNet-Captions dataset.

[0098] The second step is to perform frame processing on the sample video to obtain a sample video frame sequence, wherein the video is converted into a video frame sequence by a preset sampling frequency.

[0099] The third step is to perform highlight processing on the sample video frame sequence to obtain a final highlight video frame sequence and a sample video highlight feature. The final highlight video frame sequence is a sequence consisting of a portion of the sample video frames whose highlight probability values ​​are greater than a threshold in the sample video frame sequence and the video frames within the portion. Here, the highlight probability value is a probability value that a video frame is a start frame or an end frame.

[0100] Specifically, the execution subject performs highlight processing on the sample video frame sequence to obtain a final highlight video frame sequence and a sample video highlight feature, which may include the following steps:

[0101] Sub-step one, inputting the above sample video frame sequence into the initial state space module, the initial cross-modal interaction module and the initial feedforward network module in the initial surveillance video positioning model in sequence to obtain a fused sample video frame feature sequence.

[0102] Sub-step 2, determining the sample text features of the sample text, and splicing the sample text features and the fused sample video frame feature sequence to obtain multimodal sample features. First, the sample text features of the sample text can be determined by the text feature extraction method in the embodiments of the above step 101, which will not be described in detail. After that, the sample text features are spliced ​​after the fused sample video frame feature sequence to obtain multimodal sample features.

[0103] In practice, the formula for obtaining multimodal sample features is: .

[0104] in, Represents a sample video. It is a multimodal sample feature. Characterize the fused sample video frame feature sequence. For text features. Represents sample text features.

[0105] Sub-step three, based on the above multimodal sample features, determine the highlight probability value of each sample video frame in the above sample video frame sequence, and use the sample video frames corresponding to the highlight probability values ​​greater than the preset threshold as highlight video frames to generate a highlight video frame set. First, through the convolutional neural network and the activation function, the highlight probability value of each video frame corresponding to the above multimodal sample features can be obtained. Then, the sample video frames corresponding to the highlight probability values ​​greater than the preset threshold are determined as highlight video frames to generate a highlight video frame set.

[0106] In practice, the formula for determining the highlight probability value is: .

[0107] in, Indicates the highlight probability value. Represents the activation function Sigmoid. Represents a convolutional neural network.

[0108] Sub-step 4: determining the highlight time point corresponding to each highlight video frame in the highlight video frame set to obtain a highlight time period, wherein the highlight time period is a time period consisting of the time points between the earliest time point and the latest time point among the highlight time points.

[0109] Sub-step 5: determining each sample video frame in the above-mentioned highlight time period in the above-mentioned sample video frame sequence as a final highlight video frame to generate a final highlight video frame sequence. Wherein, if the above-mentioned sample video frame is in the highlight time period, the sample video frame is determined as the final highlight video frame to generate a final highlight video frame sequence.

[0110] Sub-step 6: According to the highlight probability value, determine the highlight feature of the fused sample video frame feature corresponding to each final highlight video frame in the final highlight video frame sequence to generate a sample video frame highlight feature sequence. The formula can be used: , determine the highlight feature of the fused sample video frame feature corresponding to each final highlight video frame in the above final highlight video frame sequence to generate a sample video frame highlight feature sequence. is the highlight feature sequence of the sample video frame.

[0111] The fourth step is to determine the first frame in the sample video frame sequence as a static frame, and insert a static frame into the sample video frame sequence to obtain a noisy video frame sequence. First, the first frame in the sample video frame sequence is determined as a static frame. Then, the sample video frames in the sample video frame sequence are randomly replaced with the static frames to obtain a noisy video frame sequence. The noisy video frame sequence and the sample video frame sequence constitute a positive sample pair.

[0112] Step 5: Perform highlight processing on the above-mentioned noisy video frame sequence to obtain a highlight feature sequence of the noisy video frame. The highlight processing method in the above-mentioned embodiments of the third step can be used to perform highlight processing on the above-mentioned noisy video frame sequence to obtain a highlight feature sequence of the noisy video frame. No further details will be given.

[0113] The sixth step is to obtain a negative sample video frame sequence, and perform highlight processing on the negative sample video frame sequence to obtain a negative sample video frame highlight feature sequence. Among them, the negative sample video frame sequence can be highlight processed by the highlight processing method in the above-mentioned embodiments of the third step to obtain a negative sample video frame highlight feature sequence. No further details are given. Among them, the negative sample video frame sequence is a negative sample sample video frame sequence in the negative sample set. Here, the negative sample set is a set of videos other than the sample video in the public data set. At the same time, the negative sample video frame sequence and the sample video frame sequence constitute a negative sample pair;

[0114] Step 7: Determine the contrastive learning loss value based on the highlight feature sequence of the sample video frame, the highlight feature sequence of the noisy video frame, and the highlight feature sequence of the negative sample video frame. The contrastive learning loss value is determined by the contrastive learning loss function. Here, the contrastive learning loss function is:

[0115] .

[0116] in, Represents the loss value. is the contrastive learning loss value. For a sample video , add a static frame to it , and get the noisy video .in, is the number of frames to insert static frames. and Considered as a positive sample pair. Through highlight processing, the highlight feature sequence of the sample video is obtained and highlight feature sequences of noisy videos .Will Considered as Negative sample pairs. , represents the negative sample set. After highlight processing, the highlight feature sequence of the negative sample video frame is obtained .

[0117] Step 8: Determine the highlight loss value between the highlight probability value of each sample video frame in the above sample video frame sequence and the corresponding sample label. The highlight loss value between the highlight probability value of each sample video frame in the above sample video frame sequence and the corresponding sample label is determined by a highlight loss function. Here, the highlight loss function is:

[0118] .

[0119] in, is the high light loss value. Represents a true value. Indicates the actual true value label corresponding to the video frame, Represents an operation that computes the cross entropy.

[0120] The ninth step is to input the highlight feature sequence of the sample video frame into the initial prediction head module in the initial monitoring video positioning model to obtain the video positioning segment information, and determine the positioning loss value between the video positioning segment information and the sample label. First, the video positioning segment information can be obtained according to the highlight feature sequence of the sample video frame through the implementation method of the above-mentioned prediction head module. No more details are given. Afterwards, the cross entropy loss function can be used to construct the corresponding positioning loss function to determine the positioning loss value between the video positioning segment information and the sample label. Here, the positioning loss function is:

[0121] .

[0122] in, is the positioning loss value, and Labels representing the start and end frames, respectively.

[0123] In the tenth step, the weighted sum of the contrast learning loss value, the highlight loss value, and the positioning loss value is determined as the main loss value. The main loss value is determined as the weighted sum of the contrast learning loss value, the highlight loss value, and the positioning loss value through the main loss function. Here, the main loss function is:

[0124] .

[0125] in, The main loss value. is a hyperparameter used as the weight of the self-supervised loss function.

[0126] In the eleventh step, in response to determining that the subject loss value is less than or equal to the preset loss threshold, and the training of the sample video and the sample text meets the preset training end condition, the initial surveillance video positioning model is determined as the trained surveillance video positioning model. Here, if the subject loss value is less than or equal to the preset loss threshold, or the training of the sample video and the sample text meets the preset training end condition, the initial surveillance video positioning model is determined as the trained surveillance video positioning model. The training end condition may be that the sample video and the sample text have completed a preset round of training. Here, the preset rounds may be 200 rounds.

[0127] The above step 106 and its related contents, as an inventive point of an embodiment of the present disclosure, solve the technical problem mentioned in the background technology that "increases manual intervention and data processing time, thereby reducing the efficiency and accuracy of the monitoring video query positioning task". The factors that lead to the low efficiency and accuracy of the monitoring video query positioning task are often as follows: the natural language video positioning model may be negatively affected by background deception, pay more attention to the background and easily ignore the foreground entity, resulting in difficulty in learning robust foreground entity feature representation, and weak anti-background interference ability, so that it is difficult to accurately locate the corresponding monitoring video timestamp according to the query statement. If the above factors are solved, the occurrence of the background deception problem can be alleviated, thereby improving the efficiency and accuracy of monitoring video query positioning. In order to achieve this effect, static frames are inserted into the above sample video frame sequence during model training. In this way. A sample of interference video, that is, a noisy video frame sequence, is constructed to form a positive sample pair with the original sample video frame sequence. Then, through self-supervised contrast learning, the interference sample and the original feature are forced to be closer in the common subspace. Thereby, the model focuses on the changes in motion rather than the background. Through self-supervised contrast learning, the influence of the background on the monitoring video query positioning can be eliminated as much as possible. Furthermore, the efficiency and accuracy of surveillance video query and positioning are improved.

[0128] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the natural language-based industrial surveillance video query and positioning method of some embodiments of the present disclosure, the efficiency and accuracy of the surveillance video query and positioning task can be improved. Specifically, the reason for the reduction in the efficiency and accuracy of the surveillance video query and positioning task is that the natural language video positioning model may be negatively affected by background deception, pay more attention to the background and easily ignore the foreground entity, resulting in difficulty in learning robust foreground entity feature representation, and weak anti-background interference ability, so that it is difficult to accurately locate the corresponding surveillance video timestamp according to the query statement, thereby increasing manual intervention and data processing time. Based on this, some embodiments of the present disclosure are based on the natural language industrial surveillance video query and positioning method. First, in response to receiving the query text input by the user for the surveillance video, the query text is subjected to text feature extraction to obtain the query text feature. Among them, the query text is a description of the action of the entity in the surveillance video. Through text feature extraction, the matching accuracy between the query statement and the video content can be improved. Secondly, the video image of the continuous frames in the surveillance video is subjected to feature extraction to obtain a video frame feature vector sequence. Thus, the feature understanding of each image frame of the video is improved. Again, according to the query text features, each video frame feature vector in the video frame feature vector sequence is separated from the foreground and background to obtain a video frame foreground feature vector sequence. In this way, the model can focus on the foreground information related to the query text, especially when processing surveillance videos with complex backgrounds, the influence of background interference information on the accuracy of the model can be reduced. Then, global information is extracted from each video frame feature vector in the video frame feature vector sequence to obtain a video frame space feature vector sequence. In this way, the overall spatial features can be obtained from each video frame image, and the global perception ability of the action in the video frame can be improved. Afterwards, according to the video frame foreground feature vector sequence, each video frame space feature vector in the video frame space feature vector sequence is foreground enhanced to obtain an enhanced video frame feature vector sequence. By enhancing the foreground features, it is helpful to avoid the occurrence of background deception problems, thereby improving the ability to locate the video segments related to the query text. Finally, according to the query text features, each enhanced video frame feature vector in the enhanced video frame feature vector sequence is video frame positioned to generate video positioning segment information, and the video positioning segment corresponding to the video positioning segment information is displayed and stored. The video positioning segment information includes the start timestamp and end timestamp of the corresponding query text feature appearing in the surveillance video, and the video positioning segment is the video segment between the start timestamp and the end timestamp. By applying the foreground enhanced spatial feature vector to the positioning of the video frame, the exact time period related to the query text can be accurately generated, reducing manual intervention and data processing time, thereby improving the efficiency and accuracy of the surveillance video query positioning task.

[0129] Further references Figure 2 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of an industrial surveillance video query and positioning device based on natural language. These device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0130] like Figure 2 As shown, some embodiments of the natural language-based industrial surveillance video query and positioning device 200 include: a text feature extraction unit 201, a feature extraction unit 202, a foreground and background separation unit 203, a global information extraction unit 204, a foreground enhancement unit 205, and a video frame positioning unit 206. The text feature extraction unit 201 is configured to extract text features from the query text in response to receiving a query text input by a user for a surveillance video, and obtain query text features, wherein the query text is an action description of an entity in the surveillance video; the feature extraction unit 202 is configured to extract features from the video images of consecutive frames in the surveillance video, and obtain a video frame feature vector sequence; the foreground and background separation unit 203 is configured to perform foreground and background separation on each video frame feature vector in the video frame feature vector sequence according to the query text features, and obtain a video frame foreground feature vector sequence; the global information extraction unit 204 is configured to perform global information extraction on each video frame feature vector in the video frame feature vector sequence, and obtain a video frame foreground feature vector sequence. A frame space feature vector sequence; a foreground enhancement unit 205 is configured to perform foreground enhancement on each video frame space feature vector in the video frame space feature vector sequence according to the video frame foreground feature vector sequence to obtain an enhanced video frame feature vector sequence; a video frame positioning unit 206 is configured to perform video frame positioning on each enhanced video frame feature vector in the enhanced video frame feature vector sequence according to the query text feature to generate video positioning segment information, and display and store the video positioning segment corresponding to the video positioning segment information, wherein the video positioning segment information includes the start timestamp and end timestamp of the corresponding query text feature appearing in the monitoring video, and the video positioning segment is the video segment between the start timestamp and the end timestamp.

[0131] It is understood that the units described in the device 200 are similar to those described in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 200 and the units included therein, and will not be described in detail here.

[0132] Reference below Figure 3, which shows a structural schematic diagram of an electronic device (eg, a computing device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0133] like Figure 3 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory 302 or a program loaded from a storage device 308 into a random access memory 303. Various programs and data required for the operation of the electronic device 300 are also stored in the random access memory 303. The processing device 301, the read-only memory 302, and the random access memory 303 are connected to each other via a bus 304. An input / output interface 305 is also connected to the bus 304.

[0134] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0135] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the read-only memory 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0136] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0137] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0138] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: in response to receiving a query text input by a user for a surveillance video, extract text features from the query text to obtain query text features, wherein the query text is an action description of an entity in the surveillance video; extract features from the video images of consecutive frames in the surveillance video to obtain a sequence of video frame feature vectors; perform foreground and background separation on each video frame feature vector in the video frame feature vector sequence based on the query text features to obtain a sequence of video frame foreground feature vectors; perform global information extraction on each video frame feature vector in the video frame feature vector sequence. Extract to obtain a video frame spatial feature vector sequence; based on the above video frame foreground feature vector sequence, perform foreground enhancement on each video frame spatial feature vector in the above video frame spatial feature vector sequence to obtain an enhanced video frame feature vector sequence; based on the above query text features, perform video frame positioning on each enhanced video frame feature vector in the above enhanced video frame feature vector sequence to generate video positioning segment information, and display and store the video positioning segment corresponding to the video positioning segment information, wherein the above video positioning segment information includes the start timestamp and end timestamp of the corresponding query text feature appearing in the above surveillance video, and the above video positioning segment is the video segment between the start timestamp and the end timestamp.

[0139] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0140] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0141] The units described in some embodiments of the present disclosure may be implemented by software or hardware. The described units may also be provided in a processor, for example, may be described as: a processor including a text feature extraction unit, a feature extraction unit, a foreground and background separation unit, a global information extraction unit, a foreground enhancement unit, and a video frame positioning unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the feature extraction unit may also be described as a "unit for extracting features from surveillance videos".

[0142] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0143] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A natural language-based industrial surveillance video query and positioning method. include: In response to receiving a query text input by a user for a surveillance video, extracting text features from the query text to obtain query text features, wherein the query text is an action description of an entity in the surveillance video; Extracting features from the video images of consecutive frames in the surveillance video to obtain a sequence of video frame feature vectors; According to the query text feature, foreground and background separation is performed on each video frame feature vector in the video frame feature vector sequence to obtain a video frame foreground feature vector sequence; Performing global information extraction on each video frame feature vector in the video frame feature vector sequence to obtain a video frame spatial feature vector sequence; According to the video frame foreground feature vector sequence, foreground enhancement is performed on each video frame spatial feature vector in the video frame spatial feature vector sequence to obtain an enhanced video frame feature vector sequence; According to the query text feature, video frame positioning is performed on each enhanced video frame feature vector in the enhanced video frame feature vector sequence to generate video positioning segment information, and the video positioning segment corresponding to the video positioning segment information is displayed and stored, wherein the video positioning segment information includes the start timestamp and end timestamp of the corresponding query text feature appearing in the surveillance video, and the video positioning segment is the video segment between the start timestamp and the end timestamp.

2. The method according to claim 1, in, The step of extracting text features from the query text to obtain query text features in response to receiving a query text input by a user for a surveillance video includes: Performing a word segmentation operation on the query text to obtain a query word segmentation set; Extracting characters from the query text to obtain a query character set; Embedding each query word in the query word set with a word vector to obtain a query word vector set; Performing a convolution operation on each query character in the query character set to obtain a query character feature vector set; Each query word segmentation vector in the query word segmentation vector set and each query character feature vector in the query character feature vector set are encoded to obtain query text features.

3. The method according to claim 1, in, The step of extracting features from the video images of consecutive frames in the surveillance video to obtain a video frame feature vector sequence includes: Performing frame processing on the surveillance video to obtain a video frame image sequence; Performing image normalization on each video frame image in the video frame image sequence to obtain a standard video frame vector sequence, wherein each standard video frame vector in the standard video frame vector sequence has the same vector dimension; Performing standardization processing on each standard video frame vector in the standard video frame vector sequence to obtain a standard video frame vector sequence; Normalizing the vector value of each standard video frame vector in the standard video frame vector sequence to obtain a normalized video frame vector sequence; Based on a preset time window, video features are extracted from each normalized video frame vector in the normalized video frame vector sequence to obtain a video frame feature vector sequence.

4. The method according to claim 1, in, The step of separating the foreground and background of each video frame feature vector in the video frame feature vector sequence according to the query text feature to obtain a video frame foreground feature vector sequence includes: Determine the target text in the query text according to the query text feature to generate a target text set, wherein the target text represents an entity in the query text; For each video frame feature vector in the video frame feature vector sequence, the following foreground and background separation step is performed to generate a video frame foreground feature vector in the video frame foreground feature vector sequence: Performing semantic segmentation on the video frame feature vector to obtain a semantic segmentation result map and a segmentation result label set, wherein the semantic segmentation result map includes at least one segmentation region, each segmentation region corresponds to a segmentation result label, and the segmentation region is a set of pixels in the semantic segmentation result map; Determine the segmentation result labels in the segmentation result label set that do not match any target text in the target text set as background labels to obtain a background label set; The segmented areas corresponding to the background labels in the background label set in the semantic segmentation result map are determined, and the vector values ​​corresponding to the segmented areas in the video frame feature vector are set to zero to obtain the video frame foreground feature vector.

5. The method according to claim 1, in, The step of extracting global information from each video frame feature vector in the video frame feature vector sequence to obtain a video frame spatial feature vector sequence includes: Input each video frame feature vector in the video frame feature vector sequence into a preset visual state space model, and perform the following steps to generate a video frame spatial feature vector in the video frame spatial feature vector sequence: Performing image block segmentation on the video frame feature vector to obtain a video frame image block set; Performing linear processing on each video frame image block in the video frame image block set to generate a one-dimensional image block vector set; Sequentially transforming each one-dimensional image block vector in the one-dimensional image block vector set to obtain a corresponding transformed image block vector set; According to the sequence number corresponding to each video frame image block in the video frame image block set, position embedding is performed on each transformed image block vector in the transformed image block vector set to obtain a position embedding matrix; Perform spatial feature extraction on the position embedding matrix to obtain a video frame spatial feature vector.

6. The method according to claim 1, in, The performing video frame positioning on each enhanced video frame feature vector in the enhanced video frame feature vector sequence according to the query text feature to generate video positioning segment information includes: For each enhanced video frame feature vector in the enhanced video frame feature vector sequence, the following steps are performed to generate a video segment start frame and a video segment end frame: Inputting the enhanced video frame feature vector sequence and the query text feature into a state space module in a preset surveillance video positioning model to perform multimodal feature mapping to obtain a video frame modal feature sequence and a query text modal feature, wherein the surveillance video positioning model comprises a state space module, a cross-modal interaction module, a feedforward network module, and a prediction head module; Inputting the video frame modal feature sequence and the query text modal feature into the cross-modal interaction module to obtain text-to-video attention features and video-to-text attention features; Determine the result of dot multiplication of the text-to-video attention feature and each video frame modality feature in the video frame modality feature sequence as the text-to-video attention feature to generate a text-to-video attention feature sequence; Determine the result of dot multiplication of the video-to-text attention feature and each video frame modality feature in the video frame modality feature sequence as the video-to-text attention feature to generate a video-to-text attention feature sequence; Inputting the video frame modal feature sequence, the video-to-text attention feature, the video-to-text attention feature sequence and the text-to-video attention feature sequence into the feedforward network module for forward feature extraction to obtain a fused video frame feature sequence; Input each fused video frame feature in the fused video frame feature sequence into a prediction header module to obtain a video segment start frame and a video segment end frame, wherein the video segment start frame and the video segment end frame are the first frame and the last frame in which the corresponding query text feature appears in the surveillance video; The start timestamp and the end timestamp corresponding to the start frame of the video segment and the end frame of the video segment are determined as the video positioning segment information.

7. A natural language-based industrial surveillance video query and positioning device, include: A text feature extraction unit is configured to extract text features from a query text input by a user for a surveillance video to obtain a query text feature, wherein the query text is an action description of an entity in the surveillance video; A feature extraction unit is configured to extract features from video images of consecutive frames in the surveillance video to obtain a video frame feature vector sequence; a foreground and background separation unit configured to separate the foreground and background of each video frame feature vector in the video frame feature vector sequence according to the query text feature, so as to obtain a video frame foreground feature vector sequence; A global information extraction unit is configured to perform global information extraction on each video frame feature vector in the video frame feature vector sequence to obtain a video frame spatial feature vector sequence; a foreground enhancement unit configured to perform foreground enhancement on each video frame spatial feature vector in the video frame spatial feature vector sequence according to the video frame foreground feature vector sequence to obtain an enhanced video frame feature vector sequence; The video frame positioning unit is configured to perform video frame positioning on each enhanced video frame feature vector in the enhanced video frame feature vector sequence according to the query text feature to generate video positioning segment information, and display and store the video positioning segment corresponding to the video positioning segment information, wherein the video positioning segment information includes the start timestamp and end timestamp of the corresponding query text feature appearing in the surveillance video, and the video positioning segment is the video segment between the start timestamp and the end timestamp.

8. An electronic device, include: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, in, When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images