Red dot video display method and system for alarm events

Abstract

The present application discloses a red dot video display method and system for alarm events, and relates to the field of video processing technology. The method includes: extracting image frame semantic features from video clips anchored by alarm tags (used to represent video clips during alarms) and the previous frame of the video clip through intelligent algorithms and image processing technology, so as to understand the status of the alarm event and the picture status in the previous frame of the video clip, and further use the image frame semantic features of the previous frame as query information, and perform scanning and interactive matching with the image frame semantic association feature pattern of the video clip, so as to intelligently determine whether the previous frame should be included in the video clip, and update the video clip in real time. The present application can filter backward image frames based on the same logic to achieve semantic-based two-way intelligent expansion of video clips, provide richer context information for review work, and avoid introducing irrelevant content.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular, to a red dot video display method and system for alarm events. Background Art

[0002] The traditional alarm system links video images for review, and uses digital video recorders and network cameras with memory cards in the monitoring system. However, because cached data is not used in data management applications, when an event occurs, the 3 minutes of video footage before the event has not been generated. The alarm images received by the platform are seriously delayed, and the video information that occurs at the moment of the alarm cannot be accurately located.

[0003] With the growing demand for real-time monitoring and post-event review, it is critical to be able to quickly and accurately identify and respond to abnormal events in areas such as security monitoring, traffic management, and industrial automation. The alarm tag (reflected as a red dot in the video) is recorded at the timestamp when the alarm occurs, so that the alarm can be accurately matched to the moment it occurs based on the timestamp. In addition, during the review process, only the video clip containing the red dot is transmitted instead of the entire video stream, which can reduce bandwidth usage.

[0004] However, focusing only on the video clip at the moment of the alarm may not be enough to provide a comprehensive background of the event. That is, in actual application scenarios, it is still necessary to pay attention to the situation before the alarm occurs and after the alarm disappears, so as to have a more comprehensive understanding of the cause and effect and scope of the event. However, if the video clip time is simply extended, such as extending the predetermined time along the forward and backward directions of the video clip, this may introduce irrelevant content and increase bandwidth usage.

[0005] Therefore, an optimized solution is needed to solve the above technical problems. Summary of the invention

[0006] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0007] In a first aspect, the present application provides a red dot video display method for an alarm event, the method comprising:

[0008] Step 1: Extract video segments anchored by alarm labels from the original video;

[0009] Step 2: extracting a previous frame of the video clip from the original video as a candidate image frame;

[0010] Step 3: extracting image frame semantic association features and enhancing image frame context semantic association features on the video clip to obtain a sequence of implicit feature vectors of enhanced semantic associations between image frames;

[0011] Step 4: inputting the candidate image frame into an image feature extractor to obtain a semantic feature vector of the candidate image frame;

[0012] Step 5: using the semantic feature vector of the candidate image frame as a query feature vector, inputting the query feature vector and the sequence of the implicit feature vector of the semantic association between the enhanced image frames into a fine-grained interaction module based on the attention mechanism to obtain a one-way matching result representation vector;

[0013] Step 6: determining an evaluation result based on the one-way matching result representation vector, wherein the evaluation result is used to indicate whether to include the candidate image frame into the video segment;

[0014] Step 7: In response to the evaluation result that the candidate image frame is to be included in the video segment, the candidate image frame is used as the first frame of the video segment to obtain an updated video segment;

[0015] Step 8: extracting the previous frame of the previous frame of the image from the original video as a candidate image frame, and looping steps 2 to 7 until the evaluation result is that the candidate image frame is not included in the video segment;

[0016] Step 9: Display the updated video clip.

[0017] Optionally, step 3 includes: passing each image frame in the video clip through an image feature extractor based on a convolutional neural network model to obtain a sequence of image frame semantic feature vectors; inputting the sequence of image frame semantic feature vectors into a video semantic feature extractor based on an RNN model to obtain a sequence of implicit feature vectors of semantic associations between image frames; and inputting the sequence of implicit feature vectors of semantic associations between image frames into a global context semantic saliency enhancement module to obtain a sequence of implicit feature vectors of enhanced semantic associations between image frames.

[0018] Optionally, the sequence of implicit feature vectors of semantic associations between image frames is input into a global context semantic significance enhancement module to obtain a sequence of enhanced implicit feature vectors of semantic associations between image frames, including: calculating the maximum value of each implicit feature vector of semantic associations between image frames in the sequence of implicit feature vectors of semantic associations between image frames to obtain a significant feature vector of semantic associations between image frames composed of each maximum value; calculating the average value of each implicit feature vector of semantic associations between image frames in the sequence of implicit feature vectors of semantic associations between image frames to obtain a global feature vector of semantic associations between image frames composed of each average value; performing neighborhood encoding and activation on the significant feature vector of semantic associations between image frames and the global feature vector of semantic associations between image frames to obtain a significant feature context-associated activation feature vector of semantic associations between image frames and a global feature context-associated activation feature vector of semantic associations between image frames. Feature vector; after fusing the image frame semantic association implicit feature significant feature context association activation feature vector and the image frame semantic association implicit feature global feature context association activation feature vector to obtain the image frame semantic association implicit feature significant-global feature context association activation feature vector, nonlinearly activate the image frame semantic association implicit feature significant-global feature context association activation feature vector to obtain the image frame semantic association implicit feature significant-global feature context association weight feature vector; using the image frame semantic association implicit feature significant-global feature context association weight feature vector as the weight vector, perform positional dot multiplication on each image frame semantic association implicit feature vector in the sequence of image frame semantic association implicit feature vectors, and add the sequence of vectors obtained by positional dot multiplication to the sequence of image frame semantic association implicit feature vectors to obtain the sequence of enhanced image frame semantic association implicit feature vectors.

[0019] Optionally, neighborhood coding and activation are performed on the significant feature vector of the semantic association implicit feature between image frames and the global feature vector of the semantic association implicit feature between image frames to obtain the significant feature context-associated activation feature vector of the semantic association implicit feature between image frames and the global feature context-associated activation feature vector of the semantic association implicit feature between image frames, including: performing one-dimensional convolution coding, non-linear activation and linear transformation on the significant feature vector of the semantic association implicit feature between image frames to obtain the significant feature context-associated activation feature vector of the semantic association implicit feature between image frames; performing one-dimensional convolution coding, non-linear activation and linear transformation on the global feature vector of the semantic association implicit feature between image frames to obtain the global feature context-associated activation feature vector of the semantic association implicit feature between image frames.

[0020] Optionally, the activation feature vector of the semantic association implicit feature significance-global feature context association between image frames is nonlinearly activated to obtain a weight feature vector of the semantic association implicit feature significance-global feature context association between image frames, including: after respectively passing the activation feature vector of the semantic association implicit feature significance-global feature context association between image frames through a tanh activation function and a sigmoid activation function, the two obtained vectors are point multiplied to obtain a weight feature vector of the semantic association implicit feature significance-global feature context association between image frames.

[0021] Optionally, the step 4 includes: inputting the candidate image frame into the image feature extractor based on the convolutional neural network model to obtain the semantic feature vector of the candidate image frame.

[0022] Optionally, step 5 includes: calculating the semantic similarity between the query feature vector and each enhanced image frame semantic association implicit feature vector in the sequence of enhanced image frame semantic association implicit feature vectors to obtain a sequence of one-way matching semantic measurement coefficients; performing normalization processing on the sequence of one-way matching semantic measurement coefficients based on a soft maximum function to obtain a sequence of one-way matching semantic measurement weights; using the sequence of one-way matching semantic measurement weights as weights, calculating the weighted sum of the sequence of enhanced image frame semantic association implicit feature vectors to obtain a one-way matching interactive reshaping feature vector; calculating the differential feature vector between the query feature vector and the one-way matching interactive reshaping feature vector as the one-way matching result representation vector.

[0023] Optionally, the semantic similarity between the query feature vector and each enhanced image frame semantic association implicit feature vector in the sequence of the enhanced image frame semantic association implicit feature vector is calculated to obtain a sequence of one-way matching semantic measurement coefficients, including: after cascading the query feature vector and the enhanced image frame semantic association implicit feature vector, multiplying the weight vector and the cascaded feature vector; adding the multiplied value to the bias parameter; and performing nonlinear activation processing based on the sigmoid function on the added value to obtain the one-way matching semantic measurement coefficient.

[0024] Optionally, the step 6 includes: inputting the one-way matching result representation vector into a classifier-based image frame association degree evaluator to obtain the evaluation result.

[0025] In a second aspect, the present application provides a red dot video display system for alarm events, the system comprising:

[0026] A video segment extraction module is used to extract video segments anchored based on alarm tags from original videos;

[0027] A previous frame image extraction module, used to extract a previous frame image of the video clip from the original video as a candidate image frame;

[0028] A semantic association feature enhancement module, used for extracting image frame semantic association features and enhancing image frame context semantic association features of the video clip to obtain a sequence of enhanced semantic association implicit feature vectors between image frames;

[0029] An image feature extraction module, used for inputting the candidate image frame into an image feature extractor to obtain a semantic feature vector of the candidate image frame;

[0030] A fine-grained scanning interaction module is used to use the semantic feature vector of the candidate image frame as a query feature vector, and input the sequence of the query feature vector and the implicit feature vector of the semantic association between the enhanced image frames into a fine-grained interaction module based on an attention mechanism to obtain a one-way matching result representation vector;

[0031] An evaluation result determination module, used to determine an evaluation result based on the one-way matching result representation vector, wherein the evaluation result is used to indicate whether to include the candidate image frame into the video segment;

[0032] A candidate image frame inclusion module, configured to include the candidate image frame into the video segment in response to the evaluation result, and use the candidate image frame as the first frame of the video segment to obtain an updated video segment;

[0033] A candidate image frame loop module is used to extract the previous frame image of the previous frame image from the original video as a candidate image frame, and loop until the evaluation result is that the candidate image frame is not included in the video segment;

[0034] The updated video clip display module is used to display the updated video clip.

[0035] The present application has at least the following technical effects: compared with the prior art, the present application uses intelligent algorithms and image processing technology to extract image frame semantic features of video clips anchored based on alarm tags (used to represent video clips during alarm periods) and the previous frame of the video clip, so as to understand the status of the alarm event and the picture status in the previous frame of the video clip, and further uses the image frame semantic features of the previous frame as query information, and scans and interactively matches the image frame semantic association feature patterns of the video clip, so as to intelligently determine whether the previous frame should be included in the video clip, and update the video clip in real time, and can filter backward image frames based on the same logic to achieve semantic-based two-way intelligent expansion of the video clip, providing richer contextual information for review work while avoiding the introduction of irrelevant content.

[0036] Other features and advantages of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The above and other features, advantages and aspects of the embodiments of the present application will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings:

[0038] Figure 1 The figure is a flow chart showing a red dot video display method for alarm events according to an exemplary embodiment.

[0039] Figure 2 It is a block diagram of a red dot video display system for alarm events according to an exemplary embodiment. DETAILED DESCRIPTION

[0040] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be interpreted as being limited to the embodiments specifically described. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0041] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0042] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0043] It should be noted that the concepts such as "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0044] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0045] The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0046] The specific implementation of the present application is described in detail below with reference to the accompanying drawings.

[0047] Figure 1 is a flow chart showing a red dot video display method for alarm events according to an exemplary embodiment. Figure 1 As shown, the method includes:

[0048] Step 1: Extract video segments anchored by alarm labels from the original video;

[0049] Step 2: extracting a previous frame of the video clip from the original video as a candidate image frame;

[0050] Step 3: extracting image frame semantic association features and enhancing image frame context semantic association features on the video clip to obtain a sequence of implicit feature vectors of enhanced semantic associations between image frames;

[0051] Step 4: inputting the candidate image frame into an image feature extractor to obtain a semantic feature vector of the candidate image frame;

[0052] Step 5: using the semantic feature vector of the candidate image frame as a query feature vector, inputting the query feature vector and the sequence of the implicit feature vector of the semantic association between the enhanced image frames into a fine-grained interaction module based on the attention mechanism to obtain a one-way matching result representation vector;

[0053] Step 6: determining an evaluation result based on the one-way matching result representation vector, wherein the evaluation result is used to indicate whether to include the candidate image frame into the video segment;

[0054] Step 7: In response to the evaluation result that the candidate image frame is to be included in the video segment, the candidate image frame is used as the first frame of the video segment to obtain an updated video segment;

[0055] Step 8: extracting the previous frame of the previous frame of the image from the original video as a candidate image frame, and looping steps 2 to 7 until the evaluation result is that the candidate image frame is not included in the video segment;

[0056] Step 9: Display the updated video clip.

[0057] In response to the above technical problems, the technical concept of the present application is: through intelligent algorithms and image processing technology, the image frame semantic features of the video clip anchored by the alarm tag (used to represent the video clip during the alarm period) and the previous frame of the video clip are extracted to understand the status of the alarm event and the picture status in the previous frame of the video clip, and further the image frame semantic features of the previous frame of the image are used as query information to scan and interactively match with the image frame semantic association feature pattern of the video clip, so as to intelligently determine whether the previous frame of the image should be included in the video clip, and update the video clip in real time. In the embodiment of the present application, the backward image frame can be screened based on the same logic to realize the semantic-based two-way intelligent expansion of the video clip, provide richer context information for the review work, and avoid the introduction of irrelevant content.

[0058] Based on this, in the technical solution of the present application, first, a video clip anchored based on an alarm tag is extracted from the original video. Among them, the video clip anchored based on the alarm tag (red dot) is a direct indication of the alarm event, which marks the specific time point when the alarm occurs. At the same time, the previous frame of the video clip is extracted from the original video as an alternative image frame. In the technical solution of the present application, the previous frame of the image may provide important contextual information for the alarm event. For example, it may contain the initial conditions or background that led to the occurrence of the alarm, which is crucial to understanding the overall picture of the entire event, and also provides an important basis for realizing intelligent expansion and visual display.

[0059] Then, each image frame in the video clip is passed through an image feature extractor based on a convolutional neural network model to obtain a sequence of image frame semantic feature vectors; and the sequence of image frame semantic feature vectors is input into a video semantic feature extractor based on an RNN model to obtain a sequence of implicit feature vectors of semantic associations between image frames. That is, the image feature extractor is constructed using a convolutional neural network model to extract features and abstractly express each image frame in the video clip; and the video semantic feature extractor is constructed using an RNN model to learn the associated feature pattern of the video clip in the time dimension.

[0060] In one embodiment of the present application, step 3 includes: passing each image frame in the video clip through an image feature extractor based on a convolutional neural network model to obtain a sequence of image frame semantic feature vectors; inputting the sequence of image frame semantic feature vectors into a video semantic feature extractor based on an RNN model to obtain a sequence of implicit feature vectors of semantic associations between image frames; and inputting the sequence of implicit feature vectors of semantic associations between image frames into a global context semantic saliency enhancement module to obtain a sequence of implicit feature vectors of enhanced semantic associations between image frames.

[0061] Considering that the sequence of the implicit feature vectors of the semantic association between the image frames expresses the event temporal semantic feature pattern of the video clip. However, since the video clip itself contains background information, it may be retained or even amplified during the feature extraction process, which interferes with the in-depth understanding of the overall picture of the event. Therefore, in the technical solution of the present application, the sequence of the implicit feature vectors of the semantic association between the image frames is input into the semantic saliency enhancement module of the global context to obtain a sequence of the implicit feature vectors of the semantic association between the image frames. Among them, the semantic saliency enhancement module of the global context calculates the maximum value of each implicit feature vector of the semantic association between the image frames in the sequence of the implicit feature vectors of the semantic association between the image frames to characterize the saliency features therein, and calculates the average value of each implicit feature vector of the semantic association between the image frames in the sequence of the implicit feature vectors of the semantic association between the image frames to characterize the global features therein. Then, the saliency features and the global features are further processed by convolution and nonlinear activation to obtain weights, and the sequence of the implicit feature vectors of the semantic association between the image frames is optimized and expressed based on the weights.

[0062] In one embodiment of the present application, the sequence of implicit feature vectors of semantic associations between image frames is input into a semantic significance enhancement module of a global context to obtain a sequence of implicit feature vectors of semantic associations between image frames that are enhanced, including: calculating the maximum value of each implicit feature vector of semantic associations between image frames in the sequence of implicit feature vectors of semantic associations between image frames to obtain a significant feature vector of semantic associations between image frames composed of each maximum value; calculating the average value of each implicit feature vector of semantic associations between image frames in the sequence of implicit feature vectors of semantic associations between image frames to obtain a global feature vector of semantic associations between image frames composed of each average value; performing neighborhood encoding and activation on the significant feature vector of semantic associations between image frames and the global feature vector of semantic associations between image frames to obtain a significant feature context-associated activation feature vector of semantic associations between image frames and a global feature context-associated activation feature vector of semantic associations between image frames. Associating activation feature vector; fusing the semantic association implicit feature significant feature context association activation feature vector between image frames and the semantic association implicit feature global feature context association activation feature vector between image frames to obtain the semantic association implicit feature significant-global feature context association activation feature vector between image frames, and then performing nonlinear activation on the semantic association implicit feature significant-global feature context association activation feature vector between image frames to obtain the semantic association implicit feature significant-global feature context association weight feature vector between image frames; using the semantic association implicit feature significant-global feature context association weight feature vector between image frames as the weight vector, performing positional dot multiplication on each image frame semantic association implicit feature vector in the sequence of image frame semantic association implicit feature vectors, and adding the sequence of vectors obtained by positional dot multiplication to the sequence of image frame semantic association implicit feature vectors element by element to obtain the sequence of enhanced image frame semantic association implicit feature vectors.

[0063] Furthermore, in one embodiment of the present application, neighborhood coding and activation are performed on the significant feature vector of the semantic association implicit feature between image frames and the global feature vector of the semantic association implicit feature between image frames to obtain the significant feature context-associated activation feature vector of the semantic association implicit feature between image frames and the global feature context-associated activation feature vector of the semantic association implicit feature between image frames, including: performing one-dimensional convolution coding, non-linear activation and linear transformation on the significant feature vector of the semantic association implicit feature between image frames to obtain the significant feature context-associated activation feature vector of the semantic association implicit feature between image frames; performing one-dimensional convolution coding, non-linear activation and linear transformation on the global feature vector of the semantic association implicit feature between image frames to obtain the global feature context-associated activation feature vector of the semantic association implicit feature between image frames.

[0064] Furthermore, in one embodiment of the present application, the activation feature vector of the semantic association implicit feature significance between image frames-global feature context association is nonlinearly activated to obtain a weight feature vector of the semantic association implicit feature significance between image frames-global feature context association, including: after respectively passing the activation feature vector of the semantic association implicit feature significance between image frames-global feature context association through a tanh activation function and a sigmoid activation function, the two obtained vectors are point multiplied to obtain a weight feature vector of the semantic association implicit feature significance between image frames-global feature context association.

[0065] Specifically, the sequence of implicit feature vectors of semantic associations between image frames is input into the semantic saliency enhancement module of the global context according to the following context formula to obtain the sequence of implicit feature vectors of semantic associations between enhanced image frames; wherein the context formula is:

[0066] ;

[0067] ;

[0068] ;

[0069] ;

[0070] ;

[0071] in, is a sequence of implicit feature vectors of semantic associations between the image frames, is the global feature vector of implicit features of semantic associations between image frames, is the salient feature vector of the implicit features of the semantic association between image frames, It means taking the average value, Indicates taking the maximum value, represents a one-dimensional convolution operation, express Activation function, and are the first weight matrix and the second weight matrix for linear transformation, and They are the semantic association implicit feature between image frames, the global feature context association activation feature vector, and the semantic association implicit feature between image frames, the salient feature context association activation feature vector. express function, express Activation function, represents dot product, is a sequence of implicit feature vectors of semantic associations between the enhanced image frames.

[0072] At the same time, the candidate image frame is input into the image feature extractor based on the convolutional neural network model to obtain a semantic feature vector of the candidate image frame. That is, the image feature extractor based on the convolutional neural network model is used to extract independent image frame semantic features of the candidate image frame, so that the model can understand the picture status in the candidate image frame.

[0073] In one embodiment of the present application, step 4 includes: inputting the candidate image frame into the image feature extractor based on the convolutional neural network model to obtain the semantic feature vector of the candidate image frame.

[0074] Next, the candidate image frame semantic feature vector is used as a query feature vector, and the query feature vector and the sequence of the enhanced image frame semantic association implicit feature vector are input into a fine-grained interaction module based on an attention mechanism to obtain a one-way matching result representation vector. In particular, when the candidate image frame semantic feature vector is used as a query feature vector and the sequence of the enhanced image frame semantic association implicit feature vector are input into a fine-grained interaction module based on an attention mechanism, the semantic association between the candidate image frame semantic feature vector and each enhanced image frame semantic association implicit feature vector in the sequence of the enhanced image frame semantic association implicit feature vector is used as a local semantic matching factor, and the local semantic matching factor is used as a class attention weight to comprehensively represent the one-way attention granular scanning interaction matching result of the candidate image frame semantic feature vector relative to the sequence of the enhanced image frame semantic association implicit feature vector, that is, the one-way matching result representation vector. In this way, the attention mechanism-based fine-grained interaction module allows the model to evaluate the degree of matching between the candidate image frame and the existing video clip at a fine-grained level to identify whether there is a strong correlation between the candidate image frame and the existing video clip.

[0075] In one embodiment of the present application, step 5 includes: calculating the semantic similarity between the query feature vector and each enhanced image frame semantic association implicit feature vector in the sequence of enhanced image frame semantic association implicit feature vectors to obtain a sequence of one-way matching semantic measurement coefficients; performing a soft maximum function-based normalization process on the sequence of one-way matching semantic measurement coefficients to obtain a sequence of one-way matching semantic measurement weights; using the sequence of one-way matching semantic measurement weights as weights, calculating the weighted sum of the sequence of enhanced image frame semantic association implicit feature vectors to obtain a one-way matching interactive reshaping feature vector; calculating the differential feature vector between the query feature vector and the one-way matching interactive reshaping feature vector as the one-way matching result representation vector.

[0076] Furthermore, in one embodiment of the present application, the semantic similarity between the query feature vector and each enhanced image frame semantic association implicit feature vector in the sequence of the enhanced image frame semantic association implicit feature vector is calculated to obtain a sequence of one-way matching semantic measurement coefficients, including: after cascading the query feature vector and the enhanced image frame semantic association implicit feature vector, multiplying the weight vector and the cascaded feature vector; adding the multiplied value to the bias parameter; and performing nonlinear activation processing based on the sigmoid function on the added value to obtain the one-way matching semantic measurement coefficient.

[0077] Specifically, the semantic feature vector of the candidate image frame is used as the query feature vector, and the sequence of the query feature vector and the implicit feature vector of the semantic association between the enhanced image frames is input into the fine-grained interaction module based on the attention mechanism by the following attention formula to obtain a one-way matching result representation vector; wherein the attention formula is:

[0078] ;

[0079] ;

[0080] ;

[0081] ;

[0082] in, is the query feature vector, is the jth enhanced image frame semantic association implicit feature vector in the sequence of enhanced image frame semantic association implicit feature vectors, and are the weight vector and bias parameter respectively, represents the sigmoid function, is the j-th one-way matching semantic metric coefficient, represents an exponential function with a natural constant as base, The value of is the number of implicit feature vectors of semantic association between enhanced image frames in the sequence of implicit feature vectors of semantic association between enhanced image frames, is the j-th one-way matching semantic metric weight, is the one-way matching interaction reshape feature vector, is the one-way matching result representation vector, Indicates point reduction.

[0083] Specifically, the one-way matching result representation vector is input into an image frame association degree evaluator based on a classifier to obtain an evaluation result, and the evaluation result is used to indicate whether the candidate image frame is included in the video segment. That is, the image frame association degree evaluator is constructed using a classifier to provide an automatically measurable standard to quantify and determine the matching association degree between the candidate image frame and the existing video segment.

[0084] In one embodiment of the present application, the step 6 includes: inputting the one-way matching result representation vector into a classifier-based image frame association degree evaluator to obtain the evaluation result.

[0085] Subsequently, in response to the evaluation result that the alternative image frame is to be included in the video clip, the alternative image frame is used as the first frame of the video clip to obtain an updated video clip. In this way, the video clip is made more comprehensive and coherent, and the transmission and display of information are optimized. In the technical solution of the present application, further, the previous frame image of the previous frame image is extracted from the original video as an alternative image frame, and the above steps are repeated until the evaluation result is not to include the alternative image frame in the video clip; and then the updated video clip is displayed. In this way, through continuous backtracking, the system can deeply analyze the background and environment before the event occurs, help understand the cause and background conditions of the event, and realize the automatic capture and display of dynamic scenes.

[0086] Specifically, the sequence of the candidate image frame semantic feature vectors and the enhanced image frame semantic association implicit feature vectors respectively express the image semantic features of the previous frame image of the video clip and each image frame in the video clip. However, the sequence of the enhanced image frame semantic association implicit feature vectors performs semantic saliency based on the global context on the basis of the encoded semantic feature representation of the source image semantics. Therefore, after the sequence of the candidate image frame semantic feature vectors and the enhanced image frame semantic association implicit feature vectors are input into the unidirectional attention fine-grained scanning interaction module, the obtained unidirectional matching result representation vector will also have a complex semantic spatial structure due to the difference in unidirectional attention interaction matching weights under different feature distribution representations of the candidate image frame semantic feature vectors and the enhanced image frame semantic association implicit feature vectors. Therefore, it is expected to improve its classification regression convergence and generalization effects under complex aggregated spatial structures.

[0087] Preferably, in an example of the present application, inputting the one-way matching result representation vector into a classifier-based image frame association degree evaluator to obtain an evaluation result comprises:

[0088] Calculate the sum of the absolute values ​​of each eigenvalue of the one-way matching result representation vector to obtain a first one-way matching result space structure value, and calculate the square root of the sum of the squares of each eigenvalue of the one-way matching result representation vector to obtain a second one-way matching result space structure value;

[0089] Multiplying each eigenvalue of the one-way matching result representation vector by the first one-way matching result space structure value and the second one-way matching result space structure value respectively to obtain a first one-way matching result structure reference value and a second one-way matching result structure reference value corresponding to each eigenvalue;

[0090] Multiplying each eigenvalue of the one-way matching result representation vector by the length of the one-way matching result representation vector and the square root of the length to obtain a first one-way matching result scale transformation value and a second one-way matching result scale transformation value corresponding to each eigenvalue;

[0091] Dividing the first one-way matching result structure reference value by the difference between the first one-way matching result space structure value and the first one-way matching result scale transformation value to obtain a first one-way matching result transformation adjustment value;

[0092] Dividing the second one-way matching result structure reference value by the difference between the second one-way matching result space structure value and the second one-way matching result scale transformation value to obtain a second one-way matching result transformation adjustment value;

[0093] Calculating a weighted sum of the first one-way matching result transformation adjustment value and the second one-way matching result transformation adjustment value to obtain each eigenvalue of the optimized one-way matching result representation vector;

[0094] The optimized one-way matching result representation vector is input into a classifier-based image frame association degree evaluator to obtain an evaluation result.

[0095] Specifically, the one-way matching result represents the vector The optimization expression is:

[0096] ;

[0097] ;

[0098] ;

[0099] ;

[0100] ;

[0101] ;

[0102] ;

[0103] ;

[0104] in, Indicates that the one-way matching result represents the first eigenvalues, Indicates the length of the one-way matching result vector, represents the set of real numbers, represents the spatial structure value of the first one-way matching result, represents the second one-way matching result space structure value, It means point multiplication by position. It means adding by position. Indicates that the one-way matching result represents the first The first one-way matching result corresponding to the feature value is transformed into an adjustment value, a first one-way matching result transformation adjustment vector composed of the first one-way matching result transformation adjustment values, Indicates that the one-way matching result represents the first The second one-way matching result corresponding to the feature value is transformed into an adjustment value, a second one-way matching result transformation adjustment vector composed of the second one-way matching result transformation adjustment values, is the weight hyperparameter, The optimized one-way matching result representation vector.

[0105] Therefore, in the preferred example, for the spatial structure information of the feature set of the one-way matching result representation vector in the high-dimensional space, the scale-based frame transformation of each eigenvalue of the one-way matching result representation vector is performed by taking the class norm space structured representation of the one-way matching result representation vector as a reference window, and the spatial structure-based frame attention weight adjustment of each eigenvalue of the one-way matching result representation vector is implemented to ensure the invariance of the spatial transformation (translation, scaling and rotation) of the one-way matching result representation vector under the interaction of the feature space, thereby improving the convergence and generalization effect of the classification regression of the feature set of the one-way matching result representation vector under the complex spatial structure representation, and improving the accuracy of the evaluation result obtained by passing it through the classifier-based image frame association evaluator.

[0106] In summary, the above scheme is adopted to extract image frame semantic features of the video clip anchored by the alarm tag (used to represent the video clip during the alarm period) and the previous frame of the video clip through intelligent algorithms and image processing technology, so as to understand the status of the alarm event and the picture status in the previous frame of the video clip, and further use the image frame semantic features of the previous frame as query information, and scan and interactively match with the image frame semantic association feature pattern of the video clip, so as to intelligently determine whether the previous frame should be included in the video clip, and update the video clip in real time. In an embodiment of the present application, the backward image frame can be screened based on the same logic to realize the semantic-based two-way intelligent expansion of the video clip, provide richer context information for the review work, and avoid the introduction of irrelevant content.

[0107] Figure 2 FIG. 1 is a block diagram of a red dot video display system for alarm events according to an exemplary embodiment. Figure 2 As shown, the red dot video display system 200 includes:

[0108] A video segment extraction module 201 is used to extract video segments anchored based on alarm tags from original videos;

[0109] A previous frame image extraction module 202, configured to extract a previous frame image of the video clip from the original video as a candidate image frame;

[0110] The semantic association feature enhancement module 203 is used to extract the image frame semantic association features and enhance the image frame context semantic association features of the video clip to obtain a sequence of enhanced semantic association implicit feature vectors between image frames;

[0111] An image feature extraction module 204, configured to input the candidate image frame into an image feature extractor to obtain a semantic feature vector of the candidate image frame;

[0112] A fine-grained scanning interaction module 205 is used to use the candidate image frame semantic feature vector as a query feature vector, and input the query feature vector and the sequence of the semantic association implicit feature vector between the enhanced image frames into a fine-grained interaction module based on an attention mechanism to obtain a one-way matching result representation vector;

[0113] An evaluation result determination module 206, configured to determine an evaluation result based on the one-way matching result representation vector, wherein the evaluation result is used to indicate whether to include the candidate image frame into the video segment;

[0114] A candidate image frame inclusion module 207 is configured to include the candidate image frame in the video segment in response to the evaluation result, and use the candidate image frame as the first frame of the video segment to obtain an updated video segment;

[0115] A candidate image frame loop module 208 is used to extract the previous frame of the previous frame of the original video as a candidate image frame, and loop until the evaluation result is that the candidate image frame is not included in the video segment;

[0116] The updated video segment display module 209 is used to display the updated video segment.

[0117] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

[0118] In addition, although each operation is described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or to be performed in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the application. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0119] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Claims

1. A red dot video display method for alarm events, characterized in that: include: Step 1: Extract video segments anchored by alarm labels from the original video; Step 2: extracting a previous frame of the video clip from the original video as a candidate image frame; Step 3: extracting image frame semantic association features and enhancing image frame context semantic association features on the video clip to obtain a sequence of implicit feature vectors of enhanced semantic associations between image frames; Step 4: inputting the candidate image frame into an image feature extractor to obtain a semantic feature vector of the candidate image frame; Step 5: using the semantic feature vector of the candidate image frame as a query feature vector, inputting the query feature vector and the sequence of the implicit feature vector of the semantic association between the enhanced image frames into a fine-grained interaction module based on the attention mechanism to obtain a one-way matching result representation vector; Step 6: determining an evaluation result based on the one-way matching result representation vector, wherein the evaluation result is used to indicate whether to include the candidate image frame into the video segment; Step 7: In response to the evaluation result that the candidate image frame is to be included in the video segment, the candidate image frame is used as the first frame of the video segment to obtain an updated video segment; Step 8: extracting the previous frame of the previous frame of the image from the original video as a candidate image frame, and looping steps 2 to 7 until the evaluation result is that the candidate image frame is not included in the video segment; Step 9: Displaying the updated video clip; Wherein, the step 3 comprises: Passing each image frame in the video clip through an image feature extractor based on a convolutional neural network model to obtain a sequence of image frame semantic feature vectors; Inputting the sequence of image frame semantic feature vectors into a video semantic feature extractor based on an RNN model to obtain a sequence of implicit feature vectors of semantic associations between image frames; Inputting the sequence of implicit feature vectors of semantic associations between image frames into a semantic saliency enhancement module of a global context to obtain the sequence of implicit feature vectors of semantic associations between enhanced image frames; The step of inputting the sequence of implicit feature vectors of semantic associations between image frames into a semantic saliency enhancement module of a global context to obtain the sequence of implicit feature vectors of semantic associations between enhanced image frames comprises: Calculating the maximum value of each of the image frame semantic association implicit feature vectors in the sequence of the image frame semantic association implicit feature vectors to obtain an image frame semantic association implicit feature salient feature vector composed of each of the maximum values; Calculating the average value of each of the image frame semantic association implicit feature vectors in the sequence of the image frame semantic association implicit feature vectors to obtain an image frame semantic association implicit feature global feature vector composed of each of the average values; Performing neighborhood coding and activation on the semantically associated implicit feature salient feature vector between image frames and the semantically associated implicit feature global feature vector between image frames to obtain the semantically associated implicit feature salient feature context-associated activated feature vector between image frames and the semantically associated implicit feature global feature context-associated activated feature vector between image frames; After fusing the inter-image-frame semantic association implicit feature significant feature context association activation feature vector and the inter-image-frame semantic association implicit feature global feature context association activation feature vector to obtain the inter-image-frame semantic association implicit feature significant-global feature context association activation feature vector, nonlinearly activate the inter-image-frame semantic association implicit feature significant-global feature context association activation feature vector to obtain the inter-image-frame semantic association implicit feature significant-global feature context association weight feature vector; Taking the inter-image-frame semantic association implicit feature saliency-global feature context association weight feature vector as the weight vector, performing positional dot multiplication on each inter-image-frame semantic association implicit feature vector in the sequence of the inter-image-frame semantic association implicit feature vectors, and adding the sequence of vectors obtained by positional dot multiplication to the sequence of the inter-image-frame semantic association implicit feature vectors element by element to obtain the sequence of the enhanced inter-image-frame semantic association implicit feature vectors; Among them, neighborhood coding and activation are performed on the significant feature vector of the semantic association implicit feature between image frames and the global feature vector of the semantic association implicit feature between image frames to obtain the significant feature context-associated activation feature vector of the semantic association implicit feature between image frames and the global feature context-associated activation feature vector of the semantic association implicit feature between image frames, including: Performing one-dimensional convolution coding, nonlinear activation and linear transformation on the semantic association implicit feature salient feature vector between the image frames to obtain the semantic association implicit feature salient feature contextual association activation feature vector between the image frames; One-dimensional convolution coding, nonlinear activation and linear transformation are performed on the global feature vector of the semantic association implicit feature between the image frames to obtain the global feature context-associated activation feature vector of the semantic association implicit feature between the image frames.

2. The red dot video display method for alarm events according to claim 1, characterized in that: The nonlinear activation is performed on the image frame semantic association implicit feature significant-global feature context association activation feature vector to obtain the image frame semantic association implicit feature significant-global feature context association weight feature vector, including: After the activation feature vectors of the semantic association implicit feature between image frames and the global feature context association are respectively passed through the tanh activation function and the sigmoid activation function, the two vectors obtained are point multiplied to obtain the weighted feature vector of the semantic association implicit feature between image frames and the global feature context association.

3. The red dot video display method for alarm events according to claim 2, characterized in that: The step 4 comprises: The candidate image frame is input into the image feature extractor based on the convolutional neural network model to obtain the semantic feature vector of the candidate image frame.

4. The red dot video display method for alarm events according to claim 3, characterized in that: The step 5 comprises: Calculating the semantic similarity between the query feature vector and each enhanced image frame semantic association implicit feature vector in the sequence of enhanced image frame semantic association implicit feature vectors to obtain a sequence of one-way matching semantic measurement coefficients; Performing a normalization process based on a soft maximum function on the sequence of one-way matching semantic metric coefficients to obtain a sequence of one-way matching semantic metric weights; Using the sequence of one-way matching semantic measurement weights as weights, calculating the weighted sum of the sequence of semantic association implicit feature vectors between enhanced image frames to obtain a one-way matching interactive reshaping feature vector; A differential feature vector between the query feature vector and the one-way matching interactive reshaping feature vector is calculated as the one-way matching result representation vector.

5. The red dot video display method for alarm events according to claim 4, characterized in that: Calculating the semantic similarity between the query feature vector and each enhanced image frame semantic association implicit feature vector in the sequence of enhanced image frame semantic association implicit feature vectors to obtain a sequence of one-way matching semantic measurement coefficients, including: After cascading the query feature vector and the implicit feature vector of the semantic association between the enhanced image frames, the weight vector and the feature vector obtained by the cascade are multiplied; the multiplied value is added to the bias parameter; the added value is subjected to nonlinear activation processing based on the sigmoid function to obtain the one-way matching semantic measurement coefficient.

6. The red dot video display method for alarm events according to claim 5, characterized in that: The step 6 comprises: The one-way matching result representation vector is input into a classifier-based image frame association degree evaluator to obtain the evaluation result.

7. A red dot video display system for alarm events, characterized in that: include: A video segment extraction module is used to extract video segments anchored based on alarm tags from original videos; A previous frame image extraction module, used to extract a previous frame image of the video clip from the original video as a candidate image frame; A semantic association feature enhancement module, used for extracting image frame semantic association features and enhancing image frame context semantic association features of the video clip to obtain a sequence of enhanced semantic association implicit feature vectors between image frames; An image feature extraction module, used for inputting the candidate image frame into an image feature extractor to obtain a semantic feature vector of the candidate image frame; A fine-grained scanning interaction module is used to use the semantic feature vector of the candidate image frame as a query feature vector, and input the sequence of the query feature vector and the implicit feature vector of the semantic association between the enhanced image frames into a fine-grained interaction module based on an attention mechanism to obtain a one-way matching result representation vector; An evaluation result determination module, used to determine an evaluation result based on the one-way matching result representation vector, wherein the evaluation result is used to indicate whether to include the candidate image frame into the video segment; A candidate image frame inclusion module, configured to include the candidate image frame into the video segment in response to the evaluation result, and use the candidate image frame as the first frame of the video segment to obtain an updated video segment; A candidate image frame loop module is used to extract the previous frame image of the previous frame image from the original video as a candidate image frame, and loop until the evaluation result is that the candidate image frame is not included in the video segment; An updated video clip display module, used to display the updated video clip; The step of extracting image frame semantic association features and enhancing image frame context semantic association features on the video clip to obtain a sequence of implicit feature vectors of enhanced semantic associations between image frames includes: Passing each image frame in the video clip through an image feature extractor based on a convolutional neural network model to obtain a sequence of image frame semantic feature vectors; Inputting the sequence of image frame semantic feature vectors into a video semantic feature extractor based on an RNN model to obtain a sequence of implicit feature vectors of semantic associations between image frames; Inputting the sequence of implicit feature vectors of semantic associations between image frames into a semantic saliency enhancement module of a global context to obtain the sequence of implicit feature vectors of semantic associations between enhanced image frames; The step of inputting the sequence of implicit feature vectors of semantic associations between image frames into a semantic saliency enhancement module of a global context to obtain the sequence of implicit feature vectors of semantic associations between enhanced image frames comprises: Calculating the maximum value of each of the image frame semantic association implicit feature vectors in the sequence of the image frame semantic association implicit feature vectors to obtain an image frame semantic association implicit feature salient feature vector composed of each of the maximum values; Calculating the average value of each of the image frame semantic association implicit feature vectors in the sequence of the image frame semantic association implicit feature vectors to obtain an image frame semantic association implicit feature global feature vector composed of each of the average values; Performing neighborhood coding and activation on the semantically associated implicit feature salient feature vector between image frames and the semantically associated implicit feature global feature vector between image frames to obtain the semantically associated implicit feature salient feature context-associated activated feature vector between image frames and the semantically associated implicit feature global feature context-associated activated feature vector between image frames; After fusing the inter-image-frame semantic association implicit feature significant feature context association activation feature vector and the inter-image-frame semantic association implicit feature global feature context association activation feature vector to obtain the inter-image-frame semantic association implicit feature significant-global feature context association activation feature vector, nonlinearly activate the inter-image-frame semantic association implicit feature significant-global feature context association activation feature vector to obtain the inter-image-frame semantic association implicit feature significant-global feature context association weight feature vector; Taking the inter-image-frame semantic association implicit feature saliency-global feature context association weight feature vector as the weight vector, performing positional dot multiplication on each inter-image-frame semantic association implicit feature vector in the sequence of the inter-image-frame semantic association implicit feature vectors, and adding the sequence of vectors obtained by positional dot multiplication to the sequence of the inter-image-frame semantic association implicit feature vectors element by element to obtain the sequence of the enhanced inter-image-frame semantic association implicit feature vectors; Among them, neighborhood coding and activation are performed on the significant feature vector of the semantic association implicit feature between image frames and the global feature vector of the semantic association implicit feature between image frames to obtain the significant feature context-associated activation feature vector of the semantic association implicit feature between image frames and the global feature context-associated activation feature vector of the semantic association implicit feature between image frames, including: Performing one-dimensional convolution coding, nonlinear activation and linear transformation on the semantic association implicit feature salient feature vector between the image frames to obtain the semantic association implicit feature salient feature contextual association activation feature vector between the image frames; One-dimensional convolution coding, nonlinear activation and linear transformation are performed on the global feature vector of the semantic association implicit feature between the image frames to obtain the global feature context-associated activation feature vector of the semantic association implicit feature between the image frames.

Images