Video detection method, system, medium and device based on zero-sample discrete hashing

Abstract

The present invention belongs to the field of information retrieval technology, and provides a video detection method, system, medium and device based on zero-sample discrete hashing, including obtaining a video to be detected, averaging frames of the video to be detected to generate a frame data set to be detected, using a trained visual feature extraction network model to extract features of the frame data set to be detected, and obtaining visual features of the frame data set; based on the visual features of the frame data set, using a trained hash function and a hash mapping matrix to obtain a visual hash code; performing calculations and judgments on the visual hash code and the hash code in the retrieval database to determine whether the video to be detected is abnormal; if the judgment result is abnormal, an alarm signal is issued. The present invention compresses and reduces the dimension of data through zero-sample hashing technology, detects unfamiliar abnormal situations from the perspective of hash codes and issues warnings, and realizes efficient and discriminative data dimension reduction and timely warning and detection of unfamiliar abnormal situations.

Description

Technical Field

[0001] The present invention belongs to the technical field of information retrieval, and in particular relates to a video detection method, system, medium and device based on zero-sample discrete hashing. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] The amount of data in surveillance videos of electric fields is often huge. Whether it is storing and transmitting surveillance videos, or warning and retrieving abnormal situations in surveillance videos, it is a very challenging problem. The acquisition, storage, analysis and retrieval of video images are difficult because both images and videos are high-dimensional data. It is very difficult and time-consuming to sort and retrieve data in the original space of high-dimensional data. However, hashing, as a fast nearest neighbor search technology, can easily realize video image processing and retrieval due to its great advantages in storage and search efficiency. The hashing method can convert high-dimensional data into compact binary codes while maintaining the similarity and structural information of the original data. By using XOR operations in Hamming space, pairwise comparisons can be performed very effectively between data. Therefore, the hashing method can achieve effective storage and fast retrieval of surveillance videos.

[0004] In addition, in the actual management scenarios of electric fields, the emergence of many new situations has brought new challenges to traditional hash retrieval methods. Traditional supervised hashing methods, such as supervised discrete hashing, can only identify "old situations" (known classes) and cannot identify "new situations" (unknown classes) in open real-world scenarios, so their performance is often unsatisfactory. Taking the photovoltaic power generation system that uses solar energy as an example, solar panels may be damaged due to aging of the equipment, and the wiring may be loose or broken at the cable connection points and junction boxes. Extreme weather such as wind, snow, rain and hail may damage the brackets and cells of photovoltaic equipment. Birds may build nests on the panels and cause component damage. Human operational errors and intrusions by outsiders will also bring safety hazards to the management and normal operation of electric fields. In many different scenarios with different situations, the hash algorithm model cannot pre-train all possible abnormal situations of the electric field. When dealing with unfamiliar situations, it is also difficult for traditional hash algorithm models to determine whether the situation will threaten the safe operation of the electric field, and it is impossible to make timely warnings. Timely monitoring of various unfamiliar abnormal situations in the electric field and issuing early warnings to ensure the normal operation of the electric field has become an electric field management issue that needs to be urgently resolved.

[0005] Traditional supervised hashing methods can only process known classes, not unknown classes, so they cannot be well applied to open real-world scenarios in electric fields. Zero-shot hashing can quickly identify "new situations" by transferring supervised knowledge from known classes to unknown classes, thereby effectively identifying unfamiliar situations and providing timely warnings for unfamiliar abnormal situations in electric fields. However, zero-shot hashing methods generally have poor fitting capabilities for nonlinear relationships, resulting in low retrieval performance, or complex optimization problems, which cause the performance of hashing methods to quickly collapse when the hash code length increases. Summary of the invention

[0006] In order to solve the above problems, the present invention proposes a video detection method, system, medium and device based on zero-sample discrete hashing. The present invention uses visual Transformer to generate visual features, fully utilizes the advantages of visual Transformer in capturing long-distance dependencies, and makes up for the shortcomings of zero-sample hashing method in fitting nonlinear relationships. Then, the visual features are input into the attribute-based strong discrete zero-sample hashing to generate hash codes, so that it can fully mine and utilize the label information and attribute features of known classes in zero-sample scenarios, maintain the structure and similarity of the original space to a greater extent, and adopt a strong discrete optimization strategy to minimize the negative impact of quantization loss on the model, and generate a more discriminative hash code. After the video image is reduced in dimension, the amount of data is greatly reduced, and an early warning can be issued in time when the hash code of the abnormal situation is detected, and the retrieved abnormal situation video summary is returned to the terminal, so that the staff can manually intervene and handle the abnormal situation in time.

[0007] According to some embodiments, a first solution of the present invention provides a video detection method based on zero-sample discrete hashing, which adopts the following technical solution:

[0008] The video detection method based on zero-sample discrete hashing includes:

[0009] Obtain the video to be detected, average the frames of the video to be detected to generate a frame data set to be detected, use the trained visual feature extraction network model to extract features from the frame data set to obtain the visual features of the frame data set;

[0010] Based on the visual features of the frame dataset, the visual hash code is obtained using the trained hash mapping matrix;

[0011] The visual hash code is compared with the hash code in the retrieval database to determine whether the video to be detected is abnormal;

[0012] If the judgment result is abnormal, an alarm signal is issued, and the video frame data sets before and after the abnormal frame are input into the trained visual feature extraction network model and hash mapping matrix again to generate a hash code for the abnormal frame data set, determine whether it is an abnormal frame, and generate a collection of abnormal frames as the video summary of the abnormal frame.

[0013] Furthermore, the visual hash code is operated and judged with the hash code in the retrieval database to determine whether the video to be detected is abnormal, specifically:

[0014] The visual hash code is XORed with the hash code in the retrieval database by 0 / 1, the Hamming distance is calculated, and similar data are returned in descending order of Hamming distance;

[0015] If the similar data returned is abnormal data, the visual hash code is classified as abnormal data, that is, an abnormal frame is retrieved.

[0016] Furthermore, the video frame datasets before and after the abnormal frame are input into the trained visual feature extraction network model and hash mapping matrix again to generate hash codes of the abnormal frame datasets, determine whether they belong to abnormal frames, and generate a collection of abnormal frames as the video summary of the abnormal frames, specifically:

[0017] The video of 1 minute before and 1 minute after the abnormal frame is averaged and the generated frame data set is input into the visual feature extraction network model and the trained hash mapping matrix in sequence to generate a hash code for it;

[0018] The hash code is input into the retrieval database for retrieval, and whether the frame is an abnormal frame is determined by judging whether the hash code is an abnormal hash code. If so, the frame with the abnormal hash code is merged as the video summary of the abnormal frame and transmitted back to the server terminal.

[0019] Furthermore, during training, the method processes the data of the video training set as follows:

[0020] The video training set is averaged and frame extracted to generate a training frame data set, and the frame data of the training frame data set is divided into a known class frame data set and an unknown class frame data set. A part is extracted from the known class frame data set as a known class training set, and a part is extracted from the unknown class frame data set as an unknown class test set. The remaining parts of the known class frame data set and the unknown class frame data set are used as retrieval sets, wherein the category labels of the known class frame data set and the unknown class frame data set do not overlap;

[0021] For each known class training sample in the known class frame dataset, extract the Word2Vec attribute real-valued vector according to the category name to form an attribute matrix;

[0022] According to the known class information, class labels are added to the known class training samples to form a label matrix; the label matrix of the known class training samples is used to generate a similarity matrix;

[0023] The training frame data set is input into the visual feature extraction network model to obtain the visual features of the training frame data set, including the visual features of known training class samples, the visual features of unknown class test samples and the visual features of the retrieval set samples.

[0024] Furthermore, the visual features of the training frame data set are used to train the hash mapping matrix to obtain a trained hash mapping matrix, which is specifically:

[0025] Based on the visual features of the known class training set samples and with the hash function as a constraint, the initial hash mapping matrix is ​​iteratively trained to obtain the hash codes of the known class training set samples and the trained hash code mapping matrix;

[0026] Store the hash codes corresponding to the known class training set into the retrieval database;

[0027] Processing the visual features of the retrieval set samples with the trained hash code mapping matrix to obtain the hash code of the retrieval set, and storing it in the retrieval database;

[0028] The visual features of the unknown class test samples are processed by the trained hash code mapping matrix to obtain the hash code of the unknown class test set.

[0029] Furthermore, the hash function is an objective function of a strong discrete sample hash algorithm based on attributes, specifically:

[0030] The loss function term of projecting visual features and label information into hash codes, the loss function term of projecting attribute features into another hash code, the loss function term of projecting visual features, label information and attribute features into the common subspace of two hash codes, the loss function term of maintaining spatial structure, the loss function term of maintaining intra-class similarity, and the regularization term.

[0031] Furthermore, the visual features based on the frame data set are used to obtain the visual hash code using the trained hash mapping matrix, specifically:

[0032] Mapping matrix according to hash code The hash code of the visual feature X of the frame dataset is calculated to generate the hash code B of the video frame dataset to be retrieved, that is, .

[0033] According to some embodiments, a second solution of the present invention provides a video detection system based on zero-sample discrete hashing, which adopts the following technical solution:

[0034] The video detection system based on zero-shot discrete hashing includes:

[0035] The feature extraction module is configured to obtain a video to be detected, average the frames of the video to be detected to generate a frame data set to be detected, and use the trained visual feature extraction network model to extract features from the frame data set to be detected to obtain visual features of the frame data set;

[0036] A hash code generation module is configured to obtain a visual hash code using a trained hash mapping matrix based on visual features of a frame dataset;

[0037] The abnormal frame retrieval module is configured to perform calculation and judgment on the visual hash code and the hash code in the retrieval database to determine whether the video to be detected is abnormal;

[0038] The video summary generation module is configured to send out an alarm signal if the judgment result is abnormal, and input the video frame data set before and after the abnormal frame into the trained visual feature extraction network model and hash mapping matrix again, generate a hash code for the abnormal frame data set, judge whether it is an abnormal frame and generate a collection of abnormal frames as the video summary of the abnormal frame.

[0039] According to some embodiments, a third aspect of the present invention provides a computer-readable storage medium.

[0040] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the video detection method based on zero-sample discrete hashing as described in the first aspect above.

[0041] According to some embodiments, a fourth aspect of the present invention provides a computer device.

[0042] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the video detection method based on zero-sample discrete hashing as described in the first aspect above are implemented.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] The present invention uses visual Transformer to generate visual features, fully utilizes the advantages of visual Transformer in capturing long-distance dependencies, and makes up for the shortcomings of the crossover of the ability of hashing methods to fit nonlinear relationships. Then, a strong discrete zero-sample hash based on attributes is used to generate hash codes from visual features, and visual features, label information and attribute features are projected into a common subspace to generate hash codes, fully mining and utilizing limited pairwise similarity relationships and attribute information in the supervision scenario, improving the ability of the algorithm to transfer supervision knowledge and identify "old situation" unknown classes, and using a strong discrete strategy in the hash code generation process to reduce the negative impact of quantization loss to a certain extent, improving the performance of hash learning, overcoming the limitations of the poor ability of traditional supervised hashing methods to fit nonlinear relationships, and exploring the original space structure and similarity preservation problems that are not considered by traditional supervised hashing methods when constructing common subspaces, achieving the dual goals of enhancing semantic information to maintain the original space structure and reducing quantization loss. In addition, the similarity information within the sample is embedded when learning the hash function, so that the linear hash projection is biased towards a direction that is more conducive to retrieval. Compared with the existing methods, the computational complexity is reduced while the retrieval accuracy is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0046] Figure 1 is a flow chart of a video detection method based on zero-sample discrete hashing in an embodiment of the present invention;

[0047] Figure 2 is a processing flow chart of a visual feature extraction network model in an embodiment of the present invention;

[0048] Figure 3 is a processing flow chart of a zero-sample hash module in an embodiment of the present invention;

[0049] Figure 4 It is a flow chart of generating a summary of an abnormal frame video in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0051] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0052] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0053] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0054] Embodiment 1

[0055] like Figure 1 As shown, this embodiment provides a video detection method based on zero-sample discrete hashing. This embodiment uses the method applied to a server as an example for illustration. It can be understood that the method can also be applied to a terminal, and can also be applied to a terminal, a server, and a system, and is implemented through the interaction between the terminal and the server. The server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communications, middleware services, domain name services, security services CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application. In this embodiment, the method includes the following steps:

[0056] Obtain the video to be detected, average the frames of the video to be detected to generate a frame data set to be detected, use the trained visual feature extraction network model to extract features from the frame data set to obtain the visual features of the frame data set;

[0057] Based on the visual features of the frame dataset, the visual hash code is obtained using the trained hash mapping matrix;

[0058] The visual hash code is compared with the hash code in the retrieval database to determine whether the video to be detected is abnormal;

[0059] If the judgment result is abnormal, an alarm signal is issued, and the video frame data sets before and after the abnormal frame are input into the trained visual feature extraction network model and hash mapping matrix again to generate a hash code for the abnormal frame data set, determine whether it is an abnormal frame, and generate a collection of abnormal frames as the video summary of the abnormal frame.

[0060] like Figure 2 and Figure 3 As shown, this embodiment provides a training process of a video detection method based on zero-sample discrete hashing, which specifically includes the following steps:

[0061] S1: Average the frames of the video to be retrieved and the video training set to generate frame data sets respectively, and input the frame data sets into the visual feature extraction network model to obtain the visual features of the frame data sets.

[0062] The specific steps are as follows: average the frames of the video to be retrieved and the video training set to generate frame data sets respectively, and input the images in the frame data sets of the video to be retrieved and the video training set into the visual feature extraction network model - visual transformer to obtain the visual features of the video to be retrieved and the video training set respectively;

[0063] The frame data set of the video training set is divided into a known class frame data set and an unknown class frame data set. A part is extracted from the known class frame data set as the known class training set, and a part is extracted from the unknown class frame data set as the unknown class test set. The remaining parts of the known class frame data set and the unknown class true data set are used as retrieval sets, wherein the category labels of the known class and the unknown class do not overlap. Since the data categories used in the training process and the test process are different, the division of the data set enables the algorithm to identify "new situations" that did not appear in the training process. That is to say, compared with other video monitoring methods, this embodiment has more advantages in identifying completely new abnormal situations in real scenes.

[0064] For each known class training sample in the known class frame dataset, extract the Word2Vec attribute real-valued vector according to the category name to form an attribute matrix;

[0065] According to the known class information, class labels are added to the known class training samples to form a label matrix;

[0066] Generate a similarity matrix using the label matrix of training samples of known classes;

[0067] The training frame data set is input into the visual feature extraction network model to obtain the visual features of the training frame data set, including the visual features of known training class samples, the visual features of unknown class test samples and the visual features of the retrieval set samples.

[0068] like Figure 2 As shown, in a specific implementation process, the processing flow of the visual feature extraction network model is as follows:

[0069] Assume that there are n pictures in the frame data set, and the i-th picture is , and is the height and width of the image, is the number of channels of the image, is a real number. First, the image is segmented to obtain 9 images after segmentation. , ,in is the resolution of the image. Then the segmented image is mapped into 9 D-dimensional vectors through a linear projection layer , .at this time, Convert the image into D dimensions.

[0070] Set D-dimensional learnable embedding vector , the embedding vector With 9 segmented image vectors Combine and add position encoding Get the embedding vector ,in , and then input the embedded vector into the visual Transformer, which outputs D-dimensional visual features.

[0071] Visual feature extraction network model - Visual Transformer consists of Transformer encoder, linear projection layer and other conventional structures. The encoder uses Vision Transformer as the network structure, which is composed of multiple multi-layer self-attention layers and multi-layer perceptrons. Layer normalization and residual connection are used before and after the multi-layer self-attention layer and multi-layer perceptron respectively. Using Visual Transformer to generate visual features is to make full use of the advantage of Visual Transformer in capturing long-distance dependencies, while making up for the disadvantage of hashing method's poor ability to fit nonlinear relationships.

[0072] In a specific implementation process, the frame data set division process of the video training set is as follows: the visual features of the image data are divided into and ,in Belongs to a known class category , Belongs to unknown category ,and For simplicity, we will use A matrix representing the visual features of an image.

[0073] In a specific implementation, the sample label is a one-hot vector (1 if the sample belongs to the class, 0 otherwise), and the label matrix is ​​used to generate the similarity matrix. , similarity matrix Indicates Pictures and Whether the pictures belong to the same category, if they are the same category, it is 1, otherwise it is -1.

[0074] S2: Based on the visual features of the video training set, a hash mapping matrix is ​​trained to obtain a trained hash mapping matrix and generate a retrieval database.

[0075] Establish the objective function of attribute-based strong discrete zero-sample hashing as the hash function, optimize the hash function to generate a closed-form solution; take the hash function as a constraint, use the visual features of the known class training set samples to train the initial hash mapping matrix, and obtain the hash code of the known class training set samples and the trained hash code mapping matrix;

[0076] Store the hash codes corresponding to the training set samples of known classes into the retrieval database;

[0077] The visual features of the retrieval set samples are processed by the trained hash code mapping matrix to obtain the hash code of the retrieval set samples and store them in the retrieval database; the hash code in the retrieval database is composed of the hash code corresponding to the known class training set and the hash code of the retrieval set;

[0078] The visual features of the unknown class test set samples are processed by the trained hash code mapping matrix to obtain the hash codes of the unknown class test set samples, so as to test the performance of the zero-sample hash module.

[0079] In a specific implementation, the hash function is the objective function of an attribute-based strong discrete zero-shot hash algorithm, which is composed of a loss function term for projecting visual features and label information into a hash code, a loss function term for projecting attribute features into another hash code, a loss function term for projecting visual features, label information, and attribute features into a common subspace of two hash codes, a loss function term for preserving spatial structure, a loss function term for preserving intra-class similarity, and a regularization term.

[0080] In the following formula, Represents the visual features of image data, Represents the label information of the image data, Represents the attribute characteristics of image data, represents the hash code obtained by mapping visual features and label information, and Indicates the amount of image data, represents the dimension of the attribute vector, represents the dimension of the attribute vector, and Indicates the length of the hash code, represents the mapping matrix from visual features to hash codes, represents the mapping matrix from visual features to hash codes, Represents the hash code obtained by attribute feature mapping, Represents the mapping matrix from attribute features to hash codes, represents the similarity matrix, The Laplacian matrix representing visual features, represents a vector of all 1s with dimension N, represents a zero vector of dimension K, represents the identity matrix of dimension K, is a hyperparameter used to balance the loss terms of each function.

[0081] The construction process of the hash function is:

[0082] The loss function term of visual features and label information projected into hash codes is:

[0083] ;

[0084] By projecting the visual features and label information of image data into hash codes, the constrained hash codes preserve the semantic information of the visual features and label information.

[0085] The attribute feature is projected into another hash code loss function term, specifically:

[0086] ;

[0087] By projecting the attribute features of image data into hash codes, constrained hash codes preserve the semantic information of the attribute features.

[0088] The loss function term of visual features, label information and attribute features projected into the common subspace of two hash codes is:

[0089] ;

[0090] The visual features and label information of the image data are mapped to a hash code Hamming space, and the attribute features of the image data are mapped to another hash code Hamming space, so that the loss function term of the subtraction of the two hash code Hamming spaces is as small as possible, so that the visual features, label information and attribute features of the image data can be mapped to a common Hamming subspace to generate hash codes. The mapped common subspace not only extracts the semantic information of the image visual features and label information, but also incorporates the supervisory information of the attribute features to generate a more discriminative hash code.

[0091] The loss function term for spatial structure preservation is:

[0092] ;

[0093] By mapping the inner product of hash codes to a similarity matrix, the hash codes from different categories must be orthogonal and the hash codes from the same category must be the same, thus preserving the structural information of the original space.

[0094] The loss function term for maintaining intra-class similarity is:

[0095] ,

[0096] Can be converted into ;

[0097] in, , express No. The loss function term of the hash codes from the same category is made as small as possible, making the intra-class distance as close as possible to maintain the intra-class similarity of the original space.

[0098] Regularization term, specifically:

[0099] ;

[0100] The overall hash objective function is, specifically:

[0101] ;

[0102] In a specific embodiment, in optimizing In the process, the hash code uses a strong discrete strategy to obtain a closed-form solution ,

[0103] and Depend on The singular value decomposition of is obtained, where , , and Obtained via the Gram-Schmidt process.

[0104] S3: Input the visual features of the video to be retrieved into the trained hash function, and obtain the hash code of the video frame data set to be retrieved through the hash mapping matrix:

[0105] Get the visual features of the video to be retrieved and map the matrix according to the hash code Calculate the hash code, that is , generate the hash code of the video frame dataset to be retrieved.

[0106] S4: inputting the hash code into the database for retrieval, and if the hash code is classified as abnormal data, that is, an abnormal frame is retrieved, an early warning is issued and an alarm signal is issued;

[0107] The specific steps are as follows: the hash code of the video frame data set to be inspected and retrieved is XORed with the hash code in the retrieval database by 0 / 1, and the Hamming distance is calculated. The smaller the Hamming distance, the greater the probability that the hash code belongs to the same category; the larger the Hamming distance, the smaller the probability that the hash code belongs to the same category. After calculating the Hamming distance, similar data will be returned in ascending order of Hamming distance. If the similar data returned is abnormal data, the hash code will be classified as abnormal data, that is, an abnormal frame has been retrieved. An early warning will be issued and an alarm signal will be issued.

[0108] Since the retrieval database contains both normal and abnormal data, and the normal data and the abnormal data belong to different categories, it is possible to determine whether the data is abnormal based on the category of the returned similar data. If the data is abnormal, it means that an abnormal frame has been detected.

[0109] S5: inputting the video frame data set before and after the abnormal frame into the trained hash function again, generating a hash code of the abnormal frame data set, determining whether it is an abnormal frame and generating a collection of abnormal frames as a video summary of the abnormal frame;

[0110] like Figure 4 As shown in the figure, the specific steps are: average the frames of the video before and after the abnormal frame, input the generated frame data set into the visual Transformer and the trained hash function in sequence, and generate a hash code for it. Input the hash code into the retrieval database for retrieval, and judge whether the frame is an abnormal frame by judging whether the hash code belongs to an abnormal hash code. The frames with abnormal hash codes are merged as the video summary of the abnormal frame and transmitted back to the server terminal.

[0111] To verify the superiority of the proposed method over the existing methods, simulation experiments with four hash code lengths (24, 48, 64, and 128 bits) were performed on two datasets. AWA2 is a dataset for attribute-based sample classification and zero-shot learning. It has 50 animal categories, 40 of which are randomly selected as known categories, and the remaining 10 categories are unknown categories. CUB is a bird dataset for zero-shot learning. It has 200 categories, 150 of which are randomly selected as known categories, and the remaining 50 categories are unknown categories. We use Word2Vec to generate category attributes based on category names, and use category attributes to generate the required attribute matrix.

[0112] Table 1 MAP performance comparison of this embodiment and other algorithms (AWA dataset)

[0113]

[0114] Table 2 MAP performance comparison of this embodiment and other algorithms (CUB dataset)

[0115]

[0116] Table 1 and Table 2 are simulation experiments of this embodiment, which are measured by MAP (average precision mean). Table 1 shows the average precision mean of this embodiment in the AWA data set, and Table 2 shows the average precision mean of this embodiment in the CUB data set. In each simulation experiment, the method we proposed is compared with one supervised hashing method (SDH, supervised discrete hashing) and three zero-sample hashing methods (AH attribute hashing, TSK migration supervised knowledge hashing, and ASZH asymmetric similarity matrix hashing). It can be seen that the method of this embodiment has superior performance under different hash code lengths.

[0117] This embodiment extracts features from the video through a visual Transformer, and compresses and reduces the dimension of the data through zero-sample hashing technology, detects unfamiliar abnormal situations from the perspective of hash codes and issues warnings, and detects relevant frame data sets to generate video summaries of abnormal frames while issuing warnings. The invention is proposed to mainly achieve efficient and discriminative data dimensionality reduction and timely warning and detection of unfamiliar abnormal situations.

[0118] Embodiment 2

[0119] This embodiment provides a video detection system based on zero-sample discrete hashing, including:

[0120] The feature extraction module is configured to obtain a video to be detected, average the frames of the video to be detected to generate a frame data set to be detected, and use the trained visual feature extraction network model to extract features from the frame data set to be detected to obtain visual features of the frame data set;

[0121] A hash code generation module is configured to obtain a visual hash code using a trained hash mapping matrix based on visual features of a frame dataset;

[0122] The abnormal frame retrieval module is configured to perform calculation and judgment on the visual hash code and the hash code in the retrieval database to determine whether the video to be detected is abnormal;

[0123] The video summary generation module is configured to send out an alarm signal if the judgment result is abnormal, and input the video frame data set before and after the abnormal frame into the trained visual feature extraction network model and hash mapping matrix again, generate a hash code for the abnormal frame data set, judge whether it is an abnormal frame and generate a collection of abnormal frames as the video summary of the abnormal frame.

[0124] The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment 1. It should be noted that the above modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0125] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0126] The proposed system can be implemented in other ways. For example, the system embodiment described above is only illustrative, and the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0127] Embodiment 3

[0128] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the video detection method based on zero-sample discrete hashing as described in the first embodiment are implemented.

[0129] Embodiment 4

[0130] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the video detection method based on zero-sample discrete hashing as described in the first embodiment are implemented.

[0131] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0132] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0133] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0134] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0135] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0136] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A video detection method based on zero-sample discrete hashing, characterized in that: include: Obtain the video to be detected, average the frames of the video to be detected to generate a frame data set to be detected, use the trained visual feature extraction network model to extract features from the frame data set to obtain the visual features of the frame data set; The frame dataset of the video training set is divided into a known class frame dataset and an unknown class frame dataset, a part is extracted from the known class frame dataset as the known class training set, a part is extracted from the unknown class frame dataset as the unknown class test set, and the remaining part of the known class frame dataset and the unknown class true dataset is used as the retrieval set; Based on the visual features of the frame dataset, the visual hash code is obtained using the trained hash mapping matrix; The video training set is averaged and frame-sampled to generate a training frame data set, and the visual features of the training frame data set are used to train the hash mapping matrix to obtain a trained hash mapping matrix, specifically: based on the visual features of the known class training set samples, the initial hash mapping matrix is ​​iteratively trained with the hash function as a constraint to obtain the hash code of the known class training set samples and the trained hash code mapping matrix; the hash code corresponding to the known class training set is stored in a retrieval database; The visual features of the retrieval set samples are processed by the trained hash code mapping matrix to obtain the hash code of the retrieval set and stored in the retrieval database; the visual features of the unknown class test samples are processed by the trained hash code mapping matrix to obtain the hash code of the unknown class test set; The hash function is an objective function of a strong discrete sample hash algorithm based on attributes, specifically, it is composed of a loss function term of projecting visual features and label information into a hash code, a loss function term of projecting attribute features into another hash code, a loss function term of projecting visual features, label information and attribute features into a common subspace of two hash codes, a loss function term of maintaining spatial structure, a loss function term of maintaining intra-class similarity, and a regularization term; the visual features and label information of the image data are mapped into a hash code Hamming space, and the attribute features of the image data are mapped into another hash code Hamming space, so that the loss function term of subtracting the two hash code Hamming spaces is as small as possible, and then the visual features, label information and attribute features of the image data are mapped into a common Hamming subspace to generate a hash code; the loss function term of maintaining spatial structure is specifically: by mapping the inner product of the hash code to a similarity matrix, constraining hash codes from different categories to be orthogonal, and hash codes from the same category to be the same, so as to maintain the structural information of the original space; The visual hash code is compared with the hash code in the retrieval database to determine whether the video to be detected is abnormal. If the judgment result is abnormal, an alarm signal is issued, and the video frame data sets before and after the abnormal frame are input into the trained visual feature extraction network model and hash mapping matrix again to generate a hash code for the abnormal frame data set, determine whether it is an abnormal frame, and generate a collection of abnormal frames as the video summary of the abnormal frame.

2. The video detection method based on zero-sample discrete hashing as claimed in claim 1, characterized in that: The visual hash code is operated and judged with the hash code in the retrieval database to determine whether the video to be detected is abnormal, specifically: The visual hash code is XORed with the hash code in the retrieval database by 0 / 1, the Hamming distance is calculated, and similar data are returned in descending order of Hamming distance; If the similar data returned is abnormal data, the visual hash code is classified as abnormal data, that is, an abnormal frame is retrieved.

3. The video detection method based on zero-sample discrete hashing as claimed in claim 1, characterized in that: The video frame datasets before and after the abnormal frame are input into the trained visual feature extraction network model and hash mapping matrix again, a hash code of the abnormal frame dataset is generated, whether it is an abnormal frame is determined and a collection of abnormal frames is generated as the video summary of the abnormal frame, specifically: The video of 1 minute before and 1 minute after the abnormal frame is averaged and the generated frame data set is input into the visual feature extraction network model and the trained hash mapping matrix in sequence to generate a hash code for it; The hash code is input into the retrieval database for retrieval, and whether the frame is an abnormal frame is determined by judging whether the hash code is an abnormal hash code. If so, the frame with the abnormal hash code is merged as the video summary of the abnormal frame and transmitted back to the server terminal.

4. The video detection method based on zero-sample discrete hashing as claimed in claim 1, characterized in that: During training, the method processes the data of the video training set as follows: The video training set is averaged and frame extracted to generate a training frame data set, and the frame data of the training frame data set is divided into a known class frame data set and an unknown class frame data set. A part is extracted from the known class frame data set as a known class training set, and a part is extracted from the unknown class frame data set as an unknown class test set. The remaining parts of the known class frame data set and the unknown class frame data set are used as retrieval sets, wherein the category labels of the known class frame data set and the unknown class frame data set do not overlap; For each known class training sample in the known class frame dataset, extract the Word2Vec attribute real-valued vector according to the category name to form an attribute matrix; According to the known class information, class labels are added to the known class training samples to form a label matrix; the label matrix of the known class training samples is used to generate a similarity matrix; The training frame data set is input into the visual feature extraction network to obtain the visual features of the training frame data set, including the visual features of known training class samples, the visual features of unknown class test samples and the visual features of the retrieval set samples.

5. The video detection method based on zero-sample discrete hashing as claimed in claim 1, characterized in that: Based on the visual features of the frame data set, the visual hash code is obtained by using the trained hash mapping matrix, specifically: Mapping matrix according to hash code The hash code of the visual feature X of the frame dataset is calculated to generate the hash code B of the video frame dataset to be retrieved, that is, .

6. A video detection system based on zero-sample discrete hashing, characterized in that: include: The feature extraction module is configured to obtain a video to be detected, average the frames of the video to be detected to generate a frame data set to be detected, and use the trained visual feature extraction network model to extract features from the frame data set to be detected to obtain visual features of the frame data set; The frame dataset of the video training set is divided into a known class frame dataset and an unknown class frame dataset, a part is extracted from the known class frame dataset as the known class training set, a part is extracted from the unknown class frame dataset as the unknown class test set, and the remaining part of the known class frame dataset and the unknown class true dataset is used as the retrieval set; A hash code generation module is configured to obtain a visual hash code using a trained hash mapping matrix based on visual features of a frame dataset; The video training set is averaged and frame-sampled to generate a training frame data set, and the visual features of the training frame data set are used to train the hash mapping matrix to obtain a trained hash mapping matrix, specifically: based on the visual features of the known class training set samples, the initial hash mapping matrix is ​​iteratively trained with the hash function as a constraint to obtain the hash code of the known class training set samples and the trained hash code mapping matrix; the hash code corresponding to the known class training set is stored in a retrieval database; The visual features of the retrieval set samples are processed by the trained hash code mapping matrix to obtain the hash code of the retrieval set and stored in the retrieval database; the visual features of the unknown class test samples are processed by the trained hash code mapping matrix to obtain the hash code of the unknown class test set; The hash function is an objective function of a strong discrete sample hash algorithm based on attributes, specifically, it is composed of a loss function term of projecting visual features and label information into a hash code, a loss function term of projecting attribute features into another hash code, a loss function term of projecting visual features, label information and attribute features into a common subspace of two hash codes, a loss function term of maintaining spatial structure, a loss function term of maintaining intra-class similarity, and a regularization term; the visual features and label information of the image data are mapped into a hash code Hamming space, and the attribute features of the image data are mapped into another hash code Hamming space, so that the loss function term of subtracting the two hash code Hamming spaces is as small as possible, and then the visual features, label information and attribute features of the image data are mapped into a common Hamming subspace to generate a hash code; the loss function term of maintaining spatial structure is specifically: by mapping the inner product of the hash code to a similarity matrix, constraining hash codes from different categories to be orthogonal, and hash codes from the same category to be the same, so as to maintain the structural information of the original space; The abnormal frame retrieval module is configured to perform calculation and judgment on the visual hash code and the hash code in the retrieval database to determine whether the video to be detected is abnormal; The video summary generation module is configured to send out an alarm signal if the judgment result is abnormal, and input the video frame data set before and after the abnormal frame into the trained visual feature extraction network model and hash mapping matrix again, generate a hash code for the abnormal frame data set, judge whether it is an abnormal frame and generate a collection of abnormal frames as the video summary of the abnormal frame.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the video detection method based on zero-sample discrete hashing as described in any one of claims 1 to 5 are implemented.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the video detection method based on zero-sample discrete hashing as described in any one of claims 1 to 5 are implemented.

Images