Video image difference detection method, device and equipment and readable storage medium

Abstract

The invention discloses a video image difference detection method. The method comprises the following steps: acquiring a reference image and a to-be-detected image; blocking the reference image and the to-be-detected image according to a preset image blocking principle to obtain block image pairs; performing feature point extraction and matching on each block image pair to obtain the confidence coefficient of the similarity of each block image pair; and when the difference between the reference image and the to-be-detected image is determined according to each confidence coefficient, performing difference detection on the reference image and the to-be-detected image by using a twin neural network image detection model obtained by pre-training to obtain a video image difference region. By applying the video image difference detection method provided by the invention, the accuracy of image difference detection is improved, a lightweight twin neural network image detection model can be deployed to edge equipment, and the model is ensured to have relatively good performance. The invention also discloses a video image difference detection device and equipment, and a storage medium, which have corresponding technical effects.

Description

Technical Field

[0001] The present application relates to the field of video monitoring technology, and in particular to a video image difference detection method, device, equipment and computer-readable storage medium. Background Art

[0002] Video image difference detection is one of the important applications in the field of video surveillance. It is a technology used to detect the difference areas between video images. Currently, there are two main solutions: one is the traditional image processing method combined with machine learning; the other is the deep learning method.

[0003] Traditional image processing technology is a video event detection technology based on the difference between adjacent steady-state video images. It uses digital video and image processing technology to measure the pixel points and similarity of the image to detect the difference position. Since the video acquisition process is greatly affected by the environment, different lighting and equipment will cause pixel differences, resulting in poor stability and low accuracy, which has certain limitations.

[0004] The deep learning algorithm is used to automatically detect the difference areas between video images. The effect of deep learning neural network requires a large amount of video data as support. Video data requires a large amount of different types of materials so that the trained model can be generalized. The structure of neural network model is often complex and has a large number of parameters. It is difficult to guarantee the performance when deployed on edge devices.

[0005] In summary, how to effectively solve the problems of poor stability and low accuracy of current video image difference detection methods, complex neural network model structures, and difficulty in ensuring performance when deployed on edge devices is an issue that technical personnel in this field urgently need to solve. Summary of the invention

[0006] The purpose of this application is to provide a video image difference detection method, which improves the accuracy of image difference detection, enables a lightweight twin neural network image detection model to be deployed to edge devices, and ensures that the model has good performance; another purpose of this application is to provide a video image difference detection device, equipment and computer-readable storage medium.

[0007] In order to solve the above technical problems, this application provides the following technical solutions:

[0008] A video image difference detection method, comprising:

[0009] Acquire a reference image and an image to be detected;

[0010] Divide the reference image and the image to be detected into blocks according to a preset image dividing principle to obtain each block image pair;

[0011] Extract and match the feature points of each block image pair respectively to obtain the confidence of the similarity of each block image pair;

[0012] When it is determined according to each confidence level that there is a difference between the reference image and the image to be detected, the pre-trained twin neural network image detection model is used to perform difference detection on the reference image and the image to be detected to obtain the difference area of ​​the video image.

[0013] In a specific implementation of the present application, the reference image and the image to be detected are divided into blocks according to a preset image block principle to obtain each block image pair, including:

[0014] Divide the reference image into blocks according to a preset filter window size to obtain a reference block image set;

[0015] Numbering each reference block image according to the target arrangement order of each reference block image in the reference image in the reference block image set to obtain each number value;

[0016] Divide the image to be detected into blocks according to the preset filter window size to obtain a set of block images to be detected;

[0017] Using each number value to number each of the to-be-detected block images in the to-be-detected block image set according to the target arrangement order;

[0018] The reference block images and the block images to be detected with the same number value are grouped into image pairs to obtain each block image pair.

[0019] In a specific implementation of the present application, feature point extraction and matching are performed on each block image pair respectively, including:

[0020] Using a fast corner point detection algorithm to perform feature contour detection on each block image pair, the feature contour of each block image pair is obtained;

[0021] The binary relative intensity keypoint descriptor is used to match the feature points of the feature contours of each block image pair.

[0022] In a specific implementation of the present application, obtaining a reference image and an image to be detected includes:

[0023] Performing frame extraction operation on the pre-captured video image according to a preset frame extraction frequency;

[0024] The extracted previous frame image is determined as the reference image, and the extracted next frame image is determined as the image to be detected.

[0025] In a specific implementation of the present application, a pre-trained twin neural network image detection model is used to perform difference detection on the reference image and the image to be detected to obtain a video image difference area, including:

[0026] Using the twin neural network feature encoder in the twin neural network image detection model to extract features through downsampling feature layers of each preset scale, respectively, to obtain twin network feature values ​​of each scale;

[0027] The hybrid attention mask block in the twin neural network image detection model is used to fuse the twin network eigenvalues ​​of each scale to obtain a fused eigenvalue matrix;

[0028] Using the feature vector classifier in the twin neural network image detection model to classify each fused eigenvalue in the fused eigenvalue matrix to obtain each classification result;

[0029] The video image difference area is determined according to each classification result.

[0030] In a specific implementation of the present application, the twin neural network feature encoder in the twin neural network image detection model is used to extract features through downsampling feature layers of each preset scale, including:

[0031] The twin neural network feature encoder trained with the Yolov5s-based backbone network structure performs feature extraction through downsampling feature layers of each preset scale.

[0032] In a specific implementation of the present application, a difference detection process between the reference image and the image to be detected is further included according to each confidence level, and the difference detection process between the reference image and the image to be detected according to each confidence level includes:

[0033] Get the preset confidence threshold;

[0034] Compare each confidence level with the confidence level threshold, and obtain the number of confidence levels that are less than the confidence level threshold;

[0035] When it is determined that the number of confidence levels that are less than the confidence threshold exceeds a preset value, it is determined that there is a difference between the reference image and the image to be detected.

[0036] A video image difference detection device, comprising:

[0037] An image acquisition module, used for acquiring a reference image and an image to be detected;

[0038] A block image pair obtaining module, used to divide the reference image and the image to be detected into blocks according to a preset image block principle to obtain each block image pair;

[0039] The confidence obtaining module is used to extract and match the feature points of each block image pair respectively to obtain the confidence of the similarity of each block image pair;

[0040] The video image difference area acquisition module is used to use the pre-trained twin neural network image detection model to perform difference detection on the reference image and the image to be detected when it is determined according to each confidence level that there is a difference between the reference image and the image to be detected, so as to obtain the video image difference area.

[0041] A video image difference detection device, comprising:

[0042] Memory for storing computer programs;

[0043] A processor is used to implement the steps of the video image difference detection method as described above when executing the computer program.

[0044] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the video image difference detection method described above are implemented.

[0045] The video image difference detection method provided in the present application obtains a reference image and an image to be detected; the reference image and the image to be detected are respectively divided into blocks according to a preset image segmentation principle to obtain pairs of divided images; feature points are extracted and matched for each pair of divided images to obtain confidence levels for the similarities of each pair of divided images; when it is determined according to each confidence level that there is a difference between the reference image and the image to be detected, a pre-trained twin neural network image detection model is used to perform difference detection on the reference image and the image to be detected to obtain a video image difference area.

[0046] It can be seen from the above technical solution that by dividing the reference image and the image to be detected into blocks, and extracting and matching the feature points of each pair of block images respectively, the confidence of the similarity of each pair of block images is obtained, and the reference image and the image to be detected are detected according to each confidence, and the reference image and the image to be detected with differences are screened out, and the images without differences are automatically filtered, thereby greatly reducing the amount of image data for the detection of the difference area, reducing the performance requirements for the deep learning neural network, reducing the complexity of the neural network model structure, and realizing the lightweight processing of the deep learning neural network. The pre-trained twin neural network image detection model is used to perform difference detection on the reference image and the image to be detected to obtain the difference area of ​​the video image, and the feature extraction and feature comparison capabilities of the twin neural network are fully utilized. By combining the advantages of the high efficiency of computer image processing and the accuracy of deep learning, the difference area detection of video images is realized in real time, and the accuracy of image difference detection is improved. The lightweight twin neural network image detection model can be deployed to edge devices, and the model is guaranteed to have good performance.

[0047] Correspondingly, the present application also provides a video image difference detection device, equipment and computer-readable storage medium corresponding to the above-mentioned video image difference detection method, which have the above-mentioned technical effects and are not repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] FIG1 is a flowchart of an implementation method of a video image difference detection method in an embodiment of the present application;

[0050] FIG2 is another implementation flow chart of the video image difference detection method in the embodiment of the present application;

[0051] FIG3 is an architecture diagram of a twin neural network image detection model in an embodiment of the present application;

[0052] FIG4 is an architecture diagram of a backbone network structure feature layer in an embodiment of the present application;

[0053] FIG5 is a structural block diagram of a video image difference detection device according to an embodiment of the present application;

[0054] FIG6 is a structural block diagram of a video image difference detection device in an embodiment of the present application;

[0055] FIG. 7 is a schematic diagram of the specific structure of a video image difference detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0057] Referring to FIG. 1 , FIG. 1 is a flowchart of an implementation method of a video image difference detection method in an embodiment of the present application. The method may include the following steps:

[0058] S101: Acquire a reference image and an image to be detected.

[0059] When video image difference detection is required, a reference image and an image to be detected are obtained. The reference image is an image used as a reference standard for difference detection, and the image to be detected is an image whose difference with the reference image needs to be detected.

[0060] S102: Divide the reference image and the image to be detected into blocks according to a preset image dividing principle to obtain block image pairs.

[0061] The image segmentation principle is preset. After the reference image and the image to be detected are acquired, the reference image and the image to be detected are segmented according to the preset image segmentation principle to obtain each segmented image pair. The two segmented images in each segmented image pair have the same position in the reference image and the image to be detected to which they belong.

[0062] S103: Extracting and matching feature points of each pair of block images respectively to obtain the confidence level of the similarity of each pair of block images.

[0063] After the blocks are divided to obtain each block image pair, feature points are extracted from each block image pair, and feature point matching is performed to obtain the confidence level of the similarity of each block image pair.

[0064] S104: When it is determined according to each confidence level that there is a difference between the reference image and the image to be detected, the pre-trained twin neural network image detection model is used to perform difference detection on the reference image and the image to be detected to obtain the difference area of ​​the video image.

[0065] Pre-train the twin neural network image detection model, and use the same network structure in the twin neural network image detection model to extract the features of the reference image and the image to be detected. After obtaining the confidence of the similarity of each block image pair, judge whether there is a difference between the reference image and the image to be detected according to each confidence. When it is determined that there is a difference between the reference image and the image to be detected according to each confidence, use the pre-trained twin neural network image detection model to perform difference detection on the reference image and the image to be detected to obtain the difference area of ​​the video image. By performing difference detection on the reference image and the image to be detected according to each confidence, the reference image and the image to be detected with differences are screened out, and the images without differences are automatically filtered, thereby greatly reducing the amount of image data for the detection of the difference area, reducing the performance requirements for the deep learning neural network, reducing the complexity of the neural network model structure, and realizing the lightweight processing of the deep learning neural network, so that the lightweight twin neural network image detection model can be deployed to edge devices, such as edge device terminals of the RK3568 chip, and ensuring that the model has good performance.

[0066] It can be seen from the above technical scheme that by dividing the reference image and the image to be detected into blocks, and extracting and matching the feature points of each pair of block images respectively, the confidence of the similarity of each pair of block images is obtained, and the reference image and the image to be detected are detected according to each confidence, and the reference image and the image to be detected with differences are screened out, and the images without differences are automatically filtered, thereby greatly reducing the amount of image data for the detection of the difference area, reducing the performance requirements for the deep learning neural network, reducing the complexity of the neural network model structure, and realizing the lightweight processing of the deep learning neural network. The pre-trained twin neural network image detection model is used to perform difference detection on the reference image and the image to be detected to obtain the difference area of ​​the video image, and fully utilizes the feature extraction and feature comparison capabilities of the twin neural network. By combining the advantages of the high efficiency of computer image processing and the accuracy of deep learning, the difference area detection of video images is realized in real time, and the accuracy of image difference detection is improved. The lightweight twin neural network image detection model can be deployed to edge devices, and the model is guaranteed to have good performance.

[0067] It should be noted that, based on the above embodiments, the embodiments of the present application also provide corresponding improved solutions. In the subsequent embodiments, the same steps or corresponding steps as those in the above embodiments can be referenced to each other, and the corresponding beneficial effects can also be referenced to each other, which will not be repeated one by one in the following improved embodiments.

[0068] Referring to FIG. 2 , FIG. 2 is another implementation flow chart of the video image difference detection method in an embodiment of the present application. The method may include the following steps:

[0069] S201: Performing a frame extraction operation on a pre-captured video image according to a preset frame extraction frequency.

[0070] The frame extraction frequency is preset, and the video image of the current environment to be monitored is captured by a video acquisition device, and the frame extraction operation is performed on the pre-captured video image according to the preset frame extraction frequency.

[0071] S202: Determine the extracted previous frame image as the reference image, and determine the extracted next frame image as the image to be detected.

[0072] After the pre-captured video image is subjected to a frame extraction operation according to a preset frame extraction frequency, the first frame of the two extracted frames is determined as a reference image, and the second frame of the extracted frame is determined as an image to be detected.

[0073] S203: Divide the reference image into blocks according to a preset filter window size to obtain a reference block image set.

[0074] The filter window size is pre-set, such as 5*5, 7*7, etc. Generally, the larger the filter window, the lower the processing efficiency, but the higher the accuracy. After determining the reference image, the reference image is patched according to the preset filter window size to obtain a reference patch image set, thereby achieving mean filtering processing on the reference image, and then standardizing and normalizing each reference patch image in the reference patch image set.

[0075] S204: Numbering each reference block image in the reference block image set according to the target arrangement order of each reference block image in the reference image to obtain each number value.

[0076] After obtaining the reference block image set by block division, each reference block image is numbered according to the target arrangement order of each reference block image in the reference image in the reference block image set, for example, each reference block image can be numbered from left to right, or from top to bottom, to obtain each number value.

[0077] S205: Divide the image to be detected into blocks according to a preset filter window size to obtain a set of block images to be detected.

[0078] After the image to be detected is determined, the image to be detected is divided into blocks according to the filter window size used when dividing the reference image into blocks, so as to obtain a set of divided images to be detected.

[0079] S206: Numbering each of the to-be-detected block images in the to-be-detected block image set according to the target arrangement order by using each numbering value.

[0080] After the image to be detected is divided into blocks to obtain a set of block images to be detected, the block images to be detected in the set of block images to be detected are numbered according to the target arrangement order using the numbering values ​​used when numbering the reference block images.

[0081] S207: Grouping the reference block images and the block images to be detected with the same serial number into image pairs to obtain block image pairs.

[0082] After the reference block images and the block images to be detected are numbered respectively, the reference block images and the block images to be detected with the same number value are grouped into image pairs to obtain each block image pair, so that each reference block image and each block image to be detected are aligned.

[0083] S208: Perform feature contour detection on each block image pair using a fast corner point detection algorithm to obtain a feature contour of each block image pair.

[0084] After obtaining each block image pair, the oriented fast feature detection and rotated binary descriptor algorithm (Oriented Fast and Rotated BRIEF, ORB) algorithm is used to compare the pixel difference values ​​of the block image pairs, so as to realize the extraction and matching of feature points. Specifically, the fast corner detection (FAST) algorithm is used to perform feature contour detection on each block image pair to obtain the feature contour of each block image pair, thereby realizing efficient detection of the feature contours of the reference block image and the block image to be detected in each block image pair.

[0085] S209: performing feature point matching on the feature contours of each block image pair using the binary relative intensity key point descriptor to obtain the confidence level of the similarity of each block image pair.

[0086] After the characteristic contours of each block image pair are detected, the binary relative intensity keypoint (BRIEF) descriptor is used to match the characteristic contours of each block image pair to obtain the confidence of the similarity of each block image pair.

[0087] S210: When it is determined according to each confidence level that there is a difference between the reference image and the image to be detected, the twin neural network feature encoder in the twin neural network image detection model is used to extract features through the downsampling feature layers of each preset scale to obtain the twin network feature values ​​of each scale.

[0088] The twin neural network feature encoder in the twin neural network image detection model includes multiple down-sampling feature layers of preset scales, such as 1 / 2, 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original image size. When it is determined that there is a difference between the reference image and the image to be detected according to each confidence level, the twin neural network feature encoder in the twin neural network image detection model is used to extract features through the down-sampling feature layers of each preset scale to obtain the twin network feature values ​​of each scale.

[0089] S211: Use the hybrid attention mask block in the twin neural network image detection model to fuse the twin network eigenvalues ​​of each scale to obtain a fused eigenvalue matrix.

[0090] After extracting the twin network eigenvalues ​​of each scale, the hybrid attention mask block in the twin neural network image detection model is used to fuse (concat) the twin network eigenvalues ​​of each scale to obtain a fused eigenvalue matrix, so that the fused eigenvalue matrix fuses the features of multiple scales.

[0091] S212: Using the feature vector classifier in the twin neural network image detection model, classify each fused eigenvalue in the fused eigenvalue matrix to obtain each classification result.

[0092] After obtaining the fused eigenvalue matrix, the eigenvector classifier in the twin neural network image detection model is used to classify the fused eigenvalues ​​in the fused eigenvalue matrix to obtain the classification results.

[0093] The fused eigenvalue matrix of two block images in the block image pair is calculated through the twin neural network image detection model. The fused eigenvalue matrix is ​​one-fourth the size of the input original image. Each eigenvalue represents the difference value of four pixels in the original image. The difference value is a probability value between 0 and 1. The larger the value, the more likely there is a difference. The value close to 0 is classified as 0, and the value close to 1 is classified as 1. During the model training process, the training sample annotation information is the area where there are differences between the video image sample pairs. It is usually represented by a mask. The difference area is represented by 1, and the no-difference area is represented by 0, so as to obtain the classification results.

[0094] S213: Determine the video image difference area according to each classification result.

[0095] After obtaining each classification result, the video image difference area is determined according to each classification result.

[0096] In a specific implementation of the present application, using the twin neural network feature encoder in the twin neural network image detection model to extract features through downsampling feature layers of each preset scale may include the following steps:

[0097] The twin neural network feature encoder trained with the Yolov5s-based backbone network structure performs feature extraction through downsampling feature layers of each preset scale.

[0098] The backbone network (Backbone) structure of Yolov5s can be used to train the twin neural network feature encoder in the twin neural network image detection model, and the twin neural network feature encoder obtained by training based on the backbone network structure of Yolov5s is used to extract features through the downsampling feature layers of each preset scale. The advantages of Yolov5s such as small number of parameters, small amount of calculation, less memory usage, and faster reasoning speed are fully utilized.

[0099] In a specific implementation of the present application, the method may further include a difference detection process between the reference image and the image to be detected according to each confidence level, and the difference detection process between the reference image and the image to be detected according to each confidence level may include the following steps:

[0100] Step 1: Obtain a preset confidence threshold;

[0101] Step 2: Compare each confidence level with the confidence threshold to obtain the number of confidence levels that are less than the confidence threshold;

[0102] Step 3: When it is determined that the number of confidence levels less than the confidence threshold exceeds a preset value, it is determined that there is a difference between the reference image and the image to be detected.

[0103] For the convenience of description, the above three steps can be combined for explanation.

[0104] A confidence threshold is set in advance, a preset confidence threshold is obtained, each confidence is compared with the confidence threshold, and the number of confidences less than the confidence threshold is obtained. When it is determined that the number of confidences less than the confidence threshold exceeds the preset value, it is determined that there is a difference between the reference image and the image to be detected. For example, when the filter window size is set to 5*5, the confidence threshold is set to 0.8. When the number of confidences less than the confidence threshold exceeds 6, it indicates that there is no difference between the two images and the comparison is completed. Otherwise, it is determined that there is a difference between the reference image and the image to be detected. By setting the confidence threshold in advance, after obtaining the confidence of the similarity of each block image pair, by comparing the confidence of the similarity of each block image pair with the confidence threshold, it is determined whether there is a difference between the reference image and the image to be detected according to the comparison result obtained by statistics, thereby realizing fast and accurate difference detection between the reference image and the image to be detected.

[0105] In a specific example application of the present application, a method for constructing and training a lightweight twin neural network image detection model includes the following steps:

[0106] Step 1: Prepare a training sample set. Extract 2 frames of images per second from the video captured by the cloud camera. Use Segment Anything 2 (SAM2) to annotate the difference areas of the video images. The previous frame and the next frame are a pair of image samples. The previous frame is used as the reference image, and the next frame is used as the image of the difference area to be detected. The annotation information indicates the difference area between the reference image and the sample image to be tested.

[0107] Step 2: Build a lightweight twin neural network image detection model.

[0108] See Figure 3, which is an architecture diagram of a twin neural network image detection model in an embodiment of the present application. The model network is mainly divided into three parts: a twin neural network feature encoder, a hybrid attention mask block, and a feature vector classifier.

[0109] Part 1: Twin neural network feature encoder, using the backbone network structure of Yolov5s. Because Yolov5s has fewer parameters, less calculation, less memory usage, and faster inference speed, the twin neural network feature encoder uses the same network structure to extract features of the reference image and the image to be tested, respectively, corresponding to five downsampling feature layers of 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image size, and adds the features extracted by the twin neural network from the 1 / 4, 1 / 8, 1 / 16, and 1 / 32 layers of the twin network feature values.

[0110] See Figure 4, which is an architecture diagram of a backbone network structure feature layer in an embodiment of the present application. Figure 4 shows the network feature details of the twin neural network feature encoder, where CBS is a combination of convolution (Conv), batch normalization (BatchNorm) and activation function SiLU, and CSP1_X is a residual connection with CBS as the basic component, which can better extract the feature value of the image.

[0111] The second part: the hybrid attention mask block aims to fuse the features extracted from the twin neural network. It can be understood as a pixel-level attention mechanism. It is mainly divided into two modules, one is the hybrid module and the other is the pixel-level mask generation module. The hybrid attention mask block can achieve multi-scale fusion of different feature layers. The hybrid module uses group convolution. The size of the group is the size of the output feature dimension. The convolution kernel is 3x3. Group convolution can fuse regional features. It can be expressed as:

[0112] MixC=PReLU[BatchNorm[Conv(input=cin, output=cout, kernel=(3, 3), groups=cin)]];

[0113] Among them, cin is the input feature dimension and cout is the output feature dimension.

[0114] The hybrid module is followed by a pixel-level mask generation module, which uses a 1x1 convolution operation and an activation function. The cout output is transformed into a 1-dimensional feature with a feature size of 512×512. A hybrid attention mask operation is performed on each layer of the backbone output feature to obtain the fused feature value of each layer.

[0115] Part 3: Upsampling encoder feature fusion block and feature vector classifier, upsampling each fused output feature layer by 8 times, 4 times and 2 times respectively, and finally fused into 1x128x128 size. Finally, the binary cross entropy loss (BCE) function is used to calculate the loss. The loss function is calculated as follows:

[0116] ;

[0117] in, represents the confidence of the i-th feature point, represents the target value 0 or 1, the threshold is set to 0.5, and N represents the number of pixels in each matrix.

[0118] As shown in Figure 4, the neural network architecture adopts the FPN (Feature Pyramid Network) structure. The downsampling layer uses a twin neural network. Two contrasting images are input and downsampled to 1 / 2, 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original image. The dimensions of the input and output feature layers of each downsampling layer are (3, 48), (48, 96), (96, 192), (192, 384), (384, 512). After the features of the two images are fused, each layer is upsampled by 2 times, and feature fusion is performed. The pixel values ​​are added to obtain feature values ​​of 1 / 32, 1 / 16, 1 / 8 and 1 / 4 times the size of the original image. Finally, these feature values ​​are upsampled by 8 times, 4 times, and 2 times, respectively, and finally fused into a feature layer of 1 / 4 the size of the original image. Finally, the binary cross entropy function is used to calculate the loss.

[0119] The backbone of the twin neural network uses the feature extractor of yolov5, takes 1 / 4, 1 / 8, 1 / 16 and 1 / 32 downsampling layers, and adds the feature vectors of the twin neural network to obtain the eigenvalues.

[0120] The algorithm of this application is deployed on the edge device of the RK3568 chip. The operating system uses Debain, which is commonly used on edge devices such as industrial detection, IoT gateways, industrial control tablets, cloud terminals, etc. The device is configured with a 4-core central processing unit (CPU), 16G memory, and a maximum main frequency of 2.0GHz. It supports the full video format H.264 decoder 4K@60fps and H.265 decoder 4K@60fps. The traditional deep learning model has relatively large parameters and calculation amount and cannot be deployed on edge devices. This application reduces the size of the model, the number of inference floating-point operations is 2FLOPS, and the ORB algorithm is used to compare the similarity of block image pairs, which greatly improves the inference speed.

[0121] Corresponding to the above method embodiment, the present application further provides a video image difference detection device. The video image difference detection device described below and the video image difference detection method described above can refer to each other.

[0122] Referring to FIG. 5 , FIG. 5 is a structural block diagram of a video image difference detection device in an embodiment of the present application, and the device may include:

[0123] An image acquisition module 51 is used to acquire a reference image and an image to be detected;

[0124] A block image pair obtaining module 52 is used to divide the reference image and the image to be detected into blocks according to a preset image block principle to obtain each block image pair;

[0125] The confidence obtaining module 53 is used to extract and match the feature points of each block image pair respectively to obtain the confidence of the similarity of each block image pair;

[0126] The video image difference area acquisition module 54 is used to perform difference detection on the reference image and the image to be detected using the pre-trained twin neural network image detection model when it is determined according to each confidence level that there is a difference between the reference image and the image to be detected, so as to obtain the video image difference area.

[0127] It can be seen from the above technical scheme that by dividing the reference image and the image to be detected into blocks, and extracting and matching the feature points of each pair of block images respectively, the confidence of the similarity of each pair of block images is obtained, and the reference image and the image to be detected are detected according to each confidence, and the reference image and the image to be detected with differences are screened out, and the images without differences are automatically filtered, thereby greatly reducing the amount of image data for the detection of the difference area, reducing the performance requirements for the deep learning neural network, reducing the complexity of the neural network model structure, and realizing the lightweight processing of the deep learning neural network. The pre-trained twin neural network image detection model is used to perform difference detection on the reference image and the image to be detected to obtain the difference area of ​​the video image, and fully utilizes the feature extraction and feature comparison capabilities of the twin neural network. By combining the advantages of the high efficiency of computer image processing and the accuracy of deep learning, the difference area detection of video images is realized in real time, and the accuracy of image difference detection is improved. The lightweight twin neural network image detection model can be deployed to edge devices, and the model is guaranteed to have good performance.

[0128] In a specific implementation of the present application, the block image pair obtaining module 52 includes:

[0129] The reference block image set acquisition submodule is used to divide the reference image into blocks according to a preset filter window size to obtain a reference block image set;

[0130] A first image numbering submodule is used to number each reference block image according to the target arrangement order of each reference block image in the reference image in the reference block image set to obtain each number value;

[0131] The submodule for obtaining a set of block images to be detected is used to divide the image to be detected into blocks according to a preset filter window size to obtain a set of block images to be detected;

[0132] The second image numbering submodule is used to number each of the to-be-detected block images in the to-be-detected block image set according to the target arrangement order by using each numbering value;

[0133] The block image pair obtaining submodule is used to form image pairs with reference block images and block images to be detected having the same serial number value, so as to obtain each block image pair.

[0134] In a specific implementation of the present application, the confidence obtaining module 53 includes:

[0135] The feature contour acquisition submodule is used to perform feature contour detection on each block image pair using a fast corner point detection algorithm to obtain the feature contour of each block image pair;

[0136] The feature point matching submodule is used to perform feature point matching on the feature contours of each block image pair using the binary relative intensity key point descriptor.

[0137] In a specific implementation of the present application, the image acquisition module 51 includes:

[0138] A frame extraction submodule, used for performing a frame extraction operation on the pre-captured video image according to a preset frame extraction frequency;

[0139] The submodule for determining the image to be detected is used to determine the extracted previous frame image as the reference image, and to determine the extracted next frame image as the image to be detected.

[0140] In a specific implementation of the present application, the video image difference area obtaining module 54 includes:

[0141] The twin network feature value acquisition submodule is used to use the twin neural network feature encoder in the twin neural network image detection model to extract features through the downsampling feature layers of each preset scale to obtain the twin network feature values ​​of each scale;

[0142] The fused eigenvalue matrix acquisition submodule is used to classify each fused eigenvalue in the fused eigenvalue matrix using the eigenvector classifier in the twin neural network image detection model to obtain each classification result;

[0143] The video image difference region determination submodule is used to determine the video image difference region according to each classification result.

[0144] In a specific embodiment of the present application, the twin network feature value acquisition submodule is specifically a module that uses the twin neural network feature encoder obtained by training based on the backbone network structure of Yolov5s to perform feature extraction through downsampling feature layers of each preset scale.

[0145] In a specific implementation of the present application, the device may further include an image difference detection module, which may include:

[0146] A confidence threshold acquisition submodule is used to obtain a preset confidence threshold;

[0147] A confidence number obtaining submodule is used to compare each confidence with a confidence threshold, and obtain the number of confidences that are less than the confidence threshold;

[0148] The image difference determination submodule is used to determine that there is a difference between the reference image and the image to be detected when the number of confidence levels that are determined to be less than the confidence threshold exceeds a preset value.

[0149] Corresponding to the above method embodiment, referring to FIG. 6 , FIG. 6 is a schematic diagram of a video image difference detection device provided by the present application, and the device may include:

[0150] A memory 332, for storing computer programs;

[0151] The processor 322 is configured to implement the steps of the video image difference detection method of the above method embodiment when executing a computer program.

[0152] Specifically, please refer to FIG. 7, which is a schematic diagram of the specific structure of a video image difference detection device provided in this embodiment. The video image difference detection device may have relatively large differences due to different configurations or performances, and may include a processor (central processing units, CPU) 322 (for example, one or more processors) and a memory 332, and the memory 332 stores one or more computer programs 342 or data 344. Among them, the memory 332 can be a temporary storage or a permanent storage. The program stored in the memory 332 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the data processing device. Furthermore, the processor 322 can be configured to communicate with the memory 332, and execute a series of instruction operations in the memory 332 on the video image difference detection device 301.

[0153] The video image difference detection device 301 may further include one or more power supplies 326 , one or more wired or wireless network interfaces 350 , one or more input and output interfaces 358 , and / or one or more operating systems 341 .

[0154] The steps in the video image difference detection method described above can be implemented by the structure of a video image difference detection device.

[0155] Corresponding to the above method embodiment, the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the following steps can be implemented:

[0156] Acquire a reference image and an image to be detected; divide the reference image and the image to be detected into blocks according to a preset image segmentation principle to obtain pairs of block images; extract and match feature points of each pair of block images to obtain the confidence level of the similarity of each pair of block images; when it is determined that there is a difference between the reference image and the image to be detected according to each confidence level, use the pre-trained twin neural network image detection model to perform difference detection on the reference image and the image to be detected to obtain the difference area of ​​the video image.

[0157] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0158] For an introduction to the computer-readable storage medium provided in this application, please refer to the above method embodiment, and this application will not go into details here.

[0159] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the devices, equipment and computer-readable storage media disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0160] Specific examples are used herein to illustrate the principles and implementation methods of the present application, and the description of the above embodiments is only used to help understand the technical solution and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the present application.

Claims

1. A video image difference detection method, characterized in that: include: Acquire a reference image and an image to be detected; Divide the reference image and the image to be detected into blocks according to a preset image dividing principle to obtain each block image pair; Extract and match the feature points of each block image pair respectively to obtain the confidence of the similarity of each block image pair; When it is determined according to each confidence level that there is a difference between the reference image and the image to be detected, the pre-trained twin neural network image detection model is used to perform difference detection on the reference image and the image to be detected to obtain the difference area of ​​the video image.

2. The video image difference detection method according to claim 1, characterized in that: The reference image and the image to be detected are divided into blocks according to a preset image division principle to obtain each pair of divided images, including: Divide the reference image into blocks according to a preset filter window size to obtain a reference block image set; Numbering each reference block image according to the target arrangement order of each reference block image in the reference image in the reference block image set to obtain each number value; Divide the image to be detected into blocks according to the preset filter window size to obtain a set of block images to be detected; Using each number value to number each of the to-be-detected block images in the to-be-detected block image set according to the target arrangement order; The reference block images and the block images to be detected with the same number value are grouped into image pairs to obtain each block image pair.

3. The video image difference detection method according to claim 1, characterized in that: Feature points are extracted and matched for each block image pair, including: Using a fast corner point detection algorithm to perform feature contour detection on each block image pair, the feature contour of each block image pair is obtained; The binary relative intensity keypoint descriptor is used to match the feature points of the feature contours of each block image pair.

4. The video image difference detection method according to claim 1, characterized in that: Obtain the reference image and the image to be detected, including: Performing frame extraction operation on the pre-captured video image according to a preset frame extraction frequency; The extracted previous frame image is determined as the reference image, and the extracted next frame image is determined as the image to be detected.

5. The video image difference detection method according to any one of claims 1 to 4, characterized in that: The pre-trained twin neural network image detection model is used to perform difference detection on the reference image and the image to be detected to obtain the difference area of ​​the video image, including: Using the twin neural network feature encoder in the twin neural network image detection model to extract features through downsampling feature layers of each preset scale, respectively, to obtain twin network feature values ​​of each scale; The hybrid attention mask block in the twin neural network image detection model is used to fuse the twin network eigenvalues ​​of each scale to obtain a fused eigenvalue matrix; Using the feature vector classifier in the twin neural network image detection model to classify each fused eigenvalue in the fused eigenvalue matrix to obtain each classification result; The video image difference area is determined according to each classification result.

6. The video image difference detection method according to claim 5, characterized in that: The twin neural network feature encoder in the twin neural network image detection model is used to extract features through downsampling feature layers of each preset scale, including: The twin neural network feature encoder trained with the Yolov5s-based backbone network structure performs feature extraction through downsampling feature layers of each preset scale.

7. The video image difference detection method according to claim 1, characterized in that: The method further includes a process of detecting the difference between the reference image and the image to be detected according to each confidence level, wherein the process of detecting the difference between the reference image and the image to be detected according to each confidence level includes: Get the preset confidence threshold; Compare each confidence level with the confidence level threshold, and obtain the number of confidence levels that are less than the confidence level threshold; When it is determined that the number of confidence levels that are less than the confidence threshold exceeds a preset value, it is determined that there is a difference between the reference image and the image to be detected.

8. A video image difference detection device, characterized in that: include: An image acquisition module, used for acquiring a reference image and an image to be detected; A block image pair obtaining module, used to divide the reference image and the image to be detected into blocks according to a preset image block principle to obtain each block image pair; The confidence obtaining module is used to extract and match the feature points of each block image pair respectively to obtain the confidence of the similarity of each block image pair; The video image difference area acquisition module is used to use the pre-trained twin neural network image detection model to perform difference detection on the reference image and the image to be detected when it is determined according to each confidence level that there is a difference between the reference image and the image to be detected, so as to obtain the video image difference area.

9. A video image difference detection device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the video image difference detection method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the video image difference detection method according to any one of claims 1 to 7 are implemented.

Images