Moving target detection method based on multiple prompt messages

By fusing multiple prompt information in motion target detection, including grayscale processing and contextual difference information, combining time dimension and semantic information, using convolutional neural network to detect moving targets in video streams, motion information capture in the prior art is solved The problems of insufficient and high computational complexity achieve efficient and real-time motion object detection.

CN118967741APending Publication Date: 2024-11-15HANGZHOU CHUANGLIAN ELECTRONICS TECH
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411029859.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

When processing video streams, existing motion object detection algorithms cannot effectively capture motion information in the image, especially in complex and changeable scenarios, and have high computational complexity or poor real-time performance.

Method used

A motion object detection method based on multi-prompt information is proposed. By acquiring multi-frame images, grayscale processing and context differential calculations are performed, time dimensions and semantic information are fused, and motion object detection is performed using a convolutional neural network.

Benefits of technology

It realizes accurate detection of moving targets in the video stream, improves the robustness and real-time detection, is suitable for edge device deployment, simplifies the network structure, and reduces the amount of computing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118967741A_ABST
    Figure CN118967741A_ABST
Patent Text Reader

Abstract

The invention provides a moving target detection method based on multi-prompt information. The method comprises the following steps: acquiring N frames of images from monitoring equipment; performing graying processing on the current image frame to obtain an image containing semantic information; calculating the N frames of images through a CNT algorithm to obtain an image containing context difference information; calculating an average frame of nearest neighbor frames of the N frames of images, and performing graying processing on the average frame to obtain an image containing time dimension motion information; splicing and fusing the image containing the semantic information, the image containing the context difference information and the image containing the time dimension motion information to obtain a multi-prompt information image; a pre-trained convolutional neural network model extracts a moving target segmentation mask graph from the multi-prompt information image; according to the method, the moving target in the video stream is well extracted, and the method can be widely applied to the field of video monitoring, in particular to detection of intrusion of unspecific foreign matters on edge equipment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of moving target detection, and in particular to a moving target detection method based on multiple prompt information. Background Art

[0002] Detecting moving targets in video streams is one of the key basic research contents in the field of computer vision. It is widely used in daily production and life, such as intelligent video surveillance, intelligent transportation systems, and autonomous driving. Detecting moving targets of interest in surveillance videos can remove a large amount of redundant background information and accurately extract valuable information from the video, which is conducive to more efficient processing and analysis of subsequent advanced tasks, such as human and vehicle recognition, behavior understanding, target tracking, and abnormal alarms. Traditional moving target detection methods include optical flow, frame difference, and background subtraction based on background modeling. The optical flow method has high computational complexity and large amount of calculation, and cannot meet the real-time requirements; the frame difference method is simple and fast, but the determination of the pixel difference of consecutive frames is prone to missed detection and incomplete targets. Background subtraction methods based on background modeling, such as the Gaussian mixture model GMM (Gaussian Mixture Model), can dynamically complete the background update and detect foreground moving targets in real time and accurately, but the modeling complexity is high and cannot adapt to complex and changing application scenarios. With the rapid development of deep learning, the application of convolutional neural networks to moving target detection algorithms has achieved good results. The motion target detection algorithm based on deep learning can well extract the feature information of video image frames and detect the moving targets based on the extracted semantic information. However, most of the current motion target detection algorithms based on deep learning predict single image frames, ignoring the rich contextual time dimension information, and therefore cannot well capture the motion information in the image (such as FgSegNet), and lack generalization for complex and changing scenes. Summary of the invention

[0003] The present invention solves the shortcomings and deficiencies of motion target detection algorithms based on traditional detection algorithms and deep learning, and proposes a motion target detection method based on multi-prompt information. It utilizes multiple information of motion targets to well extract motion targets in video streams. The method can be widely used in the field of video surveillance, especially for detecting intrusion of unspecific foreign objects on edge devices.

[0004] In order to achieve the above objectives, the following technical solutions are proposed: A moving target detection method based on multiple prompt information comprises the following steps: S1, obtain N frames of images from the monitoring device; S2, grayscale the current image frame to obtain an image containing semantic information; S3, N frames of images are calculated by CNT algorithm to obtain images containing context difference information; S4, calculating the average frame of the nearest neighbor frames of the N frames of the image, graying the average frame, and obtaining an image containing motion information in the time dimension; S5, splicing and fusing the image containing semantic information, the image containing context difference information, and the image containing time dimension motion information to obtain a multi-hint information image; S6, the pre-trained convolutional neural network model extracts the moving object segmentation mask map from the multi-cue information image.

[0005] The present invention utilizes a variety of information of moving targets, including differential information, timing information, and context information, and can well extract moving targets in video streams with high accuracy, providing a basis for more efficient processing and analysis of subsequent advanced tasks; the network structure used in the present invention is simple, and the general operator structure is convenient for edge deployment and rapid reasoning. This method integrates motion information in the time dimension, differential information of the context, and semantic information of the image frame, and uses a convolutional neural network to detect moving targets in video streams. Based on the fusion of multi-prompt information, moving targets can be well extracted with good robustness and real-time performance.

[0006] Preferably, the sequence length of the image frames acquired in S1 is always maintained at N frames, and the image frames are 24-bit 3-channel RGB images.

[0007] The image frames acquired by S1 from the monitoring device are all 3-channel, 24-bit RGB images, and the sequence length needs to be maintained at N frames. The selection of the N value can be adjusted according to the actual application scenario. The factors generally considered include the movement speed of the moving target, the computing power of the edge device, etc. Considering that the speed of the moving target is relatively fast, sufficient time intervals need to be considered when acquiring the motion information in the time dimension, and N should be greater than 40. When the computing power of the edge device is relatively weak, N frames can be acquired at intervals, and the number of calculated average frames can be reduced accordingly. As the number of acquired image frames increases, the N frames are updated and replaced. As a new frame is added to the sequence, the first frame is discarded, and the sequence length is always maintained, that is, the N frames are iteratively updated according to the first-in-first-out principle.

[0008] Preferably, the grayscale processing in S2 and S4 is the same, and weights are assigned to the three color channels of the color RGB image respectively.

[0009] The grayscale processing of the image frame in S2 and S4 of the present invention is the same. Since the sensitivity of the human eye to the three colors of red, green and blue is different, the weight assigned to each color during grayscale conversion is also different. The conversion formula is: Gray=0.299*Red+0.587*Green+0.114*Blue Among them, Red, Green, and Blue are the three color channels of the color RGB image.

[0010] Preferably, the CNT algorithm obtains an image containing context difference information by calling an opencv library API.

[0011] The CNT algorithm in S3 of the present invention is a method that does not require Gaussian modeling of background points. It can very quickly extract context differential information using only the information of pixel point values ​​in the past N consecutive frames. The CNT algorithm of the present invention actually obtains the bgsub frame containing context differential information by calling the opencv library API, namely bgsegm.createBackgroundSubtractorCNT.

[0012] Preferably, the nearest neighbor frame is the neighboring historical image frame before the current frame, and the process of obtaining the average frame is as follows: calculate the pixel average of the nearest neighbor frames between the i-th frame before the current frame of each channel and the (i+n)-th frame before the current frame, and form an average frame by the pixel averages.

[0013] The nearest neighbor frame in S4 refers to the neighboring historical image frame before the current frame. Figure 2 As shown, the average frame means that according to the timing of the current frame, the nearest neighbor frames between the 20th frame before the current frame and the 40th frame before the current frame are calculated, a total of 20 image frames, and the pixel average value is calculated according to the RGB channels. The calculation formula for the Red channel is as follows: The calculation method for the Green and Blue channels is the same. Considering the computing power of the edge device, when the N frames of images are selected at intervals, the selection of the nearest neighbor frame can be halved to 10 or less. After obtaining the nearest neighbor average frame, grayscale processing is performed to obtain the recent frame containing the motion information of the time dimension.

[0014] Preferably, the stitching and fusion process is as follows: stitching the image containing semantic information, the image containing contextual difference information and the image containing time dimension motion information according to the RGB merging mode to obtain a 24-bit deep 3-channel multi-prompt information image frame.

[0015] The three single-channel images of the current frame, the nearest neighbor average frame, and the difference bgsub frame obtained in the previous step of S5 of the present invention are spliced ​​into a 3-channel, 24-bit deep multi-prompt information image frame according to the RGB merging mode.

[0016] Preferably, the convolutional neural network is a UNet type full convolutional neural network, including an encoder, a residual skip connection and a decoder. The convolutional neural network converts the multi-prompt information image into a mask map. After the mask map passes through the sigmoid layer, a probability distribution map of each pixel being the foreground is obtained. The probability distribution map is threshold judged, and the judgment basis is as follows: Where: mask(x) is the pixel value, x is the probability value of the pixel being the foreground, and τ is the threshold.

[0017] The input size of the convolutional neural network in S6 is the common size of surveillance cameras, 1080*1920*3. In order to meet the requirement of 32 times downsampling of the convolutional neural network, the surveillance image size is adjusted to 1088*1920*3. The convolutional neural network is a UNet type full convolutional neural network. Figure 3 As shown in the figure, the network structure consists of three parts: encoder, decoder and residual jump connection. The multi-prompt information frame is pre-trained and then convolutional neural network is used to obtain the mask map. The mask map size is H*W*1 and then passes through the sigmoid layer to obtain the probability distribution map of each pixel being the foreground, with a size of H*W*1. Combined with the set threshold τ, it is judged whether each pixel belongs to the foreground or the background. The judgment basis is as follows: Where mask(x) is the pixel value, x is the probability value of the pixel being the foreground, if the probability value is less than the threshold τ, then the pixel value is the background and is set to 0; if the probability value is greater than or equal to the threshold τ, then the pixel value is the moving target foreground and is set to 255. The extracted moving target mask is as follows: Figure 4 shown.

[0018] Preferably, the encoder uses ResNet-18 to extract image features, downsamples the image size by a factor of 32, and increases the number of channels from 3 to 512.

[0019] The encoder part of the present invention uses ResNet-18 to extract image features. It contains 5 blocks in total. Each block contains a basic convolution layer, a batch normalization layer, and an activation layer. After each block, the image size is halved, and the number of channels is doubled (except the first one). The encoder part downsamples the image size by 32 times and increases the number of channels from 3 to 512.

[0020] Preferably, the residual jump connection is provided with 5 lateral connections for adjusting the number of feature channels.

[0021] The residual jump connection part of the present invention includes 5 lateral connections, each of which includes a basic convolution layer and an activation layer. The lateral connection starting from the image adjusts the number of feature channels, while the other connections do not. When connecting, it is necessary to ensure that the size of the feature map matches.

[0022] Preferably, the decoder splices the feature map from the residual jump connection, and upsamples the image size by 32 times to generate a mask map with 1 channel.

[0023] The decoder part of the present invention contains 11 blocks in total, including basic convolutional layers, upsampling layers, activation layers, and needs to splice the feature maps from the residual jump connection in the channel dimension. The decoder part upsamples the image size by 32 times, restores the original width and height of the image, and the number of channels is 1, and generates a mask map.

[0024] The beneficial effects of the present invention are: (1) Multiple prompt information: The present invention utilizes multiple information of moving targets, including differential information, timing information, and context information, which can well extract moving targets in video streams with high accuracy, providing a basis for more efficient processing and analysis of subsequent advanced tasks; (2) Fast reasoning speed: The network structure used in the present invention is simple, the general operator structure is convenient for edge deployment, and the reasoning is fast. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the complete moving target detection algorithm of the present invention.

[0026] Figure 2 It is a schematic diagram of the nearest neighbor frame selection of the present invention.

[0027] Figure 3 It is a schematic diagram of the Encoder-Decoder network structure of the present invention.

[0028] Figure 4 It is a schematic diagram of the algorithm input and moving target extraction results of the present invention. DETAILED DESCRIPTION

[0029] Embodiment 1: This embodiment proposes a moving target detection method based on multiple prompt information, such as Figure 1 As shown, the following steps are included: S1. Obtain a sequence of N frames of images from the monitoring device; the image frames obtained by S1 from the monitoring device are all 3-channel, 24-bit deep RGB images, and the sequence length needs to be maintained at N frames. The selection of the N value can be adjusted according to the actual application scenario. General factors to be considered include the movement speed of the moving target, the computing power of the edge device, etc. Considering that the speed of the moving target is relatively fast, sufficient time intervals need to be considered when obtaining the motion information in the time dimension, and N should be greater than 40. As the number of acquired image frames increases, the N frames of images are updated and replaced. As a new frame is added to the sequence, the first frame is discarded, and the sequence length is always maintained, that is, the N frames of images are iteratively updated according to the first-in-first-out principle.

[0030] S2. grayscale the current image frame to obtain a current frame containing semantic information; S3. Calculate the bgsub frame containing context difference information based on N frames of images using the CNT algorithm; S4. Calculate the average frame of the nearest neighbor frames based on N frames of images, and obtain the recent frame containing the motion information of the time dimension after grayscale processing; S5. The current frame containing semantic information, the bgsub frame containing contextual difference information, and the recent frame containing motion information in the time dimension are spliced ​​and fused to obtain a multi-prompt information image; The three single-channel images of the current frame, the nearest neighbor average frame, and the difference bgsub frame obtained in the previous step of S5 of the present invention are spliced ​​into a 3-channel, 24-bit deep multi-prompt information image frame according to the RGB merging mode.

[0031] S6. Figure 4 As shown, the spliced ​​image frames use the trained convolutional neural network model to extract the moving targets in the multi-cue information image.

[0032] The input size of the convolutional neural network in S6 is 1080*1920*3, which is a common size for surveillance cameras. To meet the requirement of 32 times downsampling of the convolutional neural network, the surveillance image size is adjusted to 1088*1920*3. The convolutional neural network is a UNet type full convolutional neural network. Figure 3 As shown in the figure, the network structure consists of three parts: encoder, decoder and residual jump connection.

[0033] The encoder part of the present invention uses ResNet-18 to extract image features. It contains 5 blocks in total. Each block contains a basic convolution layer, a batch normalization layer, and an activation layer. After each block, the image size is halved, and the number of channels is doubled (except the first one). The encoder part downsamples the image size by 32 times and increases the number of channels from 3 to 512.

[0034] The residual jump connection part of the present invention includes 5 lateral connections, each of which includes a basic convolution layer and an activation layer. The lateral connection starting from the image adjusts the number of feature channels, while the other connections do not. When connecting, it is necessary to ensure that the size of the feature map matches.

[0035] The decoder part of the present invention contains 11 blocks in total, including basic convolutional layers, upsampling layers, activation layers, and needs to splice the feature maps from the residual jump connection in the channel dimension. The decoder part upsamples the image size by 32 times, restores the original width and height of the image, and the number of channels is 1, and generates a mask map.

[0036] After the multi-prompt information frame is pre-trained and convolutional neural network is used to obtain a mask map. The size of the mask map is H*W*1. After passing through the sigmoid layer, the probability distribution map of each pixel being the foreground is obtained, and the size is H*W*1. Combined with the set threshold τ, it is judged whether each pixel belongs to the foreground or the background. The judgment basis is as follows: Where mask(x) is the pixel value, x is the probability value of the pixel being the foreground, if the probability value is less than the threshold τ, then the pixel value is the background and is set to 0; if the probability value is greater than or equal to the threshold τ, then the pixel value is the moving target foreground and is set to 255. The extracted moving target mask is as follows: Figure 4 shown.

[0037] The present invention utilizes a variety of information of moving targets, including differential information, timing information, and context information, and can well extract moving targets in video streams with high accuracy, providing a basis for more efficient processing and analysis of subsequent advanced tasks; the network structure used in the present invention is simple, and the general operator structure is convenient for edge deployment and rapid reasoning. This method integrates motion information in the time dimension, differential information of the context, and semantic information of the image frame, and uses a convolutional neural network to detect moving targets in video streams. Based on the fusion of multi-prompt information, the moving target can be well extracted, with good robustness and real-time performance. The present invention has high accuracy and low computational complexity, is convenient for edge device deployment, and provides accurate and efficient results for subsequent advanced tasks.

[0038] Embodiment 2: This embodiment improves S2 on the basis of embodiment 1 and proposes a moving target detection method based on multiple prompt information, such as Figure 1 As shown, the following steps are included: S1. Obtain a sequence of N frames of images from the monitoring device; the image frames obtained by S1 from the monitoring device are all 3-channel, 24-bit RGB images, and the sequence length needs to be maintained at N frames. The selection of the N value can be adjusted according to the actual application scenario. The factors generally considered include the movement speed of the moving target, the computing power of the edge device, etc. Considering that the speed of the moving target is fast, it is necessary to consider sufficient time intervals when obtaining the motion information in the time dimension, and N should be greater than 40. When the computing power of the edge device is weak, N frames can be obtained at intervals, and the number of calculated average frames can be reduced accordingly. As the number of acquired image frames increases, the N frames of images are updated and replaced. As a new frame is added to the sequence, the first frame is removed, and the sequence length is always maintained, that is, the N frames of images are iteratively updated according to the first-in-first-out principle.

[0039] S2. grayscale the current image frame to obtain a current frame containing semantic information; The grayscale processing of the image frame in S2 and S4 of the present invention is the same. Since the sensitivity of the human eye to the three colors of red, green and blue is different, the weight assigned to each color during grayscale conversion is also different. The conversion formula is: Gray=0.299*Red+0.587*Greem+0.114*Blue Among them, Red, Green, and Blue are the three color channels of the color RGB image.

[0040] S3. Calculate the bgsub frame containing context difference information based on N frames of images using the CNT algorithm; S4. Calculate the average frame of the nearest neighbor frames based on N frames of images, and obtain the recent frame containing the motion information of the time dimension after grayscale processing; S5. The current frame containing semantic information, the bgsub frame containing contextual difference information, and the recent frame containing motion information in the time dimension are spliced ​​and fused to obtain a multi-prompt information image; The three single-channel images of the current frame, the nearest neighbor average frame, and the difference bgsub frame obtained in the previous step of S5 of the present invention are spliced ​​into a 3-channel, 24-bit deep multi-prompt information image frame according to the RGB merging mode.

[0041] S6. Figure 4As shown, the spliced ​​image frames use the trained convolutional neural network model to extract the moving targets in the multi-cue information image.

[0042] The input size of the convolutional neural network in S6 is 1080*1920*3, which is a common size for surveillance cameras. To meet the requirement of 32 times downsampling of the convolutional neural network, the surveillance image size is adjusted to 1088*1920*3. The convolutional neural network is a UNet type full convolutional neural network. Figure 3 As shown in the figure, the network structure consists of three parts: encoder, decoder and residual jump connection.

[0043] The encoder part of the present invention uses ResNet-18 to extract image features. It contains 5 blocks in total. Each block contains a basic convolution layer, a batch normalization layer, and an activation layer. After each block, the image size is halved, and the number of channels is doubled (except the first one). The encoder part downsamples the image size by 32 times and increases the number of channels from 3 to 512.

[0044] The residual jump connection part of the present invention includes 5 lateral connections, each of which includes a basic convolution layer and an activation layer. The lateral connection starting from the image adjusts the number of feature channels, while the other connections do not. When connecting, it is necessary to ensure that the size of the feature map matches.

[0045] The decoder part of the present invention contains 11 blocks in total, including basic convolutional layers, upsampling layers, activation layers, and needs to splice the feature maps from the residual jump connection in the channel dimension. The decoder part upsamples the image size by 32 times, restores the original width and height of the image, and the number of channels is 1, and generates a mask map.

[0046] After the multi-prompt information frame is pre-trained and convolutional neural network is used to obtain a mask map. The size of the mask map is H*W*1. After passing through the sigmoid layer, the probability distribution map of each pixel being the foreground is obtained, and the size is H*W*1. Combined with the set threshold τ, it is judged whether each pixel belongs to the foreground or the background. The judgment basis is as follows: Where mask(x) is the pixel value, x is the probability value of the pixel being the foreground, if the probability value is less than the threshold τ, then the pixel value is the background and is set to 0; if the probability value is greater than or equal to the threshold τ, then the pixel value is the moving target foreground and is set to 255. The extracted moving target mask is as follows: Figure 4 shown.

[0047] The present invention utilizes a variety of information of moving targets, including differential information, timing information, and context information, and can well extract moving targets in video streams with high accuracy, providing a basis for more efficient processing and analysis of subsequent advanced tasks; the network structure used in the present invention is simple, and the general operator structure is convenient for edge deployment and rapid reasoning. This method integrates motion information in the time dimension, differential information of the context, and semantic information of the image frame, and uses a convolutional neural network to detect moving targets in video streams. Based on the fusion of multi-prompt information, the moving target can be well extracted, with good robustness and real-time performance. The present invention has high accuracy and low computational complexity, is convenient for edge device deployment, and provides accurate and efficient results for subsequent advanced tasks.

[0048] Embodiment 3: This embodiment improves S3 on the basis of embodiment 2, and proposes a moving target detection method based on multiple prompt information, such as Figure 1 As shown, the following steps are included: S1. Obtain a sequence of N frames of images from the monitoring device; the image frames obtained by S1 from the monitoring device are all 3-channel, 24-bit RGB images, and the sequence length needs to be maintained at N frames. The selection of the N value can be adjusted according to the actual application scenario. The factors generally considered include the movement speed of the moving target, the computing power of the edge device, etc. Considering that the speed of the moving target is fast, it is necessary to consider sufficient time intervals when obtaining the motion information in the time dimension, and N should be greater than 40. When the computing power of the edge device is weak, N frames can be obtained at intervals, and the number of calculated average frames can be reduced accordingly. As the number of acquired image frames increases, the N frames of images are updated and replaced. As a new frame is added to the sequence, the first frame is removed, and the sequence length is always maintained, that is, the N frames of images are iteratively updated according to the first-in-first-out principle.

[0049] S2. grayscale the current image frame to obtain a current frame containing semantic information; The grayscale processing of the image frame in S2 and S4 of the present invention is the same. Since the sensitivity of the human eye to the three colors of red, green and blue is different, the weight assigned to each color during grayscale conversion is also different. The conversion formula is: Gray=0.299*Red+0.587*Green+0.114*Blue Among them, Red, Green, and Blue are the three color channels of the color RGB image.

[0050] S3. According to N frames of images, a bgsub frame containing context differential information is calculated using a CNT algorithm; the CNT algorithm in S3 of the present invention is a method that does not require Gaussian modeling of background points, and it can very quickly extract context differential information using only information about pixel point values ​​in the past N consecutive frames. The CNT algorithm of the present invention actually obtains a bgsub frame containing context differential information by calling an opencv library API, namely bgsegm.createBackgroundSubtractorCNT.

[0051] S4. Calculate the average frame of the nearest neighbor frames based on N frames of images, and obtain the recent frame containing the motion information of the time dimension after grayscale processing; S5. The current frame containing semantic information, the bgsub frame containing contextual difference information, and the recent frame containing motion information in the time dimension are spliced ​​and fused to obtain a multi-prompt information image; The three single-channel images of the current frame, the nearest neighbor average frame, and the difference bgsub frame obtained in the previous step of S5 of the present invention are spliced ​​into a 3-channel, 24-bit deep multi-prompt information image frame according to the RGB merging mode.

[0052] S6. Figure 4 As shown, the spliced ​​image frames use the trained convolutional neural network model to extract the moving targets in the multi-cue information image.

[0053] The input size of the convolutional neural network in S6 is 1080*1920*3, which is a common size for surveillance cameras. To meet the requirement of 32 times downsampling of the convolutional neural network, the surveillance image size is adjusted to 1088*1920*3. The convolutional neural network is a UNet type full convolutional neural network. Figure 3 As shown in the figure, the network structure consists of three parts: encoder, decoder and residual jump connection.

[0054] The encoder part of the present invention uses ResNet-18 to extract image features. It contains 5 blocks in total. Each block contains a basic convolution layer, a batch normalization layer, and an activation layer. After each block, the image size is halved, and the number of channels is doubled (except the first one). The encoder part downsamples the image size by 32 times and increases the number of channels from 3 to 512.

[0055] The residual jump connection part of the present invention includes 5 lateral connections, each of which includes a basic convolution layer and an activation layer. The lateral connection starting from the image adjusts the number of feature channels, while the other connections do not. When connecting, it is necessary to ensure that the size of the feature map matches.

[0056] The decoder part of the present invention contains 11 blocks in total, including basic convolutional layers, upsampling layers, activation layers, and needs to splice the feature maps from the residual jump connection in the channel dimension. The decoder part upsamples the image size by 32 times, restores the original width and height of the image, and the number of channels is 1, and generates a mask map.

[0057] After the multi-prompt information frame is pre-trained and convolutional neural network is used to obtain a mask map. The size of the mask map is H*W*1. After passing through the sigmoid layer, the probability distribution map of each pixel being the foreground is obtained, and the size is H*W*1. Combined with the set threshold τ, it is judged whether each pixel belongs to the foreground or the background. The judgment basis is as follows: Where mask(x) is the pixel value, x is the probability value of the pixel being the foreground, if the probability value is less than the threshold τ, then the pixel value is the background and is set to 0; if the probability value is greater than or equal to the threshold τ, then the pixel value is the moving target foreground and is set to 255. The extracted moving target mask is as follows: Figure 4 shown.

[0058] The present invention utilizes a variety of information of moving targets, including differential information, timing information, and context information, and can well extract moving targets in video streams with high accuracy, providing a basis for more efficient processing and analysis of subsequent advanced tasks; the network structure used in the present invention is simple, and the general operator structure is convenient for edge deployment and rapid reasoning. This method integrates motion information in the time dimension, differential information of the context, and semantic information of the image frame, and uses a convolutional neural network to detect moving targets in video streams. Based on the fusion of multi-prompt information, the moving target can be well extracted, with good robustness and real-time performance. The present invention has high accuracy and low computational complexity, is convenient for edge device deployment, and provides accurate and efficient results for subsequent advanced tasks.

[0059] Embodiment 4: This embodiment improves S4 on the basis of embodiment 3, and proposes a moving target detection method based on multiple prompt information, such as Figure 1 As shown, the following steps are included: S1. Obtain a sequence of N frames of images from the monitoring device; the image frames obtained by S1 from the monitoring device are all 3-channel, 24-bit RGB images, and the sequence length needs to be maintained at N frames. The selection of the N value can be adjusted according to the actual application scenario. The factors generally considered include the movement speed of the moving target, the computing power of the edge device, etc. Considering that the speed of the moving target is fast, it is necessary to consider sufficient time intervals when obtaining the motion information in the time dimension, and N should be greater than 40. When the computing power of the edge device is weak, N frames can be obtained at intervals, and the number of calculated average frames can be reduced accordingly. As the number of acquired image frames increases, the N frames of images are updated and replaced. As a new frame is added to the sequence, the first frame is removed, and the sequence length is always maintained, that is, the N frames of images are iteratively updated according to the first-in-first-out principle.

[0060] S2. grayscale the current image frame to obtain a current frame containing semantic information; The grayscale processing of the image frame in S2 and S4 of the present invention is the same. Since the sensitivity of the human eye to the three colors of red, green and blue is different, the weight assigned to each color during grayscale conversion is also different. The conversion formula is: Gray=0.299*Red+0.587*Green+0.114*Blue Among them, Red, Green, and Blue are the three color channels of the color RGB image.

[0061] S3. According to N frames of images, a bgsub frame containing context differential information is calculated using a CNT algorithm; the CNT algorithm in S3 of the present invention is a method that does not require Gaussian modeling of background points, and it can very quickly extract context differential information using only information about pixel point values ​​in the past N consecutive frames. The CNT algorithm of the present invention actually obtains a bgsub frame containing context differential information by calling an opencv library API, namely bgsegm.createBackgroundSubtractorCNT.

[0062] S4. Calculate the average frame of the nearest neighbor frames based on N frames of images, and obtain the recent frame containing the motion information of the time dimension after grayscale processing; The nearest neighbor frame in S4 refers to the neighboring historical image frame before the current frame. Figure 2 As shown, the average frame means that according to the timing of the current frame, the nearest neighbor frames between the 20th frame before the current frame and the 40th frame before the current frame are calculated, a total of 20 image frames, and the pixel average value is calculated according to the RGB channels. The calculation formula for the Red channel is as follows: The Green and Blue channels are calculated in the same way.

[0063] Taking into account the computing power of edge devices, when the acquired N-frame images are selected at intervals, the selection of the nearest neighbor frames can be halved to 10 or less.

[0064] After obtaining the nearest neighbor average frame, grayscale processing is performed to obtain the recentframe containing the motion information of the time dimension.

[0065] S5. The current frame containing semantic information, the bgsub frame containing contextual difference information, and the recent frame containing motion information in the time dimension are spliced ​​and fused to obtain a multi-prompt information image; The three single-channel images of the current frame, the nearest neighbor average frame, and the difference bgsub frame obtained in the previous step of S5 of the present invention are spliced ​​into a 3-channel, 24-bit deep multi-prompt information image frame according to the RGB merging mode.

[0066] S6. Figure 4 As shown, the spliced ​​image frames use the trained convolutional neural network model to extract the moving targets in the multi-cue information image.

[0067] The input size of the convolutional neural network in S6 is 1080*1920*3, which is a common size for surveillance cameras. To meet the requirement of 32 times downsampling of the convolutional neural network, the surveillance image size is adjusted to 1088*1920*3. The convolutional neural network is a UNet type full convolutional neural network. Figure 3 As shown in the figure, the network structure consists of three parts: encoder, decoder and residual jump connection.

[0068] The encoder part of the present invention uses ResNet-18 to extract image features. It contains 5 blocks in total. Each block contains a basic convolution layer, a batch normalization layer, and an activation layer. After each block, the image size is halved, and the number of channels is doubled (except the first one). The encoder part downsamples the image size by 32 times and increases the number of channels from 3 to 512.

[0069] The residual jump connection part of the present invention includes 5 lateral connections, each of which includes a basic convolution layer and an activation layer. The lateral connection starting from the image adjusts the number of feature channels, while the other connections do not. When connecting, it is necessary to ensure that the size of the feature map matches.

[0070] The decoder part of the present invention contains 11 blocks in total, including basic convolutional layers, upsampling layers, activation layers, and needs to splice the feature maps from the residual jump connection in the channel dimension. The decoder part upsamples the image size by 32 times, restores the original width and height of the image, and the number of channels is 1, and generates a mask map.

[0071] After the multi-prompt information frame is pre-trained and convolutional neural network is used to obtain a mask map. The size of the mask map is H*W*1. After passing through the sigmoid layer, the probability distribution map of each pixel being the foreground is obtained, and the size is H*W*1. Combined with the set threshold τ, it is judged whether each pixel belongs to the foreground or the background. The judgment basis is as follows: Where mask(x) is the pixel value, x is the probability value of the pixel being the foreground, if the probability value is less than the threshold τ, then the pixel value is the background and is set to 0; if the probability value is greater than or equal to the threshold τ, then the pixel value is the moving target foreground and is set to 255. The extracted moving target mask is as follows: Figure 4 shown.

[0072] The present invention utilizes a variety of information of moving targets, including differential information, timing information, and context information, and can well extract moving targets in video streams with high accuracy, providing a basis for more efficient processing and analysis of subsequent advanced tasks; the network structure used in the present invention is simple, and the general operator structure is convenient for edge deployment and rapid reasoning. This method integrates motion information in the time dimension, differential information of the context, and semantic information of the image frame, and uses a convolutional neural network to detect moving targets in video streams. Based on the fusion of multi-prompt information, the moving target can be well extracted, with good robustness and real-time performance. The present invention has high accuracy and low computational complexity, is convenient for edge device deployment, and provides accurate and efficient results for subsequent advanced tasks.

[0073] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A moving target detection method based on multi-hint information, Its characteristics are: The following steps are involved: S1, obtain N frames of images from the monitoring device; S2, grayscale the current image frame to obtain an image containing semantic information; S3, N frames of images are calculated by CNT algorithm to obtain images containing context difference information; S4, calculating the average frame of the nearest neighbor frames of the N frames of the image, graying the average frame, and obtaining an image containing motion information in the time dimension; S5, splicing and fusing the image containing semantic information, the image containing context difference information, and the image containing time dimension motion information to obtain a multi-hint information image; S6, the pre-trained convolutional neural network model extracts the moving object segmentation mask map from the multi-cue information image.

2. A moving target detection method based on multiple prompt information according to claim 1, Its characteristics are: The sequence length of the image frames acquired in S1 is always maintained at N frames, and the image frames are 24-bit 3-channel RGB images.

3. A moving target detection method based on multiple prompt information according to claim 1 or 2, Its characteristics are: The grayscale processing in S2 and S4 is the same, and weights are assigned to the three color channels of the color RGB image respectively.

4. A moving target detection method based on multiple prompt information according to claim 1 or 2, Its characteristics are: The CNT algorithm obtains an image containing context difference information by calling the OpenCV library API.

5. A moving target detection method based on multiple prompt information according to claim 1 or 2, Its characteristics are: The nearest neighbor frame is the adjacent historical image frame before the current frame. The process of obtaining the average frame is as follows: calculate the pixel average of the nearest neighbor frames between the i-th frame before the current frame of each channel and the (i+n)-th frame before the current frame, and form an average frame by the pixel averages.

6. A moving target detection method based on multiple prompt information according to claim 2, Its characteristics are: The stitching and fusion process is as follows: stitching the image containing semantic information, the image containing context difference information and the image containing time dimension motion information according to the RGB merging mode to obtain a 24-bit deep 3-channel multi-prompt information image frame.

7. A moving target detection method based on multiple prompt information according to any one of claims 2 or 6, Its characteristics are: The convolutional neural network is a UNet type full convolutional neural network, including an encoder, a residual skip connection and a decoder. The convolutional neural network converts the multi-prompt information image into a mask map. After the mask map passes through the sigmoid layer, a probability distribution map of each pixel being the foreground is obtained. The probability distribution map is threshold judged, and the judgment basis is as follows: Where: mask(x) is the pixel value, x is the probability value of the pixel being the foreground, and τ is the threshold.

8. A moving target detection method based on multiple prompt information according to claim 7, Its characteristics are: The encoder uses ResNet-18 to extract image features, downsamples the image size by a factor of 32, and increases the number of channels from 3 to 512.

9. A moving target detection method based on multiple prompt information according to claim 8, Its characteristics are: The residual skip connection is provided with 5 lateral connections for adjusting the number of feature channels.

10. A moving target detection method based on multiple prompt information according to claim 9, Its characteristics are: The decoder splices the feature map from the residual jump connection and upsamples the image size by 32 times to generate a mask map with a channel number of 1.

Citation Information

Patent Citations

  • Vehicle search method and device

    CN106469299A

  • Moving object detection method based on deep full-convolution neural network

    CN108492319A

  • Moving target extraction method based on difference and semantic information fusion

    CN112907621A

  • Moving target detection method based on data distribution difference and multi-scale feature fusion

    CN115294177A

  • Tobacco leaf image active contour segmentation method based on deep learning

    CN116740119A