A scale-invariant based co-salient object detection method
By extracting multi-scale features from the monitoring system and enhancing and fusing them, and by using residual refinement and semantic feedback to optimize the saliency map, the problem of detection bias at different resolutions is solved, and high-accuracy and consistent target detection is achieved.
Patent Information
- Application Number
- CN202511309755.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-15
- Publication Date
- 2025-12-26
- Estimated Expiration
- 2045-09-15
AI Technical Summary
Existing collaborative saliency target detection technologies suffer from prediction bias and lack semantic guidance in monitoring equipment at different resolutions, making it difficult to meet the accuracy and generalization requirements of monitoring equipment.
By acquiring image sequences from the monitoring system, multi-scale features are extracted using a pre-trained model, and feature enhancement and fusion are performed. The saliency map is recovered by refining the structure using residuals, and the target saliency map is optimized by applying scale consistency constraints and semantic feature feedback.
It improves the accuracy and robustness of object detection, maintains consistency across different resolutions and scales, and significantly enhances the model's generalization ability and robustness.
Smart Images
Figure CN120808244B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of monitoring security technology, and in particular to a scale-invariant collaborative saliency target detection method. BACKGROUND
[0002] In recent years, monitoring systems have become an important tool for ensuring public safety and social stability. Traditional video monitoring systems rely on manpower to analyze events and make judgments, and then issue alarms. However, with the large-scale growth of monitoring data, traditional monitoring systems cannot meet the needs of efficiency and real-time performance. In order to free the monitoring system from the demand for manpower, intelligent monitoring systems have emerged. Among them, collaborative saliency target detection technology, as one of the key technologies in intelligent monitoring systems, has received extensive attention by simulating the human visual system to automatically detect and identify salient targets in video streams.
[0003] Current collaborative saliency target detection techniques can be mainly divided into three categories: collaborative saliency target detection using bottom-level consistency, collaborative saliency target detection using depth consistency, and collaborative saliency target detection using salient region refinement. Most of these methods focus on describing multi-scale features to find consistent representations to extract collaborative features. Some methods simply train a network for each resolution data to make collaborative saliency predictions, which brings a large number of redundant training parameters and is time-consuming and laborious. However, even if attention is paid to multi-scale features, these methods still have prediction biases for monitoring devices at different resolutions, and lack semantic guidance, making it difficult to meet the accuracy and generalization requirements of monitoring devices. SUMMARY
[0004] The technical problem to be solved by the present application is to overcome the deficiencies of the prior art. Specifically, a scale-invariant collaborative saliency target detection method is provided, which is as follows:
[0005] 1) In a first aspect, the present application provides a scale-invariant collaborative saliency target detection method, and the specific technical solutions are as follows:
[0006] An image sequence converted from a video stream in any monitoring system is obtained, and the image sequence is taken as input data to extract multi-scale features corresponding to the input data through a backbone network in a pre-trained model;
[0007] The multi-scale features are subjected to feature enhancement processing and fusion processing to generate an initial saliency map;
[0008] The initial saliency map is subjected to recovery processing through a residual refinement structure to generate an accurate saliency map;
[0009] The precise saliency map is optimized by applying the scale consistency constraint and the semantic feature feedback, and a target saliency map is obtained.
[0010] The scale-invariant cooperative saliency target detection method has the following advantages:
[0011] The multi-scale features are extracted by the backbone network, which can capture different levels of details and semantic information in the image, providing a rich feature basis for subsequent saliency target detection. The multi-scale features are enhanced and fused to generate an initial saliency map. This process not only enhances the expression ability of the features, but also enables the model to better understand the global and local features of the target by fusing features of different scales. Further, the initial saliency map is recovered by the residual refinement structure, which can effectively recover the details and edges in the image, making the saliency map more accurate and improving the accuracy of target detection. By applying the scale consistency constraint and the semantic feature feedback to optimize the precise saliency map, the target saliency map obtained not only has high accuracy, but also maintains consistency at different resolutions and scales, significantly improving the generalization ability and robustness of the model. Therefore, the present application can stably output high-quality target saliency maps and their saliency degrees in various monitoring scenarios, providing an efficient and reliable saliency target detection solution for intelligent monitoring systems. It effectively solves the problem of lack of semantic guidance and poor generalization ability of the model for different resolution images in the prior art, and has significant innovation and practicality.
[0012] On the basis of the above-mentioned scheme, the present application can also be improved as follows.
[0013] Further, the training process of the pre-trained model is as follows:
[0014] An arbitrary scale pixel alignment data enhancement method is constructed, and a target multi-scale data set of pixel alignment in the same scene is obtained;
[0015] A target backbone network corresponding to the training requirements is selected, a target feature extraction module, a target residual refinement structure and a scale consistency constraint module are constructed, and a loss function is determined;
[0016] The training parameters of the model are set, the target multi-scale data set is taken as the target input data, and the target output result is obtained after the target input data is sequentially passed through the target backbone network, the target feature extraction module, the target residual refinement structure and the scale consistency constraint module. The target input data and the target output result are processed by the loss function, and the training parameters are updated based on the processing result until the processing result meets the preset requirements.
[0017] The beneficial effects of the above further scheme are:
[0018] By constructing an arbitrary scale pixel alignment data enhancement method, a variety of training data can be generated to enhance the adaptability of the model to different resolution images. Selecting a suitable target backbone network and constructing related modules provides the model with powerful feature extraction and processing capabilities. Determining the loss function and setting the training parameters enable the model to continuously optimize its parameters through back propagation until the preset requirements are met, thereby ensuring that the model can efficiently learn the key features and patterns of saliency object detection during the training process, laying a solid foundation for subsequent detection tasks and improving the accuracy and generalization ability of the model.
[0019] Further, the preset requirements include:
[0020] The performance of the model is evaluated by the validation dataset, and an evaluation result is obtained;
[0021] The performance indicators of the model are calculated, and the detection effect of the model is determined based on the performance indicators, and a detection result is obtained;
[0022] When the evaluation result meets the evaluation requirement and the detection result meets the detection requirement, it is determined that the preset requirement is met.
[0023] The beneficial effects of the above further scheme are:
[0024] By evaluating the performance of the model through the validation dataset, the performance of the model in actual application can be objectively measured to ensure the reliability and stability of the model. Calculating the performance indicators of the model and determining the detection effect further clarifies the advantages and disadvantages of the model, facilitating targeted optimization and improvement. When the evaluation result and the detection result both meet the corresponding requirements, it is determined that the preset requirement is met, which provides a scientific basis for the final determination of the model and ensures that the model can achieve the expected detection effect in actual application, effectively improving the practicality and application value of the model.
[0025] Further, the process of obtaining the pixel-aligned target multi-scale dataset under the same scene is specifically:
[0026] In each iteration process, a scale factor is randomly generated, and the original public data is scaled based on the scale factor to obtain a low-resolution image, a medium-resolution image, and a high-resolution image;
[0027] The low-resolution image and the high-resolution image are spliced and cropped according to the scale factor to obtain a splicing and cropping result, and the splicing and cropping result is adjusted so that the adjusted splicing and cropping result is consistent in size with the medium-resolution image.
[0028] The beneficial effects of the above further scheme are:
[0029] Randomly generating the scale factor and scaling the original data in each iteration can generate images with different resolutions, increasing the diversity and richness of the training data. By splicing and cropping low-resolution and high-resolution images to make them consistent with the size of the medium-resolution image, pixel alignment is achieved, allowing the model to better learn the correlation and differences between images of different resolutions, further enhancing the model's robustness to scale changes and improving the detection performance and accuracy of the model when processing multi-scale images.
[0030] Further, the determination process of the initial saliency map comprises:
[0031] The multi-scale features are enhanced and fused by the CBR block in the backbone network;
[0032] The feature enhancement and fusion results are decoded to obtain the initial saliency map.
[0033] The beneficial effects of the above further scheme are:
[0034] Enhancing and fusing the multi-scale features by the CBR block in the backbone network can effectively improve the quality and expression ability of the features, allowing the model to more accurately capture the saliency information in the image. Decoding the processing results to obtain the initial saliency map further optimizes the generation process of the saliency map, allowing the initial saliency map to more clearly reflect the location and contour of the salient object in the image, providing a better foundation for subsequent refinement and optimization processing, and helping to improve the quality and accuracy of the final target saliency map.
[0035] Further, the residual refinement structure comprises:
[0036] Each stage contains a convolution layer, a batch normalization layer, an activation function, and a bridge;
[0037] The bridge includes 64 convolution layers, batch normalization layers, and activation functions.
[0038] The beneficial effects of the above further scheme are:
[0039] The convolutional layer, batch normalization layer, activation function and bridge included in each stage of the residual refinement structure can perform layer-by-layer refinement processing on the initial saliency map. In particular, the 64 convolutional layers, batch normalization layers and activation functions included in the bridge part further enhance the model's ability to recover detailed information, making the detailed edges in the saliency map clearer and more accurate. Through the design of this structure, the model can better recover and retain important details in the image, thereby improving the precision and reliability of salient object detection, making the detection results more in line with actual needs.
[0040] Further, the process of optimizing the accurate saliency map to obtain the target saliency map is as follows:
[0041] Pre-constructing a memory bank that stores the saliency prediction map of each collaborative group;
[0042] Splicing the saliency prediction map of the collaborative group corresponding to the initial saliency map;
[0043] And performing consistency constraint processing on the spliced result to obtain the target saliency map.
[0044] The beneficial effects of the above further scheme are:
[0045] Pre-constructing a memory bank and storing the saliency prediction map of the collaborative group provides a memory mechanism for the model, enabling the model to better utilize historical information for the current detection task. Splicing the saliency prediction map of the collaborative group corresponding to the initial saliency map and performing consistency constraint processing can ensure consistency between the saliency prediction results of images of different resolutions, while combining semantic feature feedback to further optimize the quality of the target saliency map. This process not only improves the model's adaptability to scale changes, but also enhances the model's understanding and utilization of semantic information, making the final target saliency map more accurate and robust, significantly improving the model's detection performance and generalization ability in complex scenarios.
[0046] 2) In a second aspect, the present application also provides a collaborative salient object detection system based on scale invariance, and the specific technical scheme is as follows:
[0047] The acquisition module is used to acquire an image sequence converted from a video stream in any monitoring system, take the image sequence as input data, and extract multi-scale features corresponding to the input data through a backbone network in a pre-trained model;
[0048] The initial generation module is used to perform feature enhancement processing and fusion processing on the multi-scale features to generate an initial saliency map;
[0049] The precision generation module is configured to generate a precision saliency map by performing recovery processing on the initial saliency map through a residual refinement structure.
[0050] The detection module is configured to obtain a target saliency map by performing optimization processing on the precision saliency map by applying a scale consistency constraint and semantic feature feedback, and output the target saliency map and a corresponding saliency degree as output data.
[0051] 3) In a third aspect, the present application also provides an electronic device, which comprises a processor and a memory coupled to the processor, and the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor, so that the electronic device implements any of the above methods.
[0052] 4) In a fourth aspect, the present application also provides a computer readable storage medium, which stores at least one computer program, and the at least one computer program is loaded and executed by a processor, so that the computer implements any of the above methods.
[0053] It should be noted that the technical solutions of the second to fourth aspects of the present application and the corresponding possible implementation manners have the beneficial effects as described above for the first aspect and the corresponding possible implementation manners, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS
[0054] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments with reference to the attached drawings:
[0055] Figure 1 A flowchart of a cooperative saliency target detection method based on scale invariance according to an embodiment of the present application;
[0056] Figure 2 A cutting and splicing manner diagram of a cooperative saliency target detection method based on scale invariance according to an embodiment of the present application;
[0057] Figure 3 A feature fusion and transmission diagram of a cooperative saliency target detection method based on scale invariance according to an embodiment of the present application;
[0058] Figure 4 A network structure diagram of a cooperative saliency target detection method based on scale invariance according to an embodiment of the present application;
[0059] Figure 5 A sampling image diagram of a cooperative saliency target detection method based on scale invariance according to an embodiment of the present application;
[0060] Figure 6 A non-semantic association and semantic association after comparison schematic view of a scale invariance based cooperative saliency target detection method according to an embodiment of the present application is shown in the figure.
[0061] Figure 7 An effect comparison schematic view of a saliency target detection method of a scale invariance based cooperative saliency target detection method according to an embodiment of the present application is shown in the figure.
[0062] Figure 8 A preset model network structure schematic view of a scale invariance based cooperative saliency target detection method according to an embodiment of the present application is shown in the figure.
[0063] Figure 9 A structural framework diagram of an electronic device according to the present application is shown in the figure. DETAILED DESCRIPTION
[0064] In order to make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.
[0065] As shown in Figure 1 and Figure 8 A scale invariance based cooperative saliency target detection method according to an embodiment of the present application includes the following steps:
[0066] S1, obtaining an image sequence converted from a video stream in any monitoring system, taking the image sequence as input data, and extracting multi-scale features corresponding to the input data through a backbone network in a pre-trained model;
[0067] S2, performing feature enhancement processing and fusion processing on the multi-scale features to generate an initial saliency map;
[0068] S3, performing recovery processing on the initial saliency map through a residual refinement structure to generate an accurate saliency map;
[0069] S4, applying scale consistency constraints and semantic feature feedback to optimize the accurate saliency map to obtain a target saliency map, and outputting the target saliency map and the corresponding saliency degree as output data.
[0070] The beneficial effects of a scale invariance based cooperative saliency target detection method provided by the present application are as follows:
[0071] The extraction of multi-scale features through the backbone network can capture different levels of details and semantic information in the image, providing a rich feature basis for subsequent salient object detection. The multi-scale features are enhanced and fused to generate an initial saliency map. This process not only enhances the expression ability of the features, but also fuses features of different scales, enabling the model to better understand the global and local features of the target. Further, the initial saliency map is recovered using a residual refinement structure, which can effectively recover the details and edges in the image, making the saliency map more accurate and improving the accuracy of target detection. The precise saliency map is optimized by applying scale consistency constraints and semantic feature feedback, resulting in a target saliency map that not only has high accuracy but also maintains consistency across different resolutions and scales, significantly improving the model's generalization ability and robustness. Therefore, the present application can stably output high-quality target saliency maps and their saliency degrees in various monitoring scenarios, providing an efficient and reliable salient object detection solution for intelligent monitoring systems. It effectively solves the problem of lack of semantic guidance and poor generalization ability of the model for different resolution images in the prior art, and has significant innovation and practicality.
[0072] In another embodiment of the present application, the specific process of S1 is:
[0073] In processing video streams in a monitoring system, it is necessary to decompose continuous video streams into a series of discrete image frames to form an image sequence. This process is usually completed by a video decoder, which extracts images from the video stream at a set time interval (e.g. every second) or based on a key frame strategy. These image frames are then uniformly adjusted to a predefined size to adapt to the input requirements of the pre-trained model, ensuring that the model can effectively process input data of different resolutions. The image sequence after adjusting the size is input as input data into the backbone network of the pre-trained model. The backbone network is usually a deep convolutional neural network such as VGG, ResNet, etc., which extracts features from the input image sequence through a series of convolutional layers, pooling layers, and possibly batch normalization layers and activation functions. In this process, the network generates multiple feature maps of different scales, which capture local details, texture information and higher-level semantic information in the image. The extraction of multi-scale features enables the model to focus on both small and large targets in the image and their performance at different scales, providing a rich information basis for subsequent salient object detection.
[0074] In another embodiment of the present application, the specific process of S2 is:
[0075] The multi-scale features extracted from the backbone network are input into feature enhancement modules, which are composed of multiple convolution blocks (CBR blocks). Each CBR block contains a convolution layer, a ReLU activation function, and a batch normalization layer. These convolution blocks process features of different scales layer by layer, extract local features through convolution operations, introduce nonlinearity through ReLU activation functions to enable the model to learn complex feature representations, and use batch normalization layers to stabilize the training process, accelerate convergence, and reduce internal covariate shift. The feature maps processed by the CBR blocks not only enhance the expression ability of the original features, but also further highlight the feature information of the salient objects through the nonlinear transformation of the activation function.
[0076] The enhanced multi-scale features are fused. This process is usually achieved through feature concatenation, addition, or more complex feature fusion strategies, with the goal of integrating feature information of different scales to form a comprehensive feature representation. For example, low-level detail features can be combined with high-level semantic features, so that the fused features contain not only rich detail information but also high-level semantic understanding of the target. The fused feature map can more comprehensively describe the salient objects in the image, providing a more accurate basis for further saliency prediction.
[0077] The fused feature map is input into the decoder for decoding operation. The decoder usually uses upsampling techniques (such as bilinear interpolation, transpose convolution, etc.) to gradually restore the spatial resolution of the feature map, making it close to the size of the original input image. During the decoding process, jump connections may be combined to fuse features from the encoder stage with features from the decoder stage, further enhancing the detail information of the features. After decoding, an initial saliency map is generated, which represents the saliency degree of each pixel in the image in the form of a heat map, and the area with high saliency corresponds to the possible target object. This initial saliency map provides a basis for subsequent residual refinement and optimization, enabling the model to further improve the accuracy and robustness of salient object detection.
[0078] In another embodiment of the present scheme, the specific process of S3 is:
[0079] The initial saliency map is input into a carefully designed residual refinement network structure. This structure is usually composed of multiple stages, each containing a series of convolution layers, batch normalization layers, and ReLU activation functions, which work together to gradually optimize the details and edge information of the saliency map. Between each stage, there is a bridging module composed of multiple convolution layers, batch normalization layers, and ReLU activation functions, which further strengthens the transmission and fusion of features, ensuring that multi-scale feature information can be fully utilized during the refinement process.
[0080] The encoder part of the refinement network uses non-overlapping max-pooling layers for downsampling operations to gradually reduce the spatial resolution of feature maps, extracting higher-level semantic information. At the same time, the decoder part uses upsampling techniques such as bilinear interpolation to gradually restore the resolution of the feature maps, making them eventually match the size of the original input image. Throughout the refinement process, the features from the encoder stage are effectively combined with the features from the decoder stage through a residual connection mechanism. This residual connection not only helps to alleviate the gradient vanishing problem in deep network training, but also ensures that the network can learn more detailed information, especially in the edge and texture parts of the salient objects.
[0081] After a series of residual refinement operations, the blurred or inaccurate edges in the initial saliency map are clarified, and the detailed information is enhanced, resulting in a more accurate saliency map. This accurate saliency map can more accurately highlight the salient objects in the image while suppressing background noise and other non-target area interference, providing a high-quality basis for subsequent scale consistency constraints and semantic feature feedback optimization, significantly improving the overall performance and reliability of salient object detection.
[0082] In another embodiment of the present scheme, the specific process of S4 is:
[0083] The scale consistency constraint mechanism is introduced. The core of this mechanism is to ensure that the model has consistency in predicting the saliency of the same target at different resolutions. Specifically, by constructing a continuously updated memory bank, the memory bank is used to store the saliency prediction map of each collaborative group. In each prediction, the memory bank of the current collaborative group is mapped and spliced with the four-stage encoder output to enhance the collaborative features. The memory bank will be updated according to the new prediction results and provide rough target location clues for the next group prediction, thereby realizing dynamic tracking and optimization of salient objects.
[0084] At the same time, in order to further improve the quality of the saliency map, the semantic feature feedback mechanism is introduced. This mechanism feeds back high-level semantic information to the generation process of the saliency map, so that the model can better understand the semantic meaning of the target, and pay more attention to semantic relevance when generating the saliency map. In specific operations, the semantic features extracted by pre-trained semantic segmentation networks or classification networks are used to correct and optimize the saliency map. In this way, the model can not only pay attention to the visual saliency of the target, but also distinguish different types of salient objects by combining semantic information, improving the accuracy and robustness of the saliency map.
[0085] After completing the optimization process of scale consistency constraints and semantic feature feedback, the obtained target saliency map can more accurately reflect the position and contour of the salient target in the image, while maintaining consistency at different resolutions. Finally, the target saliency map and the corresponding saliency degree are output as output data. The saliency degree is usually represented in the form of a heat map, where high-value areas correspond to salient targets, and low-value areas correspond to backgrounds or other non-salient regions. This output form not only intuitively shows the position of the salient target, but also provides quantitative information about the saliency of the target, facilitating subsequent analysis and application. Through this series of optimization processes, the model can stably output high-quality saliency detection results in complex scenes and under multi-resolution conditions, providing a reliable saliency target detection solution for intelligent monitoring systems and other application scenarios.
[0086] Further, the training process of the pre-trained model is:
[0087] An arbitrary scale pixel alignment data augmentation method is constructed, and a target multi-scale data set with pixel alignment in the same scene is obtained;
[0088] A target backbone network corresponding to the training requirements is selected, a target feature extraction module, a target difference refinement structure, and a scale consistency constraint module are constructed, and a loss function is determined;
[0089] The training parameters of the model are set, and the target multi-scale data set is used as the target input data. After passing through the target backbone network, the target feature extraction module, the target difference refinement structure, and the scale consistency constraint module in turn, the target output result is obtained. The target input data and the target output result are processed by the loss function, and the training parameters are updated based on the processing result until the processing result meets the preset requirements.
[0090] Further, the preset requirements include:
[0091] The performance of the model is evaluated by the validation data set, and the evaluation result is obtained;
[0092] The performance indicators of the model are calculated, and the detection effect of the model is determined based on the performance indicators, and the detection result is obtained;
[0093] When the evaluation result meets the evaluation requirements and the detection result meets the detection requirements, it is determined that the preset requirements are met.
[0094] Further, the process of obtaining a target multi-scale data set with pixel alignment in the same scene is as follows:
[0095] In each iteration, a scale factor is randomly generated, and the original public data is scaled based on the scale factor to obtain low-resolution, medium-resolution, and high-resolution images.
[0096] The low-resolution image and the high-resolution image are stitched and cropped according to the scale factor to obtain the stitching and cropping results. The stitching and cropping results are then adjusted so that the adjusted stitching and cropping results are consistent with the size of the medium-resolution image.
[0097] Furthermore, the process of determining the initial saliency map includes:
[0098] The multi-scale features are enhanced and fused using the CBR block in the backbone network.
[0099] The initial saliency map is obtained by decoding the feature enhancement and fusion processing results.
[0100] Furthermore, the residual refinement structure includes:
[0101] Each stage consists of a convolutional layer, a batch normalization layer, an activation function, and a bridging layer;
[0102] The bridging consists of 64 convolutional layers, batch normalization layers, and activation functions.
[0103] Furthermore, by applying scale consistency constraints and semantic feature feedback, the precise saliency map is optimized to obtain the target saliency map.
[0104] A pre-built memory bank is used to store salient prediction maps for each cooperating group;
[0105] The saliency prediction maps of the co-groups corresponding to the initial saliency map are stitched together;
[0106] The spliced results are then subjected to consistency constraints to obtain the target saliency map.
[0107] In the above embodiments, although the steps are numbered S1, S2, etc., they are only specific embodiments given by the present invention. Those skilled in the art can adjust the execution order of S1, S2, etc. according to the actual situation, which is also within the protection scope of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.
[0108] Example 1, as Figures 2 to 4 As shown, step 1: Construct a data augmentation method for pixel alignment at arbitrary scales to obtain a multi-scale dataset of pixel alignment in the same group and scene, thereby increasing the diversity of training data.
[0109] Step 2: Extract multi-scale fused collaborative salient features from the data set obtained in step 1 to obtain an initial saliency map.
[0110] Step 3: Use the initial saliency map obtained in step 2 to restore the detail edges through residual refinement structure to obtain an accurate salient prediction image.
[0111] Step 4: Based on the accurate prediction image obtained from the previous three steps, perform scale consistency constraint and prediction class semantic feature feedback to obtain a target saliency map.
[0112] An arbitrary scale pixel alignment data enhancement method is constructed to obtain a multi-scale pixel alignment data set under the same group and scene, and the diversity of the training data is increased. First, a scale factor is randomly generated in each iteration and , the original public data set is scaled according to the scale factor, and three different resolution images are obtained. In order to ensure that the model is sensitive to the resolution of the training data and not sensitive to the acquisition size, the low resolution image and the high resolution image are spliced and cropped according to the scale factor, so that their size is consistent with that of the medium resolution image.
[0113] Multi-level features are extracted from the multi-scale pixel alignment paired data. First, the data is input into an arbitrary backbone network, and here the VGG network is taken as an example. The initial features of 5 different scales are obtained by outputting the features after each convolution layer The CBR block composed of convolution layer, ReLU activation function and batch normalization layer is used for feature enhancement and fusion for different scale features. The process expression is as follows:
[0114]
[0115] The enhanced features are transmitted to obtain fused features The first decoding of the fused features obtains an initial saliency prediction image. The process expression is as follows:
[0116] .
[0117] The four-stage encoder-decoder structure is used for residual refinement of the initial prediction image to restore the detail edges.
[0118] The four-stage encoder-decoder structure includes a 3x3 64-channel convolution layer, a batch normalization layer and a ReLU activation function in each stage. There is a bridge between each stage of the encoder and the decoder, and the bridge stage includes a 64-filter 3x3 convolution layer, a batch normalization layer and a ReLU activation function.
[0119] The precise prediction image is subjected to scale consistency constraint and prediction class semantic feature feedback to obtain a target saliency map.
[0120] In the collaborative saliency, a memory bank storing the saliency prediction map of each collaborative group is first constructed for each type, which is connected with the four-stage decoder, and the specific connection is as shown in Figure 5 As shown in the figure, at each prediction, the memory bank mapping of the same group is spliced with the four-stage encoder output, the memory bank is updated and provides a rough target position clue for the next same group prediction. The initial prediction mapping in the memory bank is filled with zeros.
[0121] The consistency constraint is performed on the collaborative saliency prediction of different resolutions, that is, for different resolution images, the pixel-aligned part of the prediction should be as close as possible, and the collaborative saliency prediction satisfying the size invariance is obtained.
[0122] Taking the high-resolution and medium-resolution prediction images as examples, and represent the prediction pixel-aligned part of the pixel, and the consistency constraint expression is as follows:
[0123]
[0124] Target detection network training setting: In order to train and evaluate the proposed collaborative saliency target detection method, the present application is implemented using Pytorch, and is trained on a single NVIDIA GTX 4090 Ti GPU. The network is trained for 30 epochs, with a batch size of 4. All experiments use a stochastic gradient descent optimizer, with a weight decay of 0.0005, an initial learning rate of 0.001, and a momentum value of 0.9. During the training process, pre-training and initialization are performed on the ImageNet dataset. During the testing process, the input image is resized and adjusted to a resolution of 320x320.
[0125] Example 2, in order to verify the effectiveness of the collaborative saliency target detection model proposed in the present application, the saliency target detection dataset DUTS-TE benchmark dataset and the collaborative saliency target detection dataset CoCA benchmark dataset image are selected as experimental objects, and the effectiveness of resolution desensitization and collaborative correlation is verified respectively.
[0126] As shown in Figure 5 , the benchmark dataset DUTS-TE is downsampled by 2 and 4 times, denoted as DUTS-2D and DUTS-4D.
[0127] Figure 5The two groups of images are shown, the first column to the third column is a group, and the fourth column to the sixth column is a group, each group is DUTS, DUTS-2D and DUTS-4D images respectively, and the resolutions are 96*96, 48*48 and 24*24 respectively, the third row to the seventh row respectively show the change process of specific image effects, wherein with the increase of the steps such as data enhancement and consistency constraint, the semantic understanding is gradually enhanced, and the structure and details are gradually clear; and the final output binary segmentation image presents desensitization to the resolution, further verifying the effectiveness of the model.
[0128] As shown in Figure 6 For example, the alarm clock group images are selected from the benchmark dataset CoCA. The non-semantic association and semantic association are set and compared. The two groups of images are shown, and the third column and the fourth column respectively show the change process of specific image effects. The third column shows only two classes of circles without semantic association.
[0129] The embodiment shows the detection ability of the cooperative saliency target detection method, and verifies the robustness and simplicity of the method in complex scenes and different acquisition conditions.
[0130] The effect comparison chart of the present algorithm and the existing algorithm processing some saliency target detection methods is shown in Figure 7 .
[0131] From the chart, it can be seen that the existing saliency target detection method is limited by cross-domain difference and model design, and is difficult to adapt to the resolution change environment. The cooperative saliency target detection method based on scale invariance proposed in the present application can better desensitize the resolution difference, solve the problem of confusion of the model's semantic type recognition of pixels caused by the resolution cross-domain difference, simulate consistent saliency cognition, and thus achieve better target detection effect.
[0132] Compared with the traditional target detection model, the scale invariance based collaborative saliency target detection method has the advantages of strong generalization, strong adaptability and resolution desensitization of the acquisition device. The traditional method usually relies on manually designed features and heuristic rules to extract visual features, infer the salient region, lacks the correlation information between images, has limitations in feature expression, and often fails to face some complex scene objects. The application of deep learning method can effectively improve the generalization of the collaborative saliency target detection method, but there is still a lack in cross-domain semantic cognition, and it cannot fully utilize the spatial structure information. The present application only needs to add two simple steps of data enhancement branch and consistency constraint to implicitly and explicitly guide the model to pay attention to the semantic information between image groups, ignore the scale information such as resolution and size in training and testing, and does not need to make additional training on acquisition devices with different resolutions in the test scene, which greatly reduces the dependence of the model on training images and the computational complexity. At the same time, the method guides the shallow texture information of the feature extraction part back to the deep features through multiple cross-layer feature refinement modules, strengthens the semantic expression ability of deep information, and at the same time retains the detailed information of the shallow layer. Therefore, this method is not only simple to train and easy to implement, but also has the advantages of both semantics and details, and can be widely used in different device platforms and complex scenes, and has good cross-scene generalization ability and robustness.
[0133] Example 3, video preprocessing process:
[0134] Video frame extraction: Extract video frames from the video stream at fixed time intervals (such as extracting one frame per second) or key frame extraction method, and convert the video stream into an image sequence.
[0135] Image size adjustment: Adjust the extracted video frames to the uniform size required by the model input (such as 320x320 pixels) to adapt to the input requirements of the model.
[0136] Saliency target detection process:
[0137] Input model: input the preprocessed video frames into the trained model.
[0138] Feature extraction and fusion: the model extracts multi-scale features through the backbone network, and performs feature enhancement and fusion to generate an initial saliency map.
[0139] Residual refinement: use the encoder-decoder structure to refine the initial saliency map, restore the detail edges, and generate an accurate saliency prediction image.
[0140] Scale consistency constraint: apply scale consistency constraint and semantic feature feedback to optimize the target saliency map.
[0141] Output saliency map: The model outputs a target saliency map, representing the saliency degree of each pixel in the video frame.
[0142] Behavior anomaly detection process:
[0143] Saliency map analysis: Analyze the generated target saliency map to identify areas with high saliency, which may be personnel with abnormal behavior.
[0144] Behavior feature extraction: Combine behavior analysis algorithms (such as optical flow method, deep learning behavior recognition model) to extract behavior features from salient regions and determine whether there is abnormal behavior.
[0145] Abnormal behavior recognition: Through a pre-trained behavior recognition model, classify the extracted behavior features to determine whether it is an abnormal behavior (such as running, fighting, falling, etc.).
[0146] Result output and alarm process:
[0147] Detection result output: Output the location and behavior type of the detected behavior abnormal personnel to the monitoring system interface.
[0148] Alarm mechanism: If abnormal behavior is detected, trigger the alarm mechanism to notify the monitoring personnel or automatically record the event for further processing.
[0149] It should be noted that the beneficial effects of the system provided by the above embodiments are the same as those of the method, and will not be repeated here. In addition, the system provided by the above embodiments is only divided into the above functional modules for example when implementing its functions, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the system is divided into different functional modules according to actual conditions to complete all or part of the above described functions. In addition, the system and method embodiments provided by the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.
[0150] As shown in Figure 9 An electronic device 300 according to an embodiment of the present application, the electronic device 300 includes a processor 320, the processor 320 is coupled with a memory 310, the memory 310 stores at least one computer program 330, the at least one computer program 330 is loaded and executed by the processor 320, so that the electronic device 300 realizes any one of the above methods, specifically:
[0151] The electronic device 300 can have a large difference due to different configurations or performances, and can include one or more processors 320 (Central Processing Units, CPUs) and one or more memories 310, wherein the one or more memories 310 store at least one computer program 330, the at least one computer program 330 is loaded and executed by the one or more processors 320, so that the electronic device 300 implements the scale-invariant based collaborative saliency target detection method provided in the above embodiments. Of course, the electronic device 300 can also have a wired or wireless network interface, a keyboard, and an input and output interface, and the like, so as to perform input and output, and the electronic device 300 can also include other components for realizing device functions, which are not described here.
[0152] The computer readable storage medium of the embodiment of the application stores at least one computer program, and the at least one computer program is loaded and executed by the processor, so that the computer implements any one of the above methods.
[0153] Optionally, the computer readable storage medium can be a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.
[0154] In the exemplary embodiments, a computer program product or computer program is also provided, which includes computer instructions stored in a computer readable storage medium. The processor of the electronic device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the electronic device executes any one of the above methods.
[0155] It should be noted that the terms "first", "second" in the specification and claims of the present application are used to distinguish similar objects, and represent a specific order or sequence. The order of use of similar objects can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described.
[0156] Those skilled in the art know that the present application can be implemented as a system, a method, or a computer program product, therefore, the present disclosure can be embodied in the form of a complete hardware, a complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, which is generally referred to as "circuitry", "module" or "system". In addition, in some embodiments, the present application can also be embodied in the form of a computer program product in one or more computer readable media, which contains computer readable program codes.
[0157] Any combination of one or more computer readable medium can be employed. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium can be, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.
[0158] Although the embodiments of the present application have been shown and described above, it should be understood that the above-described embodiments are exemplary, and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-described embodiments within the scope of the present application.
Claims
1. A scale-invariant based co-salient object detection method, characterized in that, The method comprises the following steps: An image sequence converted from a video stream in any monitoring system is obtained as input data, and a multi-scale feature corresponding to the input data is extracted by a backbone network in a pre-trained model, specifically: When processing a video stream in a monitoring system, continuous video streams are decomposed into a series of discrete image frames to form an image sequence. This process is completed by a video decoder. Images are intercepted from the video stream at a set time interval or based on a key frame strategy. These image frames are then uniformly adjusted to a predefined size to meet the input requirements of the pre-trained model. The image sequence after size adjustment is input as input data into the backbone network of the pre-trained model. The backbone network is a deep convolutional neural network that extracts features from the input image sequence through a series of convolutional, pooling, and normalization layers and activation functions. The multi-scale features are subjected to feature enhancement and fusion processing to generate an initial saliency map. The initial saliency map is recovered by a residual refinement structure to generate an accurate saliency map. The accurate saliency map is optimized by applying scale consistency constraints and semantic feature feedback to obtain a target saliency map. The target saliency map and the corresponding saliency degree are output as output data. The process of applying scale consistency constraints and semantic feature feedback to optimize the accurate saliency map to obtain the target saliency map is as follows: A memory bank is pre-constructed, which stores the salient prediction map of each collaborative group. The salient prediction map of the collaborative group corresponding to the initial saliency map is spliced. The spliced result is subjected to consistency constraint processing to obtain the target saliency map.
2. The method according to claim 1, wherein, The training process of the pre-trained model is as follows: An arbitrary scale pixel alignment data augmentation method is constructed, and a target multi-scale data set with pixel alignment in the same scene is obtained. A target backbone network corresponding to the training requirements is selected to construct a target feature extraction module, a target residual refinement structure, and a scale consistency constraint module, and a loss function is determined. The training parameters of the model are set, and the target multi-scale data set is used as the target input data. After passing through the target backbone network, the target feature extraction module, the target residual refinement structure, and the scale consistency constraint module in turn, the target output result is obtained. The target input data and the target output result are processed by the loss function. Based on the processing result, the training parameters are updated by backpropagation until the processing result meets the preset requirements.
3. The scale invariance based co-salient object detection method according to claim 2, characterized in that, The preset requirements include: The model performance is evaluated by a validation data set to obtain an evaluation result. The performance indicators of the model are calculated, and the detection effect of the model is determined based on the performance indicators to obtain a detection result. When the evaluation result meets the evaluation requirements and the detection result meets the detection requirements, it is determined that the preset requirements are met.
4. The method according to claim 2, wherein, The process of obtaining a target multi-scale data set with pixel alignment in the same scene is as follows: A scale factor is randomly generated in each iteration process, and the original public data is scaled based on the scale factor to obtain a low-resolution image, a medium-resolution image and a high-resolution image; The low-resolution image and the high-resolution image are spliced and cropped according to the scale factor to obtain a splicing and cropping result, and the splicing and cropping result is adjusted so that the adjusted splicing and cropping result is consistent with the size of the medium-resolution image.
5. The method according to claim 2, wherein, The determination process of the initial saliency map includes: The multi-scale features are enhanced and fused through the CBR block in the backbone network; The initial saliency map is obtained by decoding the feature enhancement and fusion result.
6. The scale invariance based co-salient object detection method according to claim 2, characterized in that, The residual refinement structure includes: Each stage contains a convolution layer, a batch normalization layer, an activation function and a bridge; The bridge includes 64 convolution layers, batch normalization layers and activation functions.
7. A system for scale-invariant co-saliency object detection, employing a scale-invariant co-saliency object detection method as claimed in claim 1, characterized in that, The system includes: The acquisition module is configured to: acquire an image sequence converted from a video stream in any monitoring system, take the image sequence as input data, and extract multi-scale features corresponding to the input data through a backbone network in a pre-trained model; The initial generation module is configured to: perform feature enhancement and fusion processing on the multi-scale features to generate an initial saliency map; The accurate generation module is configured to: perform recovery processing on the initial saliency map through a residual refinement structure to generate an accurate saliency map; The detection module is configured to: apply scale consistency constraints and semantic feature feedback to optimize the accurate saliency map to obtain a target saliency map, and output the target saliency map and the corresponding saliency degree as output data.
8. An electronic device, comprising: The electronic device includes a processor coupled with a memory, and the memory stores at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement the method of any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by the processor to enable the computer to implement the method of any one of claims 1 to 6.