Cooperative saliency target detection method based on scale invariance

Abstract

The invention discloses a collaborative saliency target detection method based on scale invariance, and relates to the technical field of monitoring security, and the method comprises the steps: obtaining an image sequence converted from a video stream in a monitoring system, and inputting the image sequence into a backbone network of a pre-training model to extract multi-scale features; performing enhancement and fusion processing on the multi-scale features to generate an initial saliency map; a detail edge is recovered through a residual error refining structure, and an accurate saliency map is generated; and performing optimization by applying scale consistency constraint and semantic feature feedback to obtain a target saliency map and outputting the target saliency map. The method can effectively solve the problems that in the prior art, a model lacks semantic guidance for images with different resolutions and is poor in generalization ability, the accuracy and robustness of saliency target detection in a monitoring system are remarkably improved, and the method is suitable for various complex monitoring scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of monitoring security technology, and in particular to a cooperative saliency target detection method based on scale invariance. BACKGROUND

[0002] In recent years, monitoring systems have become an important tool for ensuring public safety and social stability. Traditional video monitoring systems rely on manpower to analyze events and issue alarms. However, with the large-scale growth of monitoring data, traditional monitoring systems cannot meet the needs of efficiency and real-time performance. In order to free the monitoring system from the demand for manpower, intelligent monitoring systems have emerged. Among them, cooperative saliency target detection technology, as one of the key technologies in intelligent monitoring systems, has received widespread attention by simulating the human visual system to automatically detect and identify salient targets in video streams.

[0003] Current cooperative saliency target detection techniques can be mainly divided into three categories: cooperative saliency target detection using bottom-level consistency, cooperative saliency target detection using depth consistency, and cooperative saliency target detection using salient region refinement. Most of these methods focus on describing multi-scale features to find consistent representations to extract cooperative features. Some methods simply train a network for each resolution data to make cooperative saliency predictions, which brings a large number of redundant training parameters and is time-consuming and laborious. However, even with the attention to multi-scale features, these methods still have prediction biases for monitoring devices at different resolutions, and lack semantic guidance, making it difficult to meet the accuracy and generalization requirements of monitoring devices. SUMMARY

[0004] The technical problem to be solved by the present application is to overcome the deficiencies of the prior art. Specifically, a cooperative saliency target detection method based on scale invariance is provided, as follows: 1) In a first aspect, the present application provides a cooperative saliency target detection method based on scale invariance, with the following technical solutions: An image sequence converted from a video stream in any monitoring system is obtained, and the image sequence is taken as input data to extract multi-scale features corresponding to the input data through a backbone network in a pre-trained model; The multi-scale features are subjected to feature enhancement processing and fusion processing to generate an initial saliency map; The initial saliency map is subjected to recovery processing through a residual refinement structure to generate an accurate saliency map; The accurate saliency map is subjected to optimization processing by applying scale consistency constraints and semantic feature feedback to obtain a target saliency map, and the target saliency map and the corresponding saliency degree are taken as output data for output.

[0005] The method has the following beneficial effects: The multi-scale features extracted by the backbone network can capture different levels of details and semantic information in the image, providing a rich feature basis for subsequent salient object detection. The multi-scale features are enhanced and fused to generate an initial saliency map. This process not only enhances the expression ability of the features, but also enables the model to better understand the global and local features of the target by fusing features of different scales. Further, the initial saliency map is recovered using a residual refinement structure, which can effectively recover the details and edges in the image, making the saliency map more accurate and improving the accuracy of object detection. The precise saliency map is optimized by applying scale consistency constraints and semantic feature feedback, resulting in a target saliency map that is not only highly accurate but also consistent across different resolutions and scales, significantly improving the model's generalization ability and robustness. Therefore, the present application can stably output high-quality target saliency maps and their saliency degrees in various monitoring scenarios, providing an efficient and reliable salient object detection solution for intelligent monitoring systems. It effectively solves the problem of lack of semantic guidance and poor generalization ability of the model for different resolution images in the prior art, and has significant innovation and practicality.

[0006] Based on the above scheme, the present application can be further improved as follows.

[0007] Further, the training process of the pre-trained model is as follows: An arbitrary scale pixel alignment data enhancement method is constructed, and a target multi-scale data set of pixel alignment in the same scene is obtained; A target feature extraction module, a target residual refinement structure, and a scale consistency constraint module are constructed by selecting a target backbone network corresponding to the training requirements, and a loss function is determined; The training parameters of the model are set, and the target multi-scale data set is used as the target input data. After passing through the target backbone network, the target feature extraction module, the target residual refinement structure, and the scale consistency constraint module in sequence, the target output result is obtained. The target input data and the target output result are processed by the loss function, and the training parameters are updated based on the processing result until the processing result meets the preset requirements.

[0008] The beneficial effects of the above further scheme are as follows: By constructing an arbitrary scale pixel alignment data enhancement method, a variety of training data can be generated to enhance the adaptability of the model to different resolution images. Selecting a suitable target backbone network and constructing related modules provides the model with powerful feature extraction and processing capabilities. Determining the loss function and setting the training parameters enables the model to continuously optimize its parameters through back propagation until the preset requirements are met, thereby ensuring that the model can efficiently learn the key features and patterns of saliency object detection during the training process, laying a solid foundation for subsequent detection tasks and improving the accuracy and generalization ability of the model.

[0009] Further, the preset requirements include: The performance of the model is evaluated by the validation dataset, and an evaluation result is obtained; The performance indicators of the model are calculated, and the detection effect of the model is determined based on the performance indicators, and a detection result is obtained; When the evaluation result meets the evaluation requirement and the detection result meets the detection requirement, it is determined that the preset requirement is met.

[0010] The beneficial effects of the above further scheme are: By evaluating the performance of the model through the validation dataset, the performance of the model in actual application can be objectively measured to ensure the reliability and stability of the model. Calculating the performance indicators of the model and determining the detection effect further clarifies the advantages and disadvantages of the model, facilitating targeted optimization and improvement. When the evaluation result and the detection result both meet the corresponding requirements, it is determined that the preset requirement is met, which provides a scientific basis for the final determination of the model and ensures that the model can achieve the expected detection effect in actual application, effectively improving the practicality and application value of the model.

[0011] Further, the process of obtaining the pixel-aligned target multi-scale dataset under the same scene is specifically: In each iteration process, a scale factor is randomly generated, and the original public data is scaled based on the scale factor to obtain a low-resolution image, a medium-resolution image and a high-resolution image; The low-resolution image and the high-resolution image are spliced and cropped according to the scale factor to obtain splicing and cropping processing results, and the splicing and cropping processing results are adjusted so that the adjusted splicing and cropping processing results are consistent with the size of the medium-resolution image.

[0012] The beneficial effects of the above further scheme are: The scale factor is randomly generated in each iteration and the original data is scaled, which can generate images with different resolutions, increase the diversity and richness of the training data. By splicing and cropping the low-resolution and high-resolution images and making them consistent with the size of the medium-resolution image, pixel alignment is achieved, which enables the model to better learn the correlation and differences between images of different resolutions, further enhances the robustness of the model to scale changes, and improves the detection performance and accuracy of the model when processing multi-scale images.

[0013] Further, the determination process of the initial saliency map comprises: The multi-scale features are enhanced and fused by the CBR block in the backbone network; The feature enhancement and fusion processing result is decoded to obtain the initial saliency map.

[0014] The beneficial effects of the above further scheme are: The multi-scale features are enhanced and fused by the CBR block in the backbone network, which can effectively improve the quality and expression ability of the features, so that the model can more accurately capture the saliency information in the image. The initial saliency map is obtained by decoding the processing result, which further optimizes the generation process of the saliency map, so that the initial saliency map can more clearly reflect the position and contour of the salient target in the image, providing a better foundation for subsequent refinement and optimization processing, which helps to improve the quality and accuracy of the final target saliency map.

[0015] Further, the residual refinement structure comprises: Each stage contains a convolution layer, a batch normalization layer, an activation function and a bridge; The bridge comprises 64 convolution layers, batch normalization layers and activation functions.

[0016] The beneficial effects of the above further scheme are: The convolution layer, batch normalization layer, activation function and bridge contained in each stage of the residual refinement structure can perform layer-by-layer refinement processing on the initial saliency map. In particular, the 64 convolution layers, batch normalization layers and activation functions contained in the bridge part further enhance the model's ability to recover detailed information, making the detailed edges in the saliency map more clear and accurate. Through the design of this structure, the model can better recover and retain important details in the image, thereby improving the precision and reliability of salient object detection, making the detection result more in line with actual needs.

[0017] Further, the process of applying scale consistency constraints and semantic feature feedback to optimize the precise saliency map to obtain the target saliency map is: A pre-constructed memory bank storing a significant prediction map of each cooperative group; stitching the significant prediction map of the cooperative group corresponding to the initial saliency map; and performing consistency constraint processing on the stitched result to obtain the target saliency map.

[0018] The beneficial effects of the above further scheme are: The pre-constructed memory bank stores the significant prediction map of the cooperative group, providing a memory mechanism for the model, enabling the model to better utilize historical information for the current detection task. Stitching the significant prediction map of the cooperative group corresponding to the initial saliency map and performing consistency constraint processing can ensure consistency between the saliency prediction results of images of different resolutions, while combining semantic feature feedback to further optimize the quality of the target saliency map. This process not only improves the model's adaptability to scale changes, but also enhances the model's understanding and utilization of semantic information, making the final target saliency map more accurate and robust, significantly improving the model's detection performance and generalization ability in complex scenes.

[0019] 2) In a second aspect, the present application also provides a cooperative saliency target detection system based on scale invariance, and the specific technical scheme is as follows: The acquisition module is configured to acquire an image sequence converted from a video stream in any monitoring system, take the image sequence as input data, and extract multi-scale features corresponding to the input data through a backbone network in a pre-trained model; The initial generation module is configured to perform feature enhancement processing and fusion processing on the multi-scale features to generate an initial saliency map; The accurate generation module is configured to perform recovery processing on the initial saliency map through a residual refinement structure to generate an accurate saliency map; The detection module is configured to apply scale consistency constraint and semantic feature feedback to optimize the accurate saliency map to obtain a target saliency map, and output the target saliency map and the corresponding saliency degree as output data.

[0020] 3) In a third aspect, the present application also provides an electronic device, which includes a processor coupled with a memory, and the memory stores at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement any of the above methods.

[0021] 4) In a fourth aspect, the present application also provides a computer readable storage medium, which stores at least one computer program, which is loaded and executed by a processor to enable a computer to implement any of the above methods.

[0022] It should be noted that the technical solutions of the second aspect to the fourth aspect of the present application and the corresponding possible implementation manners have the beneficial effects as described above for the first aspect and its corresponding possible implementation manners, which will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0023] Other features, objects, and advantages of the application will become more apparent from the following detailed description of non-limiting embodiments with reference to the drawings: Figure 1 A flowchart of a cooperative saliency target detection method based on scale invariance according to an embodiment of the present application; Figure 2 A cutting and splicing manner diagram of a cooperative saliency target detection method based on scale invariance according to an embodiment of the present application; Figure 3 A feature fusion and transmission diagram of a cooperative saliency target detection method based on scale invariance according to an embodiment of the present application; Figure 4 A network structure diagram of a cooperative saliency target detection method based on scale invariance according to an embodiment of the present application; Figure 5 A sampling image diagram of a cooperative saliency target detection method based on scale invariance according to an embodiment of the present application; Figure 6 A non-semantic association and semantic association after comparison diagram of a cooperative saliency target detection method based on scale invariance according to an embodiment of the present application; Figure 7 An effect comparison diagram of a saliency target detection method of a cooperative saliency target detection method based on scale invariance according to an embodiment of the present application; Figure 8 A preset model network structure diagram of a cooperative saliency target detection method based on scale invariance according to an embodiment of the present application; Figure 9 A structural framework diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] To make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0025] As shown in Figure 1 and Figure 8 A cooperative saliency target detection method based on scale invariance according to an embodiment of the present application includes the following steps: S1, obtaining an image sequence converted from a video stream in any monitoring system, taking the image sequence as input data, and extracting multi-scale features corresponding to the input data through a backbone network in a pre-trained model; S2, performing feature enhancement processing and fusion processing on the multi-scale features to generate an initial saliency map; S3, performing recovery processing on the initial saliency map through a residual refinement structure to generate an accurate saliency map; S4, applying scale consistency constraints and semantic feature feedback to optimize the accurate saliency map to obtain a target saliency map, and outputting the target saliency map and the corresponding saliency degree as output data.

[0026] The scale-invariant cooperative saliency target detection method provided by the present application has the following beneficial effects: The multi-scale features extracted by the backbone network can capture different levels of details and semantic information in the image, providing a rich feature basis for subsequent saliency target detection. The multi-scale features are enhanced and fused to generate an initial saliency map, which not only enhances the expression ability of the features, but also makes the model better understand the global and local features of the target by fusing features of different scales. Further, the residual refinement structure is used to recover the initial saliency map, which can effectively recover the detail edges in the image, making the saliency map more accurate and improving the accuracy of target detection. By applying scale consistency constraints and semantic feature feedback to optimize the accurate saliency map, the target saliency map obtained not only has high accuracy, but also maintains consistency at different resolutions and scales, significantly improving the generalization ability and robustness of the model. Therefore, the present application can stably output high-quality target saliency map and saliency degree in various monitoring scenarios, providing an efficient and reliable saliency target detection solution for intelligent monitoring systems, effectively solving the problem of lack of semantic guidance and poor generalization ability of the model for different resolution images in the prior art, and having significant innovation and practicality.

[0027] In another embodiment of the present application, the specific process of S1 is as follows: In processing video streams in a monitoring system, it is necessary to decompose the continuous video stream into a series of discrete image frames, forming an image sequence. This process is usually done by a video decoder, which extracts images from the video stream at a set time interval (e.g. every second) or based on a keyframe strategy. These image frames are then uniformly adjusted to a predefined size to adapt to the input requirements of the pre-trained model, ensuring that the model can effectively process input data of different resolutions. The resized image sequence is sent as input data into the backbone network of the pre-trained model. The backbone network is usually a deep convolutional neural network, such as VGG, ResNet, etc., which extracts features from the input image sequence through a series of convolutional layers, pooling layers, and possibly batch normalization layers and activation functions. In this process, the network generates multiple feature maps of different scales, which capture local details, texture information, and higher-level semantic information in the image. The extraction of multi-scale features allows the model to focus on both small and large targets in the image and their performance at different scales, providing a rich information base for subsequent salient object detection.

[0028] In another embodiment of the present scheme, the specific process of S2 is: The multi-scale features extracted by the backbone network are input into the feature enhancement module, which is composed of multiple convolutional blocks (CBR blocks). Each CBR block contains a convolutional layer, a ReLU activation function, and a batch normalization layer. These convolutional blocks process features of different scales layer by layer, extract local features through convolution operations, introduce nonlinearity through ReLU activation functions to enable the model to learn complex feature representations, and use batch normalization layers to stabilize the training process, speed up convergence, and reduce internal covariate shift. The feature maps processed by the CBR blocks not only enhance the expression ability of the original features, but also further highlight the feature information of salient objects through the nonlinear transformation of the activation function.

[0029] The enhanced multi-scale features are fused. This process is usually achieved through feature concatenation, addition, or more complex feature fusion strategies, with the goal of integrating feature information of different scales to form a comprehensive feature representation. For example, low-level detail features can be combined with high-level semantic features, so that the fused features contain not only rich detail information but also high-level semantic understanding of the target. The fused feature maps can more comprehensively describe the salient objects in the image, providing a more accurate basis for further salient prediction.

[0030] The fused feature map is input into the decoder for decoding operation. The decoder usually adopts upsampling techniques (such as bilinear interpolation, transpose convolution, etc.) to gradually restore the spatial resolution of the feature map, so that it approaches the size of the original input image. During the decoding process, the features of the encoder stage and the features of the decoder stage can be combined through the jump connection to further enhance the details of the features. After the decoding operation, an initial saliency map is generated, which represents the saliency degree of each pixel in the image in the form of a heat map. The area with high saliency corresponds to the possible target object. This initial saliency map provides a basis for subsequent residual refinement and optimization processing, enabling the model to further improve the accuracy and robustness of salient object detection.

[0031] In another embodiment of the present scheme, the specific process of S3 is: The initial saliency map is input into a carefully designed residual refinement network structure. This structure is usually composed of multiple stages, each of which contains a series of convolutional layers, batch normalization layers, and ReLU activation functions that work together to gradually optimize the details and edge information of the saliency map. Between each stage, there is also a bridging module composed of multiple convolutional layers, batch normalization layers, and ReLU activation functions, which further strengthens the transmission and fusion of features to ensure that multi-scale feature information can be fully utilized during the refinement process.

[0032] The encoder part of the refinement network uses non-overlapping max pooling layers for downsampling operations to gradually reduce the spatial resolution of the feature map, thereby extracting higher-level semantic information. At the same time, the decoder part uses upsampling techniques such as bilinear interpolation to gradually restore the resolution of the feature map, so that it eventually matches the size of the original input image. During the entire refinement process, the features of the encoder stage and the features of the decoder stage are effectively combined through the residual connection mechanism. This residual connection not only helps to alleviate the gradient vanishing problem in deep network training, but also ensures that the network can learn more detailed information, especially in the edge and texture parts of the salient object.

[0033] After a series of residual refinement operations, the blurred or inaccurate edges in the initial saliency map are clarified, and the detailed information is enhanced, resulting in a more accurate saliency map. This accurate saliency map can more accurately highlight the salient object in the image while suppressing background noise and other non-target area interference, providing a high-quality basis for subsequent scale consistency constraints and semantic feature feedback optimization, significantly improving the overall performance and reliability of salient object detection.

[0034] In another embodiment of the present scheme, the specific process of S4 is: A scale consistency constraint mechanism is introduced. The core of this mechanism lies in ensuring that the model has consistency in predicting the saliency of the same target at different resolutions. Specifically, by constructing a continuously updated memory bank that stores the saliency prediction map of each collaborative group, the memory bank of the current collaborative group is spliced with the four-stage encoder output at each prediction to enhance the collaborative features. The memory bank is updated according to the new prediction results and provides rough target location clues for the next group prediction, thereby achieving dynamic tracking and optimization of salient targets.

[0035] At the same time, in order to further improve the quality of the saliency map, a semantic feature feedback mechanism is introduced. This mechanism feeds high-level semantic information into the generation process of the saliency map, enabling the model to better understand the semantic meaning of the target and focus more on semantic relevance when generating the saliency map. In specific operations, semantic features extracted by a pre-trained semantic segmentation network or classification network are used to correct and optimize the saliency map. In this way, the model can not only focus on the visual saliency of the target, but also distinguish different types of salient targets by combining semantic information, improving the accuracy and robustness of the saliency map.

[0036] After completing the scale consistency constraint and semantic feature feedback optimization, the resulting target saliency map can more accurately reflect the location and contour of the salient target in the image, while maintaining consistency at different resolutions. Finally, the target saliency map and the corresponding saliency degree are output as output data. The saliency degree is usually represented in the form of a heat map, where high-value areas correspond to salient targets and low-value areas correspond to backgrounds or other non-salient areas. This output form not only visually displays the location of the salient target, but also provides quantitative information about the saliency of the target, facilitating subsequent analysis and application. Through this series of optimization processes, the model can stably output high-quality saliency detection results in complex scenes and under multiple resolution conditions, providing a reliable salient target detection solution for intelligent monitoring systems and other application scenarios.

[0037] Further, the training process of the pre-trained model is: An arbitrary scale pixel alignment data augmentation method is constructed, and a pixel-aligned multi-scale target data set under the same scene is obtained; A target feature extraction module, a target difference refinement structure, and a scale consistency constraint module are constructed by selecting a target backbone network corresponding to the training requirements, and a loss function is determined; The training parameters of the model are set, the target multi-scale data set is taken as target input data, and the target output result is obtained after the target input data sequentially passes through the target backbone network, the target feature extraction module, the target residual refinement structure and the scale consistency constraint module. The target input data and the target output result are processed by the loss function, and the training parameters are updated based on the processing result until the processing result meets the preset requirement.

[0038] Further, the preset requirement includes: The model performance is evaluated by the validation data set, and an evaluation result is obtained. The performance indicators of the model are calculated, and the detection effect of the model is determined based on the performance indicators, and a detection result is obtained. When the evaluation result meets the evaluation requirement and the detection result meets the detection requirement, it is determined that the preset requirement is met.

[0039] Further, the process of obtaining the pixel-aligned target multi-scale data set under the same scene is: In each iteration process, a scale factor is randomly generated, and the original public data is scaled based on the scale factor to obtain a low-resolution image, a medium-resolution image and a high-resolution image. The low-resolution image and the high-resolution image are spliced and cropped according to the scale factor to obtain splicing and cropping processing results, and the splicing and cropping processing results are adjusted so that the adjusted splicing and cropping processing results are consistent with the size of the medium-resolution image.

[0040] Further, the determination process of the initial saliency map includes: The multi-scale features are enhanced and fused by the CBR block in the backbone network. The feature enhancement and fusion processing result is decoded to obtain the initial saliency map.

[0041] Further, the residual refinement structure includes: Each stage contains a convolution layer, a batch normalization layer, an activation function and a bridge. The bridge includes 64 convolution layers, batch normalization layers and activation functions.

[0042] Further, the process of optimizing the accurate saliency map by applying scale consistency constraints and semantic feature feedback to obtain the target saliency map is: A memory bank is pre-constructed, and the saliency prediction map of each cooperative group is stored in the memory bank. The saliency prediction map of the cooperative group corresponding to the initial saliency map is spliced. and consistency constraint processing is performed on the spliced result to obtain the target saliency map.

[0043] In the above embodiments, although the steps are numbered S1, S2, etc., the execution order of S1, S2, etc. can be adjusted according to actual conditions by those skilled in the art, which is also within the protection scope of the present application. It can be understood that in some embodiments, some or all of the above embodiments can be included.

[0044] Embodiment 1, as shown in the following steps: Figures 2 to 4 Step 1: Construct a data enhancement method for pixel alignment of any scale to obtain a multi-scale data set with pixel alignment in the same group and the same scene, and increase the diversity of training data.

[0045] Step 2: Extract multi-scale fusion collaborative salient features from the data set obtained in step 1 to obtain an initial saliency map.

[0046] Step 3: Use the initial saliency map obtained in step 2 to restore details and edges through a residual refinement structure to obtain an accurate salient prediction image.

[0047] Step 4: Based on the accurate prediction image obtained in the above three steps, scale consistency constraint and prediction semantic feature feedback are performed to obtain a target saliency map.

[0048] Construct a data enhancement method for pixel alignment of any scale to obtain a multi-scale data set with pixel alignment in the same group and the same scene, and increase the diversity of training data. First, randomly generate scale factors and in each iteration, scale the original public data set according to the scale factor to obtain three images with different resolutions. In order to ensure that the model is sensitive to the resolution of the training data and not sensitive to the acquisition size, the low-resolution image and the high-resolution image are spliced and cropped according to the scale factor to make their size consistent with that of the medium-resolution image.

[0049] Extract multi-level features from the multi-scale pixel-aligned paired data. First, input the data into any backbone network, here taking VGG network as an example. After each convolutional layer, output 5 initial features with different scales , and use CBR blocks composed of convolutional layers, ReLU activation functions and batch normalization layers to enhance and fuse the features. The process expression is as follows: Pass the enhanced features to obtain fusion features , and decode the fusion features for the first time to obtain an initial saliency prediction image. The process expression is as follows: .

[0050] The initial prediction map is refined by a four-stage encoder-decoder structure to recover the detail edges.

[0051] The four-stage encoder-decoder structure, each stage includes a 3x3 64-channel convolution layer, a batch normalization layer and a ReLU activation function. There is a bridge between each stage of the encoder and the decoder, and the bridge stage includes a 3x3 64-filter convolution layer, a batch normalization layer and a ReLU activation function.

[0052] The accurate prediction image is subjected to scale consistency constraint and prediction class semantic feature feedback to obtain a target saliency map.

[0053] In collaborative saliency, a sustainable memory bank is first constructed for each type to store the saliency prediction map of each collaborative group, and the memory bank is connected with the four-stage decoder, as shown in Figure 5 At each prediction, the memory bank mapping of the same group is spliced with the four-stage encoder output, the memory bank is updated and provides rough target location clues for the next same group prediction. The initial prediction mapping in the memory bank is zero-filled.

[0054] The consistency constraint of collaborative saliency prediction of different resolutions is that the pixel-aligned part of the prediction should be as close as possible, and the collaborative saliency prediction satisfying the size invariance is obtained.

[0055] Taking high-resolution and medium-resolution prediction images as examples, and represent the prediction pixel-aligned part of the pixel, and the consistency constraint expression is as follows: Target detection network training setting: In order to train and evaluate the proposed collaborative saliency target detection method, the present application is implemented using Pytorch and trained on a single NVIDIA GTX 4090 Ti GPU. The network is trained for 30 epochs, with a batch size of 4. All experiments use the stochastic gradient descent optimizer, with a weight decay of 0.0005, an initial learning rate of 0.001, and a momentum value of 0.9. During the training process, pre-training and initialization are performed on the ImageNet dataset. During the testing process, the input image is resized and adjusted to a resolution of 320x320.

[0056] In order to verify the effectiveness of the synergistic salient object detection model proposed in the present application, the salient object detection dataset DUTS-TE benchmark dataset and the synergistic salient object detection dataset CoCA benchmark dataset image are selected as the experimental objects to verify the effectiveness of resolution desensitization and synergistic association respectively.

[0057] In Figure 5 For example, the benchmark dataset DUTS-TE is down-sampled by 2 and 4 times, denoted as DUTS-2D and DUTS-4D.

[0058] Figure 5 Two groups of images are shown in the figure, the first column to the third column are a group, and the fourth column to the sixth column are a group, each group is a DUTS, DUTS-2D, DUTS-4D image, and the resolution is 96x96, 48x48 and 24x24 respectively. The third row to the seventh row show the change process of the specific image effect, wherein the semantic understanding is gradually enhanced with the increase of the data enhancement and consistency constraint steps, and the structure and details are gradually clear; and the final output binary segmentation image presents resolution desensitization, further verifying the effectiveness of the model.

[0059] In Figure 6 For example, the alarm clock group image is selected from the benchmark dataset CoCA. The non-semantic association and semantic association are set and compared. Two groups of images are shown in the figure, and the third column and the fourth column show the change process of the specific image effect. The third column shows only two classes of circles without semantic association.

[0060] This embodiment shows the detection capability of the synergistic salient object detection method, and verifies the robustness and simplicity of the method in complex scenes and different acquisition conditions.

[0061] The effect comparison chart of the present algorithm and the existing algorithm in processing some salient object detection methods is shown in Figure 7 .

[0062] From the figure, it can be seen that the existing salient object detection method is limited by cross-domain difference and model design, and it is difficult to adapt to the resolution change environment. The synergistic salient object detection method based on scale invariance proposed in the present application can better desensitize the resolution difference, solve the problem of confusion of the semantic type recognition of the model caused by the resolution cross-domain difference, simulate consistent salient recognition, and thus achieve better target detection effect.

[0063] Compared with the traditional target detection model, the scale invariance based collaborative saliency target detection method has the advantages of strong generalization, strong adaptability and resolution desensitization of the acquisition device. The traditional method usually relies on manually designed features and heuristic rules to extract visual features, infer the salient region, lacks the correlation information between images, and has limitations in feature expression, and often fails to face some complex scene objects. The application of deep learning method can effectively improve the generalization of the collaborative saliency target detection method, but there is still a lack in cross-domain semantic cognition, and it cannot fully utilize the spatial structure information. The present application only needs to add two simple steps of data enhancement branch and consistency constraint to implicitly and explicitly guide the model to pay attention to the semantic information between image groups, ignore the scale information such as resolution and size in training and testing, and does not need to make additional training on acquisition devices with different resolutions in the test scene, which greatly reduces the dependence of the model on training images and the computational complexity. At the same time, the method guides the shallow texture information of the feature extraction part back to the deep features through multiple cross-layer feature refinement modules, strengthens the semantic expression ability of the deep information, and at the same time retains the detailed information of the shallow layer. Therefore, this method is not only simple to train and easy to implement, but also has the advantages of both semantics and details, and can be widely used in different device platforms and complex scenes, and has good cross-scene generalization ability and robustness.

[0064] Example 3, video preprocessing process: Video frame extraction: Extract video frames from the video stream at fixed time intervals (such as extracting one frame per second) or key frame extraction method, and convert the video stream into an image sequence.

[0065] Image size adjustment: Adjust the extracted video frames to the uniform size required by the model input (such as 320x320 pixels) to adapt to the input requirements of the model.

[0066] Saliency target detection process: Input model: input the preprocessed video frames into the trained model.

[0067] Feature extraction and fusion: the model extracts multi-scale features through the backbone network, and performs feature enhancement and fusion to generate an initial saliency map.

[0068] Residual refinement: use the encoder-decoder structure to refine the initial saliency map, restore the details and edges, and generate an accurate saliency prediction image.

[0069] Scale consistency constraint: apply scale consistency constraint and semantic feature feedback to optimize the target saliency map.

[0070] Output saliency map: the model outputs the target saliency map, which represents the saliency degree of each pixel in the video frame.

[0071] Behavioral anomaly detection process: Saliency map analysis: Analyze the generated target saliency map to identify areas with high saliency, which may be people with abnormal behavior.

[0072] Behavioral feature extraction: Combined with behavioral analysis algorithms (such as optical flow and deep learning behavior recognition models), behavioral features are extracted from salient areas to determine whether abnormal behavior exists.

[0073] Abnormal behavior recognition: Using a pre-trained behavior recognition model, the extracted behavior features are classified to determine whether the behavior is abnormal (such as running, fighting, falling, etc.).

[0074] Result output and alarm process: Detection result output: The location and behavior type of the detected person with abnormal behavior are output to the monitoring system interface.

[0075] Alarm mechanism: If abnormal behavior is detected, the alarm mechanism is triggered to notify monitoring personnel or automatically record the event for further processing.

[0076] It should be noted that the beneficial effects of the scale-invariant collaborative salient target detection system provided by the above embodiment are the same as the beneficial effects of the scale-invariant collaborative salient target detection method provided by the above embodiment, and will not be repeated here. In addition, when implementing its functions, the system provided by the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided by the above embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0077] like Figure 9 As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 320, which is coupled to a memory 310. The memory 310 stores at least one computer program 330. The at least one computer program 330 is loaded and executed by the processor 320 to enable the electronic device 300 to implement any of the above methods. Specifically: The electronic device 300 can have a large difference due to different configurations or performances, and can include one or more processors 320 (Central Processing Units, CPUs) and one or more memories 310, wherein the one or more memories 310 store at least one computer program 330, the at least one computer program 330 is loaded and executed by the one or more processors 320, so that the electronic device 300 implements the scale-invariant based collaborative saliency target detection method provided in the above embodiments. Of course, the electronic device 300 can also have a wired or wireless network interface, a keyboard, and an input and output interface, and the like, so as to perform input and output, and the electronic device 300 can also include other components for realizing device functions, which are not described here.

[0078] The computer readable storage medium of the embodiment of the present application stores at least one computer program, and the at least one computer program is loaded and executed by the processor, so that the computer implements any one of the above methods.

[0079] Optionally, the computer readable storage medium can be a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0080] In the exemplary embodiments, a computer program product or computer program is also provided, which includes computer instructions stored in a computer readable storage medium. The processor of the electronic device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the electronic device executes any one of the above methods.

[0081] It should be noted that the terms "first", "second" in the specification and claims of the present application are used to distinguish similar objects, and represent a specific order or sequence. The order of use of similar objects can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described.

[0082] Those skilled in the art know that the present application can be implemented as a system, a method, or a computer program product, therefore, the present disclosure can be embodied in the form of a complete hardware, a complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, which is generally referred to as "circuitry", "module" or "system". In addition, in some embodiments, the present application can also be embodied in the form of a computer program product in one or more computer readable media, which contains computer readable program codes.

[0083] Any combination of one or more computer readable medium can be employed. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0084] Although the embodiments of the present application have been shown and described above, it should be understood that the above-described embodiments are exemplary, and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-described embodiments within the scope of the present application.

Claims

1. A scale-invariant collaborative salient object detection method, characterized in that: include: Obtain an image sequence converted from a video stream in any monitoring system, use the image sequence as input data, and extract multi-scale features corresponding to the input data through the backbone network in the pre-trained model; Performing feature enhancement and fusion processing on the multi-scale features to generate an initial saliency map; Restoring the initial saliency map by refining the residual structure to generate an accurate saliency map; The precise saliency map is optimized by applying scale consistency constraints and semantic feature feedback to obtain a target saliency map, and the target saliency map and the corresponding saliency degree are output as output data.

2. The scale-invariant collaborative salient object detection method according to claim 1, characterized in that: The training process of the pre-trained model is: Construct a data augmentation method for pixel alignment at any scale and obtain a target multi-scale dataset with pixel alignment in the same scene; Select the target backbone network corresponding to the training requirements, build the target feature extraction module, target difference refinement structure and scale consistency constraint module, and determine the loss function; The training parameters of the model are set, and the target multi-scale dataset is used as the target input data. After passing through the target backbone network, the target feature extraction module, the target difference refinement structure and the scale consistency constraint module in sequence, the target output result is obtained. The target input data and the target output result are processed by the loss function, and the training parameters are updated based on the processing results by backpropagation until the processing results meet the preset requirements.

3. The scale-invariant collaborative salient object detection method according to claim 2, wherein: The preset requirements include: Evaluate the model performance using the validation data set and obtain the evaluation results; Calculating the performance index of the model, and determining the detection effect of the model based on the performance index to obtain a detection result; When the evaluation result meets the evaluation requirement and the detection result meets the detection requirement, it is determined that the preset requirement is met.

4. The scale-invariant collaborative salient object detection method according to claim 2, wherein: The specific process of obtaining the target multi-scale dataset with pixel alignment in the same scene is as follows: In each round of iteration, a scale factor is randomly generated, and the original public data is scaled based on the scale factor to obtain a low-resolution image, a medium-resolution image, and a high-resolution image; The low-resolution image and the high-resolution image are stitched and cropped according to the scale factor to obtain a stitching and cropping result, and the stitching and cropping result is adjusted so that the adjusted stitching and cropping result is consistent with the size of the medium-resolution image.

5. The scale-invariant collaborative salient object detection method according to claim 2, wherein: The process of determining the initial saliency map includes: Performing feature enhancement and fusion processing on the multi-scale features through the CBR block in the backbone network; The feature enhancement and fusion processing results are decoded to obtain the initial saliency map.

6. The scale-invariant collaborative salient object detection method according to claim 2, wherein: The residual refinement structure includes: Each stage consists of a convolutional layer, batch normalization layer, activation function, and bridge; The bridge consists of 64 convolutional layers, batch normalization layers, and activation functions.

7. The scale-invariant collaborative salient object detection method according to claim 5, wherein: Applying scale consistency constraints and semantic feature feedback to optimize the precise saliency map to obtain the target saliency map is as follows: A pre-built memory bank storing a significant prediction map of each collaborative group; splicing the saliency prediction maps of the collaborative group corresponding to the initial saliency map; The concatenated results are subjected to consistency constraint processing to obtain the target saliency map.

8. A scale-invariant collaborative salient object detection system, characterized in that: include: The acquisition module is used to: acquire an image sequence converted from a video stream in any monitoring system, use the image sequence as input data, and extract multi-scale features corresponding to the input data through the backbone network in the pre-trained model; The initial generation module is used to perform feature enhancement and fusion processing on the multi-scale features to generate an initial saliency map; The precise generation module is used to: restore the initial saliency map through the residual refinement structure to generate a precise saliency map; The detection module is used to apply scale consistency constraints and semantic feature feedback to optimize the precise saliency map to obtain a target saliency map, and output the target saliency map and the corresponding saliency degree as output data.

9. An electronic device, characterized in that: The electronic device includes a processor coupled to a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to enable a computer to implement the method according to any one of claims 1 to 7.

Images