Salient object detection method and device, computer equipment and storage medium
By using the first pyramid network and the second pyramid network to perform multi-scale analysis on high-resolution and low-resolution images in significant object detection in the financial insurance field, the high-resolution image detection misjudgment and downsampling are solved by using the first pyramid network and the second pyramid network to perform multi-scale analysis on high-resolution image and share parameters. The problem of reduced accuracy achieves high-quality significant object detection.
Patent Information
- Application Number
- CN202411178106.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-12-06
AI Technical Summary
The existing significant object detection technology is prone to misjudgment of false positive examples on high-resolution images, and downsampling high-resolution images to low resolution will lead to a decrease in detection accuracy, affecting the risk assessment and decision-making accuracy in the financial and insurance field. sex.
The first pyramid network is used to perform multi-scale analysis of high-resolution images, and the second pyramid network is used to perform multi-scale analysis of low-resolution images. The two share parameters and fuse features to generate a significance map for detection.
It improves the accuracy of detection of significant objects, captures saliency characteristics at different scales, integrates local details and global saliency cues, generates high-quality saliency maps, and accurately identify and locates significant objects.
Smart Images

Figure CN119091208A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computer equipment and storage medium for detecting a salient object. Background Art
[0002] In the financial insurance sector, tasks such as automated risk assessment and loss assessment rely on accurate analysis of image data, which may include photos of damaged vehicles or property in insurance claims. Salient object detection aims to detect the mask of the most salient objects in an image and is a research area that has received much attention in computer vision.
[0003] When existing salient object detection technologies directly perform salient object detection on high-resolution images, although they can obtain more accurate edges, they are also prone to misjudging more false positives. Such misjudgments are particularly disadvantageous in the financial and insurance fields, and may lead to incorrect risk assessments and inaccurate claims decisions. In addition, many existing salient object detection networks are more suitable for images with lower resolutions. If high-resolution images are downsampled to low resolution for detection, the accuracy of detection will be reduced. This approach is also undesirable in the financial and insurance fields, because the loss of details may lead to missed detection or misjudgment of key damage or important objects, thereby affecting the accuracy of risk assessment and decision-making. Summary of the invention
[0004] The purpose of the embodiments of the present application is to provide a method, apparatus, computer device and storage medium for detecting a salient object, so as to improve the accuracy of detecting a salient object.
[0005] In order to solve the above technical problems, the present application embodiment provides a salient object detection method, which adopts the following technical solution:
[0006] Acquire a high-resolution image to be detected and its corresponding low-resolution image;
[0007] Inputting the high-resolution image into a first pyramid network in a salient object detection model to perform multi-scale analysis on the high-resolution image to obtain high-resolution image features, and inputting the low-resolution image into a second pyramid network in the salient object detection model to perform multi-scale analysis on the low-resolution image to obtain low-resolution image features; the first pyramid network and the second pyramid network share parameters;
[0008] Performing feature fusion on the high-resolution image features and the low-resolution image features to obtain a saliency map;
[0009] Perform salient object detection according to the saliency map to obtain a salient object detection result.
[0010] In order to solve the above technical problems, the embodiment of the present application further provides a salient object detection device, which adopts the following technical solution:
[0011] An image acquisition module, used to acquire a high-resolution image to be detected and its corresponding low-resolution image;
[0012] A feature extraction module, configured to input the high-resolution image into a first pyramid network in a salient object detection model to perform multi-scale analysis on the high-resolution image to obtain high-resolution image features, and input the low-resolution image into a second pyramid network in the salient object detection model to perform multi-scale analysis on the low-resolution image to obtain low-resolution image features; the first pyramid network and the second pyramid network share parameters;
[0013] A feature fusion module, used for fusing the high-resolution image features and the low-resolution image features to obtain a saliency map;
[0014] The object detection module is used to perform salient object detection according to the saliency map to obtain a salient object detection result.
[0015] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:
[0016] Acquire a high-resolution image to be detected and its corresponding low-resolution image;
[0017] Inputting the high-resolution image into a first pyramid network in a salient object detection model to perform multi-scale analysis on the high-resolution image to obtain high-resolution image features, and inputting the low-resolution image into a second pyramid network in the salient object detection model to perform multi-scale analysis on the low-resolution image to obtain low-resolution image features; the first pyramid network and the second pyramid network share parameters;
[0018] Performing feature fusion on the high-resolution image features and the low-resolution image features to obtain a saliency map;
[0019] Perform salient object detection according to the saliency map to obtain a salient object detection result.
[0020] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:
[0021] Acquire a high-resolution image to be detected and its corresponding low-resolution image;
[0022] Inputting the high-resolution image into a first pyramid network in a salient object detection model to perform multi-scale analysis on the high-resolution image to obtain high-resolution image features, and inputting the low-resolution image into a second pyramid network in the salient object detection model to perform multi-scale analysis on the low-resolution image to obtain low-resolution image features; the first pyramid network and the second pyramid network share parameters;
[0023] Performing feature fusion on the high-resolution image features and the low-resolution image features to obtain a saliency map;
[0024] Perform salient object detection according to the saliency map to obtain a salient object detection result.
[0025] Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects: obtaining a high-resolution image to be detected and its corresponding low-resolution image, the high-resolution and low-resolution images ensure the diversity of input data, so that the model can utilize more image information; inputting the high-resolution image into the first pyramid network in the salient object detection model to perform multi-scale analysis on the high-resolution image to obtain high-resolution image features, and inputting the low-resolution image into the second pyramid network in the salient object detection model to perform multi-scale analysis on the low-resolution image to obtain low-resolution image features, and the multi-scale analysis can capture salient features of different scales; the first pyramid network and the second pyramid network share parameters, so that the feature extraction process of the high-resolution and low-resolution images remains consistent, and the effect of subsequent feature fusion is enhanced; the high-resolution image features and the low-resolution image features are feature fused to obtain a saliency map, and the captured local details and global saliency clues can be integrated to generate a high-quality saliency map; performing saliency object detection based on the saliency map can accurately identify and locate salient objects, and significantly improve the accuracy of detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0027] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;
[0028] Figure 2 is a flowchart of an embodiment of a significant object detection method according to the present application;
[0029] Figure 3is a schematic structural diagram of an embodiment of a salient object detection device according to the present application;
[0030] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION
[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.
[0032] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0033] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.
[0034] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables.
[0035] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.
[0036] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers and desktop computers, etc.
[0037] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .
[0038] It should be noted that the salient object detection method provided in the embodiment of the present application is generally executed by a server, and accordingly, the salient object detection device is generally arranged in the server.
[0039] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.
[0040] Continue to refer Figure 2 , shows a flow chart of an embodiment of a salient object detection method according to the present application. The salient object detection method comprises the following steps:
[0041] Step S201: obtaining a high-resolution image to be detected and its corresponding low-resolution image.
[0042] In this embodiment, the electronic device (eg, Figure 1 The server shown in the figure) can communicate with the terminal device through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.
[0043] Specifically, a high-resolution image to be detected and its corresponding low-resolution image are obtained. The low-resolution image can be obtained by downsampling the high-resolution image. The high-resolution image can provide more detailed local information, and the low-resolution image can provide more comprehensive global information. The combination of high-resolution and low-resolution images provides a rich information basis for subsequent saliency detection.
[0044] Step S202: Input the high-resolution image into the first pyramid network in the salient object detection model to perform multi-scale analysis on the high-resolution image to obtain high-resolution image features, and input the low-resolution image into the second pyramid network in the salient object detection model to perform multi-scale analysis on the low-resolution image to obtain low-resolution image features. The first pyramid network and the second pyramid network share parameters.
[0045] Specifically, the present application performs salient object detection through a salient object detection model. The salient object detection model can be obtained by combining multiple neural networks. It can include a first pyramid network and a second pyramid network. The pyramid network is a multi-scale feature extraction method, which is mainly used to capture information of different scales in an image and can process the image into representations of different scales, so as to better capture local and global features in the image. In the salient object detection task, the pyramid network can better identify salient areas in the image through multi-scale feature extraction.
[0046] The high-resolution image is input into the first pyramid network in the salient object detection model to perform multi-scale analysis on the high-resolution image to obtain high-resolution image features. High-resolution image features are feature representations obtained by processing and analyzing high-resolution images. These features contain detailed information of the image, including fine edges, textures, and features of small-sized objects. High-resolution image features can provide richer detailed information, which is very important for accurately locating salient targets. In the first pyramid network, high-resolution images are subjected to a series of convolution operations and multi-scale analysis to extract feature maps of different scales. These feature maps can describe local details and global structures in the image.
[0047] The low-resolution image is input into the second pyramid network in the salient object detection model to perform multi-scale analysis on the low-resolution image and obtain low-resolution image features. Low-resolution image features are feature representations obtained by processing and analyzing low-resolution images. These features mainly reflect the global information and large-scale structure of the image. Low-resolution image features can summarize the overall layout of the image and the distribution of major objects, reducing the computational overhead when processing high-resolution images. In the second pyramid network, the low-resolution image is also subjected to a series of convolution operations and multi-scale analysis to extract feature maps of different scales. These feature maps can describe the global information and large-scale features in the image.
[0048] The first pyramid network and the second pyramid network share parameters, or they share weights. That is, the first pyramid network and the second pyramid network are actually the same network with the same weights and structure, but the input image resolution is different. The parameters (weights) of the two networks are shared and will not change due to the different input image resolutions, ensuring that the responses of the two networks are consistent when processing images of different resolutions, ensuring the consistency of feature extraction, and improving the effect of feature fusion.
[0049] Step S203: performing feature fusion on the high-resolution image features and the low-resolution image features to obtain a saliency map.
[0050] Specifically, the high-resolution image features and low-resolution image features are fused to obtain a saliency map. The role of feature fusion is to effectively integrate the features of images with different resolutions. Low-resolution image representation captures global saliency clues, and high-resolution image representation captures local details. Feature fusion can effectively integrate this information; by combining multi-scale features, the model can better understand the relationship between salient objects and their context, thereby improving the quality of saliency maps.
[0051] A saliency map is an image that represents salient regions in an image. Each pixel value in the saliency map represents the probability or importance of that location belonging to a salient object.
[0052] Step S204: performing salient object detection according to the saliency map to obtain a salient object detection result.
[0053] Specifically, according to the generated saliency map, the salient object detection model can accurately identify and locate the salient objects in the image and obtain salient object detection results. The salient object detection results can be the bounding box or mask of the salient objects in the image.
[0054] In this embodiment, a high-resolution image to be detected and its corresponding low-resolution image are obtained, and the high-resolution and low-resolution images ensure the diversity of input data, so that the model can use more image information; the high-resolution image is input into the first pyramid network in the salient object detection model to perform multi-scale analysis on the high-resolution image to obtain high-resolution image features, and the low-resolution image is input into the second pyramid network in the salient object detection model to perform multi-scale analysis on the low-resolution image to obtain low-resolution image features, and the multi-scale analysis can capture salient features of different scales; the first pyramid network and the second pyramid network share parameters, so that the feature extraction process of the high-resolution and low-resolution images remains consistent, and the effect of subsequent feature fusion is enhanced; the high-resolution image features and the low-resolution image features are feature fused to obtain a saliency map, which can integrate the captured local details and global saliency clues, thereby generating a high-quality saliency map; salient object detection based on the saliency map can accurately identify and locate salient objects, and significantly improve the accuracy of detection.
[0055] Furthermore, the above-mentioned step of performing multi-scale analysis on the high-resolution image to obtain high-resolution image features may include: iteratively downsampling the high-resolution image through a first pyramid network to obtain a first image pyramid structure of the high-resolution image; for the first image of each scale in the first image pyramid structure, performing feature extraction through a convolutional neural network, and generating high-resolution image features based on the features extracted at multiple scales.
[0056] Specifically, the high-resolution image is iteratively downsampled through the first pyramid network, that is, iteratively downsampled multiple times to generate multiple image representations of different scales. The first image obtained by downsampling at each scale can constitute the first image pyramid structure of the high-resolution image. The downsampling operation reduces the image resolution layer by layer, and each layer represents a coarser version of the original image. In this way, the first pyramid network can capture different scale information in the image.
[0057] For the first image of each scale in the first image pyramid, a convolutional neural network (CNN) is used to extract features and generate corresponding feature maps, which contain the key information of the image at that scale. Convolutional neural networks have powerful feature learning capabilities and can extract rich feature representations from images.
[0058] The features extracted at multiple scales are integrated to generate the final high-resolution image features. This multi-scale feature fusion method can capture information at different scales, ensuring that both details and global information can be considered when detecting salient objects.
[0059] It should be noted that the processing performed by the second image pyramid on the low-resolution image is the same as the processing performed by the first image pyramid on the high-resolution image.
[0060] In this embodiment, the high-resolution image is iteratively downsampled through the first pyramid network, and the image is observed from perspectives of different resolutions, so that the content in the image can be fully understood and the first image pyramid structure can be obtained; for the first image of each scale in the first image pyramid structure, feature extraction is performed through a convolutional neural network to extract rich and representative feature maps, covering various details and structural information of the image; high-resolution image features are generated based on the features extracted at multiple scales, ensuring that the model can pay attention to the details of the image while taking into account the overall global information when detecting significant objects, thereby ensuring the accuracy of subsequent detection.
[0061] Furthermore, the above step S203 may include: calculating the difference between the high-resolution image features and the low-resolution image features to obtain a feature difference value; adjusting the high-resolution image features and the low-resolution image features for consistency according to the feature difference value to obtain a high-resolution adjusted feature and a low-resolution adjusted feature, respectively; and performing feature fusion on the high-resolution adjusted feature and the low-resolution adjusted feature to obtain a saliency map.
[0062] Specifically, the difference between the high-resolution image features and the low-resolution image features is calculated to obtain feature difference values, which can be measured by various metrics (such as L2 distance, etc.). The feature difference value reflects the inconsistency of the two resolution images in the feature space.
[0063] According to the feature difference value, the high-resolution image features and the low-resolution image features are adjusted for consistency. The adjustment can be performed through linear transformation or nonlinear transformation, and high-resolution adjustment features and low-resolution adjustment features are generated. The purpose of the adjustment is to make the distribution of the two features in the same feature space more consistent, and the content expression more consistent, so as to better integrate them.
[0064] The high-resolution adjusted features and the low-resolution adjusted features are fused. The fusion can be achieved by weighted averaging, cascading or through a specially designed fusion network, and a saliency map is generated.
[0065] In this embodiment, the difference between the high-resolution image features and the low-resolution image features is calculated to obtain a feature difference value, thereby identifying the inconsistency between the two in the feature space; the high-resolution image features and the low-resolution image features are adjusted for consistency according to the feature difference value to obtain high-resolution adjusted features and low-resolution adjusted features, respectively, which are more consistent in the same feature space and eliminate the deviation between image features of different resolutions; the high-resolution adjusted features and the low-resolution adjusted features are feature fused to obtain a more accurate saliency map.
[0066] Furthermore, the above-mentioned step of calculating the difference between the high-resolution image features and the low-resolution image features to obtain the feature difference value may include: obtaining a preset difference evaluation method, the difference evaluation method including distance difference, statistical difference and similarity metric difference; and according to the determined difference evaluation method, performing difference calculation on the high-resolution image features and the low-resolution image features to obtain the feature difference value.
[0067] Specifically, the present application calculates the difference between the high-resolution image features and the low-resolution image features according to a preset difference evaluation method to obtain a feature difference value. The difference evaluation method includes three main types: distance difference, statistical difference and similarity measurement difference.
[0068] Among them, the distance difference usually uses L1 distance or L2 distance, which measures the difference by calculating the Euclidean distance or Manhattan distance between two feature vectors. Statistical difference evaluates the difference by comparing the statistics of the two feature distributions, such as the difference in statistics such as mean and variance. Similarity metric difference uses measurement methods such as cosine similarity or mutual information to evaluate the difference by calculating the degree of similarity.
[0069] In this embodiment, the preset difference evaluation methods include distance difference, statistical difference and similarity measurement difference, which can accurately evaluate the differences between high-resolution image features and low-resolution image features from multiple dimensions, and calculate the feature difference values according to the difference evaluation methods, thereby ensuring the accuracy of the feature difference values.
[0070] Furthermore, the above-mentioned step of adjusting the consistency of the high-resolution image features and the low-resolution image features according to the feature difference value to obtain the high-resolution adjustment feature and the low-resolution adjustment feature respectively may include: calculating the first weighting coefficient of the high-resolution image feature and the second weighting coefficient of the low-resolution image feature according to the feature difference value; calculating the high-resolution image feature and the low-resolution image feature based on the first weighting coefficient and the second weighting coefficient to obtain the high-resolution adjustment feature and the low-resolution adjustment feature respectively; or, combining the feature difference value, the high-resolution image feature and the low-resolution image feature to obtain a perceptron input combination; inputting the perceptron input combination into a first multi-layer perceptron to obtain the high-resolution adjustment feature, and inputting the perceptron input combination into a second multi-layer perceptron to obtain the low-resolution adjustment feature.
[0071] Specifically, a first weighting coefficient of the high-resolution image feature and a second weighting coefficient of the low-resolution image feature are calculated according to the feature difference value. In one embodiment, the feature difference value can be mapped to the first weighting coefficient and the second weighting coefficient by linear mapping. In one embodiment:
[0072]
[0073] α low =1-α high (2)
[0074] Among them, d is the characteristic difference value, e is the natural constant, α high is the first weighting coefficient, α low is the second weighting coefficient.
[0075] Based on the first weighting coefficient and the second weighting coefficient, the high-resolution image feature and the low-resolution image feature are calculated to obtain the high-resolution adjustment feature and the low-resolution adjustment feature respectively. In one embodiment, there are:
[0076] F′ high =α high ·F high +(1-α high )·F low (3)
[0077] F′ low =α low ·F low +(1-α low )·F high (4)
[0078] Among them, α high is the first weighting coefficient, α low is the second weighting coefficient. high is the high-resolution image feature, Flow is the low-resolution image feature, F′ high To adjust the features for high resolution, F′ low Adjust features for low resolution.
[0079] The above is a linear adjustment method for image features; the following is a nonlinear adjustment method for image features.
[0080] The feature difference value d, high-resolution image feature F high and low-resolution image features F low Combine them to get the perceptron input combination (F high ,F low ,d). The perceptron input combination is input into the first multilayer perceptron (MLP) to obtain high-resolution adjustment features, and the perceptron input combination is input into the second multilayer perceptron (MLP) to obtain low-resolution adjustment features. Through the deep learning processing of the multilayer perceptron, the input combination is subjected to nonlinear transformation and feature extraction to generate high-resolution adjustment features and low-resolution adjustment features. In one embodiment, there are:
[0081] F′ high =MLP high (F high ,F low ,d) (5)
[0082] F′ low =MLP low (F high ,F low ,d) (6)
[0083] Among them, d is the characteristic difference value, F high is the high-resolution image feature, F low is the low-resolution image feature, F′ high To adjust the features for high resolution, F l ' ow Adjust features for low resolution, MLP high () represents the first multi-layer perceptron, MLP, that processes high-resolution images low () represents the second multi-layer perceptron that processes low-resolution images.
[0084] In this embodiment, the weighting coefficient can be calculated by the feature difference value, and the high-resolution image features and the low-resolution image features can be linearly adjusted according to the weighting coefficient. The perceptron input combination consisting of the feature difference value, the high-resolution image features and the low-resolution image features can also be nonlinearly adjusted through a multi-layer perceptron, thereby enriching the implementation methods of consistency adjustment.
[0085] Furthermore, the step of fusing the high-resolution adjustment features and the low-resolution adjustment features to obtain a saliency map may include: inputting the high-resolution adjustment features and the low-resolution adjustment features into a feature fusion network in the salient object detection model, and obtaining context information of each scale through the feature fusion network; based on the high-resolution adjustment features, the low-resolution adjustment features and the context information of each scale, performing feature fusion through the feature fusion network to obtain a comprehensive feature map; and inputting the comprehensive feature map into a saliency map generation network in the salient object detection model to obtain a saliency map.
[0086] Specifically, the high-resolution adjusted features and low-resolution adjusted features are input into the feature fusion network in the salient object detection model, and the context information of each scale is obtained through the feature fusion network. The context information includes the context of the feature map of each scale and the relationship between the salient object and its context.
[0087] The feature fusion network integrates the high-resolution adjusted features and the low-resolution adjusted features, while considering the role of contextual information, thereby improving the effect of feature fusion and generating a comprehensive feature map. The comprehensive feature map is input into the saliency map generation network in the salient object detection model to obtain a saliency map for salient object detection.
[0088] In this embodiment, the high-resolution adjustment features and the low-resolution adjustment features are combined with the context information of each scale, so that the generated comprehensive feature map contains a variety of information; a saliency map is generated based on the comprehensive feature map, so that the saliency map can more accurately reflect the position and form of salient objects, thereby ensuring accurate detection of salient objects.
[0089] Furthermore, after the above step S203, the following steps may be further included: performing edge refinement processing on the saliency map; performing noise reduction processing on the saliency map after the edge refinement processing; and performing salient object detection based on the saliency map after the noise reduction processing to obtain a salient object detection result.
[0090] Specifically, edge refinement processing can be performed on the saliency map to enhance the edge details of the salient objects in the saliency map through edge detection algorithms, such as Canny edge detection or Sobel operator. These algorithms can identify pixel areas with strong changes in the image, thereby outlining clearer object boundaries, making the edges of salient objects sharper and reducing the difficulty of recognition caused by blurred edges.
[0091] Since additional noise may be introduced during edge refinement, the saliency map after edge refinement can be subjected to noise reduction. For example, image smoothing techniques such as Gaussian blur or median filtering can be used to remove noise from the saliency map, smoothing the image while retaining the clear edges of salient objects. This process helps reduce false detections and false positives, and improves the accuracy of salient object detection.
[0092] Based on the saliency map after noise reduction, salient object detection is performed. For example, methods such as threshold segmentation and connected component analysis can be applied, or the saliency map can be input into a salient object detection network in a salient object detection model to obtain salient object detection results.
[0093] In this embodiment, the saliency map is subjected to edge refinement processing to enhance the edge details of the salient objects, making them clearer and more accurate in the saliency map, thereby reducing the detection difficulty and error caused by edge blur; the saliency map after edge refinement processing is subjected to noise reduction processing to improve the signal-to-noise ratio of salient object detection, reduce the incidence of false detection and false positive examples, further improve the quality of the saliency map, and greatly improve the accuracy of salient object detection.
[0094] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned high-resolution images and low-resolution images, the above-mentioned high-resolution images and low-resolution images can also be stored in a node of a blockchain.
[0095] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.
[0096] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0097] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0098] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0099] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0100] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a salient object detection device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0101] like Figure 3 As shown, the salient object detection device 300 described in this embodiment includes: an image acquisition module 301, a feature extraction module 302, a feature fusion module 303 and an object detection module 304, wherein:
[0102] The image acquisition module 301 is used to acquire a high-resolution image to be detected and its corresponding low-resolution image.
[0103] The feature extraction module 302 is used to input the high-resolution image into the first pyramid network in the salient object detection model to perform multi-scale analysis on the high-resolution image to obtain high-resolution image features, and input the low-resolution image into the second pyramid network in the salient object detection model to perform multi-scale analysis on the low-resolution image to obtain low-resolution image features. The first pyramid network and the second pyramid network share parameters.
[0104] The feature fusion module 303 is used to fuse the high-resolution image features and the low-resolution image features to obtain a saliency map.
[0105] The object detection module 304 is used to perform salient object detection according to the saliency map to obtain a salient object detection result.
[0106] In this embodiment, a high-resolution image to be detected and its corresponding low-resolution image are obtained, and the high-resolution and low-resolution images ensure the diversity of input data, so that the model can use more image information; the high-resolution image is input into the first pyramid network in the salient object detection model to perform multi-scale analysis on the high-resolution image to obtain high-resolution image features, and the low-resolution image is input into the second pyramid network in the salient object detection model to perform multi-scale analysis on the low-resolution image to obtain low-resolution image features, and the multi-scale analysis can capture salient features of different scales; the first pyramid network and the second pyramid network share parameters, so that the feature extraction process of the high-resolution and low-resolution images remains consistent, and the effect of subsequent feature fusion is enhanced; the high-resolution image features and the low-resolution image features are feature fused to obtain a saliency map, which can integrate the captured local details and global saliency clues, thereby generating a high-quality saliency map; salient object detection based on the saliency map can accurately identify and locate salient objects, and significantly improve the accuracy of detection.
[0107] In some optional implementations of this embodiment, the feature extraction module 302 may include: a downsampling submodule and a feature extraction submodule, wherein:
[0108] The downsampling submodule is used to iteratively downsample the high-resolution image through a first pyramid network to obtain a first image pyramid structure of the high-resolution image.
[0109] The feature extraction submodule is used to extract features of the first image at each scale in the first image pyramid structure through a convolutional neural network, and generate high-resolution image features based on the features extracted at multiple scales.
[0110] In this embodiment, the high-resolution image is iteratively downsampled through the first pyramid network, and the image is observed from perspectives of different resolutions, so that the content in the image can be fully understood and the first image pyramid structure can be obtained; for the first image of each scale in the first image pyramid structure, feature extraction is performed through a convolutional neural network to extract rich and representative feature maps, covering various details and structural information of the image; high-resolution image features are generated based on the features extracted at multiple scales, ensuring that the model can pay attention to the details of the image while taking into account the overall global information when detecting significant objects, thereby ensuring the accuracy of subsequent detection.
[0111] In some optional implementations of this embodiment, the feature fusion module 303 may include: a difference calculation submodule, a feature adjustment submodule and a feature fusion submodule, wherein:
[0112] The difference calculation submodule is used to calculate the difference between the high-resolution image features and the low-resolution image features to obtain a feature difference value.
[0113] The feature adjustment submodule is used to adjust the consistency of the high-resolution image features and the low-resolution image features according to the feature difference values to obtain the high-resolution adjusted features and the low-resolution adjusted features respectively.
[0114] The feature fusion submodule is used to fuse the high-resolution adjusted features and the low-resolution adjusted features to obtain a saliency map.
[0115] In this embodiment, the difference between the high-resolution image features and the low-resolution image features is calculated to obtain a feature difference value, thereby identifying the inconsistency between the two in the feature space; the high-resolution image features and the low-resolution image features are adjusted for consistency according to the feature difference value to obtain high-resolution adjusted features and low-resolution adjusted features, respectively, which are more consistent in the same feature space and eliminate the deviation between image features of different resolutions; the high-resolution adjusted features and the low-resolution adjusted features are feature fused to obtain a more accurate saliency map.
[0116] In some optional implementations of this embodiment, the difference calculation submodule may include: a mode acquisition unit and a difference calculation unit, wherein:
[0117] The method acquisition unit is used to acquire a preset difference evaluation method, where the difference evaluation method includes distance difference, statistical difference and similarity measurement difference.
[0118] The difference calculation unit is used to calculate the difference between the high-resolution image feature and the low-resolution image feature according to the determined difference evaluation method to obtain a feature difference value.
[0119] In this embodiment, the preset difference evaluation methods include distance difference, statistical difference and similarity measurement difference, which can accurately evaluate the differences between high-resolution image features and low-resolution image features from multiple dimensions, and calculate the feature difference values according to the difference evaluation methods, thereby ensuring the accuracy of the feature difference values.
[0120] In some optional implementations of this embodiment, the feature adjustment submodule may include: a coefficient calculation unit, a feature calculation unit, a combination unit, and a combination input unit, wherein:
[0121] The coefficient calculation unit is used to calculate a first weighting coefficient of the high-resolution image feature and a second weighting coefficient of the low-resolution image feature according to the feature difference value.
[0122] The feature calculation unit is used to calculate the high-resolution image feature and the low-resolution image feature based on the first weighting coefficient and the second weighting coefficient to obtain the high-resolution adjustment feature and the low-resolution adjustment feature respectively.
[0123] The combining unit is used to combine the feature difference value, the high-resolution image feature and the low-resolution image feature to obtain a perceptron input combination.
[0124] The combined input unit is used to input the combined perceptron inputs into the first multi-layer perceptron to obtain high-resolution adjustment features, and to input the combined perceptron inputs into the second multi-layer perceptron to obtain low-resolution adjustment features.
[0125] In this embodiment, the weighting coefficient can be calculated by the feature difference value, and the high-resolution image features and the low-resolution image features can be linearly adjusted according to the weighting coefficient. The perceptron input combination consisting of the feature difference value, the high-resolution image features and the low-resolution image features can also be nonlinearly adjusted through a multi-layer perceptron, thereby enriching the implementation methods of consistency adjustment.
[0126] In some optional implementations of this embodiment, the feature fusion submodule may include: a feature input unit, a feature fusion unit, and a graph generation unit, wherein:
[0127] The feature input unit is used to input the high-resolution adjusted features and the low-resolution adjusted features into the feature fusion network in the salient object detection model, and obtain the context information of each scale through the feature fusion network.
[0128] The feature fusion unit is used to perform feature fusion through a feature fusion network based on high-resolution adjustment features, low-resolution adjustment features and context information of each scale to obtain a comprehensive feature map.
[0129] The map generation unit is used to input the comprehensive feature map into the saliency map generation network in the salient object detection model to obtain a saliency map.
[0130] In this embodiment, the high-resolution adjustment features and the low-resolution adjustment features are combined with the context information of each scale, so that the generated comprehensive feature map contains a variety of information; a saliency map is generated based on the comprehensive feature map, so that the saliency map can more accurately reflect the position and form of salient objects, thereby ensuring accurate detection of salient objects.
[0131] In some optional implementations of this embodiment, the salient object detection device 300 may further include: an edge processing module and a noise reduction processing module, wherein:
[0132] The edge processing module is used to perform edge refinement processing on the saliency map.
[0133] The denoising processing module is used to perform denoising on the saliency map after edge refinement processing.
[0134] The object detection module 304 is further configured to perform salient object detection based on the salient map after the noise reduction process, and obtain a salient object detection result.
[0135] In this embodiment, the saliency map is subjected to edge refinement processing to enhance the edge details of the salient objects, making them clearer and more accurate in the saliency map, thereby reducing the detection difficulty and error caused by edge blur; the saliency map after edge refinement processing is subjected to noise reduction processing to improve the signal-to-noise ratio of salient object detection, reduce the incidence of false detection and false positive examples, further improve the quality of the saliency map, and greatly improve the accuracy of salient object detection.
[0136] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.
[0137] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.
[0138] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.
[0139] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of a significant object detection method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.
[0140] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the salient object detection method.
[0141] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.
[0142] The computer device provided in this embodiment can execute the above-mentioned salient object detection method. Here, the salient object detection method can be the salient object detection method of each of the above-mentioned embodiments.
[0143] In this embodiment, a high-resolution image to be detected and its corresponding low-resolution image are obtained, and the high-resolution and low-resolution images ensure the diversity of input data, so that the model can use more image information; the high-resolution image is input into the first pyramid network in the salient object detection model to perform multi-scale analysis on the high-resolution image to obtain high-resolution image features, and the low-resolution image is input into the second pyramid network in the salient object detection model to perform multi-scale analysis on the low-resolution image to obtain low-resolution image features, and the multi-scale analysis can capture salient features of different scales; the first pyramid network and the second pyramid network share parameters, so that the feature extraction process of the high-resolution and low-resolution images remains consistent, and the effect of subsequent feature fusion is enhanced; the high-resolution image features and the low-resolution image features are feature fused to obtain a saliency map, which can integrate the captured local details and global saliency clues, thereby generating a high-quality saliency map; salient object detection based on the saliency map can accurately identify and locate salient objects, and significantly improve the accuracy of detection.
[0144] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the significant object detection method as described above.
[0145] In this embodiment, a high-resolution image to be detected and its corresponding low-resolution image are obtained, and the high-resolution and low-resolution images ensure the diversity of input data, so that the model can use more image information; the high-resolution image is input into the first pyramid network in the salient object detection model to perform multi-scale analysis on the high-resolution image to obtain high-resolution image features, and the low-resolution image is input into the second pyramid network in the salient object detection model to perform multi-scale analysis on the low-resolution image to obtain low-resolution image features, and the multi-scale analysis can capture salient features of different scales; the first pyramid network and the second pyramid network share parameters, so that the feature extraction process of the high-resolution and low-resolution images remains consistent, and the effect of subsequent feature fusion is enhanced; the high-resolution image features and the low-resolution image features are feature fused to obtain a saliency map, which can integrate the captured local details and global saliency clues, thereby generating a high-quality saliency map; salient object detection based on the saliency map can accurately identify and locate salient objects, and significantly improve the accuracy of detection.
[0146] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0147] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.
Claims
1. A salient object detection method, It is characterized in that The steps include: Acquire a high-resolution image to be detected and its corresponding low-resolution image; Inputting the high-resolution image into a first pyramid network in a salient object detection model to perform multi-scale analysis on the high-resolution image to obtain high-resolution image features, and inputting the low-resolution image into a second pyramid network in the salient object detection model to perform multi-scale analysis on the low-resolution image to obtain low-resolution image features; The first pyramid network and the second pyramid network share parameters; Performing feature fusion on the high-resolution image features and the low-resolution image features to obtain a saliency map; Perform salient object detection according to the saliency map to obtain a salient object detection result.
2. The salient object detection method according to claim 1, It is characterized in that The step of performing multi-scale analysis on the high-resolution image to obtain high-resolution image features comprises: Iteratively downsample the high-resolution image through the first pyramid network to obtain a first image pyramid structure of the high-resolution image; For the first image at each scale in the first image pyramid structure, feature extraction is performed through a convolutional neural network, and high-resolution image features are generated according to the features extracted at multiple scales.
3. The salient object detection method according to claim 1, It is characterized in that The step of fusing the high-resolution image features and the low-resolution image features to obtain a saliency map comprises: Calculating the difference between the high-resolution image feature and the low-resolution image feature to obtain a feature difference value; Performing consistency adjustment on the high-resolution image feature and the low-resolution image feature according to the feature difference value to obtain a high-resolution adjustment feature and a low-resolution adjustment feature respectively; The high-resolution adjustment features and the low-resolution adjustment features are subjected to feature fusion to obtain a saliency map.
4. The salient object detection method according to claim 3, It is characterized in that The step of calculating the difference between the high-resolution image feature and the low-resolution image feature to obtain a feature difference value comprises: Obtaining a preset difference evaluation method, wherein the difference evaluation method includes distance difference, statistical difference and similarity measurement difference; According to the determined difference evaluation method, the difference between the high-resolution image feature and the low-resolution image feature is calculated to obtain a feature difference value.
5. The salient object detection method according to claim 3, It is characterized in that The step of adjusting the consistency of the high-resolution image feature and the low-resolution image feature according to the feature difference value to obtain a high-resolution adjustment feature and a low-resolution adjustment feature respectively comprises: Calculating a first weighting coefficient of the high-resolution image feature and a second weighting coefficient of the low-resolution image feature according to the feature difference value; Based on the first weighting coefficient and the second weighting coefficient, the high-resolution image feature and the low-resolution image feature are calculated to obtain a high-resolution adjustment feature and a low-resolution adjustment feature respectively; or, Combining the feature difference value, the high-resolution image feature, and the low-resolution image feature to obtain a sensor input combination; The perceptron input combination is input into a first multi-layer perceptron to obtain a high-resolution adjusted feature, and the perceptron input combination is input into a second multi-layer perceptron to obtain a low-resolution adjusted feature.
6. The salient object detection method according to claim 3, It is characterized in that The step of fusing the high-resolution adjustment features and the low-resolution adjustment features to obtain a saliency map comprises: Inputting the high-resolution adjusted features and the low-resolution adjusted features into a feature fusion network in the salient object detection model, and acquiring context information of each scale through the feature fusion network; Based on the high-resolution adjustment features, the low-resolution adjustment features and context information of each scale, feature fusion is performed through the feature fusion network to obtain a comprehensive feature map; The comprehensive feature map is input into a saliency map generation network in the salient object detection model to obtain a saliency map.
7. The salient object detection method according to claim 1, It is characterized in that After the step of fusing the high-resolution image features and the low-resolution image features to obtain a saliency map, the method further includes: Performing edge refinement processing on the saliency map; Perform noise reduction on the saliency map after edge refinement; Based on the saliency map after the noise reduction process, the step of performing salient object detection according to the saliency map to obtain a salient object detection result is performed.
8. A salient object detection device, It is characterized in that include: An image acquisition module, used to acquire a high-resolution image to be detected and its corresponding low-resolution image; a feature extraction module, configured to input the high-resolution image into a first pyramid network in a salient object detection model to perform multi-scale analysis on the high-resolution image to obtain high-resolution image features, and input the low-resolution image into a second pyramid network in the salient object detection model to perform multi-scale analysis on the low-resolution image to obtain low-resolution image features; The first pyramid network and the second pyramid network share parameters; A feature fusion module, used for fusing the high-resolution image features and the low-resolution image features to obtain a saliency map; The object detection module is used to perform salient object detection according to the saliency map to obtain a salient object detection result.
9. A computer device, It is characterized in that The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the salient object detection method according to any one of claims 1 to 7 when executing the computer-readable instructions.
10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the salient object detection method according to any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
High-resolution saliency target detection method based on graffiti supervision
CN114332490A
