A method, device, equipment and storage medium for detecting significant objects in an image

Abstract

The present invention is applicable to the field of image processing technology, and provides a method, device, equipment and storage medium for detecting significant objects in an image. The method comprises: first acquiring an image to be detected, then detecting the image to be detected by using a significant detection model to obtain all significant objects in the image to be detected, then calculating the significance score of each significant object respectively, and finally sorting all significant objects by significance according to the significance score, and determining the significant object with the largest significance score as the target significant object in the image to be detected, thereby improving the recognition speed and recognition accuracy of significant objects in multi-scene images.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to a method, device, equipment and storage medium for detecting significant objects in an image. Background Art

[0002] With the rapid development of information technology, the continuous upgrading and application of cameras and portable cameras on mobile electronic devices have made it a normal practice for people to use images to record and share information. Images have become the main data resource in the information society, which has led to an increasing demand for data processing. The increasing demand for data processing inevitably requires improving information processing efficiency. For an image, people are often only interested in the areas that best represent the image content and arouse the most interest of users. These areas are called salient areas. Therefore, how to automatically obtain salient areas on images is becoming more and more important.

[0003] In recent years, convolutional neural networks have been widely used in the field of machine vision. They have the ability to automatically extract image features. In particular, the use of fully convolutional neural networks has greatly improved the performance of salient target detection. However, the current saliency detection methods based on deep learning neural networks generally perform image transformations on images containing salient objects, such as scaling and feature extraction. In these image transformation processes, it is easy to cause pollution to small-scale salient objects, resulting in missed detection of small-target explicit objects. In addition, the existing saliency detection types mainly focus on specific objects in specific fields such as humans, animals, and plants, and lack the recognition of richer salient objects in daily life scenes. At the same time, in the case of multiple salient targets, there is a lack of comparative analysis between multiple salient targets, resulting in unclear saliency problems. Summary of the invention

[0004] The purpose of the present invention is to provide a method, device, equipment and storage medium for detecting salient objects in an image, aiming to solve the problem that the prior art cannot provide an effective method for detecting salient objects in an image, resulting in slow detection speed and low recognition efficiency of salient objects in various scene images.

[0005] In one aspect, the present invention provides a method for detecting salient objects in an image, the method comprising the following steps:

[0006] Acquire the image to be detected;

[0007] Detecting the image to be detected by using a saliency detection model to obtain all salient objects in the image to be detected;

[0008] Calculating a saliency score for each of the salient objects respectively;

[0009] All the significant objects are sorted according to the significance scores, and the significant object with the largest significance score is determined as the target significant object in the image to be detected.

[0010] Preferably, the step of respectively calculating the saliency score of each salient object comprises:

[0011] Calculating a first saliency score of each of the salient objects respectively, and calculating a first saliency mean according to the obtained first saliency scores of all the salient objects;

[0012] Determining a significance threshold according to the first significance mean;

[0013] Cutting the contour area of ​​each of the salient objects according to the saliency threshold;

[0014] Calculating a second saliency mean value according to all the salient objects after clipping;

[0015] The second significance score of each of the significant objects is calculated according to the calculated second significance mean value and a preset proportionality coefficient, and the obtained second significance score is determined as the significance score.

[0016] Preferably, the method further comprises:

[0017] The saliency detection model is obtained by training a preset neural network to learn the mapping relationship between an image and a salient object in the image through preset training data, wherein the training data includes an image data set without salient objects and an image data set containing salient objects.

[0018] Further preferably, the preset neural network is a U-Net network and / or a classic saliency detection network.

[0019] Further preferably, the U-Net network includes a downsampling layer, the downsampling layer includes a skip connection module, and the skip connection module includes a depthwise separable convolutional layer and a maximum pooling layer.

[0020] In another aspect, the present invention provides a device for detecting salient objects in an image, the device comprising:

[0021] A detection image acquisition unit, used to acquire an image to be detected;

[0022] A salient object obtaining unit, configured to detect the image to be detected by using a salient detection model, and obtain all salient objects in the image to be detected;

[0023] a saliency score calculation unit, configured to calculate a saliency score of each of the salient objects; and

[0024] A saliency ranking unit is used to saliency-rank all the salient objects according to the saliency scores, and determine the salient object with the largest saliency score as the target salient object in the image to be detected.

[0025] Preferably, the significance score calculation unit comprises:

[0026] a first mean value calculation unit, configured to calculate a first significance score of each of the significant objects respectively, and calculate a first significance mean value according to the obtained first significance scores of all the significant objects;

[0027] a threshold determination unit, configured to determine a significance threshold according to the first significance mean;

[0028] A region clipping unit, used for clipping the contour region of each of the salient objects according to the saliency threshold;

[0029] a second mean value calculation unit, configured to calculate a second saliency mean value according to all the cropped salient objects; and

[0030] A score calculation unit is used to calculate the second significance score of each of the significant objects according to the calculated second significance mean value and a preset proportionality coefficient, and determine the obtained second significance score as the significance score.

[0031] Preferably, the device further comprises:

[0032] The detection model training unit is used to train a preset neural network to learn the mapping relationship between an image and a salient object in the image through preset training data to obtain the salient detection model, wherein the training data includes an image data set without salient objects and an image data set containing salient objects.

[0033] On the other hand, the present invention also provides an image processing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps described in the above-mentioned method for detecting significant objects in an image when executing the computer program.

[0034] On the other hand, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps described in the above-mentioned method for detecting significant objects in an image are implemented.

[0035] The present invention first obtains an image to be detected, and then detects the image to be detected through a saliency detection model to obtain all salient objects in the image to be detected, then calculates the saliency score of each salient object respectively, and finally sorts all salient objects according to the saliency score, and determines the salient object with the largest saliency score after sorting as the target salient object in the image to be detected, thereby improving the recognition speed and recognition accuracy of salient objects in multi-scene images. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flowchart of an implementation method for detecting significant objects in an image provided by the first embodiment of the present invention;

[0037] Figure 2 is a flowchart of a method for detecting significant objects in an image provided by Embodiment 2 of the present invention;

[0038] Figure 3 is a flowchart of an implementation method for detecting significant objects in an image provided by Embodiment 3 of the present invention;

[0039] Figure 4 is a schematic diagram of a skip connection module in a method for detecting salient objects in an image provided by Embodiment 3 of the present invention;

[0040] Figure 5 is a schematic diagram of the structure of a salient object detection device for an image provided by a fourth embodiment of the present invention;

[0041] Figure 6 is a schematic diagram of a preferred structure of a salient object detection device for an image provided by Embodiment 5 of the present invention; and

[0042] Figure 7 It is a structural schematic diagram of an image processing device provided in Embodiment 6 of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0044] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments:

[0045] Embodiment 1:

[0046] Figure 1 The implementation process of the image salient object detection method provided by the first embodiment of the present invention is shown. For the convenience of description, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0047] In step S101, an image to be detected is obtained.

[0048] The embodiment of the present invention is applicable to image processing devices such as image display and acquisition. In the embodiment of the present invention, the image to be detected can be taken in real time by a mobile electronic device with a camera, or can be obtained from a preset storage location (e.g., cloud storage space) by an image processing device.

[0049] In step S102, the image to be detected is detected by using a saliency detection model to obtain all salient objects in the image to be detected.

[0050] In an embodiment of the present invention, the acquired image to be detected is detected by a saliency detection model to obtain all salient objects on the image to be detected, and relevant attribute information (for example, contour area, position information, color, etc.) of each salient object is obtained.

[0051] When detecting an image to be detected by a saliency detection model, preferably, feature extraction and image segmentation are performed on the input image to be detected by a U-Net network and / or a classic saliency detection network to obtain all salient objects on the image to be detected, thereby improving the saliency and accuracy of saliency detection.

[0052] In step S103, the saliency score of each salient object is calculated respectively.

[0053] In the embodiment of the present invention, the saliency score of each salient object is calculated according to the relevant attribute information of the salient object, and the saliency score value is at the pixel level.

[0054] Preferably, the calculation of the saliency score of each salient object is achieved by the following steps:

[0055] (1) Calculate the first saliency score of each salient object respectively, and calculate the first saliency mean according to the obtained first saliency scores of all salient objects.

[0056] In the embodiment of the present invention, the first saliency score of each salient object is calculated according to the relevant attribute information of the salient object, and the first saliency mean is calculated according to the obtained first saliency scores of all salient objects.

[0057] As an example, the relative relationship between the size of each salient object and the image to be detected, as well as the color difference between each salient object and the image to be detected can be determined according to the contour area, position information, and color of the salient object, and then the first saliency score of each salient object is determined according to the relative relationship between the sizes and the color difference, and finally the average of the first saliency scores of all salient objects is calculated to obtain the first saliency mean.

[0058] (2) Determine the significance threshold based on the first significance mean.

[0059] In the embodiment of the present invention, the significance threshold is determined according to the first significance mean, and the significance threshold is smaller than the first significance mean. For example, if the first significance mean is M0, then the significance threshold M1=0.2×M0.

[0060] (3) The contour area of ​​each salient object is cropped according to the saliency threshold.

[0061] In the embodiment of the present invention, the contour region of each salient object is clipped according to the saliency threshold, and only the contour regions of the salient objects with a value higher than the current saliency threshold are retained.

[0062] (4) Calculate the second saliency mean based on all the salient objects after cropping.

[0063] In the embodiment of the present invention, the second saliency score of each salient object is recalculated according to the cropped contour region retained by each salient object, and the second saliency mean is calculated according to the obtained second saliency score.

[0064] (5) Calculate the second significance score of each significant object according to the calculated second significance mean value and the preset proportional coefficient, and determine the obtained second significance score as the significance score.

[0065] In the embodiment of the present invention, the proportional coefficient is determined according to the area size of the significant object. The larger the area, the larger the proportional coefficient. The second significance score of each significant object is obtained by multiplying the calculated second significance mean by the proportional coefficient of each significant object.

[0066] The above steps (1) to (5) are used to calculate the saliency score of each salient object, thereby clarifying the priority of salient objects by comparative analysis between multiple salient objects in an image.

[0067] In step S104, all the significant objects are sorted according to their significance scores, and the significant object with the largest significance score after sorting is determined as the target significant object in the image to be detected.

[0068] In an embodiment of the present invention, all significant objects are sorted by significance according to the significance score. The significant objects can be sorted in ascending / descending order according to the score, wherein the significant object with the largest significance score is the most significant target object in the current image to be detected, and the target object is determined as the target significant object in the image to be detected.

[0069] In an embodiment of the present invention, all salient objects in the image to be detected are detected by a saliency detection model, and the saliency score of each salient object is calculated respectively. All salient objects are saliency sorted according to the saliency score to obtain the most salient target object in the image to be detected, thereby improving the recognition speed and recognition accuracy of salient objects in multi-scene images.

[0070] Embodiment 2:

[0071] Figure 2 The implementation process of the image salient object detection method provided by the second embodiment of the present invention is shown. For the convenience of explanation, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0072] In step S201, a preset neural network is trained to learn the mapping relationship between an image and a salient object in the image using preset training data to obtain a saliency detection model, wherein the training data includes an image dataset without salient objects and an image dataset with salient objects.

[0073] The embodiment of the present invention is applicable to image processing devices such as image display and acquisition. In the embodiment of the present invention, the training data composed of the image data set without significant objects and the image data set containing significant objects can adopt a standard data set, such as the Imagenet data set, or a customized image training data set, wherein the significant objects in the image of the image data set containing significant objects can be one or more. When training the preset neural network, the fine contours of the significant objects on the image are firstly manually annotated for the image data set containing significant objects, but the annotated significant objects are not divided into specific categories, that is, all significant objects are classified into one category, and other non-significant areas on the image are classified into another category, so as to obtain an image pair of an image and a significant result, and then the mapping relationship between the image and the significant object in the image is learned and trained for the preset neural network by using these annotated image data sets and the image data sets without significant objects, so as to obtain a significant detection model, thereby improving the training speed and training effect of the network.

[0074] Preferably, the preset neural network is a U-Net network and / or a classical saliency detection network, thereby improving the saliency and accuracy of the neural network for saliency detection.

[0075] Further preferably, the U-Net network is an improved U-Net network including a skip connection module in the downsampling layer, and the skip connection module includes a depthwise separable convolution layer (SepConv for short) and a maximum pooling layer (Max Pooling), so as to avoid excessive loss of details of small target explicit objects in the image during the downsampling process of the saliency detection model and reduce the probability of missed detection of small target explicit objects.

[0076] In step S202, an image to be detected is obtained.

[0077] In step S203, the image to be detected is detected by using a saliency detection model to obtain all salient objects in the image to be detected.

[0078] In step S204, the saliency score of each salient object is calculated respectively.

[0079] In step S205, all the significant objects are sorted according to their significance scores, and the significant object with the largest significance score after sorting is determined as the target significant object in the image to be detected.

[0080] In the embodiment of the present invention, the specific implementation of steps S202 to S205 can refer to the description of steps S101 to S104 in the first embodiment, which will not be repeated here.

[0081] In an embodiment of the present invention, a preset neural network is first trained using training data consisting of an image data set without salient objects and an image data set containing salient objects to obtain a salient detection model, and then all salient objects in the image to be detected are detected using the salient detection model, and a salient score of each salient object is calculated respectively, and all salient objects are ranked according to the salient score to obtain the most salient target object in the image to be detected, thereby improving the recognition speed and recognition accuracy of salient objects in multi-scene images.

[0082] Embodiment three:

[0083] Figure 3 The implementation process of the method for detecting salient objects in an image provided by the third embodiment of the present invention is shown. For the convenience of explanation, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0084] In step S301, an image to be detected is obtained.

[0085] The embodiment of the present invention is applicable to image processing devices such as image display and acquisition. In the embodiment of the present invention, the image to be detected can be taken in real time by a mobile electronic device with a camera, or can be obtained from a preset storage location (e.g., cloud storage space) by an image processing device.

[0086] In step S302, the image to be detected is detected by an improved U-Net network to obtain all significant objects in the image to be detected, wherein the downsampling of the improved U-Net network includes a skip connection module.

[0087] In an embodiment of the present invention, the saliency detection model is an improved U-Net network including a skip connection module in the downsampling layer. The improved U-Net network is used to perform feature extraction and image segmentation on the input image to be detected, so as to obtain all salient objects on the image to be detected, and obtain relevant attribute information (for example, contour area, position information, and color, etc.) of each salient object, wherein the skip connection module does not change the overall U-Net structure, and there is a skip connection module in the downsampling process of each layer of the U-Net network U-shaped structure.

[0088] Preferably, the jump connection module included in the downsampling structure of the improved U-Net network includes a depthwise separable convolution layer (Depthwise Separable Convolution, referred to as SepConv) and a maximum pooling layer (Max Pooling), so as to avoid excessive loss of details of small target explicit objects in the image during the downsampling process and reduce the probability of missed detection of small target explicit objects.

[0089] More preferably, Figure 4 The structure of the jump connection module is shown, which includes 2 SepConv layers, a leaky rectified linear unit (Leaky ReLU) function and a Max Pooling layer. The jump connection module implemented by the Max Pooling layer compresses the features before downsampling and directly transmits them to the feature extraction module after downsampling, retaining more original features before downsampling, thereby further avoiding excessive loss of details of small target explicit objects in the image during the downsampling process and reducing the probability of missed detection of small target explicit objects. As an example, after feature a is input into the jump connection module, it is subjected to a depth-separable convolution through 2 layers of SepConv layers to obtain feature b, and at the same time, the Max Pooling layer performs a maximum pooling operation on feature a to obtain feature c. Finally, the jump connection module fuses features b and c to obtain and output feature d.

[0090] In step S303, the saliency score of each salient object is calculated respectively.

[0091] In step S304, all the significant objects are sorted according to their significance scores, and the significant object with the largest significance score after sorting is determined as the target significant object in the image to be detected.

[0092] In the embodiment of the present invention, the specific implementation of step S303-step S304 can refer to the description of step S103-step S104 in embodiment 1, which will not be repeated here.

[0093] In an embodiment of the present invention, all salient objects in the image to be detected are detected by an improved U-Net network that includes a skip connection module in downsampling, and the saliency score of each salient object is calculated respectively. All salient objects are saliency sorted according to the saliency score to obtain the most salient target object in the image to be detected, thereby improving the recognition speed and recognition accuracy of salient objects in multi-scene images.

[0094] Embodiment 4:

[0095] Figure 5 The structure of the apparatus for detecting salient objects in an image provided by the fourth embodiment of the present invention is shown. For the sake of convenience, only the part related to the embodiment of the present invention is shown, which includes:

[0096] The detection image acquisition unit 51 is used to acquire the image to be detected.

[0097] The embodiment of the present invention is applicable to image processing devices such as image display and acquisition. In the embodiment of the present invention, the image to be detected can be taken in real time by a mobile electronic device with a camera, or can be obtained from a preset storage location (e.g., cloud storage space) by an image processing device.

[0098] The salient object acquisition unit 52 is used to detect the image to be detected by using a salient detection model to obtain all salient objects in the image to be detected.

[0099] In an embodiment of the present invention, the acquired image to be detected is detected by a saliency detection model to obtain all salient objects on the image to be detected, and relevant attribute information (for example, contour area, position information, color, etc.) of each salient object is obtained.

[0100] The salient score calculation unit 53 is used to calculate the salient score of each salient object respectively.

[0101] In the embodiment of the present invention, the saliency score of each salient object is calculated according to the relevant attribute information of the salient object, and the saliency score value is at the pixel level.

[0102] The saliency ranking unit 54 is used to saliency-rank all salient objects according to the saliency scores, and determine the salient object with the largest saliency score after ranking as the target salient object in the image to be detected.

[0103] In an embodiment of the present invention, all significant objects are sorted by significance according to the significance score. The significant objects can be sorted in ascending / descending order according to the score, wherein the significant object with the largest significance score is the most significant target object in the current image to be detected, and the target object is determined as the target significant object in the image to be detected.

[0104] In the embodiment of the present invention, each unit of the image salient object detection device can be implemented by a corresponding hardware or software unit. Each unit can be an independent software or hardware unit or integrated into one software or hardware unit, which is not intended to limit the present invention.

[0105] Embodiment five:

[0106] Figure 6 The structure of the apparatus for detecting salient objects in an image provided by the fifth embodiment of the present invention is shown. For the sake of convenience, only the part related to the embodiment of the present invention is shown, including:

[0107] The detection model training unit 61 is used to train a preset neural network to learn the mapping relationship between an image and a salient object in the image through preset training data to obtain a salient detection model, wherein the training data includes an image data set without salient objects and an image data set containing salient objects.

[0108] The embodiment of the present invention is applicable to image processing devices such as image display and acquisition. In the embodiment of the present invention, the training data composed of the image data set without significant objects and the image data set containing significant objects can adopt a standard data set, such as the Imagenet data set, or a customized image training data set, wherein the significant objects in the image of the image data set containing significant objects can be one or more. When training the preset neural network, the fine contours of the significant objects on the image are firstly manually annotated for the image data set containing significant objects, but the annotated significant objects are not divided into specific categories, that is, all significant objects are classified into one category, and other non-significant areas on the image are classified into another category, so as to obtain an image pair of an image and a significant result, and then the mapping relationship between the image and the significant object in the image is learned and trained for the preset neural network by using these annotated image data sets and the image data sets without significant objects, so as to obtain a significant detection model, thereby improving the training speed and training effect of the network.

[0109] The detection image acquisition unit 62 is used to acquire the image to be detected.

[0110] In the embodiment of the present invention, the image to be detected may be captured in real time by a mobile electronic device with a camera, or may be acquired from a preset storage location (eg, a cloud storage space) by an image processing device.

[0111] The salient object acquisition unit 63 is used to detect the image to be detected by using a salient detection model to obtain all salient objects in the image to be detected.

[0112] In an embodiment of the present invention, the acquired image to be detected is detected by a saliency detection model to obtain all salient objects on the image to be detected, and relevant attribute information (for example, contour area, position information, color, etc.) of each salient object is obtained.

[0113] When detecting an image to be detected by a saliency detection model, preferably, feature extraction and image segmentation are performed on the input image to be detected by a U-Net network and / or a classic saliency detection network, thereby improving the saliency and accuracy of saliency detection.

[0114] In another preferred embodiment, the saliency detection model is an improved U-Net network including a skip connection module in the downsampling layer, wherein the skip connection module includes a depthwise separable convolution layer (SepConv) and a maximum pooling layer (Max Pooling), and the skip connection module does not change the overall U-Net structure. At the same time, there is a skip connection module in the downsampling process of each layer of the U-net network U-shaped structure, thereby avoiding excessive loss of details of small target explicit objects in the image during the downsampling process and reducing the probability of missed detection of small target explicit objects.

[0115] Further preferably, the jump connection module includes 2 SepConv layers, a leaky rectified linear unit (Leaky ReLU) function and a Max Pooling layer. The jump connection module implemented by the MaxPooling layer compresses the features before downsampling and directly transmits them to the feature extraction module after downsampling, retaining more original features before downsampling, thereby further avoiding excessive loss of details of small target explicit objects in the image during the downsampling process, and reducing the probability of missed detection of small target explicit objects. As an example, after feature a is input into the jump connection module, it is subjected to a depth-separable convolution through 2 layers of SepConv layers to obtain feature b, and at the same time, the Max Pooling layer performs a maximum pooling operation on feature a to obtain feature c. Finally, the jump connection module performs feature fusion on features b and c to obtain and output feature d.

[0116] The salient score calculation unit 64 is used to calculate the salient score of each salient object.

[0117] In the embodiment of the present invention, the saliency score of each salient object is calculated according to the relevant attribute information of the salient object, and the saliency score value is at the pixel level.

[0118] The saliency ranking unit 65 is used to saliency-rank all salient objects according to the saliency scores, and determine the salient object with the largest saliency score after ranking as the target salient object in the image to be detected.

[0119] In an embodiment of the present invention, all significant objects are sorted by significance according to the significance score. The significant objects can be sorted in ascending / descending order according to the score, wherein the significant object with the largest significance score is the most significant target object in the current image to be detected, and the target object is determined as the target significant object in the image to be detected.

[0120] Preferably, the significant score calculation unit 64 includes:

[0121] The first mean value calculation unit 641 is used to calculate the first significance score of each significant object respectively, and calculate the first significance mean value according to the obtained first significance scores of all significant objects.

[0122] In the embodiment of the present invention, the first saliency score of each salient object is calculated according to the relevant attribute information of the salient object, and the first saliency mean is calculated according to the obtained first saliency scores of all salient objects.

[0123] As an example, the relative relationship between the size of each salient object and the image to be detected, as well as the color difference between each salient object and the image to be detected can be determined according to the contour area, position information, and color of the salient object, and then the first saliency score of each salient object is determined according to the relative relationship between the sizes and the color difference, and finally the average of the first saliency scores of all salient objects is calculated to obtain the first saliency mean.

[0124] The threshold determination unit 642 is used to determine a significance threshold according to the first significance mean value.

[0125] In the embodiment of the present invention, the significance threshold is determined according to the first significance mean, and the significance threshold is smaller than the first significance mean. For example, if the first significance mean is M0, then the significance threshold M1=0.2×M0.

[0126] The region clipping unit 643 is used to clip the contour region of each salient object according to the saliency threshold.

[0127] In the embodiment of the present invention, the contour region of each salient object is clipped according to the saliency threshold, and only the contour regions of the salient objects with a value higher than the current saliency threshold are retained.

[0128] The second mean value calculation unit 644 is configured to calculate a second saliency mean value according to all the cropped salient objects.

[0129] In the embodiment of the present invention, the second saliency score of each salient object is recalculated according to the cropped contour region retained by each salient object, and the second saliency mean is calculated according to the obtained second saliency score.

[0130] The score calculation unit 645 is used to calculate the second significance score of each significant object according to the calculated second significance mean value and the preset proportional coefficient, and determine the obtained second significance score as the significance score.

[0131] In the embodiment of the present invention, the proportional coefficient is determined according to the area size of the significant object. The larger the area, the larger the proportional coefficient. The second significance score of each significant object is obtained by multiplying the calculated second significance mean by the proportional coefficient of each significant object.

[0132] In the embodiment of the present invention, each unit of the image salient object detection device can be implemented by a corresponding hardware or software unit. Each unit can be an independent software or hardware unit or integrated into one software or hardware unit, which is not intended to limit the present invention.

[0133] Embodiment six:

[0134] Figure 7 The structure of the image processing device provided by the sixth embodiment of the present invention is shown. For the convenience of explanation, only the part related to the embodiment of the present invention is shown.

[0135] The image processing device 7 of the embodiment of the present invention includes a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70. When the processor 70 executes the computer program 72, the steps of the above-mentioned method for detecting salient objects in an image are implemented, for example: Figure 1 Alternatively, when the processor 70 executes the computer program 72, the functions of each unit in the above-mentioned device embodiments are realized, for example Figure 5 The functions of the units 51 to 54 are shown.

[0136] In an embodiment of the present invention, all salient objects in the image to be detected are detected by a saliency detection model, and the saliency score of each salient object is calculated respectively, and all salient objects are sorted according to the saliency score, and the salient object with the largest saliency score after sorting is determined as the target salient object in the image to be detected, thereby improving the recognition speed and recognition accuracy of salient objects in multi-scene images.

[0137] The image processing device of the embodiment of the present invention may be a smart phone or a personal computer. The steps implemented when the processor 70 in the image processing device 7 executes the computer program 72 to implement the image salient object detection method can refer to the description of the above method embodiment, which will not be repeated here.

[0138] Embodiment seven:

[0139] In an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned image salient object detection method embodiment are implemented, for example, Figure 1Alternatively, when the computer program is executed by a processor, the functions of each unit in the above-mentioned device embodiments are realized, for example Figure 5 The functions of the units 51 to 54 are shown.

[0140] In an embodiment of the present invention, all salient objects in the image to be detected are detected by a saliency detection model, and the saliency score of each salient object is calculated respectively, and all salient objects are sorted according to the saliency score, and the salient object with the largest saliency score after sorting is determined as the target salient object in the image to be detected, thereby improving the recognition speed and recognition accuracy of salient objects in multi-scene images.

[0141] The computer-readable storage medium of the embodiment of the present invention may include any entity or device or recording medium capable of carrying computer program code, for example, ROM / RAM, magnetic disk, optical disk, flash memory and other memories.

[0142] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting salient objects in an image, characterized in that: The method comprises the following steps: Acquire the image to be detected; Detecting the image to be detected by using a saliency detection model to obtain all salient objects in the image to be detected; Calculating a saliency score of each salient object according to the contour area, position information and color of the salient object; Sorting all the significant objects according to the significance scores, and determining the significant object with the largest significance score as the target significant object in the image to be detected; The step of respectively calculating the saliency score of each salient object comprises: Calculating a first saliency score of each of the salient objects respectively, and calculating a first saliency mean according to the obtained first saliency scores of all the salient objects; Determining a significance threshold according to the first significance mean; Cutting the contour area of ​​each of the salient objects according to the saliency threshold; Calculating a second saliency mean value according to all the salient objects after clipping; The second significance score of each of the significant objects is calculated according to the calculated second significance mean value and a preset proportionality coefficient, and the obtained second significance score is determined as the significance score.

2. The method according to claim 1, characterized in that Before detecting the image to be detected by using the saliency detection model to obtain all salient objects in the image to be detected, the method further includes: The saliency detection model is obtained by training a preset neural network to learn the mapping relationship between an image and a salient object in the image through preset training data, wherein the training data includes an image data set without salient objects and an image data set containing salient objects.

3. The method according to claim 2, characterized in that The preset neural network is a U-Net network and / or a classic saliency detection network.

4. The method according to claim 3, characterized in that The U-Net network includes a downsampling layer, the downsampling layer includes a skip connection module, and the skip connection module includes a depth-separable convolution layer and a maximum pooling layer.

5. A salient object detection device for an image, characterized in that: The device comprises: A detection image acquisition unit, used to acquire an image to be detected; A salient object obtaining unit, configured to detect the image to be detected by using a salient detection model, and obtain all salient objects in the image to be detected; a saliency score calculation unit, configured to calculate a saliency score of each of the saliency objects according to the contour area, position information and color of the saliency object; and A significance ranking unit, used for ranking all the significant objects according to the significance scores, and determining the significant object with the largest significance score as the target significant object in the image to be detected; Wherein, the significant score calculation unit includes: a first mean value calculation unit, configured to calculate a first significance score of each of the significant objects respectively, and calculate a first significance mean value according to the obtained first significance scores of all the significant objects; a threshold determination unit, configured to determine a significance threshold according to the first significance mean; A region clipping unit, used for clipping the contour region of each of the salient objects according to the saliency threshold; a second mean value calculation unit, configured to calculate a second saliency mean value according to all the cropped salient objects; and A score calculation unit is used to calculate the second significance score of each of the significant objects according to the calculated second significance mean value and a preset proportionality coefficient, and determine the obtained second significance score as the significance score.

6. The device according to claim 5, characterized in that The device also includes: The detection model training unit is used to train a preset neural network to learn the mapping relationship between an image and a salient object in the image through preset training data to obtain the salient detection model, wherein the training data includes an image data set without salient objects and an image data set containing salient objects.

7. An image processing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images