Image region localization method and target region localization model training method

Abstract

The present application relates to an image region localization method and a training method for a target region localization model, wherein the target region localization model is an initial localization model trained using a sample image set belonging to a target category. When training the target region localization model, each sample image in the sample image set is processed separately by the initial localization model to obtain a class activation map corresponding to each sample image. When the target region determined according to the class activation map of each sample image does not completely cover the region related to the target type in the corresponding sample image, the sample image is continued to be used to adjust the model parameters of the initial localization model to guide the current model to focus more attention on the target region omitted in the sample image, thereby checking for omissions and filling in the gaps during the adjustment process, so that the positioning result of the obtained target region localization model is more accurate.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method for locating an area of ​​an image, a method and apparatus for training a target area locating model, a computer-readable storage medium, and a computer device. Background Art

[0002] Computer Vision has many different applications, such as image classification and image localization. The difference between image classification and image localization is that image classification only needs to predict the category of the image through a computer model, that is, predict the specific target in the image; while image localization is a technology that uses a computer model to identify the specific target in the image and locate the position of the target in the image. Therefore, for supervised learning, the training data for image classification only needs to label the category of the sample image, while image localization needs to label the position of the target in the image and the target category. Obviously, the labeling cost of the training data required for image localization is higher.

[0003] Existing methods of image localization using neural networks usually only use image category information to locate the target area in the image due to the cost of labeling training data. Due to the lack of supervision information of the target area, the effect is poor and the positioning is inaccurate. Summary of the invention

[0004] Based on this, it is necessary to provide an image region positioning method, a target region positioning model training method, device, computer-readable storage medium and computer equipment to address the technical problem of inaccurate positioning effect in the prior art method of using image category information to locate images.

[0005] A method for locating a region of an image, comprising:

[0006] Get the image to be processed;

[0007] Acquire a class activation map corresponding to the image to be processed through a target region positioning model;

[0008] Wherein, the target region localization model is obtained by continuing to use the sample images to adjust the model parameters of the initial localization model when the target region determined according to the class activation map of each sample image does not completely cover the region related to the target category in the corresponding sample image when the sample image set belonging to the target category is used to train the initial localization model, and the class activation map corresponding to each sample image is obtained by processing the sample image with the initial localization model;

[0009] Determine a target region in the image to be processed according to a class activation map corresponding to the image to be processed.

[0010] A device for locating an area of ​​an image, the device comprising:

[0011] An image acquisition module, used for acquiring an image to be processed;

[0012] A class activation map acquisition module, used for acquiring the class activation map corresponding to the image to be processed through a target region localization model; wherein the target region localization model is obtained by continuously using the sample images to adjust the model parameters of the initial localization model when the target region determined according to the class activation map of each sample image does not completely cover the region related to the target category in the corresponding sample image when the initial localization model is trained using the sample image set belonging to the target category, and the class activation map corresponding to each sample image is obtained by processing the sample image with the initial localization model;

[0013] The target region determination module is used to determine the target region in the image to be processed according to the class activation map corresponding to the image to be processed.

[0014] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the steps of the above-mentioned image region positioning method.

[0015] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above-mentioned image region positioning method.

[0016] The above-mentioned image region localization method, device, computer-readable storage medium and computer equipment, the target region localization model is to use the sample image set belonging to the target category to train the initial localization model, when training the target region localization model, each sample image in the sample image set is processed by the initial localization model respectively, and the class activation map corresponding to each sample image is obtained, when the target region determined according to the class activation map of each sample image does not completely cover the region related to the target type in the corresponding sample image, then the sample image is continued to be used to adjust the model parameters of the initial localization model, so as to guide the current model to focus more attention on the target region omitted in the sample image, so as to check for omissions in the adjustment process, so that the positioning result of the obtained target region localization model is more accurate. In this way, when positioning the image to be processed, the class activation map corresponding to the image to be processed can be obtained through the target region localization model, and the target region in the image to be processed can be accurately determined according to the class activation map.

[0017] A training method for a target area positioning model, comprising:

[0018] Obtain a set of sample images belonging to the target category;

[0019] Acquire a class activation map corresponding to each sample image in the sample image set through an initial positioning model;

[0020] Determine the target area in the sample image according to the class activation map;

[0021] When the determined target area does not completely cover the area in the sample image related to the target category, the sample image is continued to be used to adjust the model parameters of the initial positioning model to obtain a target area positioning model for locating the target area in the image.

[0022] A training device for a target area positioning model, the device comprising:

[0023] A sample image acquisition module is used to acquire a sample image set belonging to a target category;

[0024] A class activation map acquisition module, used to acquire a class activation map corresponding to each sample image in the sample image set through an initial positioning model;

[0025] A target region determination module, used for determining a target region in the sample image according to the class activation map;

[0026] A model parameter adjustment module is used to continue to use the sample image to adjust the model parameters of the initial positioning model when the determined target area does not completely cover the area related to the target category in the sample image, so as to obtain a target area positioning model for locating the target area in the image.

[0027] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the steps of the training method of the target area positioning model.

[0028] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the training method of the target area positioning model.

[0029] The training method, apparatus, computer-readable storage medium and computer equipment of the above-mentioned target area positioning model, the target area positioning model is to use a sample image set belonging to the target category to train an initial positioning model. When training the target area positioning model, each sample image in the sample image set is processed separately by the initial positioning model to obtain a class activation map corresponding to each sample image. When the target area determined according to the class activation map of each sample image does not completely cover the area related to the target type in the corresponding sample image, the sample image is continued to be used to adjust the model parameters of the initial positioning model to guide the current model to focus more attention on the target area that is omitted in the sample image, so as to fill in the gaps during the adjustment process, so that the positioning result of the target area positioning model obtained is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A diagram showing an application environment of a method for locating a region of an image in an embodiment;

[0031] Figure 2 A schematic diagram of a flow chart of a method for locating a region of an image in one embodiment;

[0032] Figure 3 A schematic diagram of a process of obtaining a class activation map corresponding to an image to be processed through a target region positioning model in one embodiment;

[0033] Figure 4 A schematic diagram of a network structure of a target area positioning model in one embodiment;

[0034] Figure 5 A schematic diagram of a process for determining a target area in an image to be processed according to a class activation map corresponding to the image to be processed in one embodiment;

[0035] Figure 6 is a schematic diagram showing that the target area is a rectangular area in one embodiment;

[0036] Figure 7 is a schematic diagram of fusing candidate target areas in one embodiment;

[0037] Figure 8 A schematic diagram of a framework of a method for locating an image region in a specific embodiment;

[0038] Fig. 9 It is a flowchart of a method for locating an area of ​​an image in a specific embodiment;

[0039] Fig.10 A schematic diagram of a flow chart of a method for training a target area positioning model in one embodiment;

[0040] Fig.11 is a schematic diagram of a visualization result in one embodiment;

[0041] Fig.12 A schematic diagram of a training process of a target area positioning model in one embodiment;

[0042] Fig.13 is a structural block diagram of a device for locating a region of an image in one embodiment;

[0043] Fig.14 is a structural block diagram of a training device for a target area positioning model in one embodiment;

[0044] Fig.15 FIG. 4 is a structural block diagram of a computer device in one embodiment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0046] Figure 1 FIG. 1 is an application environment diagram of an image region positioning method in an embodiment. Figure 1 The image regional positioning method is applied to the image regional positioning system. The image regional positioning system includes a terminal 110 and a server 120. The terminal 110 and the server 120 are connected via a network. The terminal 110 can be a desktop terminal or a mobile terminal, and the mobile terminal can be at least one of a mobile phone, a tablet computer, a laptop computer, etc. The server 120 can be implemented by an independent server or a server cluster composed of multiple servers.

[0047] In one embodiment, an application is running on the terminal 110, and the user can use the application to watch videos or view pictures, and the watched videos or pictures are obtained by processing the server 120. The server 120 can obtain the image to be processed, and obtain the class activation map corresponding to the image to be processed through the target area positioning model; determine the target area in the image to be processed according to the class activation map corresponding to the image to be processed; determine the target area in the video or image; wherein, the target area positioning model is obtained by continuing to use the sample images to adjust the model parameters of the initial positioning model when the target area determined according to the class activation map of each sample image does not completely cover the area related to the target category in the corresponding sample image when the initial positioning model is trained using the sample image set belonging to the target category, and the class activation map corresponding to each sample image is obtained by processing the sample image with the initial positioning model.

[0048] Optionally, the server 120 may also return the determined target area to the terminal 110, and the terminal 110 may mark the target area in the video or image and then display it, or the terminal 110 may blur or mosaic the target area and then display it.

[0049] It can be understood that in other embodiments, the terminal 110 can also directly process the image to be processed, obtain the class activation map corresponding to the image to be processed through the target area positioning model; determine the target area in the image to be processed according to the class activation map corresponding to the image to be processed; wherein, the target area positioning model is obtained by continuing to use the sample image to adjust the model parameters of the initial positioning model when the target area determined according to the class activation map of each sample image does not completely cover the area related to the target category in the corresponding sample image when the initial positioning model is trained using the sample image set belonging to the target category, and the class activation map corresponding to each sample image is obtained by processing the sample image with the initial positioning model. In addition, the target area positioning model can be obtained by training the server 120, or by training the terminal 110.

[0050] like Figure 2 As shown, in one embodiment, a method for locating an area of ​​an image is provided. This embodiment applies the method to a computer device (such as the above Figure 1 The terminal 110 or server 120 in FIG. 1 is used as an example to illustrate. Figure 2 , the region positioning method of the image specifically includes the following steps:

[0051] S202: Obtain an image to be processed.

[0052] The image to be processed is an image to be imaged for image localization. Image localization refers to classifying an image into a target category in a candidate category according to a target object in the image and locating a target area where the target object is located. In practical applications, the candidate category can be set according to actual application requirements. The image to be processed is an unprocessed original image, and the image to be processed can be a video frame in a video or a picture.

[0053] Specifically, the computer device may obtain an image transmitted by another computer device and use the image as an image to be processed, such as the above Figure 1 The server 120 in the computer obtains the image transmitted by the terminal 110 and uses the image as the image to be processed. The computer device can also obtain the image generated on the computer and use the image as the image to be processed. The computer device can also download the image from the network and use the image as the image to be processed.

[0054] In other embodiments, the image to be processed may be an image including sensitive areas. The target area determined by the computer device after performing image positioning on the image to be processed is the sensitive area. The sensitive area may refer to pornographic areas, bloody areas, disgusting areas, etc. The computer device may obtain the sensitive areas in the image after positioning the image to be processed, and may further perform blur processing (such as Gaussian blur processing) or mosaic processing (such as a small area) on the sensitive areas, thereby reducing the user's discomfort when watching videos or pictures.

[0055] S204, obtaining a class activation map corresponding to the image to be processed through the target region positioning model.

[0056] Among them, the target area localization model is obtained by continuing to use the sample image set belonging to the target category to train the initial localization model. When the target area determined according to the class activation map of each sample image does not completely cover the area related to the target category in the corresponding sample image, the sample image is continued to be used to adjust the model parameters of the initial localization model. The class activation map corresponding to each sample image is obtained by processing the sample image with the initial localization model.

[0057] The initial positioning model can be learned through sample images, so that it has the ability to locate image regions. The initial positioning model can adopt a neural network model, such as a convolutional neural network model. The sample image set used to train the initial positioning model is a sample of the target category, that is, a sample with labeled category information. It can be understood that the image region positioning model obtained by training the sample images of the target category has the ability to locate the target region belonging to the target category from the image. For example, the target region positioning model obtained by training the initial positioning model with a bloody sample image set has the ability to locate the bloody region from the image, and the target region positioning model obtained by training the initial positioning model with a sample image set including human faces has the ability to locate human faces from the image.

[0058] In one embodiment, the computer device may set the model structure of the deep learning model in advance to obtain an initial positioning model, and then train the model parameters of the initial positioning model through a sample image set belonging to the target category. When it is necessary to perform image positioning on the image to be processed, the computer device may obtain the model parameters obtained by the pre-training, and then import the model parameters into the initial positioning model to obtain a target area positioning model that can locate the target area in the image to be processed belonging to the target category.

[0059] The activated area in the class activation map is the area related to the target category. The class activation map obtained by processing the image to be processed with the trained target region localization model makes the area related to the target category in the image to be processed visualized.

[0060] Specifically, the computer device can input the acquired image to be processed into the target area positioning model, and obtain the class activation map corresponding to the image to be processed through the target area positioning model. The target area positioning model is obtained by training the initial positioning model using a sample image set belonging to the target category. During the training process, when the target area determined according to the class activation map of each sample image does not completely cover the area related to the target category in the corresponding sample image, the sample image is continued to be used to adjust the model parameters of the initial positioning model. The class activation map corresponding to each sample image is obtained by processing the sample image with the initial positioning model. It can be understood that when the target area positioning model is obtained by training the initial positioning model using a sample image set belonging to another target category, the activation area presented by the class activation map obtained after processing the image to be processed is related to the current target category.

[0061] In one embodiment, Figure 3 As shown, the class activation map corresponding to the image to be processed is obtained through the target area positioning model, including:

[0062] S302: Input the image to be processed into the target area positioning model.

[0063] S304, extracting features of the image to be processed through the convolution layer in the target region positioning model to obtain a multi-channel feature map corresponding to the image to be processed.

[0064] Among them, the feature map is a matrix that can reflect the image features obtained after the image to be processed is convolved with the convolution kernel in the convolution layer. The image can be classified according to the image features, that is, the image features can be extracted to obtain the corresponding feature map, and the image can be classified according to the feature map. The feature map obtained by the convolution kernel of each layer is multi-channel. The feature map of each channel expresses part of the features of the image. The feature maps of all channels can jointly characterize the image features of the image to be processed. The feature maps of all channels output by the last convolution layer can more comprehensively express the image information of the image to be processed. When locating the target area of ​​the image to be processed, the image to be processed can be input into the target area positioning model. The convolution layer in the target area positioning model extracts features from the image to be processed, obtains the feature map corresponding to the image to be processed, and then can perform subsequent processing according to the feature map. Optionally, the subsequent processing is based on the feature maps of all channels output by the last convolution layer.

[0065] S306, through the classification layer in the target region positioning model, the feature maps of each channel are weighted summed with the weight corresponding to the target category to obtain a class activation map corresponding to the target category of the image to be processed.

[0066] Among them, the classification layer can be used to classify the categories to which each pixel in the processed image belongs according to the obtained feature map, so that the target area composed of pixels related to the target category can be found based on the classified pixels. The weight represents the importance of the feature map to the target category. The feature maps of different channels have different importance to the target category, that is, different weights. For example, Represents the weight of the target category C corresponding to channel k. For the pixel at position (x, y) in the feature map of channel k output by the last convolutional layer, its feature value is , the activation value of the target category C corresponding to the pixel For the entire image to be processed, in fact, the weighted summation of the feature maps of all channels is performed through the weight corresponding to the target category C to obtain the activation value of each pixel in the image to be processed corresponding to the target category C. According to the activation value of each pixel in the image to be processed corresponding to the target category C, the class activation map of the entire image to be processed corresponding to the target category C can be obtained.

[0067] like Figure 4 FIG. 1 is a schematic diagram of a network structure of a target area positioning model in an embodiment. Figure 4 The target area positioning model includes a convolution layer and a classification layer. The convolution layer includes multiple convolution kernels, and different convolution kernels have different scales. The classification layer includes a global average pooling layer (Global Average Pooling, GAP) and an output layer (softmax for classification). The convolution kernel in the convolution layer is convolved with the image to be processed to extract the image features of the image to be processed. The global average pooling layer outputs the average value of the feature maps of all channels obtained by the last convolution kernel. The output layer performs weighted summation on all feature maps obtained by the last convolution kernel to obtain the final output, which is the class activation map. Reference Figure 4 , multiply the weight corresponding to the C class by the feature map of each channel input by the last convolutional layer, that is: W1*first layer+W2*second layer+…+Wn*nth layer=class activation map corresponding to category C.

[0068] In one embodiment, the target area localization model is trained by the following steps: obtaining a set of sample images belonging to a target category; obtaining a class activation map corresponding to each sample image in the sample image set through an initial localization model; determining the target area in the sample image according to the class activation map; when the determined target area does not completely cover the area related to the target category in the sample image, continuing to use the sample image to adjust the model parameters of the initial localization model to obtain a target area localization model for locating the target area in the image.

[0069] Specifically, the computer device can obtain a set of sample images belonging to the target category, obtain the class activation map corresponding to each sample image in the sample image set through the initial positioning model, and perform visualization processing (including normalization processing, scale scaling processing and binarization processing) on ​​the class activation map to obtain a visualization result of the target area. In this way, the annotator can intuitively judge whether the target area obtained by the current model completely covers the area related to the target category in the sample image based on the visualization result. If not, the sample image is retained and added to the sample image set corresponding to the target category. The sample image set is continued to be used to adjust the model parameters of the current initial positioning model to guide the model to focus on the area belonging to the target category that is omitted in the sample image, thereby finding and filling in the gaps during the adjustment process, so that the positioning result of the target area positioning model obtained is more accurate.

[0070] Optionally, if the target area obtained by the current model has completely covered the area in the sample image that is related to the target category, the sample image may be removed from the sample image set corresponding to the target category.

[0071] In this embodiment, in the process of training the model, the annotation cost of the sample image set used only includes the category information, and the annotation cost is relatively low. In the process of adjusting the model parameters, the model structure is not changed, and it is still based on the structure of the original classification model, so that the adjusted model can more accurately locate the target area in the image. The development cost and the annotation cost are relatively low, and the target area positioning model obtained is very effective and has a high accuracy rate. For the specific embodiment of training the target area positioning model, please refer to the embodiment of the training method of the target area positioning model in the following text.

[0072] S206: Determine a target region in the image to be processed according to a class activation map corresponding to the image to be processed.

[0073] Since the class activation map preliminarily reflects the probability that the pixel points in the image to be processed belong to the target category, the computer device can determine the target area in the image to be processed according to the class activation map. In some embodiments, the computer device can determine the candidate target area in the image to be processed according to the class activation map, and continue to fuse or screen the candidate target areas to obtain the final target area.

[0074] In one embodiment, Figure 5 As shown, determining the target area in the image to be processed according to the class activation map corresponding to the image to be processed includes:

[0075] S502, normalizing and scaling the class activation map to obtain the probability that each pixel in the image to be processed belongs to the target area.

[0076] Since the pixel values ​​in the class activation map are different, in order to increase the comparability between pixels, the class activation map needs to be normalized. Computer equipment can use appropriate normalization methods to process the class activation map, such as finding the maximum value max and minimum value min in each type of activation map, and then converting each pixel value f according to the following formula:

[0077] ;

[0078] The transformed f' (a value between [0,1]) is the normalized value corresponding to f, which can express the probability that the corresponding pixel belongs to the target category.

[0079] Furthermore, the computer device also needs to scale the normalized class activation map to the same scale as the image to be processed, and finally obtain the probability that each pixel in the image to be processed belongs to the target area.

[0080] S504: Determine, from the image to be processed, pixel points whose corresponding probabilities are greater than a preset threshold.

[0081] For the class activation map after normalization and scaling, Represents the pixel value at the (x, y) coordinate, and also represents the probability that the pixel at the (x, y) coordinate belongs to the target area. The preset threshold thresh is a threshold value in the range of [0,1]. Thresh can be set according to actual needs. For example, if the target area is expected to cover a larger area in the image to be processed, a smaller threshold can be taken. If false alarms are less desired, a higher threshold can be taken. In one embodiment, thresh is 0.8. In other words, the computer device needs to find out the pixel values ​​greater than the preset threshold value from the processed class activation map, so as to determine the corresponding pixel points from the image to be processed according to the coordinates of these values.

[0082] S506: Taking the area formed by the determined pixel points as the target area in the image to be processed.

[0083] Specifically, the computer device can directly use the area formed by the determined pixel points as the target area in the image to be processed, and the obtained target area is obviously an irregularly shaped area. In one embodiment, the computer device can visualize the target area in the image to be processed in the following manner, that is, first binarize the normalized class activation map:

[0084] ,like ;

[0085] ,like .

[0086] The image to be processed is then superimposed pixel by pixel with the binarized class activation map to visualize the target area in the image to be processed.

[0087] In some embodiments, the computer device may also use the irregular shape area formed by the determined pixel points as the candidate target area of ​​the image to be processed, and further perform regional fusion or screening on the candidate target area to obtain the final target area in the image to be processed.

[0088] In this embodiment, for the class activation map corresponding to the target category of the image to be processed obtained by the target localization model, the class activation map can be directly normalized and scaled and then compared with a preset threshold to obtain the target area in the image to be processed.

[0089] In one embodiment, determining a target region in the image to be processed according to a class activation map corresponding to the image to be processed includes:

[0090] The class activation map is normalized and scaled to obtain the probability that each pixel in the image to be processed belongs to the target area; the pixel whose corresponding probability is greater than a preset threshold is determined from the image to be processed; the vertex pixel is determined from the pixel according to the coordinates of the pixel; the rectangular area is obtained according to the vertex pixel; and the rectangular area is used as the target area in the image to be processed.

[0091] Specifically, after determining the corresponding pixel points whose probability is greater than a preset threshold value from the image to be processed, at least one irregularly shaped candidate target area can be determined based on the determined pixel points. The vertex pixel points are the pixel points corresponding to the maximum or minimum value of the coordinates in each candidate target area, and the coordinates include x-coordinates and y-coordinates. For each irregularly shaped candidate target area, four vertex pixel points can be obtained based on the coordinates of the pixel points, that is, the leftmost, rightmost, topmost and bottommost pixel points. Further, the computer device can determine the corresponding rectangular area based on the four vertex pixel points of each candidate target area, and use the corresponding rectangular area as the target area in the image to be processed. The four vertex coordinates of the rectangular area can be used as the position information of the target area in the image to be processed.

[0092] like Figure 6 FIG. 1 is a schematic diagram showing a target area as a rectangular area in an embodiment. Figure 6 , the target category is "mouse", the target area is the area including "mouse", the left side is the original image to be processed, the middle is the candidate target area obtained according to the class activation map of the image to be processed, and the right side is the rectangular area determined according to the candidate target area, which is used as the target area corresponding to the image to be processed.

[0093] In this embodiment, by further processing the irregular candidate target area, a regular rectangular area is obtained, and the rectangular area is used as the positioning result of the image to be processed. That is to say, the model can be trained without the location information of the training data, and the target area positioning model obtained by training can accurately locate the location information of the image to be processed.

[0094] In one embodiment, the method also includes: when the size of the acquired rectangular area is smaller than a preset threshold, filtering out the rectangular area; when the acquired rectangular area includes multiple partially overlapping rectangular areas, determining the minimum rectangular area including the multiple rectangular areas based on the multiple rectangular areas, and using the determined minimum rectangular area as the target area in the image to be processed.

[0095] Specifically, the size of the obtained rectangular area can be measured by the side length of the rectangular area or the number of pixels included. For example, when the side length of the rectangular area is less than 5% of the side length of the image to be processed, the rectangular area can be filtered out, or when the number of pixels included in the rectangular area is less than 10% of the number of pixels included in the image to be processed, the rectangular area can be filtered out. In some other embodiments, the computer device can determine whether there are multiple partially overlapping rectangular areas based on the coordinates of the four vertex pixels of the rectangular area. If so, the multiple partially overlapping rectangular areas can be regionally merged, which is actually a minimum merging process based on the multiple partially overlapping rectangular areas to obtain the minimum rectangular area including the multiple rectangular areas, and the obtained minimum rectangular area is used as the target area in the image to be processed.

[0096] like Figure 7 FIG. 1 is a schematic diagram of fusing candidate target regions in an embodiment. Figure 7 , the left side shows two rectangular areas A and B obtained from the image to be processed. If rectangular area A partially overlaps with rectangular area B, rectangular area A and rectangular area B are merged, as shown in Figure 7 As shown on the right, a minimum rectangular area including both rectangular area A and rectangular area B is obtained, and finally the rectangular area is used as the target area in the image to be processed.

[0097] In this embodiment, the computer device can continue to filter the rectangular areas that are too small to reduce the impact of the noise generated by the algorithm on the positioning results. For partially overlapping rectangular areas, the computer device can perform fusion processing, which can more completely cover the target area, thereby improving the accuracy of locating the target area.

[0098] In one embodiment, the target area is a sensitive area in the image to be processed, and the method further includes: blurring or mosaicing the sensitive area; and displaying the processed image to be processed.

[0099] In specific application scenarios, when the current area is a sensitive area in the image to be processed, such as pornographic areas, bloody areas, and disgusting areas, in order to reduce the discomfort of users when viewing pictures, the computer equipment can further blur or mosaic the located sensitive areas.

[0100] like Figure 8 FIG. 1 is a schematic diagram of a framework of an image region positioning method in a specific embodiment. Figure 8 First, the image to be processed is input into the target region localization model, and the class activation map of the target category corresponding to the image to be processed is obtained through the target region localization model. Then, the class activation map is normalized, scaled and binarized to obtain the candidate target region. After screening and fusing the candidate target regions, the target region in the image to be processed is located.

[0101] like Fig. 9 FIG. 1 is a flow chart of a method for locating an area of ​​an image in a specific embodiment, comprising the following steps:

[0102] S902, obtaining an image to be processed;

[0103] S904, inputting the image to be processed into the target area positioning model;

[0104] S906, extracting features of the image to be processed through the convolution layer in the target region positioning model to obtain a multi-channel feature map corresponding to the image to be processed;

[0105] Among them, the target area localization model is obtained by continuing to use the sample image set belonging to the target category to train the initial localization model. When the target area determined according to the class activation map of each sample image does not completely cover the area related to the target category in the corresponding sample image, the sample image is continued to be used to adjust the model parameters of the initial localization model. The class activation map corresponding to each sample image is obtained by processing the sample image with the initial localization model.

[0106] S908, through the classification layer in the target region positioning model, the feature maps of each channel are weighted summed with the weight corresponding to the target category to obtain a class activation map corresponding to the target category of the image to be processed.

[0107] S910, normalizing and scaling the class activation map to obtain the probability that each pixel in the image to be processed belongs to the target area;

[0108] S912, determining, from the image to be processed, pixels whose corresponding probabilities are greater than a preset threshold;

[0109] S914, determining a vertex pixel point from the pixel points according to the coordinates of the pixel points;

[0110] S916, obtaining a rectangular area according to vertex pixels;

[0111] S918, when the size of the obtained rectangular area is smaller than a preset threshold, filter out the rectangular area;

[0112] S920: When the acquired rectangular area includes a plurality of partially overlapping rectangular areas, a minimum rectangular area including the plurality of rectangular areas is determined according to the plurality of rectangular areas, and the determined minimum rectangular area is used as a target area in the image to be processed.

[0113] S922, performing blur processing or mosaic processing on the target area;

[0114] S924, displaying the processed image to be processed.

[0115] In the above-mentioned image region localization method, the target region localization model is to use a sample image set belonging to the target category to train an initial localization model. When training the target region localization model, each sample image in the sample image set is processed by the initial localization model to obtain a class activation map corresponding to each sample image. When the target region determined according to the class activation map of each sample image does not completely cover the region related to the target type in the corresponding sample image, the sample image is continued to be used to adjust the model parameters of the initial localization model to guide the current model to focus more attention on the target region omitted in the sample image, thereby checking for omissions and filling in the gaps during the adjustment process, so that the positioning result of the target region localization model obtained is more accurate. In this way, when locating the image to be processed, the class activation map corresponding to the image to be processed can be obtained through the target region localization model, and the target region in the image to be processed can be accurately determined according to the class activation map.

[0116] Fig. 9 FIG. 1 is a flow chart of a method for locating an area of ​​an image in one embodiment. It should be understood that although Fig. 9 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Fig. 9At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0117] like Fig.10 As shown, in one embodiment, a method for training a target area positioning model is provided. This embodiment applies the method to a computer device (such as the above Figure 1 The terminal 110 or server 120 in FIG. 1 is used as an example to illustrate. Fig.10 , the training method of the target area positioning model specifically includes the following steps:

[0118] S1002, obtaining a set of sample images belonging to the target category.

[0119] The sample image set belonging to the target category is training data for distinguishing images belonging to the target category. Each sample image in the sample image set is an image that is only labeled with real category information, and the location information of the target area in the sample image is not known yet. The sample image set includes sample images whose real category information is the target category, and also includes sample images whose real category information is the non-target category. The computer device can obtain the sample image set and obtain the category information of each sample image. The category information corresponding to each sample image can be determined by the labeling personnel.

[0120] S1004: Obtain a class activation map corresponding to each sample image in the sample image set through the initial positioning model.

[0121] Among them, the initial positioning model is a deep learning model corresponding to the initial model parameters, and the initial positioning model can be learned through sample images, so as to have the ability to locate the image area. The initial positioning model can adopt a neural network model, such as a convolutional neural network model. In the process of continuously training the initial positioning model, the corresponding model parameters are constantly changing, until the deep learning model corresponding to the model parameters obtained when the training is completed is the target area positioning model. It can be understood that the image area positioning model obtained by training the sample images of the target category has the ability to locate the target area belonging to the target category from the image. In this embodiment, the deep learning models obtained before the end of training are referred to as the initial positioning model.

[0122] Specifically, when training the initial positioning model, the computer device can sequentially input each sample image in the sample image set belonging to the target category into the initial positioning model, and obtain the class activation map corresponding to the sample image through the initial positioning model. The activated area in the class activation map is the area related to the target category.

[0123] In one embodiment, the class activation map corresponding to each sample image in the sample image set is obtained through the initial positioning model, including: inputting the sample image into the initial positioning model; extracting features of the sample image through the convolution layer in the initial positioning model to obtain a multi-channel feature map corresponding to the sample image; and performing weighted summation of the feature maps of each channel with the weight corresponding to the target category through the classification layer in the initial positioning model to obtain the class activation map of the target category corresponding to the sample image.

[0124] Specifically, the model structure of the initial positioning model set by the computer device includes a convolution layer and a classification layer. During the process of training the model, the computer device can input each sample image into the convolution layer, extract image features of the sample image through the convolution layer, obtain feature maps of each channel corresponding to the sample image, and then use the classification layer to perform linear weighted summation on the feature maps of each channel with the weight corresponding to the target category to obtain the class activation map corresponding to the sample image.

[0125] Among them, the convolution layer is used to extract the image features of the sample image and obtain the feature map corresponding to the sample image. The feature map is a matrix that can reflect the image features obtained by convolution processing between the sample image and the convolution kernel in the convolution layer. The image can be classified according to the feature map. The feature map obtained by the convolution kernel of each layer is multi-channel. The feature map of each channel expresses part of the features of the image. The feature maps of all channels can jointly characterize the image features of the sample image. The feature maps of all channels output by the last convolution kernel of the convolution layer can more comprehensively express the image information of the sample image.

[0126] The classification layer can be used to classify the categories to which each pixel in the sample image belongs based on the obtained feature map, so that the target area composed of pixels related to the target category can be found based on the classified pixels. The weight represents the importance of the feature map to the target category. The feature maps of different channels have different importance to the target category, that is, the weights are different. For the entire sample image, the classification layer is used to perform weighted summation of the feature maps of all channels through the weights corresponding to the target category to obtain the activation value of each pixel in the sample image corresponding to the target category. According to the activation value of each pixel in the sample image corresponding to the target category, the class activation map of the entire sample image corresponding to the target category can be obtained.

[0127] S1006: Determine the target region in the sample image according to the class activation map.

[0128] The class activation map preliminarily reflects the possibility that the pixel points in the sample image belong to the target category. The computer device can determine the target area in the sample image based on the class activation map.

[0129] In one embodiment, determining a target area in a sample image based on a class activation map includes: performing normalization, scaling, and binarization on the class activation map in sequence, and then superimposing the map with the sample image to obtain a visualization result corresponding to the sample image, and the visualization result is used to display the target area in the sample image.

[0130] The values ​​of each pixel in the class activation map are different in size. In order to increase the comparability between pixels, the computer device can normalize the values ​​of each pixel in the class activation map and convert each pixel value into a value between [0,1], which can represent the probability that the corresponding pixel belongs to the target category. At the same time, after the convolution operation of multiple convolution kernels in the convolution layer, the scale of the feature map output by the convolution layer is inconsistent with the sample image, resulting in the scale of the obtained class activation map being different from the sample image. Therefore, the computer device also needs to scale the normalized class activation map to the same scale as the sample image. Each value in the scale is the probability that each pixel in the sample image belongs to the target area. Finally, in order to obtain the visualization result, the computer device needs to binarize the probability that each pixel belongs to the target area, that is, compare the probability with the preset threshold and then binarize it, and superimpose the binarized result with the original sample image to obtain the visualization result corresponding to the sample image. In this way, according to the obtained visualization result, the staff or annotation personnel can intuitively see whether the target area in the sample image determined by the currently trained model completely covers the area related to the target category in the sample image.

[0131] For example, the pixel value at the (x, y) coordinate in the normalized and scaled class activation map can be expressed as It is then compared with the preset threshold thresh for binarization:

[0132] ;

[0133] .

[0134] Finally, we get the visualization result:

[0135] ,in is the original sample image, is the product term, is the result after binarization, This is the visualization result. Obviously, the value after multiplying the pixel value by 0 is 0, so the black area in the visualization result is the target area obtained by the current positioning.

[0136] like Fig.11 As shown, it is a schematic diagram of the visualization result in one embodiment. If the initial positioning model is used to locate the "mouse" in the image, refer to Fig.11 , the left side is the original sample image, which includes "mouse" and the background area outside "mouse". The middle is the visualization result obtained by iterating the initial positioning model once and outputting the class activation map. The black area in the visualization result is the target area determined by the class activation map output by the current model. It can be seen that since the initial positioning model has not been trained, the image positioning result is not accurate enough, and the current visualization result fails to fully cover the area related to the target category ("mouse") in the sample image.

[0137] S1008, when the determined target area does not completely cover the area related to the target category in the sample image, the sample image is continued to be used to adjust the model parameters of the initial positioning model to obtain a target area positioning model for positioning the target area in the image.

[0138] Specifically, all generated visualization results are handed over to the annotators for judgment. When the target area in the sample image determined by the currently trained model fails to completely cover the area related to the target category in the sample image, the sample image needs to be retained and the current model continues to be trained using the sample image to adjust the model parameters until a target area positioning model for locating the target area in the image is obtained when the iteration conditions are met. In the process of adjusting the model parameters, the model structure is not changed. It is still based on the structure of the original classification model, so that the adjusted model can more accurately locate the target area in the image. The development cost and annotation cost are relatively low, and the obtained target area positioning model is very effective and has a high accuracy rate. Reference Fig.11 The right picture in the figure is the visualization result determined based on the class activation map output by the model after multiple iterations. It can be seen that in this visualization result, the area related to "mouse" is more completely covered.

[0139] In one embodiment, S1008 specifically includes: when the determined target area does not completely cover the area related to the target category in the sample image, the sample image is re-added to the sample image set corresponding to the target category; after adjusting the model parameters of the initial positioning model, each sample image in the sample image set continues to be input into the initial positioning model in turn to obtain the target area related to the target category in each sample image; until the adjusted initial positioning model meets the iteration stop condition, a target area positioning model for locating the target area in the image is obtained.

[0140] Specifically, when the determined target area does not completely cover the area related to the target category in the sample image, it means that the corresponding model under the current model parameters fails to fully notice all areas related to the target category in the sample image, and it is necessary to retain the sample image, add the sample image to the sample image set, and continue to use the sample images in the sample image set to train the current model until the adjusted model meets the iteration stop condition, and obtains a target area positioning model with a high image positioning accuracy. The iteration stop condition can be set according to the accuracy requirement, for example, the iteration stop condition can be whether the accuracy of locating the target area from the sample image reaches a second preset value when the intersection over union (IoU) meets the first preset value.

[0141] In one embodiment, when the determined target area completely covers the area in the sample image related to the target category, the sample image is removed from the sample image set corresponding to the target category. Specifically, when the determined target area completely covers the area in the sample image related to the target category, the sample image is discarded and there is no need to use the sample image to train the current model.

[0142] like Fig.12 FIG. 1 is a schematic diagram of a training process of a target area positioning model in an embodiment. Fig.12 , the sample image is input into the initial positioning model to obtain the corresponding class activation map, and the visualization result corresponding to the target area in the sample image is determined according to the class activation map. If it is determined according to the visualization result that the target area completely covers the target area related to the target category in the sample image, the sample image is discarded. If it does not completely cover the target area related to the target category in the sample image, the sample image is retained and added to the sample image set corresponding to the target category. The model is continued to be trained until the model after multiple iterations meets the iteration stop condition, and the target area positioning model is obtained.

[0143] In a specific embodiment, the training method of the target area positioning model includes the following steps:

[0144] Obtain a set of sample images belonging to the target category;

[0145] Inputting the sample image into the initial positioning model;

[0146] The sample image is subjected to feature extraction through the convolutional layer in the initial positioning model to obtain a multi-channel feature map corresponding to the sample image;

[0147] Through the classification layer in the initial positioning model, the feature maps of each channel are weighted and summed with the weights corresponding to the target category to obtain the class activation map of the sample image corresponding to the target category;

[0148] The class activation map is normalized, scaled, and binarized in sequence, and then superimposed with the sample image to obtain a visualization result corresponding to the sample image. The visualization result is used to display the target area in the sample image.

[0149] When the determined target area completely covers the area in the sample image related to the target category, the sample image is removed from the sample image set corresponding to the target category;

[0150] When the determined target area does not completely cover the area in the sample image related to the target category, the sample image is re-added to the sample image set corresponding to the target category;

[0151] After adjusting the model parameters of the initial positioning model, each sample image in the sample image set is input into the initial positioning model in sequence to obtain a target area related to the target category in each sample image;

[0152] When the adjusted initial positioning model satisfies the iteration stopping condition, a target region positioning model for positioning the target region in the image is obtained.

[0153] The training method of the above-mentioned target area positioning model is that the target area positioning model uses a sample image set belonging to the target category to train an initial positioning model. When training the target area positioning model, each sample image in the sample image set is processed separately by the initial positioning model to obtain a class activation map corresponding to each sample image. When the target area determined according to the class activation map of each sample image does not completely cover the area related to the target type in the corresponding sample image, the sample image is continued to be used to adjust the model parameters of the initial positioning model to guide the current model to focus more attention on the target area that is omitted in the sample image, so as to fill in the gaps in the adjustment process, so that the positioning result of the target area positioning model obtained is more accurate.

[0154] In one embodiment, Fig.13 As shown, a device 1300 for locating a region of an image is provided, the device comprising an image acquisition module 1302, a class activation map acquisition module 1304 and a target region determination module 1306, wherein:

[0155] An image acquisition module 1302 is used to acquire an image to be processed;

[0156] The class activation map acquisition module 1304 is used to acquire the class activation map corresponding to the image to be processed through the target region localization model; wherein the target region localization model is obtained by continuously using the sample images to adjust the model parameters of the initial localization model when the target region determined according to the class activation map of each sample image does not completely cover the region related to the target category in the corresponding sample image when the initial localization model is trained using the sample image set belonging to the target category, and the class activation map corresponding to each sample image is obtained by processing the sample image with the initial localization model;

[0157] The target region determination module 1306 is used to determine the target region in the image to be processed according to the class activation map corresponding to the image to be processed.

[0158] In one embodiment, the class activation map acquisition module 1304 is specifically used to input the image to be processed into the target area positioning model; perform feature extraction on the image to be processed through the convolution layer in the target area positioning model to obtain a multi-channel feature map corresponding to the image to be processed; and perform weighted summation on the feature maps of each channel using the weight corresponding to the target category through the classification layer in the target area positioning model to obtain a class activation map of the target category corresponding to the image to be processed.

[0159] In one embodiment, the target area determination module 1306 is specifically used to normalize and scale the class activation map to obtain the probability that each pixel in the image to be processed belongs to the target area; determine the pixels whose corresponding probabilities are greater than a preset threshold from the image to be processed; and use the area formed by the determined pixels as the target area in the image to be processed.

[0160] In one embodiment, the target area determination module 1306 is specifically used to normalize and scale the class activation map to obtain the probability that each pixel in the image to be processed belongs to the target area; determine the pixel points whose corresponding probabilities are greater than a preset threshold from the image to be processed; determine the vertex pixel points from the pixel points according to the coordinates of the pixel points; obtain the rectangular area according to the vertex pixel points; and use the rectangular area as the target area in the image to be processed.

[0161] In one embodiment, the target area determination module 1306 is also specifically used to filter out the rectangular area when the size of the acquired rectangular area is smaller than a preset threshold; when the acquired rectangular area includes multiple partially overlapping rectangular areas, determine the minimum rectangular area including the multiple rectangular areas based on the multiple rectangular areas, and use the determined minimum rectangular area as the target area in the image to be processed.

[0162] In one embodiment, the target area is a sensitive area in the image to be processed, and the image area positioning device 1300 further includes a display module for performing blur processing or mosaic processing on the sensitive area; and displaying the processed image to be processed.

[0163] In the above-mentioned image region positioning device 1300, the target region positioning model is to use a sample image set belonging to the target category to train an initial positioning model. When training the target region positioning model, each sample image in the sample image set is processed by the initial positioning model to obtain a class activation map corresponding to each sample image. When the target region determined according to the class activation map of each sample image does not completely cover the region related to the target type in the corresponding sample image, the sample image is continued to be used to adjust the model parameters of the initial positioning model to guide the current model to focus more attention on the target region omitted in the sample image, thereby checking for omissions and filling in the gaps during the adjustment process, so that the positioning result of the obtained target region positioning model is more accurate. In this way, when positioning the image to be processed, the class activation map corresponding to the image to be processed can be obtained through the target region positioning model, and the target region in the image to be processed can be accurately determined according to the class activation map.

[0164] In one embodiment, Fig.14 As shown, a training device 1400 for a target region positioning model is provided, the device comprising a sample image acquisition module 1402, a class activation map acquisition module 1404, a target region determination module 1406 and a model parameter adjustment module 1408, wherein:

[0165] The sample image acquisition module 1402 is used to acquire a sample image set belonging to a target category;

[0166] The class activation map acquisition module 1404 is used to acquire the class activation map corresponding to each sample image in the sample image set through the initial positioning model;

[0167] A target region determination module 1406 is used to determine a target region in a sample image according to a class activation map;

[0168] The model parameter adjustment module 1408 is used to continue to use the sample image to adjust the model parameters of the initial positioning model when the determined target area does not completely cover the area related to the target category in the sample image, so as to obtain a target area positioning model for locating the target area in the image.

[0169] In one embodiment, the class activation map acquisition module 1404 is specifically used to input the sample image into the initial positioning model; perform feature extraction on the sample image through the convolution layer in the initial positioning model to obtain a multi-channel feature map corresponding to the sample image; perform weighted summation on the feature maps of each channel using the weight corresponding to the target category through the classification layer in the initial positioning model to obtain a class activation map of the target category corresponding to the sample image.

[0170] In one embodiment, the target area determination module 1406 is specifically used to perform normalization, scale processing and binarization processing on the class activation map in sequence, and then superimpose it with the sample image to obtain a visualization result corresponding to the sample image, and the visualization result is used to display the target area in the sample image.

[0171] In one embodiment, the model parameter adjustment module 1408 is specifically used to add the sample image back to the sample image set corresponding to the target category when the determined target area does not completely cover the area related to the target category in the sample image; after adjusting the model parameters of the initial positioning model, continue to input each sample image in the sample image set into the initial positioning model in turn to obtain the target area related to the target category in each sample image; until the adjusted initial positioning model meets the iteration stop condition, a target area positioning model for locating the target area in the image is obtained.

[0172] In one embodiment, the model parameter adjustment module 1408 is further configured to remove the sample image from the sample image set corresponding to the target category when the determined target area completely covers an area in the sample image related to the target category.

[0173] The training device 1400 of the above-mentioned target area positioning model, the target area positioning model is to use a sample image set belonging to the target category to train an initial positioning model. When training the target area positioning model, each sample image in the sample image set is processed separately by the initial positioning model to obtain a class activation map corresponding to each sample image. When the target area determined according to the class activation map of each sample image does not completely cover the area related to the target type in the corresponding sample image, the sample image is continued to be used to adjust the model parameters of the initial positioning model to guide the current model to focus more attention on the target area that is omitted in the sample image, so as to find out the omissions and make up for the deficiencies in the adjustment process, so that the positioning result of the target area positioning model obtained is more accurate.

[0174] Fig.15 FIG. 2 shows an internal structure diagram of a computer device in one embodiment. Fig.15 As shown, when the computer device is Figure 1When the server 120 in the embodiment of the present invention is a computer device, the computer device may include a processor, a memory, and a network interface connected via a system bus; when the computer device is a server 120 in the embodiment of the present invention, the computer device may include a processor, a memory, and a network interface connected via a system bus; Figure 1 When the terminal 110 is used, the computer device may further include a display screen. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program, which, when executed by the processor, enables the processor to implement an image region positioning method or a target region positioning model training method. The internal memory may also store a computer program, which, when executed by the processor, enables the processor to execute an image region positioning method or a target region positioning model training method. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or may be a button, trackball or touchpad provided on the computer device housing, or may be an external keyboard, touchpad or mouse, etc.

[0175] Those skilled in the art will understand that Fig.15 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0176] In one embodiment, the image region positioning device 1300 provided in the present application can be implemented in the form of a computer program. The computer program can be used in a computer program such as Fig.15 The memory of the computer device may store various program modules constituting the region positioning device 1300 of the image, for example, Fig.13 The image acquisition module 1302, the class activation map acquisition module 1304 and the target region determination module 1306 are shown. The computer program composed of various program modules enables the processor to execute the steps of the image region location method of each embodiment of the present application described in this specification.

[0177] For example, Fig.15 The computer device shown can be Fig.13 The image acquisition module 1302 in the image region positioning device 1300 shown in the figure performs step S202. The computer device may perform step S204 through the class activation map acquisition module 1304. The computer device may perform step S206 through the target region determination module 1306.

[0178] In one embodiment, the training device 1400 of the target area positioning model provided by the present application can be implemented in the form of a computer program. The computer program can be used in Fig.15The computer device shown in the figure is run. The memory of the computer device can store various program modules of the training device 1400 constituting the target area positioning model, for example, Fig.14 The sample image acquisition module 1402, the class activation map acquisition module 1404, the target region determination module 1406 and the model parameter adjustment module 1408 are shown. The computer program composed of various program modules enables the processor to execute the steps in the training method of the target region positioning model of each embodiment of the present application described in this specification.

[0179] For example, Fig.15 The computer device shown can be Fig.14 The sample image acquisition module 1402 in the training device 1400 of the target region positioning model shown in the figure performs step S1002. The computer device can perform step S1004 through the class activation map acquisition module 1404. The computer device can perform step S1006 through the target region determination module 1406. The computer device can perform step S1008 through the model parameter adjustment module 1408.

[0180] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above-mentioned method for locating an area of ​​an image. The steps of the method for locating an area of ​​an image here may be the steps of the method for locating an area of ​​an image in the above-mentioned embodiments.

[0181] In one embodiment, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the processor executes the steps of the above-mentioned method for locating an area of ​​an image. The steps of the method for locating an area of ​​an image here may be the steps of the method for locating an area of ​​an image in the above-mentioned embodiments.

[0182] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the training method of the target area positioning model. The steps of the training method of the target area positioning model here may be the steps of the training method of the target area positioning model in each of the above embodiments.

[0183] In one embodiment, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the processor executes the steps of the training method of the target area positioning model. The steps of the training method of the target area positioning model here can be the steps in the training method of the target area positioning model in each of the above embodiments.

[0184] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0185] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0186] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for locating a region of an image, comprising: Get the image to be processed; Acquire a class activation map corresponding to the image to be processed through a target region positioning model; The class activation map is sequentially normalized and scaled to obtain the probability that each pixel in the image to be processed belongs to the target area, and the corresponding pixel whose probability is greater than a preset threshold is determined from the image to be processed, and the area formed by the determined pixel points is used as the target area in the image to be processed; The target area positioning model is obtained by training an initial positioning model based on the structure of the original classification model using a sample image set belonging to the target category, each sample image in the sample image set is only labeled with category information, and the location information of the target area is not labeled, the sample image set includes sample images whose category information is the target category, and also includes sample images whose category information is the non-target category, and the training steps include: when it is determined and displayed according to the class activation map of each sample image that the target area of ​​the sample image does not completely cover the area related to the target category in the corresponding sample image, the sample image is retained and re-added to the sample image set corresponding to the target category, and when the displayed target area completely covers the area related to the target category in the sample image, the sample image is removed from the sample image set corresponding to the target category; using the target area The sample image set and category information corresponding to the target category are obtained, and the model parameters of the initial positioning model are adjusted based on the original classification task; the aforementioned steps are continued to be iterated multiple times to guide the model to focus on the omitted area belonging to the target category in the sample image until the target area positioning model is obtained when the iteration condition is met; wherein, the class activation map corresponding to each of the sample images is obtained by processing the feature map of the sample image with the initial positioning model, wherein the class activation map of the sample image is normalized, scaled and binarized in turn, and then superimposed with the sample image to obtain the visualization result corresponding to the sample image, and the visualization result is used to display the target area in the sample image, and the visualization result is used for the annotation personnel to judge whether the target area completely covers the area related to the target category in the corresponding sample image.

2. The method according to claim 1, characterized in that The step of obtaining a class activation map corresponding to the image to be processed by using a target region positioning model includes: Inputting the image to be processed into a target area positioning model; Extracting features of the image to be processed through the convolution layer in the target area positioning model to obtain a multi-channel feature map corresponding to the image to be processed; Through the classification layer in the target region positioning model, the feature maps of each channel are weighted summed with the weight corresponding to the target category to obtain a class activation map of the image to be processed corresponding to the target category.

3. The method according to claim 1, characterized in that The step of using the determined area formed by the pixels as the target area in the image to be processed includes: Determine a vertex pixel point from the pixel points according to the coordinates of the corresponding pixel points whose probability is greater than a preset threshold; Acquire a rectangular area according to the vertex pixel points; The rectangular area is used as the target area in the image to be processed.

4. The method according to claim 3, characterized in that The step of taking the rectangular area as the target area in the image to be processed includes: When the size of the acquired rectangular area is smaller than a preset threshold, filtering out the rectangular area; When the acquired rectangular area includes a plurality of partially overlapping rectangular areas, a minimum rectangular area including the plurality of rectangular areas is determined according to the plurality of rectangular areas, and the determined minimum rectangular area is used as the target area in the image to be processed.

5. The method according to any one of claims 1 to 4, characterized in that: The target area is a sensitive area in the image to be processed, and the method further includes: Performing blurring or mosaic processing on the sensitive area; Display the processed image.

6. A method for training a target area positioning model, comprising: Acquire a sample image set corresponding to a target category, wherein each sample image in the sample image set is only marked with category information, but not with location information of a target area, and the sample image set includes sample images whose category information is a target category, and also includes sample images whose category information is a non-target category; Obtaining a class activation map corresponding to a feature map of each sample image in the sample image set through an initial positioning model, wherein the initial positioning model is a model based on the structure of an original classification model; The class activation map is subjected to normalization, scaling and binarization processing in sequence, and then superimposed with the sample image to obtain a visualization result corresponding to the sample image, wherein the visualization result is used to display the target area in the sample image, and the visualization result is used by the annotator to determine whether the target area completely covers the area related to the target category in the corresponding sample image; When the displayed target area does not completely cover the area in the sample image related to the target category, the sample image is retained and re-added to the sample image set corresponding to the target category; when the displayed target area completely covers the area in the sample image related to the target category, the sample image is removed from the sample image set corresponding to the target category; Using the sample image set and category information corresponding to the target category, adjusting the model parameters of the initial positioning model based on the original classification task; Continue to iterate the above steps multiple times to guide the model to focus on the areas belonging to the target category that are omitted in the sample image, until a target area positioning model is obtained when the iteration conditions are met, and the target area positioning model is used to obtain the class activation map corresponding to the image to be processed, and locate the target area in the image to be processed according to the class activation map.

7. The method according to claim 6, characterized in that The obtaining, by using the initial positioning model, a class activation map corresponding to each sample image in the sample image set comprises: Inputting the sample image into the initial positioning model; Extracting features of the sample image through the convolution layer in the initial positioning model to obtain a multi-channel feature map corresponding to the sample image; The feature maps of each channel are weighted summed by using the weight corresponding to the target category through the classification layer in the initial positioning model to obtain a class activation map of the sample image corresponding to the target category.

8. An image region positioning device, characterized in that: The device comprises: An image acquisition module, used for acquiring an image to be processed; A class activation map acquisition module is used to acquire the class activation map corresponding to the image to be processed through a target area positioning model; wherein the target area positioning model is obtained by training an initial positioning model based on an original classification model using a sample image set belonging to a target category, each sample image in the sample image set is only labeled with category information, and the location information of the target area is not labeled, the sample image set includes sample images whose category information is a target category, and also includes sample images whose category information is a non-target category, and the training step includes: when it is determined and displayed according to the class activation map of each sample image that the target area of ​​the sample image does not completely cover the area related to the target category in the corresponding sample image, the sample image is retained and re-added to the sample image set corresponding to the target category, and when the displayed target area completely covers the area related to the target category in the sample image, the sample image corresponding to the target category is selected from the sample image set corresponding to the target category. The method further comprises the steps of: removing the sample image from the image set; using the sample image set and category information corresponding to the target category, adjusting the model parameters of the initial positioning model based on the original classification task; continuing to iterate the aforementioned steps multiple times to guide the model to focus on the omitted area belonging to the target category in the sample image until a target area positioning model is obtained when the iteration condition is met, wherein the class activation map corresponding to each of the sample images is obtained by processing the feature map of the sample image with the initial positioning model, wherein the class activation map of the sample image is normalized, scaled and binarized in turn, and then superimposed with the sample image to obtain a visualization result corresponding to the sample image, wherein the visualization result is used to display the target area in the sample image, and the visualization result is used for the annotation personnel to judge whether the target area completely covers the area related to the target category in the corresponding sample image; The target region determination module is used to perform normalization and scale processing on the class activation map in sequence, obtain the probability that each pixel point in the image to be processed belongs to the target region, determine the corresponding pixel points whose probability is greater than a preset threshold from the image to be processed, and use the region formed by the determined pixel points as the target region in the image to be processed.

9. The device according to claim 8, characterized in that The class activation map acquisition module is specifically used to input the image to be processed into the target area positioning model; perform feature extraction on the image to be processed through the convolution layer in the target area positioning model to obtain a multi-channel feature map corresponding to the image to be processed; and perform weighted summation on the feature maps of each channel using the weight corresponding to the target category through the classification layer in the target area positioning model to obtain a class activation map of the image to be processed corresponding to the target category.

10. The device according to claim 8, characterized in that The target area determination module is also specifically used to determine vertex pixel points from the pixel points according to the coordinates of the corresponding pixel points whose probability is greater than a preset threshold; obtain a rectangular area according to the vertex pixel points; and use the rectangular area as the target area in the image to be processed.

11. The device according to claim 10, characterized in that The target area determination module is also specifically used to filter out the rectangular area when the size of the acquired rectangular area is smaller than a preset threshold; when the acquired rectangular area includes multiple partially overlapping rectangular areas, determine the minimum rectangular area including the multiple rectangular areas based on the multiple rectangular areas, and use the determined minimum rectangular area as the target area in the image to be processed.

12. The device according to any one of claims 8 to 11, characterized in that The target area is a sensitive area in the image to be processed. The device also includes a display module, which is used to perform blur processing or mosaic processing on the sensitive area; and display the processed image to be processed.

13. A training device for a target area positioning model, characterized in that: The device comprises: A sample image acquisition module is used to acquire a sample image set corresponding to a target category, wherein each sample image in the sample image set is only marked with category information, but not with location information of the target area, and the sample image set includes sample images whose category information is the target category, and also includes sample images whose category information is the non-target category; A class activation map acquisition module, used to acquire a class activation map corresponding to a feature map of each sample image in the sample image set through an initial positioning model, wherein the initial positioning model is a model based on the structure of an original classification model; A target region determination module is used to perform normalization, scale scaling and binarization on the class activation map in sequence, and then superimpose it with the sample image to obtain a visualization result corresponding to the sample image, wherein the visualization result is used to display the target region in the sample image, and the visualization result is used for the annotator to determine whether the target region completely covers the region related to the target category in the corresponding sample image; A model parameter adjustment module is used to retain the sample image and add it back into the sample image set corresponding to the target category when the displayed target area does not completely cover the area related to the target category in the sample image; and remove the sample image from the sample image set corresponding to the target category when the displayed target area completely covers the area related to the target category in the sample image; use the sample image set and category information corresponding to the target category to adjust the model parameters of the initial positioning model based on the original classification task, and continue to iterate the above steps multiple times to guide the model to focus on the omitted area belonging to the target category in the sample image until a target area positioning model is obtained when the iteration conditions are met, and the target area positioning model is used to obtain the class activation map corresponding to the image to be processed, and locate the target area in the image to be processed according to the class activation map.

14. The device according to claim 13, characterized in that The class activation map acquisition module is specifically further used to input the sample image into the initial positioning model; perform feature extraction on the sample image through the convolution layer in the initial positioning model to obtain a multi-channel feature map corresponding to the sample image; and perform weighted summation on the feature maps of each channel using the weight corresponding to the target category through the classification layer in the initial positioning model to obtain a class activation map of the sample image corresponding to the target category.

15. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 7.

16. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 7.

Images