A method for detecting hidden object scenarios in smeared images

Abstract

The present invention discloses a method for detecting hidden object scenarios in smeared images, comprising the following steps: S1, obtaining an image; S2, smeared area segmentation, using an improved DDRNet-23 segmentation network to segment the smeared area of the image obtained in S1 to determine the features and shape of the smeared area; S3, smeared feature recognition, recognizing the segmentation result of the smeared area in S2, and after preprocessing the obtained segmentation result by Resize, sending it into a classification network for classification; S4, whole image scene recognition, scaling the size of the source image corresponding to the positive class sample index in the smeared feature recognition result in S3 and sending it into a classification network for recognition to determine whether the image belongs to the corresponding category. This method for detecting hidden object scenarios in smeared images uses semantic segmentation methods and classification network models to describe the image scene from multiple dimensions respectively, so as to solve the problem of rough description of scene parsing in a single dimension.

Description

Technical Field

[0001] The present invention relates to the technical field of image scene parsing, and specifically to a method for detecting a hidden item scene in a smeared image. Background Art

[0002] With the rapid development of the Internet and the full popularity of smart phones, people's lives have been greatly facilitated. Among them, there are also some ways to trade some items through contactless methods. In real life, first, prohibited items are placed in a pre-designated place, and then the actual hiding location is informed to the other party by taking a photo with a mobile phone and smearing it later. The other party finds the prohibited items based on the picture, and finally completes the entire prohibited item transaction through online transfer.

[0003] Currently, there is no detection method for hidden items in smeared images. Existing methods are mostly one part of the detection, including: semantic segmentation of smeared images, classification methods for indoor and outdoor scenes. However, in the existing technology, there are still the following two deficiencies:

[0004] 1. A single model can only describe one dimension in an image, and it is difficult to describe the image scene parsing in multiple dimensions;

[0005] 2. There is no overall process for detecting hidden items, lacking an overall solution. Summary of the Invention

[0006] In order to more accurately and quickly detect whether an image is a hidden item scene, the present invention provides the following technical solutions:

[0007] A method for detecting a hidden item scene in a smeared image according to the present invention includes the following steps:

[0008] S1. Obtain an image;

[0009] S2. Smearing area segmentation. An improved DDRNet-23 segmentation network is used to segment the smeared area of the image obtained in S1 to determine the characteristics and shape of the smeared area. Then, training samples are generated in an automated manner, and the data is trained to obtain a trained segmentation network model;

[0010] S3. Smearing feature recognition. The segmentation result of the smeared area in S2 is recognized. The obtained segmentation result is preprocessed by Resize and then sent into a classification network for classification. It is judged whether the category is a negative class. If it is not a negative class, the index corresponding to the image is saved to the recognition result. Otherwise, the index of the image is discarded;

[0011] S4. Whole-image scene recognition: Resize the source images corresponding to the positive sample indices in the smear feature recognition results in S3 to the specified size, and feed them into the classification network for recognition to determine whether the images belong to the corresponding category. If the category meets a specific scene, it indicates that the image is a scene with hidden items.

[0012] As a preferred technical solution of the present invention, the DDRNet-23 segmentation network in S2 belongs to a real-time semantic segmentation method and uses a two-stream feature extraction method. One stream uses a lightweight backbone network ResNet-18 to extract deep abstract features of the image, and the other stream uses dilated convolution and maintains the resolution of the image to extract high-resolution detail features of the image. After summing the two-stream features, the result is upsampled to 1 / 4 of the input image resolution for prediction to improve the accuracy of the segmentation region.

[0013] As a preferred technical solution of the present invention, the specific steps for generating training samples in an automated manner in S2 are as follows:

[0014] S2-1. Select a certain number of images of indoor and outdoor scenes and other scenes. For each image S with a size of H×W, randomly select a point on the image as the center point. ;

[0015] S2-2. Determine the tail position of the arrow according to the distance interval , and randomly select a point between the area of the center and the tail of the arrow as the intersection point of the arrow head and the circle contour. ; ;

[0016] S2-3. Construct a mask image M with a size of H×W and all values being 0. Draw a circle and an arrow at the same position on S and M. The smeared area generated in S uses random colors, and the value of the smeared area generated in M is 1.

[0017] S2-4. In the mask image M, crop it according to the positions of the circle and the arrow respectively to generate the smear feature recognition training data in the detection process.

[0018] As a preferred technical solution of the present invention, the classification network in S3 uses a lightweight network MobileV3. The feature extraction therein uses an inverted residual module, and a channel attention mechanism is introduced in some modules to enhance the representation ability of the model. After layer-by-layer convolution, average pooling is performed on the feature map, and convolution is used to obtain the feature vector. Finally, softmax classification is used to obtain the probabilities of all categories.

[0019] As a preferred technical solution of the present invention, in step S3, the negative class is other scenarios, such as scenario images without items that can be hidden, such as screenshots, texts, people, etc., and in step S4, the positive class is indoor and outdoor scenario images.

[0020] The beneficial effects of the present invention are:

[0021] This method for detecting hidden item scenarios in smeared images uses semantic segmentation methods and classification network models to describe the image scenarios from multiple dimensions respectively, so as to solve the problem of rough description of scene parsing in a single dimension. An end-to-end processing flow for detecting hidden item scenarios is proposed, and a lightweight network model is adopted to improve the processing efficiency of the entire process. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the present invention, but do not constitute a limitation to the present invention. In the drawings:

[0023] Figure 1 is a schematic diagram of the detection process for hidden items in smeared images according to the present invention;

[0024] Figure 2 is a schematic diagram of the improved DDRNet-23 network structure according to the present invention;

[0025] Figure 3 is a structure diagram of the MobileV3-large network according to the present invention;

[0026] Figure 4 is a flow chart of smear feature recognition according to the present invention;

[0027] Figure 5 is a flow chart of whole image scene recognition according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0028] The following describes the preferred embodiments of the present invention with reference to the drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0029] Embodiment 1

[0030] The method for detecting hidden item scenarios proposed by the present invention mainly includes three parts, namely smear area segmentation, smear feature recognition, and whole image scene recognition.

[0031] 1. Smear area segmentation

[0032] For smear area segmentation, an efficient real-time semantic segmentation network is adopted, and the improved DDRNet-23 is used as the segmentation method for smear areas. The specific process is as follows:

[0033] (1) Segmentation network structure

[0034] Use DDRNet-23 to segment the smeared area in the image to determine the characteristics and shape of the smeared area. The network structure is as Figure 2 shown. DDRNet-23 belongs to the real-time semantic segmentation method and uses the two-stream feature extraction method. One stream uses the lightweight backbone network ResNet-18 to extract the deep abstract features of the image; the other stream uses dilated convolution and maintains the resolution of the image to extract the high-resolution detailed features of the image. At the same time, interaction and fusion are carried out between the two streams to enhance the representation ability of the network. Finally, the two-stream features are summed and sent into the segmentation layer for softmax classification and prediction. During the training stage, an auxiliary segmentation head is added to the feature map with a resolution of 1 / 8 for training to improve the training stability of the network.

[0035] On this basis, after summing the two-stream features, the smeared area segmentation method upsamples the result to 1 / 4 of the input image resolution for prediction to improve the accuracy of the segmented area.

[0036] (2) Automatic data generation

[0037] Since the smeared area has obvious color and shape characteristics, an automated method is used to generate training samples. The specific steps are as follows:

[0038] 1) Select a certain number of images of indoor and outdoor scenes and other scenes. For each image S with a size of H×W, randomly select a point on the image as the center point ;

[0039] 2) Determine the tail position of the arrow according to the distance interval , and randomly select a point between the area of the center and the arrow tail as the intersection point of the arrow head and the circle contour ; ;

[0040] 3) Construct a mask image M with a size of H×W and all values being 0. Draw a circle and an arrow at the same position on S and M. The smeared area generated in S uses random colors, and the value of the smeared area generated in M is 1;

[0041] 4) In the mask image M, crop it according to the positions of the circle and the arrow respectively to generate the smeared feature recognition training data in the detection process.

[0042] 2. Smeared feature recognition

[0043] Smear feature recognition is to recognize the segmentation results of the smeared areas in the detection process. Here, it mainly includes smeared circles, arrows, mixtures, and negative class samples. A lightweight network, MobileV3, is used to improve the overall recognition efficiency.

[0044] (1)MobileV3-large network structure

[0045] As Figure 3 shown, MobileV3 is a lightweight classification network. The feature extraction therein uses an inverted residual module, and a channel attention mechanism is introduced in some modules to enhance the model's representation ability. Through layer-by-layer convolution, average pooling is performed on the feature map, convolution is used to obtain the feature vector, and finally softmax classification is used to obtain the probabilities of all classes.

[0046] (2)Smear feature recognition process

[0047] As Figure 4 shown, the smear feature recognition process is as follows: for the obtained smear area results, they are scaled to a certain size and sent into the classification network to obtain the classification results. It is judged whether the category is a negative class. If it is not a negative class, the index corresponding to the image in the source image is saved in the recognition results; otherwise, the index of the image is discarded.

[0048] 3. Whole-image scene recognition

[0049] As Figure 5 shown, whole-image scene recognition is to scale the source image corresponding to the positive class sample index in the smear feature recognition results in the detection process to a certain size and send it into the classification network for recognition to determine whether the image belongs to the corresponding category. The positive class categories here are indoor and outdoor, and the negative class categories are other scenes such as screenshots, texts, and images of people that do not have scenes where items can be hidden. The same network structure as that for the smear feature recognition is used to improve the overall recognition efficiency.

[0050] Example 2

[0051] A method for detecting hidden items in smeared images. According to the input image, segmentation of the smeared area, recognition of smear features, and recognition of the whole-image scene are performed, and the final result of whether it is a hidden item scene is given. It can be applied to image filtering in scenarios such as illegal item transactions and illegal item hiding under a large amount of data.

[0052] The process and result schematic diagram of applying the hidden item scene detection are as Figure 1 shown. The following steps A to C are respectively executed to obtain a smear area segmentation model, a smear feature recognition model, and a whole-image scene recognition model (regardless of order), and then steps i to vi are executed to apply the hidden item scene detection to obtain the detection results.

[0053] Step A. Collect a certain number of mobile phone captured images as background pictures, adaptively generate smearing regions using the method described in this patent to obtain training samples for the segmentation model. Use the segmentation model to train the data to obtain a trained segmentation network model.

[0054] Step B. Classify the training sample labels generated in Step A into categories such as arrows, circles, etc. Use the smearing feature recognition method described in this patent to construct a multi-classification network model and train these samples to obtain a smearing feature recognition network model.

[0055] Step C. Prepare images in various scenarios as samples. The model described in this patent includes 15 types of scenarios such as indoor, outdoor, text, and screenshots. Use the whole-image scene recognition method described in this patent to construct a multi-classification network model and train these samples to obtain a whole-image recognition network model.

[0056] Execute Steps i to iv to detect the hidden item scenario.

[0057] Step i. Initialize all models and load the weight parameters; perform normalization preprocessing on the input image, and then proceed to Step ii.

[0058] Step ii. Apply the smearing region segmentation network model to the preprocessed image to obtain the smearing region segmentation result of the image, which is represented as a 0-1 array, where 0 is the background region and 1 is the smearing region. Then proceed to Step iii.

[0059] Step iii. Apply the smearing feature recognition network model to the smearing region segmentation result to obtain the classification result of the smearing region. Judge the classification result. If it is a negative class, ignore the data; otherwise, proceed to Step iv.

[0060] Step iv. Apply the whole-image scene recognition network model to the original image with the smearing feature recognized as a positive class to obtain the scene category of the original image. If the category meets a specific scene, it indicates that the image is a hidden item scene.

[0061] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing embodiments or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for detecting hidden object scenarios in smeared images, characterized in that, It includes the following steps: S1. Obtain an image; S2. Smearing area segmentation: Use an improved DDRNet-23 segmentation network to segment the smearing area of the image obtained in S1 to determine the characteristics and shape of the smearing area. Then, generate training samples in an automated manner, train the data, and obtain a trained segmentation network model; S3. Smearing feature recognition: Recognize the segmentation result of the smearing area in S2. After preprocessing the obtained segmentation result by Resize, send it into a classification network for classification to determine whether the category is a negative class. If it is not a negative class, save the index of the corresponding source image of the figure to the recognition result. Otherwise, discard the index of the figure; S4. Whole-image scene recognition: Scale the source image corresponding to the positive-class sample index in the smearing feature recognition result in S3 to a specific size, and send it into a classification network for recognition to determine whether the image belongs to the corresponding category. If the category meets a specific scene, it indicates that the image is a hidden-item scene; The specific steps of generating training samples in an automated manner in S2 are as follows: S2-1. Select a certain number of images of indoor and outdoor scenes and other scenes. For each image S with an image size of H×W, randomly select a point as the center point (x1, y1) on the image; S2-2. Determine the tail position (x2, y2) of the arrow according to the distance interval and randomly select a point between the area of the center of the circle and the tail of the arrow as the interaction point (x3, y3) between the head of the arrow and the contour of the circle; S2-3. Construct a mask image M with a size of H×W and all values being 0. Draw circles and arrows at the same positions in S and M. The generated smearing area in S uses random colors, and the value of the generated smearing area in M is 1; S2-4. In the mask image M, crop it according to the positions of the circles and arrows respectively to generate the training data for smearing feature recognition in the detection process.

2. The method for detecting a hidden item scenario for a smeared image according to claim 1, wherein The DDRNet-23 segmentation network in S2 belongs to a real-time semantic segmentation method and uses a two-stream feature extraction method. One stream uses a lightweight backbone network ResNet-18 to extract the deep abstract features of the image, and the other stream uses dilated convolution and maintains the resolution of the image to extract the high-resolution detailed features of the image. After summing the two-stream features, the result is upsampled to 1 / 4 of the input image resolution for prediction to improve the accuracy of the segmentation area.

3. A method for detecting hidden object scenarios in smeared images according to claim 1, characterized in that, The classification network in S3 uses a lightweight network MobileV3. The feature extraction uses an inverted residual module, and a channel attention mechanism is introduced in some modules to enhance the representation ability of the model. After layer-by-layer convolution, average pooling is performed on the feature map, and convolution is used to obtain the feature vector. Finally, softmax classification is used to obtain the probabilities of all categories.

4. A method for detecting hidden object scenarios in smeared images according to claim 1, characterized in that, The negative class in S3 is other scenes, such as screenshot, text, and scene images of people without hidden-item scenes. The positive class in S4 is indoor and outdoor scene images.

Images