Occlusion image detection method, device and medium

Abstract

The present disclosure relates to an occluded image detection method, device and medium. The method comprises: after capturing an image through a camera, obtaining the image and using it as an image to be detected; inputting the image to be detected into a trained occluded image detection model; wherein the occluded image detection model is trained using the original occluded image, the unoccluded image and the trained data feature enhancement network; based on the occlusion characteristics of the image in the occluded image detection model, judging whether the image to be detected is an occluded image; and outputting the image detection result. By adopting this method, a small number of finger occluded images with mask annotations can be used to stably train the generative model to generate occluded image features, thereby effectively solving the problem of insufficient positive samples in the training of the finger occlusion detection model and saving a lot of positive sample collection costs.

Description

Technical Field

[0001] The present disclosure relates to the field of camera control, and in particular to an occluded image detection method, device and medium. Background Art

[0002] Data augmentation is a technique that uses limited data to generate more equivalent data. It is an important technique to improve the accuracy and generalization ability of deep network models, and plays a huge role in computer vision tasks such as image classification, object recognition, and semantic segmentation. Most existing data augmentation techniques use image transformation methods, such as rotation, cropping, affine transformation, and color jitter.

[0003] When users hold their phones horizontally to take photos, the edge of the lens is easily blocked by fingers due to shooting habits, resulting in a small blocked area on the edge of the photo. Since this type of occlusion is not easy to detect immediately, users usually do not retake the photo, which results in a regret that they cannot get a perfect photo. The task of occlusion detection is to determine whether the image is blocked immediately after taking the photo, and to prompt the user to retake the photo in real time after detecting an blocked image.

[0004] In occlusion detection, the occluded images need to be collected by artificially simulating real scenes. Since the occluded area in real scenes occupies a small area of ​​the photo and is located in a remote location, it takes a lot of manpower to collect. At the same time, existing data enhancement methods will cause distortion of the occluded area and thus fail to generate effective training data, so these methods are not suitable for occlusion detection tasks. Summary of the invention

[0005] In order to overcome the problems existing in the related art, the present disclosure provides a method, device and medium for detecting an occluded image.

[0006] According to a first aspect of an embodiment of the present disclosure, there is provided a method for detecting an occluded image, the method comprising:

[0007] After the image is captured by the camera, the image is obtained and used as the image to be detected;

[0008] Inputting the image to be detected into the trained occlusion image detection model; wherein the occlusion image detection model is trained by using the original occlusion image, the unocclusion image and the trained data feature enhancement network;

[0009] Based on the occlusion characteristics of the image in the occlusion image detection model, determining whether the image to be detected is an occlusion image;

[0010] The image detection result is outputted.

[0011] Wherein, the method further comprises:

[0012] Acquire the original occluded image and the unoccluded image;

[0013] Training the data feature enhancement network based on the original occluded image;

[0014] The occluded image detection model is trained based on the original occluded image and the unoccluded image and using the trained data feature enhancement network.

[0015] Wherein, the step of training the data feature enhancement network based on the original occluded image comprises:

[0016] Based on the original occluded image, a finger template image and a non-finger image are generated, wherein the finger template image is an image obtained by removing the region not occluded by the finger from the original occluded image, and the non-finger image is an image obtained by removing the region occluded by the finger from the original occluded image;

[0017] Generate training data groups, each group of training data includes a finger template image, a non-finger image and an original occluded image, and use multiple groups of training data to train the data feature enhancement network in each round of training.

[0018] Wherein, generating the finger template image and the non-finger image based on the original occlusion image comprises:

[0019] Based on the original occlusion image, acquiring mask data, wherein the mask data indicates an occluded position in the original occlusion image;

[0020] Based on the original occlusion image and the mask data, a finger template image and a non-finger image are generated.

[0021] Wherein, the training of the data feature enhancement network based on the original occluded image further includes:

[0022] Inputting the finger template image and the non-finger image into the data feature enhancement network to synthesize the generated features of the original occluded image;

[0023] Inputting the original occluded image into a feature network to extract original features of the original occluded image, wherein the occluded image detection model includes the feature network;

[0024] Inputting the generated features of the original occluded image and the original features into a discriminant network for training;

[0025] Obtaining the loss function of the data feature enhancement network and the loss function of the discriminant network in each round of training;

[0026] When the loss function of the data feature enhancement network and the loss function of the discriminant network both converge, it is determined that the training of the data feature enhancement network is completed.

[0027] The step of training the occluded image detection model based on the original occluded image and the unoccluded image and using the trained data feature enhancement network includes:

[0028] Based on the mask data and the unobstructed image, obtaining a processed unobstructed image, wherein the processed unobstructed image is an image obtained by removing the area corresponding to the mask data from the unobstructed image;

[0029] Inputting the finger template image and the processed unobstructed image into the trained data feature enhancement network to synthesize the generated features of the unobstructed image;

[0030] Inputting the original occluded image into a feature network to extract original features of the original occluded image, wherein the occluded image detection model includes the feature network;

[0031] Inputting the generated features of the unobstructed image and the original features into a classification network for training, wherein the occluded image detection model includes the classification network;

[0032] When the loss function of the occluded image detection model converges, it is determined that the training of the occluded image detection model is completed.

[0033] Wherein, the occluded image detection model is a convolutional neural network model.

[0034] Among them, the data feature enhancement network is a generative adversarial network.

[0035] According to a second aspect of an embodiment of the present disclosure, there is provided an occlusion image detection device, the device comprising:

[0036] An acquisition module is configured to acquire an image as an image to be detected after the image is captured by a camera;

[0037] An input module is configured to input the image to be detected into the trained occlusion image detection model;

[0038] The occluded image detection model is configured to determine whether the image to be detected is an occluded image based on the occlusion characteristics of the image; wherein the occluded image detection model is trained using the original occluded image, the unoccluded image and the trained data feature enhancement network;

[0039] The output module is configured to output the image detection result.

[0040] The occluded image detection model is trained by a training module, and the training module is set to:

[0041] Acquire the original occluded image and the unoccluded image;

[0042] Training the data feature enhancement network based on the original occluded image;

[0043] The occluded image detection model is trained based on the original occluded image and the unoccluded image and using the trained data feature enhancement network.

[0044] The training module is further configured to train the data feature enhancement network based on the original occluded image in the following manner:

[0045] Based on the original occluded image, a finger template image and a non-finger image are generated, wherein the finger template image is an image obtained by removing the region not occluded by the finger from the original occluded image, and the non-finger image is an image obtained by removing the region occluded by the finger from the original occluded image;

[0046] Generate training data groups, each group of training data includes a finger template image, a non-finger image and an original occluded image, and use multiple groups of training data to train the data feature enhancement network in each round of training.

[0047] The training module is further configured to generate a finger template image and a non-finger image based on the original occlusion image in the following manner:

[0048] Based on the original occlusion image, acquiring mask data, wherein the mask data indicates an occluded position in the original occlusion image;

[0049] Based on the original occlusion image and the mask data, a finger template image and a non-finger image are generated.

[0050] The training module is further configured to train the data feature enhancement network based on the original occluded image in the following manner:

[0051] Inputting the finger template image and the non-finger image into the data feature enhancement network to synthesize the generated features of the original occluded image;

[0052] Inputting the original occluded image into a feature network to extract original features of the original occluded image, wherein the occluded image detection model includes the feature network;

[0053] Inputting the generated features of the original occluded image and the original features into a discriminant network for training;

[0054] Obtaining the loss function of the data feature enhancement network and the loss function of the discriminant network in each round of training;

[0055] When the loss function of the data feature enhancement network and the loss function of the discriminant network both converge, it is determined that the training of the data feature enhancement network is completed.

[0056] The training module is further configured to train the occluded image detection model based on the original occluded image and the unoccluded image and using the trained data feature enhancement network in the following manner:

[0057] Based on the mask data and the unobstructed image, obtaining a processed unobstructed image, wherein the processed unobstructed image is an image obtained by removing the area corresponding to the mask data from the unobstructed image;

[0058] Inputting the finger template image and the processed unobstructed image into the trained data feature enhancement network to synthesize the generated features of the unobstructed image;

[0059] Inputting the original occluded image into a feature network to extract original features of the original occluded image, wherein the occluded image detection model includes the feature network;

[0060] Inputting the generated features of the unobstructed image and the original features into a classification network for training, wherein the occluded image detection model includes the classification network;

[0061] When the loss function of the occluded image detection model converges, it is determined that the training of the occluded image detection model is completed.

[0062] Wherein, the occluded image detection model is a convolutional neural network model.

[0063] Among them, the data feature enhancement network is a generative adversarial network.

[0064] According to a third aspect of an embodiment of the present disclosure, there is provided an occlusion image detection device, comprising:

[0065] processor;

[0066] a memory for storing processor-executable instructions;

[0067] Wherein, the processor is configured to:

[0068] After the image is captured by the camera, the image is obtained and used as the image to be detected;

[0069] Inputting the image to be detected into the trained occlusion image detection model; wherein the occlusion image detection model is trained by using the original occlusion image, the unocclusion image and the trained data feature enhancement network;

[0070] Based on the occlusion characteristics of the image in the occlusion image detection model, determining whether the image to be detected is an occlusion image;

[0071] The image detection result is output.

[0072] According to a fourth aspect of an embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor of a router, an occluded image detection method can be executed. The method includes:

[0073] After the image is captured by the camera, the image is obtained and used as the image to be detected;

[0074] Inputting the image to be detected into the trained occlusion image detection model; wherein the occlusion image detection model is trained by using the original occlusion image, the unocclusion image and the trained data feature enhancement network;

[0075] Based on the occlusion characteristics of the image in the occlusion image detection model, determining whether the image to be detected is an occlusion image;

[0076] The image detection result is output.

[0077] The present disclosure proposes an occluded image detection method. After an image is captured by a camera, the image is used as an image to be detected and input into a trained occluded image detection model. Then, the occluded image detection model determines whether the image to be detected is an occluded image and outputs an image detection result. The occluded image detection model is trained by using an original occluded image, an unoccluded image, and a trained data feature enhancement network.

[0078] Therefore, the method disclosed in the present invention has the following beneficial effects:

[0079] (1) A small number of finger occlusion images with mask annotations can be used to stably train the generative model to generate occlusion image features, which can effectively solve the problem of insufficient positive samples in the training of finger occlusion detection models and save a lot of positive sample collection costs.

[0080] (2) Generating high-level features of the occluded image can effectively avoid the impact of noise caused by insufficient image generation details on model accuracy. Therefore, after adding the generated features to the training, the detection accuracy is significantly improved.

[0081] (3) The generated features only optimize the classification network of the finger occlusion detection model again, and the additional training cost introduced is relatively small. In addition, the existing finger occlusion detection model can be fully utilized and a small number of parameters can be fine-tuned, resulting in little additional overhead.

[0082] (4) The network model has fewer parameters, which can save a lot of storage and computing resources.

[0083] (5) This method has high scalability and universality and can be easily extended to other classification tasks.

[0084] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0086] Figure 1 The figure is a flow chart of a method for detecting an occluded image according to an exemplary embodiment.

[0087] Figure 2 It is a schematic diagram showing training of a data feature enhancement network according to an exemplary embodiment.

[0088] Figure 3 It is a schematic diagram showing training of an occluded image detection model according to an exemplary embodiment.

[0089] Figure 4 is a schematic diagram of a trained occluded image detection model according to an exemplary embodiment.

[0090] Figure 5 The figure is a flow chart of a method for detecting an occluded image according to an exemplary embodiment.

[0091] Figure 6 The figure is a block diagram of a device for detecting an occluded image according to an exemplary embodiment.

[0092] Figure 7 It is a block diagram of a device according to an exemplary embodiment.

[0093] Figure 8 It is a block diagram of a device according to an exemplary embodiment. DETAILED DESCRIPTION

[0094] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0095] When users hold their phones horizontally to take photos, the edge of the lens is easily blocked by fingers, resulting in a small blocked area at the edge of the photo. Since this type of occlusion is not easy to detect immediately, users usually do not retake the photo, which results in a regret that they cannot get a perfect photo. The task of finger occlusion detection is to determine whether the image is blocked immediately after taking the photo, and to prompt the user to retake the photo in real time after detecting an blocked image.

[0096] In order to solve the above problems, it is currently proposed to generate new images and directly add them as samples of a certain category to the model training. However, this method has the following problems: (1) This type of algorithm cannot effectively generate images with strict semantic content such as finger occlusion areas, and is therefore not suitable for finger occlusion detection tasks; (2) Directly generating images requires the generative network and the discriminative network to have a large number of parameters, which will increase the computational cost and time cost of network training.

[0097] The present disclosure proposes an occluded image detection method. In this method, after an image is captured by a camera, the image is used as an image to be detected and input into a trained occluded image detection model. Then, the occluded image detection model determines whether the image to be detected is an occluded image and outputs an image detection result. The occluded image detection model is trained by using an original occluded image, an unoccluded image, and a trained data feature enhancement network.

[0098] The method disclosed herein is particularly suitable for the following application scenarios:

[0099] (1) It is difficult to obtain positive sample images (occluded by fingers), which requires a lot of time and manpower.

[0100] (2) It is not easy to expand the positive samples using simple data augmentation techniques. That is, when a new image is generated from the original image by cropping, deformation, adding noise, etc., the finger part of the occluded image will be destroyed, causing distortion, so that the occluded part of the finger does not match the actual situation.

[0101] (3) It is difficult to generate positive sample images using generative adversarial networks, which will lose a lot of detail information and introduce a lot of noise, resulting in a decrease in the accuracy of the model. This is because in some application scenarios, finger occlusion detection has high data requirements and is very sensitive to the data distribution of the occluded image, while the adversarial generative network technology cannot realistically simulate the generation of finger occlusion images.

[0102] (4) The existing generative image augmentation algorithm has a large network size and requires a large amount of training data, thus requiring a large amount of storage and computing resources, which is not conducive to large-scale training. In particular, when it is difficult to obtain positive sample images, this generative image augmentation algorithm is not applicable.

[0103] The method disclosed herein is applicable to electronic products equipped with cameras, such as mobile phones, PADs, wearable devices, etc.

[0104] The present disclosure provides a method for detecting an occluded image. Figure 1 The flowchart of an occluded image detection according to an exemplary embodiment includes the following steps:

[0105] Step 101, after an image is captured by a camera, the image is obtained and used as an image to be detected;

[0106] Step 102, inputting the image to be detected into the trained occlusion image detection model; wherein the occlusion image detection model is trained by using the original occlusion image, the non-occlusion image and the trained data feature enhancement network;

[0107] Step 103, judging whether the image to be detected is an occlusion image based on the occlusion characteristics of the image in the occlusion image detection model;

[0108] Step 104: output the image detection result.

[0109] In this method, an image captured by a camera is obtained and used as an image to be detected, and then the image to be detected is input into a trained image detection model. The trained occlusion image detection model is used to determine whether the image to be detected is an occlusion image. It is known to those skilled in the art that the occlusion image detection model can learn about the occlusion characteristics of an image after training.

[0110] The occluded image detection model is trained using the original occluded image, the unoccluded image and the trained data feature enhancement network. When training the occluded image detection model, the data feature enhancement network is trained by the original occluded image so that the data feature enhancement network can synthesize occluded image features close to the original features of the image, that is, the occluded image features are synthesized from the original unoccluded image and the mask image. Then, the original occluded image features are obtained through the original occluded image, and the occluded image detection model is trained by the synthesized occluded image features and the original occluded image features.

[0111] The image feature here generally refers to the output obtained after taking the image as input and processing it through the feature network. It is equivalent to a representation form with high-level semantic information obtained after the image is compressed.

[0112] By adopting the above method, when synthesizing occluded image features, a small number of occluded images with mask annotations are used to stably train the generative model to synthesize occluded image features. This can effectively solve the problem of insufficient positive samples in the training of occluded detection models and save a lot of positive sample collection costs. In addition, here are synthesized occluded image features, that is, synthesized high-level features of the occluded image, which can effectively avoid the influence of noise caused by insufficient details of image generation on model accuracy. Therefore, after adding the synthesized features in the training, the detection accuracy is significantly improved.

[0113] Moreover, in this method, the occluded image features are synthesized through the data feature enhancement network instead of synthesizing the occluded image, which is equivalent to reducing the model part from image to feature. Therefore, the network model in this method is much smaller than the network model for generating images.

[0114] In the training of the occluded image detection model here, the training data set is divided into multiple batches, and the model is trained using the gradient descent method until the loss function converges. This training process can be implemented by those skilled in the art using existing training methods, and will not be described in detail here.

[0115] The method of synthesizing occlusion image features through the occlusion image annotated by the mask will be described in conjunction with the following embodiments.

[0116] In an optional embodiment, the method further comprises:

[0117] Acquire the original occluded image and the unoccluded image;

[0118] Training the data feature enhancement network based on the original occluded image;

[0119] The occluded image detection model is trained based on the original occluded image and the unoccluded image and using the trained data feature enhancement network.

[0120] As described above, the occluded image detection model is trained using the original occluded image, the unoccluded image and the trained data feature enhancement network. Therefore, before training the occluded image detection model, it is necessary to first train the data feature enhancement network. In this embodiment, the data feature enhancement network is trained based on the acquired original occluded image. The trained data feature enhancement network can synthesize occluded image features that are close to the original features of the image.

[0121] This can solve the problem of insufficient positive samples in the training of occlusion detection models and save a lot of positive sample collection costs. Moreover, by synthesizing the high-level features of the occluded image, the influence of noise caused by insufficient image generation details on the model accuracy can be reduced.

[0122] In an optional implementation, the training of the data feature enhancement network based on the original occluded image includes:

[0123] Based on the original occluded image, a finger template image and a non-finger image are generated, wherein the finger template image is an image obtained by removing the region not occluded by the finger from the original occluded image, and the non-finger image is an image obtained by removing the region occluded by the finger from the original occluded image;

[0124] Generate training data groups, each group of training data includes a finger template image, a non-finger image and an original occluded image, and use multiple groups of training data to train the data feature enhancement network in each round of training.

[0125] Figure 2 The process of training the data feature enhancement network is shown. A represents the finger template image, B represents the non-finger image, and C represents the original finger occlusion image. As mentioned above, the feature network is included in the occlusion image detection model. The occlusion image features synthesized by the data feature enhancement network and the original occlusion image features extracted by the feature network are input into the discriminant network. The discriminant network is used to assist in the training of the data feature enhancement network, that is, to reduce the difference between the synthesized occlusion image features and the original occlusion image features during the training process, so as to obtain the expanded occlusion image features through the data feature enhancement network. The discriminant network is only used to train the data feature enhancement network, and is not used for the training of the occlusion image detection model.

[0126] In an optional implementation manner, generating a finger template image and a non-finger image based on the original occlusion image includes:

[0127] Based on the original occlusion image, acquiring mask data, wherein the mask data indicates an occluded position in the original occlusion image;

[0128] Based on the original occlusion image and the mask data, a finger template image and a non-finger image are generated.

[0129] For example, based on the original occluded image, the process of generating a finger template image and a non-finger image can be realized through the mask data of the original occluded image. After obtaining the original occluded image, the mask of the finger part in each image is manually marked to obtain the occluded image data and the mask data. The mask data here represents an image of the same size as the original occluded image, with the occluded area marked as 1 and the non-finger occluded area marked as 0. For each image composed of the original occluded image and the corresponding mask data, the sizes and corresponding pixel positions of the two are made consistent through cropping and alignment operations. The original occluded image is set to 0 according to the pixels of the non-occluded area of ​​the mask data to obtain the finger template image, and the original occluded image is set to 0 according to the pixels of the finger occluded area of ​​the mask data to obtain the non-finger image.

[0130] Since the number of original occluded images is much smaller than the number of unoccluded images, it does not require too much manpower for manual labeling. However, through this method, a large number of occluded image features can be synthesized to supplement the deficiencies of the original occluded image features.

[0131] In an optional implementation manner, the training of the data feature enhancement network based on the original occluded image further includes:

[0132] Inputting the finger template image and the non-finger image into the data feature enhancement network to synthesize the generated features of the original occluded image;

[0133] Inputting the original occluded image into a feature network to extract original features of the original occluded image, wherein the occluded image detection model includes the feature network;

[0134] Inputting the generated features of the original occluded image and the original features into a discriminant network for training;

[0135] Obtaining the loss function of the data feature enhancement network and the loss function of the discriminant network in each round of training;

[0136] When the loss function of the data feature enhancement network and the loss function of the discriminant network both converge, it is determined that the training of the data feature enhancement network is completed.

[0137] As mentioned above, the finger template image, non-finger image and original occluded image are combined into a group of training data. Since the finger template image and non-finger image are generated by the original occluded image and the mask data, the finger template image, non-finger image, mask data and original occluded image can also be combined into a group of training data. That is, the data feature enhancement network is trained by multiple groups of training data. In each round of training of the data feature enhancement network, several groups of training data are randomly selected, and a round of training is performed on the data feature enhancement network. During the training process, the loss function of the data feature enhancement network and the discriminant network is obtained. When the loss function converges, it is determined that the training of the data feature enhancement network is completed. The loss function here is, for example, the adversarial loss function of the Hinge version, which will not be repeated here.

[0138] In an optional implementation, the training of the occluded image detection model based on the original occluded image and the unoccluded image and using a trained data feature enhancement network includes:

[0139] Based on the mask data and the unobstructed image, obtaining a processed unobstructed image, wherein the processed unobstructed image is an image obtained by removing the area corresponding to the mask data from the unobstructed image;

[0140] Inputting the finger template image and the processed unobstructed image into the trained data feature enhancement network to synthesize the generated features of the unobstructed image;

[0141] Inputting the original occluded image into a feature network to extract original features of the original occluded image, wherein the occluded image detection model includes the feature network;

[0142] Inputting the generated features of the unobstructed image and the original features into a classification network for training, wherein the occluded image detection model includes the classification network;

[0143] When the loss function of the occluded image detection model converges, it is determined that the training of the occluded image detection model is completed.

[0144] The training process of the occlusion image detection model in this implementation can refer to Figure 3 . Figure 3The process of training the occluded image detection model is shown. It includes a data feature enhancement network, a feature network and a classification network (also called a discriminant network), and the occluded image detection model includes a feature network and a classification network. The data feature enhancement network is used to synthesize enhanced features, that is, occluded image features. The enhanced features here are used to enhance the accuracy of the network's discrimination of image features; the feature network is used to extract image features; and the classification network is used to classify the features of the image. Wherein A represents a finger template image, B represents a non-finger image, C represents an original occluded image, and D represents an augmented feature set. The occluded image features are synthesized through the data feature enhancement network, and the original features of the original occluded image are obtained through the feature network. The synthesized features and the original features constitute an augmented feature set to train the occluded image detection model. During specific training, the gradient descent method can be used for training until the loss function of the occluded image detection model converges. It can be seen that the loss function of the occluded image detection model here is a loss function that is jointly optimized for the feature network and the classification network.

[0145] Figure 4 The trained occlusion image detection model is shown, including a feature network and a classification network. The image to be detected is input into the trained occlusion image detection model, and based on the occlusion characteristics of the image in the occlusion image detection model, it is determined whether the image to be detected is an occlusion image, and the detection result is output.

[0146] In this implementation, the generated features are only used to optimize the classification network of the occlusion detection model, and there is no need to optimize the entire network. The cost of the optimization is much smaller than that of the overall optimization, both in terms of computing resources and time costs, so the additional training cost introduced is relatively small. During training, the existing occlusion detection model can be fully utilized and a small number of parameters can be fine-tuned, resulting in little additional overhead. In addition, since the classification tasks can be modeled as "feature extraction + feature classification" problems, the method of optimizing the feature classification network after feature augmentation in the present disclosure can be adopted for other classification tasks. Therefore, this method has high scalability and universality, and can be easily extended to other classification tasks.

[0147] In an optional implementation, the occluded image detection model is a convolutional neural network model.

[0148] Specifically, the convolutional neural network model can adopt shufflenetv2. This network model has fewer parameters and can save a lot of storage and computing resources.

[0149] In an optional implementation, the data feature enhancement network is a generative adversarial network.

[0150] The data feature enhancement network is a generative adversarial network, specifically a self-attention generative adversarial network. As known to those skilled in the art, a generative adversarial network includes a generative network and a discriminative network.

[0151] The following is a detailed description of a specific embodiment of occluded image detection according to the present disclosure. The data feature enhancement network is a self-attention generative adversarial network, the occluded image detection model is a ShuffleNetv2 network model, and the occluded image is a finger occluded image. Figure 5 As shown, this embodiment includes the following steps:

[0152] Step 501 , obtaining a plurality of original finger occlusion images and unoccluded images, and obtaining mask data of these original finger occlusion images by manual labeling.

[0153] Step 502 : obtaining a finger template image and a non-finger image of each original finger occlusion image based on the original finger occlusion image and the corresponding mask data.

[0154] Step 503: Each original finger occlusion image, the corresponding finger template image and the non-finger image are combined into a set of training data.

[0155] Step 504, in each round of training of the data feature enhancement network, several groups of training data are randomly selected from the training data obtained above, the finger template image and the non-finger image are input into the data feature enhancement network, and the original finger occlusion image is input into the feature network for training.

[0156] Step 505, obtaining the loss function of the data feature enhancement network and the discriminant network in each round of training, and when the loss function converges, determining that the training of the data feature enhancement network is completed.

[0157] Step 506, synthesizing the generated features of the unobstructed image through the trained data feature enhancement network, and extracting the original features of the original occluded image through the feature network.

[0158] Step 507: Input the generated features and the original features into a classification network to train the occluded image detection model.

[0159] Step 508: When the loss function of the occluded image detection model converges, the training is completed.

[0160] Step 509: input the image to be detected into the trained finger occlusion image detection model to determine whether the image to be detected is a finger occlusion image.

[0161] In this embodiment, before using the trained occlusion image detection model to detect the image to be detected, two training processes need to be performed: one is to train the data feature enhancement network, and the other is to train the occlusion image detection model using the trained data feature enhancement network. The training of the data feature enhancement network can refer to Figure 2 , the training of occluded image detection model can refer to Figure 3 .

[0162] The present disclosure also provides an occlusion image detection device, such as Figure 6 As shown, the device comprises:

[0163] The acquisition module 601 is configured to acquire the image as the image to be detected after the image is captured by the camera;

[0164] An input module 602 is configured to input the image to be detected into the trained occlusion image detection model;

[0165] The occluded image detection model 603 is configured to determine whether the image to be detected is an occluded image based on the occlusion characteristics of the image; wherein the occluded image detection model is trained using the original occluded image, the unoccluded image and the trained data feature enhancement network;

[0166] The output module 604 is configured to output the image detection result.

[0167] In an optional implementation, the occluded image detection model 603 is trained by a training module, and the training module is configured as:

[0168] Acquire the original occluded image and the unoccluded image;

[0169] Training the data feature enhancement network based on the original occluded image;

[0170] The occluded image detection model is trained based on the original occluded image and the unoccluded image and using the trained data feature enhancement network.

[0171] In an optional implementation, the training module is further configured to train the data feature enhancement network based on the original occluded image in the following manner:

[0172] Based on the original occluded image, a finger template image and a non-finger image are generated, wherein the finger template image is an image obtained by removing the region not occluded by the finger from the original occluded image, and the non-finger image is an image obtained by removing the region occluded by the finger from the original occluded image;

[0173] Generate training data groups, each group of training data includes a finger template image, a non-finger image and an original occluded image, and use multiple groups of training data to train the data feature enhancement network in each round of training.

[0174] In an optional implementation manner, the training module is further configured to generate a finger template image and a non-finger image based on the original occlusion image in the following manner:

[0175] Based on the original occlusion image, acquiring mask data, wherein the mask data indicates an occluded position in the original occlusion image;

[0176] Based on the original occlusion image and the mask data, a finger template image and a non-finger image are generated.

[0177] In an optional implementation, the training module is further configured to train the data feature enhancement network based on the original occluded image in the following manner:

[0178] Inputting the finger template image and the non-finger image into the data feature enhancement network to synthesize the generated features of the original occluded image;

[0179] Inputting the original occluded image into a feature network to extract original features of the original occluded image, wherein the occluded image detection model includes the feature network;

[0180] Inputting the generated features of the original occluded image and the original features into a discriminant network for training;

[0181] Obtaining the loss function of the data feature enhancement network and the loss function of the discriminant network in each round of training;

[0182] When the loss function of the data feature enhancement network and the loss function of the discriminant network both converge, it is determined that the training of the data feature enhancement network is completed.

[0183] In an optional embodiment, the training module is further configured to train the occluded image detection model based on the original occluded image and the unoccluded image and using the trained data feature enhancement network in the following manner:

[0184] Based on the mask data and the unobstructed image, obtaining a processed unobstructed image, wherein the processed unobstructed image is an image obtained by removing the area corresponding to the mask data from the unobstructed image;

[0185] Inputting the finger template image and the processed unobstructed image into the trained data feature enhancement network to synthesize the generated features of the unobstructed image;

[0186] Inputting the original occluded image into a feature network to extract original features of the original occluded image, wherein the occluded image detection model includes the feature network;

[0187] Inputting the generated features of the unobstructed image and the original features into a classification network for training, wherein the occluded image detection model includes the classification network;

[0188] When the loss function of the occluded image detection model converges, it is determined that the training of the occluded image detection model is completed.

[0189] In an optional implementation, the occluded image detection model is a convolutional neural network model.

[0190] In an optional implementation, the data feature enhancement network is a generative adversarial network.

[0191] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0192] The method disclosed in the present invention has the following beneficial effects:

[0193] (1) A small number of finger occlusion images with mask annotations can be used to stably train the generative model to generate occlusion image features, which can effectively solve the problem of insufficient positive samples in the training of finger occlusion detection models and save a lot of positive sample collection costs.

[0194] (2) Directly generating high-level features of the occluded image can effectively avoid the impact of noise caused by insufficient image generation details on model accuracy. That is, after adding the generated features to the training, the detection accuracy is significantly improved.

[0195] (3) The generated features only optimize the classification network of the finger occlusion detection model again, and the additional training cost introduced is relatively small. In addition, the existing finger occlusion detection model can be fully utilized and a small number of parameters can be fine-tuned, resulting in little additional overhead.

[0196] (4) The network model has fewer parameters, which can save a lot of storage and computing resources.

[0197] (5) This method has high scalability and universality and can be easily extended to other classification tasks.

[0198] Figure 7 is a block diagram of an occlusion image detection device 700 according to an exemplary embodiment.

[0199] Reference Figure 7, the device 700 may include one or more of the following components: a processing component 702 , a memory 704 , a power component 706 , a multimedia component 708 , an audio component 710 , an input / output (I / O) interface 712 , a sensor component 714 , and a communication component 716 .

[0200] The processing component 702 generally controls the overall operation of the device 700, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 702 may include one or more modules to facilitate the interaction between the processing component 702 and other components. For example, the processing component 702 may include a multimedia module to facilitate the interaction between the multimedia component 708 and the processing component 702.

[0201] The memory 704 is configured to store various types of data to support operations on the device 700. Examples of such data include instructions for any application or method operating on the device 700, contact data, phone book data, messages, pictures, videos, etc. The memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0202] The power component 706 provides power to the various components of the device 700. The power component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 700.

[0203] The multimedia component 708 includes a screen that provides an output interface between the device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 708 includes a front camera and / or a rear camera. When the device 700 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0204] The audio component 710 is configured to output and / or input audio signals. For example, the audio component 710 includes a microphone (MIC), and when the device 700 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 704 or sent via the communication component 716. In some embodiments, the audio component 710 also includes a speaker for outputting audio signals.

[0205] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0206] The sensor assembly 714 includes one or more sensors for providing various aspects of status assessment for the device 700. For example, the sensor assembly 714 can detect the open / closed state of the device 700, the relative positioning of components, such as the display and keypad of the device 700, and the sensor assembly 714 can also detect the position change of the device 700 or a component of the device 700, the presence or absence of user contact with the device 700, the orientation or acceleration / deceleration of the device 700, and the temperature change of the device 700. The sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 714 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 714 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0207] The communication component 716 is configured to facilitate wired or wireless communication between the device 700 and other devices. The device 700 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 716 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 716 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0208] In an exemplary embodiment, the apparatus 700 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above methods.

[0209] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, and the instructions can be executed by the processor 720 of the device 700 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0210] A non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to execute an occlusion image detection method, the method comprising: after capturing an image through a camera, obtaining the image and using it as an image to be detected; inputting the image to be detected into a trained occlusion image detection model; wherein the occlusion image detection model is trained using an original occlusion image, an unoccluded image and a trained data feature enhancement network; based on the occlusion characteristics of the image in the occlusion image detection model, determining whether the image to be detected is an occlusion image; and outputting the image detection result.

[0211] Figure 8 8 is a block diagram of a device 800 for detecting an occluded image according to an exemplary embodiment. For example, the device 800 may be provided as a server. Figure 8 , the device 800 includes a processing component 822, which further includes one or more processors, and a memory resource represented by a memory 832, for storing instructions that can be executed by the processing component 822, such as an application. The application stored in the memory 832 may include one or more modules, each of which corresponds to a set of instructions. In addition, the processing component 822 is configured to execute instructions to perform the above method: after capturing an image through a camera, the image is obtained and used as an image to be detected; the image to be detected is input into a trained occlusion image detection model; wherein the occlusion image detection model is trained using an original occlusion image, an unoccluded image and a trained data feature enhancement network; based on the occlusion characteristics of the image in the occlusion image detection model, it is determined whether the image to be detected is an occlusion image; and the image detection result is output.

[0212] The device 800 may also include a power supply component 826 configured to perform power management of the device 800, a wired or wireless network interface 850 configured to connect the device 800 to a network, and an input / output (I / O) interface 858. The device 800 may operate based on an operating system stored in the memory 832, such as Windows Server™, Mac OSX™, Unix™, Linux™, FreeBSD™, or the like.

[0213] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0214] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for detecting an occluded image, characterized in that: The method comprises: After the image is captured by the camera, the image is obtained and used as the image to be detected; Input the image to be detected into the trained occlusion image detection model; wherein the occlusion image detection model is trained by using the original occlusion image, the unoccluded image and the trained data feature enhancement network; the trained data feature enhancement network is a data feature enhancement network trained based on the original occlusion image, and the trained data feature enhancement network is configured to synthesize occlusion image features from the original unoccluded image and mask data, and then obtain the original occlusion image features through the original occlusion image, and train the occlusion image detection model through the synthesized occlusion image features and the original occlusion image features, and the mask data indicates the occluded position in the original occlusion image; the trained occlusion image detection model is configured to determine whether the image to be detected is an occlusion image; the original occlusion image includes a finger occluded area and an area not occluded by a finger; Based on the occlusion characteristics of the image in the occlusion image detection model, determining whether the image to be detected is an occlusion image; wherein the occlusion characteristics of the image are learned by the occlusion image detection model after training; The image detection result is outputted.

2. The method according to claim 1, characterized in that The method further comprises: Acquire the original occluded image and the unoccluded image; Training the data feature enhancement network based on the original occluded image; The occluded image detection model is trained based on the original occluded image and the unoccluded image and using the trained data feature enhancement network.

3. The method according to claim 2, characterized in that The step of training the data feature enhancement network based on the original occluded image comprises: Generate a finger template image and a non-finger image based on the original occluded image, wherein the finger template image is an image obtained by removing the region not occluded by the finger from the original occluded image, and the non-finger image is an image obtained by removing the region occluded by the finger from the original occluded image; Generate training data groups, each group of training data includes a finger template image, a non-finger image and an original occluded image, and use multiple groups of training data to train the data feature enhancement network in each round of training.

4. The method according to claim 3, characterized in that The generating of the finger template image and the non-finger image based on the original occlusion image comprises: Based on the original occlusion image, acquiring the mask data; Based on the original occlusion image and the mask data, a finger template image and a non-finger image are generated.

5. The method according to claim 3, characterized in that The training of the data feature enhancement network based on the original occluded image further includes: Inputting the finger template image and the non-finger image into the data feature enhancement network to synthesize the generated features of the original occluded image; Inputting the original occluded image into a feature network to extract original features of the original occluded image, wherein the occluded image detection model includes the feature network; Inputting the generated features of the original occluded image and the original features into a discriminant network for training; Obtaining the loss function of the data feature enhancement network and the loss function of the discriminant network in each round of training; When the loss function of the data feature enhancement network and the loss function of the discriminant network both converge, it is determined that the training of the data feature enhancement network is completed.

6. The method according to claim 4, characterized in that The method of training the occluded image detection model based on the original occluded image and the unoccluded image and using the trained data feature enhancement network comprises: Based on the mask data and the unobstructed image, obtaining a processed unobstructed image, wherein the processed unobstructed image is an image obtained by removing the area corresponding to the mask data from the unobstructed image; Inputting the finger template image and the processed unobstructed image into the trained data feature enhancement network to synthesize the generated features of the unobstructed image; Inputting the original occluded image into a feature network to extract original features of the original occluded image, wherein the occluded image detection model includes the feature network; Inputting the generated features of the unobstructed image and the original features into a classification network for training, wherein the occluded image detection model includes the classification network; When the loss function of the occluded image detection model converges, it is determined that the training of the occluded image detection model is completed.

7. The method according to claim 1, characterized in that The occluded image detection model is a convolutional neural network model.

8. The method according to claim 1, wherein The data feature enhancement network is a generative adversarial network.

9. An occlusion image detection device, characterized in that: The device comprises: An acquisition module is configured to acquire an image as an image to be detected after the image is captured by a camera; An input module is configured to input the image to be detected into the trained occlusion image detection model; The occluded image detection model is configured to determine whether the image to be detected is an occluded image based on the occlusion characteristics of the image; wherein the occluded image detection model is trained using the original occluded image, the unoccluded image and the trained data feature enhancement network; the trained data feature enhancement network is a data feature enhancement network trained based on the original occluded image, the trained data feature enhancement network is configured to synthesize the occluded image features from the original unoccluded image and the mask data, and then obtain the original occluded image features through the original occluded image, and train the occluded image detection model through the synthesized occluded image features and the original occluded image features, the mask data indicates the occluded position in the original occluded image; the trained occluded image detection model is configured to determine whether the image to be detected is an occluded image; the original occluded image includes a finger occluded area and an area not occluded by a finger; the occlusion characteristics of the image are learned by the occluded image detection model after training; The output module is configured to output the image detection result.

10. The device according to claim 9, characterized in that The occluded image detection model is trained by a training module, and the training module is configured as follows: Acquire the original occluded image and the unoccluded image; Training the data feature enhancement network based on the original occluded image; The occluded image detection model is trained based on the original occluded image and the unoccluded image and using the trained data feature enhancement network.

11. The device according to claim 10, characterized in that The training module is further configured to train the data feature enhancement network based on the original occluded image in the following manner: Generate a finger template image and a non-finger image based on the original occluded image, wherein the finger template image is an image obtained by removing the region not occluded by the finger from the original occluded image, and the non-finger image is an image obtained by removing the region occluded by the finger from the original occluded image; Generate training data groups, each group of training data includes a finger template image, a non-finger image and an original occluded image, and use multiple groups of training data to train the data feature enhancement network in each round of training.

12. The device according to claim 11, characterized in that The training module is further configured to generate a finger template image and a non-finger image based on the original occlusion image in the following manner: Based on the original occlusion image, acquiring the mask data; Based on the original occlusion image and the mask data, a finger template image and a non-finger image are generated.

13. The device according to claim 11, characterized in that The training module is further configured to train the data feature enhancement network based on the original occluded image in the following manner: Inputting the finger template image and the non-finger image into the data feature enhancement network to synthesize the generated features of the original occluded image; Inputting the original occluded image into a feature network to extract original features of the original occluded image, wherein the occluded image detection model includes the feature network; Inputting the generated features of the original occluded image and the original features into a discriminant network for training; Obtaining the loss function of the data feature enhancement network and the loss function of the discriminant network in each round of training; When the loss function of the data feature enhancement network and the loss function of the discriminant network both converge, it is determined that the training of the data feature enhancement network is completed.

14. The device according to claim 12, characterized in that The training module is further configured to train the occluded image detection model based on the original occluded image and the unoccluded image and using the trained data feature enhancement network in the following manner: Based on the mask data and the unobstructed image, obtaining a processed unobstructed image, wherein the processed unobstructed image is an image obtained by removing the area corresponding to the mask data from the unobstructed image; Inputting the finger template image and the processed unobstructed image into the trained data feature enhancement network to synthesize the generated features of the unobstructed image; Inputting the original occluded image into a feature network to extract original features of the original occluded image, wherein the occluded image detection model includes the feature network; Inputting the generated features of the unobstructed image and the original features into a classification network for training, wherein the occluded image detection model includes the classification network; When the loss function of the occluded image detection model converges, it is determined that the training of the occluded image detection model is completed.

15. The device according to claim 9, characterized in that The occluded image detection model is a convolutional neural network model.

16. The device according to claim 9, characterized in that The data feature enhancement network is a generative adversarial network.

17. An occlusion image detection device, characterized in that: include: processor; memory for storing processor executable instructions; Wherein, the processor is configured to: After the image is captured by the camera, the image is obtained and used as the image to be detected; Input the image to be detected into the trained occlusion image detection model; wherein the occlusion image detection model is trained by using the original occlusion image, the unoccluded image and the trained data feature enhancement network; the trained data feature enhancement network is a data feature enhancement network trained based on the original occlusion image, and the trained data feature enhancement network is configured to synthesize occlusion image features from the original unoccluded image and mask data, and then obtain the original occlusion image features through the original occlusion image, and train the occlusion image detection model through the synthesized occlusion image features and the original occlusion image features, and the mask data indicates the occluded position in the original occlusion image; the trained occlusion image detection model is configured to determine whether the image to be detected is an occlusion image; the original occlusion image includes a finger occluded area and an area not occluded by a finger; Based on the occlusion characteristics of the image in the occlusion image detection model, determining whether the image to be detected is an occlusion image; wherein the occlusion characteristics of the image are learned by the occlusion image detection model after training; The image detection result is outputted.

18. A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a router, can perform an occluded image detection method, the method comprising: After the image is captured by the camera, the image is obtained and used as the image to be detected; Input the image to be detected into the trained occlusion image detection model; wherein the occlusion image detection model is trained by using the original occlusion image, the unoccluded image and the trained data feature enhancement network; the trained data feature enhancement network is a data feature enhancement network trained based on the original occlusion image, and the trained data feature enhancement network is configured to synthesize occlusion image features from the original unoccluded image and mask data, and then obtain the original occlusion image features through the original occlusion image, and train the occlusion image detection model through the synthesized occlusion image features and the original occlusion image features, and the mask data indicates the occluded position in the original occlusion image; the trained occlusion image detection model is configured to determine whether the image to be detected is an occlusion image; the original occlusion image includes a finger occluded area and an area not occluded by a finger; Based on the occlusion characteristics of the image in the occlusion image detection model, determining whether the image to be detected is an occlusion image; wherein the occlusion characteristics of the image are learned by the occlusion image detection model after training; The image detection result is outputted.

Images