Image recognition method and device, electronic equipment, storage medium and product

Abstract

Embodiments of the present disclosure disclose an image recognition method and device, electronic equipment, storage medium and product. The method comprises: obtaining a to-be-recognized image; inputting the to-be-recognized image into a preset image recognition model to obtain a recognition result of the to-be-recognized image; the training process of the preset image recognition model comprises: recognizing and extracting a region of interest image in a printed paper attack sample image; combining the region of interest image with at least one background image in a preset background image set to obtain at least one printed paper clipping attack sample image associated with the printed paper attack sample image; inputting the printed paper attack sample image and the printed paper clipping attack sample image into a to-be-trained image recognition model for training to obtain the preset image recognition model. The technical solution disclosed by the embodiments of the present disclosure can resist more types of algorithm attacks and improve security by augmenting different clipping types of printed paper attack sample images to train the image recognition model.

Description

TECHNICAL FIELD

[0001] The embodiments of the present disclosure relate to the technical field of artificial intelligence, and particularly relate to an image recognition method and device, electronic equipment, storage medium and product. BACKGROUND

[0002] At present, in a scenario requiring user identity authentication, an image recognition method of a deep learning model is usually used to determine whether a face in a to-be-identified image is a real face, that is, face liveness detection.

[0003] However, in actual application, the image recognition algorithm may be attacked by various forms of images. If more attack scenarios cannot be covered in the algorithm learning process, the image recognition algorithm will have vulnerabilities. It is difficult to cover as many attack scenarios as possible in the algorithm learning stage, especially in the case of insufficient attack sample data. SUMMARY

[0004] The present disclosure provides an image recognition method, device, electronic equipment, storage medium and product, which can enable the image recognition algorithm to resist different types of printed paper clipping attacks in the application process of image recognition, and improve the accuracy of image recognition and the security of the image recognition algorithm.

[0005] In a first aspect, the embodiments of the present disclosure provide an image recognition method, which comprises:

[0006] obtaining a to-be-identified image;

[0007] inputting the to-be-identified image into a preset image recognition model to obtain an identification result of the to-be-identified image;

[0008] The training process of the preset image recognition model comprises the following steps:

[0009] For a printed paper attack sample image in a preset model training sample image, an image of a region of interest in the printed paper attack sample image is identified and extracted;

[0010] The image of the region of interest is combined with at least one background image in a preset background image set to obtain at least one printed paper clipping attack sample image associated with the printed paper attack sample image;

[0011] The printed paper attack sample image and the printed paper clipping attack sample image are input into a to-be-trained image recognition model for model training to obtain the preset image recognition model.

[0012] In a second aspect, the embodiments of the present disclosure further provide an image recognition device, which comprises:

[0013] An image acquisition module is configured to acquire a to-be-identified image.

[0014] An image identification module is configured to input the to-be-identified image into a preset image identification model to obtain an identification result of the to-be-identified image.

[0015] The preset image identification model is obtained based on a model training device.

[0016] A printed paper sample image processing module is configured to identify and extract a region of interest image in a printed paper attack sample image in the preset model training sample image.

[0017] A sample augmentation module is configured to combine the region of interest image with at least one background image in a preset background image set to obtain at least one printed paper clipping attack sample image associated with the printed paper attack sample image.

[0018] A model training module is configured to input the printed paper attack sample image and the printed paper clipping attack sample image into a to-be-trained image identification model for model training to obtain the preset image identification model.

[0019] In a third aspect, an electronic device is provided, and the electronic device includes:

[0020] One or more processors;

[0021] A storage device configured to store one or more programs,

[0022] When the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the image identification method according to any of the embodiments of the present disclosure.

[0023] In a fourth aspect, a storage medium containing computer executable instructions is provided, and the computer executable instructions are used to execute the image identification method according to any of the embodiments of the present disclosure when executed by a computer processor.

[0024] In a fifth aspect, a computer program product is provided, and the computer program product includes a computer program, and the computer program is used to implement the image identification method according to any of the embodiments of the present disclosure when executed by a processor.

[0025] In the embodiment of the present disclosure, a to-be-recognized image is acquired, and the to-be-recognized image is input into a preset image recognition model to obtain a recognition result of the to-be-recognized image. During training of the preset image recognition model, a printed paper attack sample image in a preset model training sample image is recognized and an image of a region of interest in the printed paper attack sample image is extracted. The image of the region of interest is combined with at least one background image in a preset background image set to obtain at least one printed paper clipping attack sample image associated with the printed paper attack sample image, so as to realize online augmentation of the training sample. Then, the printed paper attack sample image and the corresponding printed paper clipping attack sample image are input into a to-be-trained image recognition model for model training to obtain the preset image recognition model, thereby solving the problem of algorithm vulnerability of an image recognition algorithm in a variety of image attack scenarios, and realizing that the image recognition algorithm can resist different types of printed paper clipping attacks, and improving the accuracy of image recognition and the security of the image recognition algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0026] The above and other features, advantages, and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals refer to the same or similar elements. It is to be understood that the drawings are schematic, and the sizes of the components and elements are not necessarily drawn to scale.

[0027] Figure 1 is a flowchart of an image recognition method provided by an embodiment of the present disclosure;

[0028] Figure 2 is a flowchart of an image recognition model training method provided by an embodiment of the present disclosure;

[0029] Figure 3 is a schematic diagram of a printed paper attack sample image provided by an embodiment of the present disclosure;

[0030] Figure 4 is a schematic diagram of a printed paper attack clipping sample image provided by an embodiment of the present disclosure;

[0031] Figure 5 is a schematic diagram of a printed paper attack clipping sample image provided by an embodiment of the present disclosure;

[0032] Figure 6 is a schematic diagram of a region of interest binary image provided by an embodiment of the present disclosure;

[0033] Figure 7 is a schematic diagram of a clipping edge mask image provided by an embodiment of the present disclosure;

[0034] Figure 8is a schematic diagram of an image recognition model training process provided by an embodiment of the present disclosure.

[0035] Figure 9 is a structural schematic diagram of an image recognition device and a model training device provided by an embodiment of the present disclosure.

[0036] Figure 10 is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] Embodiments of the present disclosure will be described in more detail with reference to the drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms, and should not be interpreted as being limited to the embodiments set forth herein, but rather, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for exemplary purposes only, and are not intended to limit the scope of protection of the present disclosure.

[0038] It should be understood that each step described in the method embodiments of the present disclosure can be performed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0039] The term "comprising" and variations thereof as used herein are open-ended, that is, "comprising but not limited to." The term "based on" is "based, at least in part, on." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments." Related definitions will be given in the description below.

[0040] It should be noted that the concepts of "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units.

[0041] It should be noted that the modification of "one" or "multiple" mentioned in the present disclosure is illustrative and not limiting, and those skilled in the art should understand that, unless otherwise explicitly indicated in the context, it should be understood as "one or more".

[0042] It can be understood that, before using the technical solutions disclosed by the embodiments of the present disclosure, the type of personal information involved in the present disclosure, the scope of use, the scenario of use, etc. should be informed to the user and the authorization of the user should be obtained in accordance with relevant laws and regulations.

[0043] For example, in response to receiving an active request of a user, prompt information is sent to the user to explicitly prompt the user that the operation requested to be performed will require obtaining and using personal information of the user. Thus, the user can autonomously select whether to provide personal information to the software or hardware, such as an electronic device, an application program, a server, or a storage medium, performing the operation of the technical solution of the present disclosure according to the prompt information.

[0044] As an optional but non-limiting implementation, in response to receiving an active request of a user, the prompt information can be sent to the user in the form of a pop-up window, for example, in which the prompt information can be presented in the form of text. In addition, the pop-up window can also carry selection controls for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0045] It can be understood that the above notification and obtaining user authorization process is only illustrative and does not limit the implementation of the present disclosure, and other ways that meet the relevant laws and regulations can also be applied to the implementation of the present disclosure.

[0046] It can be understood that the data involved in the technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of the relevant laws and regulations and the relevant provisions.

[0047] Figure 1 A flowchart of an image recognition method provided by the embodiment of the present disclosure, the embodiment of the present disclosure is applicable to a case of realizing identity authentication based on face recognition. The method can be executed by an image recognition device, which can be realized in the form of software and / or hardware, and can be realized by an electronic device, which can be a mobile terminal, a PC terminal, or a server, etc.

[0048] As shown in Figure 1 The image recognition method comprises:

[0049] S110, obtaining a to-be-recognized image.

[0050] The to-be-recognized image can be an image that needs to be judged whether it is a live face image. The face region of interest in the to-be-recognized image belongs to the foreground region that needs to be recognized in the to-be-recognized image. The to-be-recognized image can be an image collected in a scene requiring identity recognition and authentication, for example, in the scene of account login or transaction information determination, face recognition verification is required, and the image of the user's face is collected in real time through the camera of the terminal where the application client is located. Of course, the to-be-recognized image can also be a fake live face image that can attack the preset image recognition algorithm, so it is necessary to identify whether the collected image is a real live face image. Or, other detection scenes of live face images requiring image recognition can also be applicable to the technical solution of the present embodiment.

[0051] S120, input the to-be-identified image into a preset image recognition model to obtain an identification result of the to-be-identified image.

[0052] The preset image recognition model can be a neural network model for identifying whether an input image is a live face image, can extract and analyze image features according to the input image, and directly output an image recognition result to determine whether the input to-be-identified image is a live face image.

[0053] During the training process of the preset image recognition model, the model training sample images of the print paper type are subjected to online sample augmentation, the types of the sample images can be expanded, and thus the obtained preset image recognition model can resist more types of attack scenes.

[0054] Specifically, the training process of the preset image recognition model includes the following steps:

[0055] Step one, identifying and extracting a region of interest image in the print paper attack sample image in the preset model training sample image.

[0056] The preset model training sample image includes attack sample images of different media screen replay, different print paper, 3D mask and the like, and also includes real face live sample images. During the model training process, the type of the sample image input to the to-be-trained image recognition model is identified, and the class of the sample image input to the to-be-trained image recognition model can be determined through a label of the sample image or a pre-trained image classifier.

[0057] When it is identified that the input image is a print paper attack sample image, the online augmentation process of the training sample is started, the region of interest image in the print paper attack sample image is identified, that is, the to-be-identified object in the sample image is identified, and then the region of interest image can be extracted, that is, the to-be-identified object in the print paper attack image is cropped. The identification of the region of interest image can be realized through any feasible image segmentation algorithm. The region of interest image can be the head of the to-be-identified object, the head and part of the body part, or the local region such as the facial features in the print paper attack sample image.

[0058] Step two, combining the region of interest image with at least one background image in a preset background image set to obtain at least one print paper cropped attack sample image associated with the print paper attack sample image.

[0059] The preset background image set can be a background image of various scenes collected in a network, including indoor scenes, outdoor scenes, etc. A preset number of background images can be randomly selected from the preset background image set and fused with the region of interest image. By fusing the to-be-identified foreground image and the random background image, more model training sample images are obtained through augmentation. Different regions of interest extracted from the original printed paper attack sample image can obtain augmented sample images of different clipping categories.

[0060] The model training sample images are trained, and the region of interest therein is extracted to increase the diversity of the training sample. The cost of purchasing the model training sample images can be reduced to a certain extent. Meanwhile, the online training sample image augmentation manner also reduces the storage space consumption and pressure on the server memory caused by image storage.

[0061] Step three, inputting the printed paper attack sample image and the printed paper clipping attack sample image into a to-be-trained image recognition model to perform model training, and obtaining the preset image recognition model.

[0062] In this step, the original printed paper attack sample image and the printed paper clipping attack sample image obtained by expansion thereof are input into the to-be-trained sample image as sample images of a same training batch to perform model training. Thus, the preset image recognition model obtained through training can recognize the printed paper attack image and the printed paper attack image of different clipping types, and the defense performance of the image recognition model is improved as a whole.

[0063] The technical scheme of the embodiment of the disclosure comprises the following steps: after obtaining a to-be-identified image, inputting the to-be-identified image into a preset image recognition model to obtain an identification result of the to-be-identified image; and in the process of training the preset image recognition model, identifying and extracting a region of interest image in a printed paper attack sample image in the preset model training sample image; combining the region of interest image with at least one background image in a preset background image set to obtain at least one printed paper clipping attack sample image associated with the printed paper attack sample image, and realizing online augmentation of the training sample; then, inputting the printed paper attack sample image and the corresponding printed paper clipping attack sample image into a to-be-trained image recognition model to perform model training, and obtaining the preset image recognition model. The technical scheme solves the problem that the image recognition algorithm has algorithm vulnerabilities in various image attack scenes, realizes that the image recognition algorithm can resist different types of printed paper clipping attacks, and improves the precision of image recognition and the security of the image recognition algorithm.

[0064] Figure 2A flowchart of an image recognition model training method provided by an embodiment of the present disclosure is further described on the basis of the above embodiment content, which further describes the training process of a preset image recognition model. The method can be executed by a model training device, which can be implemented in the form of software and / or hardware, and can be implemented by an electronic device, which can be a mobile terminal, a PC terminal, or a server, etc.

[0065] As shown in Figure 2 The image recognition model training method includes the following steps.

[0066] S210, an image of a region of interest in the print paper attack sample image is identified and extracted.

[0067] In this embodiment, the region of interest includes a head region of a target recognition object in the print paper attack sample image or a region including the head and part of the body contour of the target recognition object, i.e., an externally closed contour region of the target recognition object. The region of interest in the print paper attack sample image can be identified and extracted by an image segmentation or pattern recognition algorithm. Different regions of interest form different types of clipping of the print paper attack sample.

[0068] S220, the image of the region of interest is combined with at least one background image in a preset background image set to obtain at least one print paper clipping attack sample image associated with the print paper attack sample image.

[0069] S230, the print paper attack sample image and the print paper clipping attack sample image are simultaneously input into a to-be-trained image recognition model to obtain a feature map output by each convolution layer in the to-be-trained image recognition model for each sample image.

[0070] The to-be-trained image recognition model can be a convolutional neural network model, and the network structure thereof includes at least one of a residual network structure, a VGG convolutional neural network structure, and an Xception convolutional neural network structure. The specific model structure and model parameters can be selected according to the efficiency and effect of model training.

[0071] The to-be-trained image recognition model includes a plurality of convolution layers, each convolution layer extracts a corresponding feature map (feature map) from each sample image input through a corresponding convolution kernel. The number of convolution kernels in each convolution layer determines the number of feature maps generated after convolution. The convolution network maintains the spatial structure of the input sample image during the learning process, i.e., each feature map and the original sample image have a corresponding spatial relationship, and the specific corresponding position and size can be measured by a receptive field.

[0072] S240, based on the print paper clipping attack sample image clipping edge mask image, the feature map of each sample image output by at least one convolutional layer is extracted to obtain the clipping edge feature of each sample image.

[0073] In order to enable the to-be-trained image recognition model to focus on learning the clipping area of the print paper clipping attack sample image in the model training process, and to mine the features of the clipping edge, the clipping edge mask image of the print paper clipping attack sample image will be used to extract the clipping edge features of the feature map of each sample image output by at least one convolutional layer, and a corresponding model loss function will be set to force the to-be-trained model to learn the clipping edge information of each input sample.

[0074] In the process of clipping edge feature extraction of the print paper clipping attack image, first, the dimension of the clipping edge region mask image is adjusted according to the dimension of the target object feature map to be extracted, that is, the spatial mapping relationship between the clipping edge region mask image and the corresponding feature map is determined.

[0075] Since the input sample image of the to-be-trained model is a color image, the feature map usually has three images of the same dimension, and the clipping edge region mask image is a binary image or a grayscale image, which has only one image. The clipping edge region mask image can be copied to obtain three identical mask images, and the feature extraction is performed on multiple feature maps to obtain the clipping edge region features corresponding to each input image, that is, the dimension-adjusted clipping edge region mask image is used to extract the features of each channel image of the feature map to obtain the clipping edge features.

[0076] The determination of the clipping edge mask image is realized in the process of extracting the region of interest from the print paper attack sample image. First, the extracted region of interest image is binarized to obtain a region of interest binary image. Then, the edges of the binary image are respectively dilated and eroded to obtain a binary edge dilation image and a binary edge erosion image. The difference between the binary edge dilation image and the binary edge erosion image is calculated, and based on the difference between the binary edge dilation image and the binary edge erosion image, the clipping edge mask image is obtained.

[0077] S250, based on each clipping edge feature, the target loss of the to-be-trained image recognition model is determined.

[0078] The mean square error loss between each cropped edge feature extracted from the target loss package of the to-be-trained image recognition model and the cross-entropy loss of the recognition result output by the to-be-trained image recognition model for each sample image can be weighted and summed to obtain the corresponding target loss. The weights of the mean square error loss and the cross-entropy loss are preset hyperparameters.

[0079] S260, iteratively updating the model parameters of the to-be-trained image recognition model based on the target loss to obtain a target image recognition model.

[0080] By inputting the training samples in the to-be-trained image recognition model, including the original printed paper attack sample data and the printed paper clipping attack sample data after clipping processing, and learning the local features corresponding to the clipping edge region of the original printed paper attack sample data and the printed paper clipping attack sample data after clipping processing, the to-be-trained image recognition model can learn the same features in the clipping edge region, thereby improving the attack image recognition accuracy of the trained image recognition model. Therefore, the target image recognition model obtained by training can resist more types of printed paper clipping attacks.

[0081] The technical scheme of the embodiment of the present disclosure realizes the sum of foreground and background by performing foreground extraction on the portrait area of the existing printed paper attack sample image data based on a portrait segmentation algorithm and fusing at least one background image in a preset background image set, so as to realize the augmentation of the model training sample image in an online form. Then, the original printed paper attack sample data and the synthesized printed clipping attack data are input into the to-be-trained model for feature extraction, and the extracted features of these images are supervised to ensure that the model can effectively mine the edge contour of the synthesized / clipping data, solve the problem of algorithm vulnerability of the image recognition algorithm in various image attack scenes, and realize that the trained image recognition algorithm can resist different types of printed paper clipping attacks, thereby improving the accuracy of image recognition and the security of the image recognition algorithm.

[0082] With a specific example, first, the training preparation work is carried out. A certain number of various indoor and outdoor scene graphs can be obtained through different search engines as background graphs, such as 100,000. For these background images, a pre-trained face detector can be used to filter out the background images with faces, so as to avoid the existence of real faces in the subsequent clipping and synthesis data. For the currently available printed paper data, a pre-set portrait segmentation algorithm can be used to extract the portrait area, and the extracted result is stored in the form of RGBA four channels. The fourth channel represents the segmentation result of the portrait. This result can be a probability value of the corresponding pixel point belonging to the foreground portrait. Whether the pixel point belongs to the foreground portrait can be determined according to the probability value.

[0083] When the image recognition model is trained, first, for the training sample image input to the training model, it is judged whether the sample image is a printed paper attack sample data. If yes, a batch of background images (such as 4) are sampled from the pre-collected background image set, and the portrait foreground and background in the printed paper attack sample data are fused to obtain 4 fused data. For example Figure 3 The printed paper attack sample image is shown in Figure 4 and Figure 5 The printed paper attack clipping sample image is shown in Figure 4 and Figure 5 It is the result of online data synthesis (other synthesis graphs are not shown).

[0084] According to the portrait segmentation result of the printed paper attack data in Figure 3 , the portrait segmentation result can be binarized using an image binarization algorithm to obtain the binary image of the region of interest shown in Figure 6 . Then, the binary image of the region of interest is respectively dilated and eroded, and the difference is calculated to obtain the edge region of the portrait and the background fusion, that is, the clipping edge mask image shown in Figure 7 . The clipping edge mask image represents the fusion area of the portrait contour and the background, that is, the clipping edge in the actual clipping attack of black production.

[0085] Specifically, the model training process can refer to the image recognition model training process shown in Figure 8 . The printed paper attack data and the synthesized printed paper clipping attack data are input as training sample images in the same batch (the same training batch can also include other types of sample data) to the image recognition model to be trained, and the features are extracted by the image recognition model to be trained. In this process, the clipping edge mask image is multiplied with the feature map extracted and output by the printed paper attack data and the synthesized printed paper clipping attack data trained together by one convolution layer to obtain the clipping edge feature.

[0086] In the process of training the image recognition model, the target loss function of the model is defined by two parts: the cross-entropy loss (CE Loss) corresponding to the original model training sample data and the cropped synthetic sample image data input into the to-be-trained image recognition model, and the similarity loss between the extracted cropped edge features, where the similarity loss is in the form of mean square error loss (MSE Loss). The features finally extracted by the to-be-trained image recognition model pass through one or more fully connected layers to obtain the corresponding results and the corresponding CEloss. The feature map in the feature extraction process passes through the extraction of the cropped edge features, and each cropped edge feature passes through one or more fully connected layers to calculate the MES Loss. The final target loss is equal to the sum of the CE Loss and the MSE Loss, and the weights of the two are superparameters, for example, they can be taken as 1:0.5. The to-be-trained image recognition model as a feature extraction tool can be any convolutional neural network model structure, such as ResNet structure, VGG structure, Xception structure, etc.

[0087] Figure 9 The structure of the image recognition device and the model training device provided by the embodiments of the present disclosure is shown in a schematic diagram. The device is suitable for realizing identity authentication in a face recognition-based manner. The image recognition device 31 and the model training device 32 can be realized in the form of software and / or hardware and can be configured in an electronic device, which can be a mobile terminal, a PC terminal, or a server, etc.

[0088] As shown in Figure 9 The image recognition device 31 specifically includes an image acquisition module 310 and an image recognition module 320.

[0089] The image acquisition module 310 is configured to acquire a to-be-recognized image, and the image recognition module 320 is configured to input the to-be-recognized image into a preset image recognition model to obtain a recognition result of the to-be-recognized image.

[0090] The model training device 32 includes a printed paper sample image processing module 330, a sample augmentation module 340, and a model training module 350.

[0091] The print paper sample image processing module 330 is configured to identify and extract a region of interest image in the print paper attack sample image in the preset model training sample image.

[0092] In any of the optional technical solutions in the embodiments of the present disclosure, the model training module 350 can be specifically configured to:

[0093] determine the target loss of the to-be-trained image recognition model based on the clipping edge features of the sample images;

[0094] update the model parameters of the to-be-trained image recognition model based on the target loss to obtain a target image recognition model.

[0095] In any of the optional technical solutions in the embodiments of the present disclosure, the model training module 350 can be further configured to:

[0096] input the print paper attack sample image and the print paper clipping attack sample image into the to-be-trained image recognition model at the same time to obtain feature maps of the sample images output by each convolution layer in the to-be-trained image recognition model;

[0097] perform feature extraction on the feature maps of the sample images output by at least one convolution layer based on the clipping edge mask image of the print paper clipping attack sample image to obtain the clipping edge features of the sample images.

[0098] In any of the optional technical solutions in the embodiments of the present disclosure, the model training module 350 can be further configured to:

[0099] calculate a mean square error loss between the clipping edge features and a cross-entropy loss of an identification result output by the to-be-trained image recognition model for each sample image;

[0100] perform weighted summation on the mean square error loss and the cross-entropy loss to obtain the target loss;

[0101] The weights of the mean square error loss and the cross-entropy loss are preset hyperparameters.

[0102] In any of the optional technical solutions in the embodiments of the present disclosure, the model training module 350 can be further configured to:

[0103] adjusting dimensions of the cropped edge region mask image according to dimensions of the feature map;

[0104] performing feature extraction on images of each channel of the feature map respectively by using the cropped edge region mask image with adjusted dimensions, to obtain cropped edge features.

[0105] In any optional technical solution in the embodiments of the present disclosure, the print paper sample image processing module 330 is further configured to obtain the cropped edge mask image, and the specific process includes:

[0106] performing binarization processing on the region of interest image to obtain a region of interest binarization image;

[0107] performing expansion processing and corrosion processing on edges of the binarization image respectively to obtain a binarization edge expansion image and a binarization edge corrosion image;

[0108] obtaining the cropped edge mask image based on a difference between the binarization edge expansion image and the binarization edge corrosion image.

[0109] In any optional technical solution in the embodiments of the present disclosure, the region of interest includes a head region of a target recognition object in the print paper attack sample image or a part or all of a body contour region of the target recognition object including the head region.

[0110] In any optional technical solution in the embodiments of the present disclosure, the image recognition model to be trained includes at least one of a residual network structure, a VGG convolutional neural network structure and an Xception convolutional neural network structure.

[0111] The above device provided by the embodiments of the present disclosure can perform the method provided by any of the embodiments of the present disclosure, and has the corresponding function modules and beneficial effects of performing the method.

[0112] The technical scheme of the embodiment of the present disclosure comprises the following steps: obtaining a to-be-recognized image; inputting the to-be-recognized image into a preset image recognition model to obtain a recognition result of the to-be-recognized image; in the process of training the preset image recognition model, identifying and extracting a region of interest image in a printed paper attack sample image in a preset model training sample image; combining the region of interest image with at least one background image in a preset background image set to obtain at least one printed paper clipping attack sample image associated with the printed paper attack sample image, thereby realizing online augmentation of the training sample; then, inputting the printed paper attack sample image and the corresponding printed paper clipping attack sample image into a to-be-trained image recognition model for model training to obtain the preset image recognition model, thereby solving the problem of algorithm vulnerabilities of the image recognition algorithm in various image attack scenarios, and realizing that the image recognition algorithm can resist different types of printed paper clipping attacks, thereby improving the accuracy of image recognition and the security of the image recognition algorithm.

[0113] It is worth noting that each unit and module included in the above apparatus is only divided according to functional logic, but is not limited to the above division, as long as the corresponding function can be implemented; in addition, the specific name of each functional unit is only for easy mutual distinction, and does not limit the protection scope of the embodiments of the present disclosure.

[0114] Figure 10 A structural schematic diagram of an electronic device provided by the embodiments of the present disclosure is shown in FIG. 4. The following refers to Figure 10 which shows a structural schematic diagram of an electronic device (for example, a terminal device or a server in Figure 10 The terminal device in the embodiments of the present disclosure can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablets), PMPs (portable multimedia players), vehicle-mounted terminals (for example, vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 10 The electronic device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present disclosure.

[0115] As Figure 10As shown, the electronic device 400 can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 402 or loaded into a random access memory (RAM) 403 from a storage device 408. Various programs and data required for the operation of the electronic device 400 are also stored in the RAM 403. The processing device 401, the ROM 402, and the RAM 403 are connected to each other through a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0116] Generally, the following devices can be connected to the I / O interface 405: input devices 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 408 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 409. The communication devices 409 can allow the electronic device 400 to communicate wirelessly or wired with other devices to exchange data. Although Figure 10 The electronic device 400 is shown with various devices, but it should be understood that not all of the shown devices are required to be implemented or possessed. More or fewer devices can alternatively be implemented or possessed.

[0117] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 409, or installed from the storage devices 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the methods of embodiments of the present disclosure are performed.

[0118] The names of messages or information exchanged between the plurality of devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0119] The electronic device provided by the embodiments of the present disclosure and the image recognition method provided by the above-mentioned embodiments belong to the same inventive concept, and the technical details not described in detail in the present embodiments can be referred to the above-mentioned embodiments, and the present embodiments have the same beneficial effects as the above-mentioned embodiments.

[0120] The embodiment of the present disclosure provides a computer storage medium, which stores a computer program, and the program is executed by a processor to implement the image recognition and image recognition model training method provided by the above embodiment.

[0121] It should be noted that the computer readable medium of the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present disclosure, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained in the computer readable medium can be transmitted by any suitable medium, including but not limited to a wire, a cable, an RF (radio frequency) or the like, or any suitable combination of the above.

[0122] In some embodiments, the client, server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.

[0123] The above computer readable medium can be included in the above electronic device; or can exist separately without being assembled into the electronic device.

[0124] The computer readable medium described above carries one or more programs, when the one or more programs are executed by the electronic device, cause the electronic device to:

[0125] obtain an image to be recognized;

[0126] input the image to be recognized into a preset image recognition model to obtain a recognition result of the image to be recognized;

[0127] The training process of the preset image recognition model includes the following steps:

[0128] For a print paper attack sample image in a preset model training sample image, an image of a region of interest in the print paper attack sample image is recognized and extracted;

[0129] The image of the region of interest is combined with at least one background image in a preset background image set to obtain at least one print paper clipping attack sample image associated with the print paper attack sample image;

[0130] The print paper attack sample image and the print paper clipping attack sample image are input into a to-be-trained image recognition model for model training to obtain the preset image recognition model.

[0131] Computer program code for carrying out operations of the present disclosure can be written in one or more programming languages or combinations of languages including object oriented programming languages such as Java, Smalltalk, C++ or conventional procedural programming languages such as "C" or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0132] The computer program product of the first aspect can include one or more non-transitory computer-readable media storing instructions that, when executed, cause one or more processors to perform the operations of the first aspect. The one or more non-transitory computer-readable media can include, for example, magnetic media such as one or more magnetic disks, magnetic tapes or cassettes; optical media such as one or more compact discs, optical discs or Blu-ray discs; magneto-optical media such as one or more floptical discs; solid state media such as one or more solid state drives or other flash memory arrays; or any suitable combination of these. The one or more non-transitory computer-readable media can be encoded with instructions that, when executed, cause one or more processors to perform the operations of the first aspect.

[0133] The units described in the embodiments of the present disclosure can be implemented by software, or by hardware, or by a combination of software and hardware. In some cases, the name of the unit does not constitute a limitation on the unit itself. For example, the first obtaining unit can also be described as a unit for obtaining at least two Internet protocol addresses.

[0134] The functions described in this document can be implemented in part or in whole using one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0135] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more of: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0136] The embodiment of the present disclosure further provides a computer program product comprising a computer program which, when executed by a processor, implements the image recognition method provided by any embodiment of the present disclosure.

[0137] The computer program product can be written in one or more programming languages or combinations of languages including object-oriented programming languages such as Java, Smalltalk, C++ and conventional procedural programming languages such as "C" or the like and combinations of languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0138] According to one or more embodiments of the present disclosure, Example One provides an image recognition method, which comprises:

[0139] obtaining an image to be recognized;

[0140] inputting the image to be recognized into a preset image recognition model to obtain a recognition result of the image to be recognized;

[0141] The training process of the preset image recognition model comprises the following steps:

[0142] For a printed paper attack sample image in a preset model training sample image, an image of a region of interest in the printed paper attack sample image is recognized and extracted;

[0143] The image of the region of interest is combined with at least one background image in a preset background image set to obtain at least one printed paper clipping attack sample image associated with the printed paper attack sample image;

[0144] The printed paper attack sample image and the printed paper clipping attack sample image are input into a to-be-trained image recognition model for model training to obtain the preset image recognition model.

[0145] According to one or more embodiments of the present disclosure, Example Two provides an image recognition method, which further comprises:

[0146] In some optional implementations, the inputting the print paper attack sample image and the print paper clipping attack sample image into the image recognition model to be trained for model training to obtain the preset image recognition model comprises:

[0147] determining a target loss of the image recognition model to be trained based on the clipping edge features of the sample images;

[0148] updating model parameters of the image recognition model to be trained based on the target loss to obtain a target image recognition model.

[0149] According to one or more embodiments of the present disclosure, Example Three provides an image recognition method, further comprising:

[0150] In some optional implementations, the determining process of the clipping edge features comprises:

[0151] inputting the print paper attack sample image and the print paper clipping attack sample image into the image recognition model to be trained to obtain feature maps output by each convolutional layer of the image recognition model to be trained for each sample image;

[0152] performing feature extraction on the feature maps of the sample images output by at least one convolutional layer based on the clipping edge mask image of the print paper clipping attack sample image to obtain the clipping edge features of the sample images.

[0153] According to one or more embodiments of the present disclosure, Example Four provides an image recognition method, comprising:

[0154] In some optional implementations, the determining a target loss of the image recognition model to be trained based on the clipping edge features of the sample images comprises:

[0155] calculating a mean square error loss between the clipping edge features and a cross-entropy loss of an identification result output by the image recognition model to be trained for each sample image;

[0156] performing weighted summation on the mean square error loss and the cross-entropy loss to obtain the target loss;

[0157] wherein the weights of the mean square error loss and the cross-entropy loss are preset hyperparameters.

[0158] According to one or more embodiments of the present disclosure, Example Five provides an image recognition method, further comprising:

[0159] In some optional implementations, the performing feature extraction on the feature maps of the sample images output by at least one convolutional layer based on the clipping edge mask image of the print paper clipping attack sample image comprises:

[0160] adjusting a dimension of the cropped edge region mask image according to a dimension of the feature map;

[0161] performing feature extraction on images of each channel of the feature map respectively by using the dimension-adjusted cropped edge region mask image, to obtain cropped edge features.

[0162] According to one or more embodiments of the present disclosure, Example Six provides an image recognition method, further comprising:

[0163] In some optional implementations, the process of obtaining the cropped edge mask image comprises:

[0164] performing binarization processing on the region-of-interest image to obtain a region-of-interest binarization image;

[0165] performing expansion processing and corrosion processing on the edges of the binarization image respectively to obtain a binarization edge expansion image and a binarization edge corrosion image;

[0166] obtaining the cropped edge mask image based on a difference between the binarization edge expansion image and the binarization edge corrosion image.

[0167] According to one or more embodiments of the present disclosure, Example Seven provides an image recognition method, further comprising:

[0168] In some optional implementations, the region of interest includes a head region of a target recognition object in the printed paper attack sample image or a partial or whole body contour region of the target recognition object including the head region.

[0169] According to one or more embodiments of the present disclosure, Example Eight provides an image recognition method, further comprising:

[0170] In some optional implementations, the to-be-trained image recognition model includes at least one of a residual network structure, a VGG convolutional neural network structure, and an Xception convolutional neural network structure.

[0171] According to one or more embodiments of the present disclosure, Example Nine provides an image recognition device, comprising:

[0172] an image acquisition module configured to acquire a to-be-recognized image;

[0173] an image recognition module configured to input the to-be-recognized image into a preset image recognition model to obtain a recognition result of the to-be-recognized image;

[0174] wherein the preset image recognition model is obtained based on a model training device, and the model training device comprises:

[0175] The paper sample image processing module is used to identify and extract the region of interest image from the paper attack sample image in the training sample image of the preset model;

[0176] The sample augmentation module is used to combine the region of interest image with at least one background image from a preset set of background images to obtain at least one paper-cutting attack sample image associated with the paper-cutting attack sample image.

[0177] The model training module is used to input the printed paper attack sample image and the printed paper cropping attack sample image into the image recognition model to be trained for model training, so as to obtain the preset image recognition model.

[0178] According to one or more embodiments of this disclosure, [Example 10] provides an image recognition device, further comprising:

[0179] In one optional implementation, the model training module is specifically used for:

[0180] The target loss of the image recognition model to be trained is determined based on the cropping edge features of each sample image.

[0181] The model parameters of the image recognition model to be trained are iteratively updated based on the target loss to obtain the target image recognition model.

[0182] According to one or more embodiments of this disclosure, [Example 11] provides an image recognition device, further comprising:

[0183] In an optional implementation, the model training module is further configured to:

[0184] The paper attack sample image and the paper trimming attack sample image are simultaneously input into the image recognition model to be trained, and feature maps of each sample image are obtained through each convolutional layer in the image recognition model to be trained.

[0185] Based on the clipping edge mask image of the paper clipping attack sample image, feature extraction is performed on the feature map of each sample image output by at least one convolutional layer to obtain the clipping edge features of each sample image.

[0186] According to one or more embodiments of this disclosure, [Example Twelve] provides an image recognition device, further comprising:

[0187] In an optional implementation, the model training module can also be used for:

[0188] compute a mean square error loss between each of the cropped edge features, and a cross-entropy loss of an identification result output by the image identification model to be trained for each sample image;

[0189] weighting and summing the mean square error loss and the cross-entropy loss to obtain the target loss;

[0190] wherein the weights of the mean square error loss and the cross-entropy loss are preset hyperparameters.

[0191] According to one or more embodiments of the present disclosure, Example Thirteen provides an image identification device, further comprising:

[0192] In an optional implementation, the model training module can be further configured to:

[0193] adjusting a dimension of the cropped edge region mask image according to a dimension of the feature map;

[0194] performing feature extraction on images of each channel of the feature map using the dimension-adjusted cropped edge region mask image to obtain cropped edge features.

[0195] According to one or more embodiments of the present disclosure, Example Fourteen provides an image identification device, further comprising:

[0196] In an optional implementation, the printed paper sample image processing module is further configured to obtain the cropped edge mask image, and the specific process includes:

[0197] performing binaryzation processing on the region of interest image to obtain a region of interest binaryzation image;

[0198] performing dilation processing and erosion processing on edges of the binaryzation image respectively to obtain a binaryzation edge dilation image and a binaryzation edge erosion image;

[0199] obtaining the cropped edge mask image based on a difference between the binaryzation edge dilation image and the binaryzation edge erosion image.

[0200] According to one or more embodiments of the present disclosure, Example Fifteen provides an image identification device, further comprising:

[0201] In an optional implementation, the region of interest includes a head region of a target identification object in the printed paper attack sample image or a part or whole body contour region of the target identification object including the head region.

[0202] According to one or more embodiments of the present disclosure, Example Sixteen provides an image identification device, further comprising:

[0203] In an alternative implementation, the image recognition model to be trained comprises at least one of a residual network structure, a VGG convolutional neural network structure, and an Xception convolutional neural network structure.

[0204] The above description is merely that of the preferred embodiments of the disclosure and of the principles thereof. It is to be understood that the disclosed scope of the disclosure is not limited to the specific configurations described above and that the other technical solutions formed by any combination of the above technical features or equivalent features thereof without departing from the above disclosed concept should also be encompassed. For example, the technical solutions formed by replacing the above features with the technical features disclosed in the disclosure (but not limited to) having similar functions.

[0205] Furthermore, although the operations have been depicted in a particular order, this should not be understood as requiring such order of operations. Rather, the operations can be performed in any order or concurrently that is advantageous. Likewise, although specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the disclosure. Rather, certain features described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented separately or in any suitable subcombination. It will be appreciated that various features described herein can form part of a system, an apparatus, and / or a method. It will be further appreciated that a data storage medium or a computer program product carrying out a computer program can be configured to cause a computer to carry out the methods described herein.

[0206] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. An image recognition method, characterized in that, include: Acquire the image to be recognized; The image to be recognized is input into a preset image recognition model to obtain the recognition result of the image to be recognized; The training process of the preset image recognition model includes the following steps: For paper attack sample images in the training sample images of a preset model, identify and extract the region of interest (ROI) image from the paper attack sample image. The region of interest image is combined with at least one background image from a preset set of background images to obtain at least one paper cutting attack sample image associated with the paper attack sample image. The paper attack sample image and the paper trimming attack sample image are simultaneously input into the image recognition model to be trained, and feature maps of each sample image are obtained through each convolutional layer in the image recognition model to be trained. Based on the clipping edge mask image of the paper clipping attack sample image, feature extraction is performed on the feature map of each sample image output by at least one convolutional layer to obtain the clipping edge features of each sample image. The target loss of the image recognition model to be trained is determined based on the cropping edge features of each sample image. The model parameters of the image recognition model to be trained are iteratively updated based on the target loss to obtain the target image recognition model.

2. The method according to claim 1, characterized in that, The determination of the target loss of the image recognition model to be trained based on the cropping edge features of each sample image includes: Calculate the mean squared error loss between each of the cropping edge features, and the cross-entropy loss of the recognition results output by the image recognition model to be trained for each sample image; The target loss is obtained by weighted summation of the mean squared error loss and the cross-entropy loss. The weights of the mean squared error loss and the cross-entropy loss are preset hyperparameters.

3. The method according to claim 1, characterized in that, The cropping edge mask image based on the printed paper cropping attack sample image is used to extract features from the feature maps of each sample image output by at least one convolutional layer, including: Adjust the dimensions of the cropping edge mask image according to the dimensions of the feature map; The clipping edge features are obtained by extracting features from each channel of the feature map using the dimension-adjusted clipping edge mask image.

4. The method according to claim 1, characterized in that, The process of acquiring the cropping edge mask image includes: The region of interest image is binarized to obtain a binarized image of the region of interest; The edges of the binarized image are subjected to dilation and erosion processing respectively to obtain a binarized edge dilation image and a binarized edge erosion image; The clipping edge mask image is obtained based on the difference between the binarized edge dilation image and the binarized edge erosion image.

5. The method according to any one of claims 1-4, characterized in that, The region of interest includes the head region of the target object in the printed paper attack sample image, or part or all of the body contour region of the target object, including the head region.

6. The method according to any one of claims 1-4, characterized in that, The image recognition model to be trained includes at least one of the following: residual network structure, VGG convolutional neural network structure, and Xception convolutional neural network structure.

7. An image recognition device, characterized in that, include: The image acquisition module is used to acquire the image to be recognized; An image recognition module is used to input the image to be recognized into a preset image recognition model to obtain the recognition result of the image to be recognized; The preset image recognition model is obtained based on a model training device, which includes: The paper sample image processing module is used to identify and extract the region of interest image from the paper attack sample image in the training sample image of the preset model; The sample augmentation module is used to combine the region of interest image with at least one background image from a preset set of background images to obtain at least one paper-cutting attack sample image associated with the paper-cutting attack sample image. The model training module is used to simultaneously input the printed paper attack sample image and the printed paper trimming attack sample image into the image recognition model to be trained, and obtain the feature map output by each convolutional layer of the image recognition model to be trained for each sample image; based on the trimming edge mask image of the printed paper trimming attack sample image, feature extraction is performed on the feature map of each sample image output by at least one convolutional layer to obtain the trimming edge features of each sample image; the target loss of the image recognition model to be trained is determined based on the trimming edge features of each sample image; and the model parameters of the image recognition model to be trained are iteratively updated based on the target loss to obtain the target image recognition model.

8. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the image recognition method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the image recognition method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image recognition method as described in any one of claims 1-6.

Images