Image-based object recognition and / or verification method and apparatus

Abstract

Provided are image-based object recognition and / or verification methods and apparatuses. The method includes receiving an image of the object to input a convolutional neural network; extracting output results of at least two convolutional layers in the convolutional neural network; combining the output results of the at least two convolutional layers to generate an identification code representing the object; and performing recognition and / or verification of the object based on the identification code. In this way, the accuracy of object recognition and / or verification is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer, in particular to the identification and / or verification of objects. BACKGROUND

[0002] The identification and / or verification of objects based on images highly depends on the surface texture and / or specific feature points of the images. Currently, algorithms using convolutional neural networks (CNN) have been developed to identify the texture features of images. However, the current methods of extracting image texture features using convolutional neural networks focus more on object classification based on the extracted image texture features. Such methods often fail to achieve satisfactory results in the application of object identification and / or verification. SUMMARY

[0003] An improved method of object identification and / or verification is provided, which not only focuses on the deep feature information obtained by convolutional neural networks, but also focuses on the shallow feature information obtained by convolutional neural networks, so as to extract feature information more suitable for object identification and / or verification, and improve the accuracy of object identification and verification.

[0004] It has been recognized that the identification and / or verification of objects based on images is different from the classification of objects. In object classification, more attention is paid to the most fundamental features that distinguish the current object from other objects, and therefore, when designing an object classification algorithm based on convolutional neural networks, a deeper and deeper network structure, more and more convolutional layers, and multiple pooling layers are adopted to extract features that can most distinguish the current object from objects of other categories, so as to facilitate the classification of objects. However, when identifying and / or verifying objects, it is not simply to distinguish the category of the object, but to identify and / or verify the identity information of the current object, i.e. not only to distinguish the current object from objects of other categories, but also to distinguish the current object from other objects of the same category, so as to identify and / or verify a specific object. Therefore, when identifying and / or verifying objects, the shallow feature information in the image is likely to help distinguish the current object from other objects of the same category, which is useful for identifying and / or verifying objects. However, the deeper and deeper network structure, more and more convolutional layers, and multiple pooling layers involved in the current convolutional neural network-based method for object classification are likely to cause the output result to ignore these shallow feature information, so that such method is difficult to achieve satisfactory results in object identification and / or verification.

[0005] According to various embodiments of the present disclosure, by extracting and combining the outputs of at least two layers in a convolutional neural network, multi-level feature information is constituted for generating an identification code of an object, whereby in object recognition and / or verification, multiple levels of feature information of an object image, including shallow feature information and deep feature information, are focused on. Sufficient information can be provided for object recognition and / or verification to accurately distinguish a current object from other objects, uniquely identify and / or verify the object.

[0006] According to one aspect, an image-based object recognition and / or verification method is provided. The method includes receiving an image of the object to input a convolutional neural network; extracting output results of at least two convolutional layers in the convolutional neural network; combining the output results of the at least two convolutional layers to generate an identification code representing the object; and performing recognition and / or verification of the object based on the identification code.

[0007] According to another aspect, an image-based object recognition and / or verification device is provided. The device includes a convolution processing unit configured to receive an image of the object to input a convolutional neural network; an extracting unit configured to extract output results of at least two convolutional layers in the convolutional neural network; a combining unit configured to combine the output results of the at least two convolutional layers to generate an identification code representing the object; and a recognition / verification unit configured to perform recognition and / or verification of the object based on the identification code.

[0008] According to another aspect, an image-based object recognition and / or verification system is provided. The system includes a memory storing computer program instructions; and a processor executing the computer program instructions to perform the method according to any one of the various embodiments of the present disclosure.

[0009] According to still another aspect, a machine-readable storage medium storing computer program instructions which, when executed, cause a computer to perform the method according to the various embodiments of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0010] In the drawings, only examples of the embodiments are illustrated by way of example and not by way of limitation, in which like reference numerals refer to similar elements.

[0011] Figure 1 A network structure framework diagram of an image-based object recognition and / or verification method according to one embodiment is shown;

[0012] Figure 2 An image-based object recognition and / or verification device according to one embodiment is shown;

[0013] Figure 3A schematic diagram of an electronic device of an image-based object recognition and / or verification system according to one embodiment is shown.

[0014] Various aspects and features of various embodiments of the present application are described with reference to the above figures. The above figures are merely schematic and are not limiting in any regard; the dimensions, shapes, labels, or appearance of various elements in the above figures can be varied without departing from the spirit of the present application; moreover, various parts of the earphone or device of the embodiments of the present application are not all labeled with reference numerals in the above figures, and only relevant components are labeled in certain figures, which does not limit the various parts to only what is shown in the figures of the specification. DETAILED DESCRIPTION

[0015] Figure 1 A network structure framework diagram of an image-based object recognition and / or verification method 100 according to one embodiment is shown.

[0016] As shown in Figure 1 , an image I of an object is input into a convolutional neural network 110 for processing. The convolutional neural network 110 can be a convolutional neural network with any known network structure, including but not limited to ResNet, ZFNet, AlexNet, or VGG16. It is contemplated that the convolutional neural network 110 is a VGG16 network for high-resolution images, which is capable of obtaining more image detail information, thereby improving the accuracy of recognition and / or verification.

[0017] The convolutional neural network includes multiple convolutional layers. As shown in Figure 1 , the convolutional neural network includes three convolutional layers C1, C2, and C3. It is also contemplated that C1, C2, and C3 as shown in Figure 1 represent three convolutional modules, each module involving more than one convolutional layer.

[0018] The output results of at least two of the multiple convolutional layers are extracted to facilitate recognition and / or verification of the object. Figure 1 A case where the output results of three convolutional layers are extracted is shown. Specifically, the output result O1 from the convolutional layer C1, the output result O2 from the convolutional layer C2, and the output result O3 from the convolutional layer C3 are extracted. The convolutional layer C3 is the deepest convolutional layer of the convolutional neural network. The output results from different convolutional layers combine to form multi-level feature information.

[0019] By extracting the outputs of at least two convolutional layers from multiple convolutional layers in a convolutional neural network structure and combining these outputs, an identification code for the object is generated based on not only the deep feature information of the image but also the shallow feature information of the image. This provides sufficient information to accurately distinguish the current object from other objects in object recognition and / or verification, and uniquely identify and / or verify the object.

[0020] The outputs from different convolutional layers are different sets of feature maps with varying dimensions. To combine the outputs from different convolutional layers, at least a portion of them is processed to ensure that the outputs from each convolutional layer have the same size for easy combination. For example, the outputs of each convolutional layer except the deepest one are processed to reduce their size to the same size as the output of the deepest convolutional layer, facilitating subsequent combination of the extracted outputs from different convolutional layers. Alternatively, it is also expected that each output from different convolutional layers is processed to reduce its size so that the outputs from different convolutional layers have the same size.

[0021] like Figure 1 As shown, the outputs O1 from convolutional layer C1 and O2 from convolutional layer C2 are downsampled by 120 so that their dimensions are the same as the output O3 from convolutional layer C3. The downsampling process includes pooling, which may include max pooling, mean pooling, etc.

[0022] After pooling, processed outputs O11, O12, and O13 of the same size are obtained. These outputs O11, O12, and O13 are then stacked (e.g., concatenated) to obtain a set of feature maps. This set of feature maps includes information from convolutional layers of different depths and can be referred to as the multi-level feature information of the object. The multi-level feature information obtained after this stacking process has a high dimensionality. For example, if the dimension of output O11 is 7*7*256, and the dimensions of outputs O12 and O13 are both 7*7*512, then the dimension of the multi-level feature information obtained after the stacking process will be 7*7*1280.

[0023] To reduce the dimensionality of the multi-level feature information for easier processing, the multi-level feature information output after overlay processing can be dimensionality reduced by 140, thereby obtaining a set of feature maps with reduced dimensionality. For example, a 1*1 convolution can be performed on the overlay output to reduce its dimensionality.

[0024] The multi-level feature information of reduced dimension is generated by the superposition processing 130 and the dimension reduction processing 140, which not only contains the deep-level feature information of the identified and / or verified object but also includes the shallow-level feature information of the object, which is helpful for the identification and / or verification of the object.

[0025] Although the multi-level feature information can be generated as an identification code representing the identified and / or verified object, it is desirable to further process to further reduce the size and / or dimension of the multi-level feature information and extract more critical feature information therein.

[0026] Specifically, as Figure 1As shown, in one embodiment, to further process the multi-level feature information to generate an identification code for the object, the multi-level feature information is processed in two branches. In the first branch, the multi-level feature information is weighted 150, giving higher weights to relatively important features in the feature map (i.e., features that better characterize the object), and relatively lower weights to relatively less important features (i.e., features that are difficult to distinguish the object from other objects). This set of weights can be set based on the experience of a technician or obtained through machine learning. For example, the weighting methods described in the following documents can be used for the aforementioned weighting processing: Jie Hu, Li Shen, et al., “Squeeze-and-excitation networks”, published in Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7132-7141) in 2018; “Stand-alone self-attention in vision models”, published in Advances in Neural Information Processing Systems (pp. 68-80) in 2019; and “Deep ten: Texture encoding network”, published in Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 708-717) in 2017. The full text of these documents can be incorporated into this disclosure for illustrative purposes. Other weighting methods are also anticipated. The weighted multi-level feature information is processed by a fully connected layer 160 to obtain the first fully connected output.

[0027] In the second branch, such as Figure 1 As shown, the multi-level feature information is downsampled 170 to further reduce the size of the output result, for example, through pooling. The downsampled multi-level feature information is then subjected to a fully connected processing 180 to obtain a second fully connected output.

[0028] Further processing 190 is performed on the first and second fully connected outputs to generate an identification code representing the object, specifically, the first fully connected output and the second fully connected output are superimposed, e.g. fully connected, to generate the identification code. It has been verified that for the identification and / or verification of the object, superimposing the first and second fully connected outputs can obtain an identification code representing the object better, thus identifying and / or verifying the object more accurately.

[0029] It is also contemplated that the above-mentioned down-sampling processing 170 is omitted, and the multi-level feature information, i.e. the set of feature maps, obtained after the dimensionality reduction processing 140 is directly subjected to the fully connected processing 180 to obtain the second fully connected output.

[0030] In the first branch, the multi-level feature information, i.e. the set of feature maps, is weighted according to experience or machine learning on previous object images, i.e. those feature information in the set of feature maps that are more capable of representing the object is weighted, while in the second branch, as much comprehensive information as possible in the multi-level feature information is included, and the results from the two branches are combined, highlighting the feature information capable of representing the object while retaining as much comprehensive information in the multi-level feature information as possible. Thus, even if the weight distribution in the weighting processing is different from the distribution of the feature information in the current object image that is more capable of representing the object, by setting the second branch, those feature information capable of representing the object will not be completely ignored, so that the generated identification code still includes the unique feature information of the object even in this case, so as to facilitate the identification and / or verification of the object.

[0031] Although the above-mentioned two branches are shown, this is not necessary, and it is also contemplated that the identification code of the object is generated only through one of the branches. After the identification code of the object is generated, the identification and / or verification of the object can be performed based on the identification code of the object.

[0032] Figure 2 An image-based object identification and / or verification device 10 according to an embodiment is shown. The device 10 includes a convolution processing unit 1, an extraction unit 2, a combination unit 3 and an identification / verification unit 4.

[0033] The convolution processing unit 1 receives an image I of an object to input a convolutional neural network. The extraction unit 2 extracts output results of at least two convolutional layers in the convolutional neural network. The combination unit 3 combines the extracted output results to generate multi-level feature information, based on which an identification code representing the object is generated, and the identification / verification unit 4 identifies and / or verifies the object based on the identification code.

[0034] In one embodiment, the combining unit 3 further comprises a down-sampling unit 31, a stacking unit 32 and an identification code generating unit 33. The down-sampling unit 31 processes the output results of a part of the at least two convolutional layers so that the output results extracted from each convolutional layer have the same size for facilitating the combining. The stacking unit 32 stacks the processed output results to obtain multi-level feature information, i.e. a set of feature maps. The identification code generating unit 33 generates an identification code of the object based on the set of feature maps.

[0035] In one embodiment, the stacking unit further performs dimension reduction on the stacked output results to obtain a set of feature maps, i.e. multi-level feature information.

[0036] The identification code generating unit 33 can output the set of feature maps as the identification code of the object. Alternatively, the identification code generating unit 33 further comprises a weighting unit 331, a first fully connected processing unit 332, an additional down-sampling unit 333, a second fully connected processing unit 334 and a generating unit 335.

[0037] The weighting unit 331 is configured to perform weighting processing on the set of feature maps from the stacking unit 32. The first fully connected processing unit 332 is configured to perform fully connected processing on the weighted set of feature maps to obtain a first fully connected output. The additional down-sampling unit 333 is configured to process the set of feature maps to reduce the size thereof. The second fully connected processing unit 334 is configured to perform fully connected processing on the set of feature maps to obtain a second fully connected output. The generating unit 335 is configured to generate the identification code of the object based on the first fully connected output and the second fully connected output, in particular by performing concatenation processing on the first fully connected output and the second fully connected output to generate the identification code.

[0038] As described above with reference to Figure 1 and 2 , the method and the device according to the embodiments of the present disclosure are described. The above device can be implemented in hardware, or implemented in software or a combination of hardware and software.

[0039] Figure 3 A schematic diagram of an electronic device 60 of an image-based object recognition and / or verification system according to embodiments of the present disclosure is shown. As shown in Figure 3 , the electronic device 60 can include at least one processor 61, a memory (e.g. a non-volatile memory) 62, an internal memory 63 and a communication interface 64, and the at least one processor 61, the memory 62, the internal memory 63 and the communication interface 64 are connected together via a bus 66. The at least one processor 61 executes at least one computer-readable instruction (i.e. the above-mentioned elements implemented in software) stored or encoded in the memory.

[0040] In one embodiment, computer-executable instructions are stored in memory that, when executed, cause at least one processor 61 to: receive a first verification result indicating a success of a verification of a first login request based on a first account system stored outside the system using a first verification unit disposed outside the system; verify a second login request based on a second account system and generate a second verification result indicating a success of the verification when the verification is successful; initialize a login state for the first login request based on the first verification result; and initialize a login state for the second login request based on the second verification result.

[0041] It should be understood that the computer-executable instructions stored in memory, when executed, cause at least one processor 61 to perform various operations and functions described above in connection with various embodiments of the present disclosure. Figure 1 and 2 It should be understood that the computer-executable instructions stored in memory, when executed, cause at least one processor 61 to perform various operations and functions described above in connection with various embodiments of the present disclosure.

[0042] According to one embodiment, a program product such as a machine-readable medium (for example, a non-transitory machine-readable medium) is provided. The machine-readable medium can have instructions (i.e., the above-mentioned elements implemented in software) stored on it that, when executed by a machine, cause the machine to perform various operations and functions described above in connection with various embodiments of the present specification. Specifically, a system or apparatus equipped with a readable storage medium on which software program codes implementing the functions of any of the above-mentioned embodiments are stored, and a computer or processor of the system or apparatus can be provided to read out and execute the instructions stored in the readable storage medium. Figure 1 2 According to one embodiment, a program product such as a machine-readable medium (for example, a non-transitory machine-readable medium) is provided. The machine-readable medium can have instructions (i.e., the above-mentioned elements implemented in software) stored on it that, when executed by a machine, cause the machine to perform various operations and functions described above in connection with various embodiments of the present specification. Specifically, a system or apparatus equipped with a readable storage medium on which software program codes implementing the functions of any of the above-mentioned embodiments are stored, and a computer or processor of the system or apparatus can be provided to read out and execute the instructions stored in the readable storage medium.

[0043] Exemplary embodiments of the present disclosure cover both creating / using a computer program / software of the present disclosure from the beginning and converting an existing program / software into a computer program / software of the present disclosure by means of updating.

[0044] A computer program for performing the methods according to the embodiments of the present disclosure can also be presented over a network like the Internet and can be downloaded into the working memory of a microprocessor.

[0045] A computer program can also be provided on a network like the Internet and can be downloaded into the working memory of a microprocessor from such a network.

[0046] ​It must be noted that embodiments of the present disclosure are described with reference to different specific terms. In particular, some embodiments are described with reference to method claims, while other embodiments are described with reference to apparatus claims. However, a person of ordinary skill in the art will gather from the above description and the following detailed description that any combination of features from different embodiments is also considered to be disclosed herein. Also, all features can be combined in one embodiment, providing a synergistic effect that is more than a simple sum of the features.

[0047] The above described specific embodiments of the present disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order and still accomplish the desired result. Also, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain embodiments, multitasking and parallel processing can be advantageous or necessary.

[0048] The present disclosure is described above with reference to specific embodiments. Those skilled in the art will understand that the technical solutions of the present disclosure can be implemented in various manners without departing from the spirit and essential characteristics of the present disclosure. The specific embodiments are merely illustrative and not restrictive. In addition, these embodiments can be combined in any manner to achieve the purpose of the present disclosure. The scope of protection of the present disclosure is defined by the appended claims.

[0049] The word "comprise" in the specification and claims does not exclude the presence of other elements or steps, the word "first", "second", "step" etc. and the order of the steps shown in the figures do not limit the order or the number of the steps. The functions of each element described in the specification or recited in the claims can be split into or combined into corresponding multiple elements or a single element.

Claims

1. An image-based object recognition and / or verification method, comprising: receiving an image of the object to input a convolutional neural network; extracting output results of at least two convolutional layers in the convolutional neural network; combining the output results from the at least two convolutional layers to obtain a set of feature maps; performing weighting processing on the set of feature maps; performing fully connected processing on the weighted set of feature maps to obtain a first fully connected output; performing fully connected processing on the set of feature maps to obtain a second fully connected output; generating a recognition code of the object based on the first fully connected output and the second fully connected output; and performing recognition and / or verification of the object based on the recognition code. combining the output results of the at least two convolutional layers to obtain a set of feature maps comprises: processing at least output results of a portion of the at least two convolutional layers so that the output results from the at least two convolutional layers have the same size; and 2. The object identification and / or verification method of claim 1, wherein, stacking the output results from the at least two convolutional layers to obtain the set of feature maps.

3. The object recognition and / or verification method of claim 2, further comprising: performing dimension reduction processing on the stacked output results to obtain the set of feature maps.

4. The object recognition and / or verification method of claim 1, further comprising: performing processing on the set of feature maps to reduce their size; performing the fully connected processing on the processed set of feature maps to obtain the second fully connected output.

5. The object recognition and / or verification method of claim 4, further comprising: performing concatenation processing on the first fully connected output and the second fully connected output to generate the recognition code of the object.

6. An image-based object recognition and / or verification apparatus, comprising: a convolution processing unit configured to receive an image of the object to input a convolutional neural network; an extracting unit configured to extract output results of at least two convolutional layers in the convolutional neural network; a combining unit configured to combine the output results from the at least two convolutional layers to obtain a set of feature maps; to perform weighting processing on the set of feature maps; to perform fully connected processing on the weighted set of feature maps to obtain a first fully connected output; to perform fully connected processing on the set of feature maps to obtain a second fully connected output; and to generate a recognition code of the object based on the first fully connected output and the second fully connected output; and a recognition / verification unit configured to perform recognition and / or verification of the object based on the recognition code. the combining unit further comprises: a down-sampling unit configured to process at least output results of a portion of the at least two convolutional layers so that the output results from the at least two convolutional layers have the same size; a stacking unit configured to stack the output results from the at least two convolutional layers to obtain a set of feature maps; and 7. Object recognition and / or verification device according to claim 6, wherein a recognition code generating unit configured to generate the recognition code of the object based on the set of feature maps. the stacking unit is further configured to perform dimension reduction processing on the stacked output results to obtain the set of feature maps. the recognition code generating unit further comprises:

8. The object recognition and / or verification device according to claim 7, wherein, ​ ​ 9. Object recognition and / or verification device according to claim 7 or 8, wherein, ​ a weighting unit configured to perform a weighting process on the set of feature maps; a first fully connected processing unit configured to perform a fully connected process on the weighted set of feature maps to obtain a first fully connected output; a second fully connected processing unit configured to perform a fully connected process on the set of feature maps to obtain a second fully connected output; and a generating unit configured to generate the identification code of the object based on the first fully connected output and the second fully connected output.

10. The object recognition and / or verification device according to claim 9, wherein, The identification code generating unit further comprises: an additional down-sampling unit configured to perform a process on the set of feature maps to reduce a size thereof; wherein the second fully connected processing unit performs the fully connected process on the processed set of feature maps to obtain the second fully connected output.

11. The object recognition and / or verification device of claim 9, wherein, The generating unit is configured to perform a concatenation process on the first fully connected output and the second fully connected output to generate the identification code of the object.

12. An image-based object identification and / or verification system comprising: a memory storing computer program instructions; and a processor running the computer program instructions to perform the method according to any one of claims 1-5.

13. A machine-readable storage medium storing computer program instructions which, when run by a computer or processor, cause the computer or processor to perform the method according to any one of claims 1-5.

Images