Method and device for identifying target object

Abstract

The embodiments of the present disclosure disclose a method, device and system for identifying a target object. The method includes: classifying the target objects to be identified in a target image to determine a predicted category of the target objects to be identified; determining whether the predicted category is correct based on the hidden layer features of the target objects to be identified; and outputting a prompt message in response to the predicted category being incorrect.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a national phase application of the PCT application with application number PCT / IB2020 / 061574 filed on December 7, 2020, and the PCT application claims priority to the Singapore patent application with application number 10202007348T filed on August 1, 2020. The entire contents of the patent application are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of computer vision technology, and in particular to a method and device for identifying a target object. Background Art

[0004] In daily production and life, it is often necessary to identify some target objects. Take the entertainment scene of board games as an example. In some board games, it is necessary to identify the game coins on the table to obtain the category and quantity information of the game coins. However, the traditional recognition method has a low recognition accuracy and cannot determine the target objects that do not belong to the current scene. Summary of the invention

[0005] The present disclosure provides a target object recognition solution.

[0006] According to one aspect of the present disclosure, a method for identifying a target object is provided, the method comprising:

[0007] Classify the target object to be identified in the target image to determine the predicted category of the target object to be identified; determine whether the predicted category is correct based on the hidden layer features of the target object to be identified; and output prompt information in response to the predicted category being incorrect.

[0008] In combination with any embodiment provided in the present disclosure, the method further includes: in response to the predicted category being correct, determining the predicted category as the final category of the target object to be identified, and outputting the final category of the target object to be identified.

[0009] In combination with any embodiment provided by the present disclosure, determining whether the predicted category is correct based on the hidden features of the target object to be identified includes: inputting the hidden features of the target object to be identified into a true or false identification model corresponding to the predicted category, so that the true or false identification model outputs a probability value; wherein the true or false identification model corresponding to the predicted category reflects the distribution law of the hidden features of the target object of the predicted category, and the probability value represents the probability that the final category of the target object to be identified is the predicted category; when the probability value is less than a probability threshold, determining that the predicted category is incorrect; when the probability value is greater than or equal to the probability threshold, determining that the predicted category is correct.

[0010] In combination with any embodiment provided by the present disclosure, the target image includes multiple superimposed target objects to be identified; classifying the target objects to be identified in the target image and determining the predicted category of the target objects to be identified, including: adjusting the height of the target image to a preset height, wherein the target image is cropped from the captured image according to the detection frames of the multiple superimposed target objects to be identified in the captured image, and the height direction of the target image is the superimposed direction of the multiple superimposed target objects to be identified; classifying the target objects to be identified in the adjusted target image and determining the predicted category of the target objects to be identified.

[0011] In combination with any embodiment provided by the present disclosure, adjusting the height of the target image to the preset height includes: scaling the height and width of the target image proportionally until the width of the target image reaches the preset width; when the width of the scaled target image reaches the preset width and the height of the scaled target image is greater than the preset height, reducing the height and width of the scaled target image proportionally until the height of the scaled target image is equal to the preset height.

[0012] In combination with any embodiment provided in the present disclosure, adjusting the height of the target image to the preset height includes: scaling the height and width of the target image proportionally until the width of the target image reaches the preset width; when the width of the scaled target image reaches the preset width and the height of the scaled target image is less than the preset height, filling the scaled target image with the first pixel so that the height of the filled target image is the preset height.

[0013] In combination with any embodiment provided by the present disclosure, the target objects to be identified in the adjusted target image are classified to determine the predicted category of the target objects to be identified, including: extracting features from the adjusted target image to obtain a feature map, wherein the height dimension of the feature map corresponds to the height direction of the target image; performing average pooling on the feature map along the width dimension of the feature map to obtain a pooled feature map; segmenting the pooled feature map along the height dimension to obtain a preset number of features; and determining the predicted category of each target object to be identified among the multiple stacked target objects to be identified based on each feature.

[0014] In combination with any embodiment provided by the present disclosure, classifying the target objects to be identified in the adjusted target image to determine the predicted categories of the target objects to be identified is performed by a neural network, and the neural network includes a classification network; wherein the classification network includes K classifiers, wherein K is the number of known categories when performing classification, and k is a positive integer; determining the predicted categories of each of the multiple superimposed target objects to be identified based on the each feature, including: respectively calculating the cosine similarity between each feature and the weight vector of each classifier; and determining the predicted categories of each of the multiple superimposed target objects to be identified based on the calculated cosine similarity.

[0015] In combination with any embodiment provided by the present disclosure, classifying the target object to be identified in the adjusted target image to determine the predicted category of the target object to be identified is performed by a neural network, the neural network includes a feature extraction network, the feature extraction network includes multiple convolutional layers, the last N convolutional layers of the multiple convolutional layers of the feature extraction network have a step size of 1 in the height dimension of the feature map, and N is a positive integer.

[0016] In combination with any embodiment provided by the present disclosure, the target objects to be identified in the target image are classified using a neural network, and the authenticity recognition model corresponding to the predicted category is established using the hidden layer features of the authentication target object of the predicted category, and the authentication target object is correctly predicted in the neural network training stage and / or testing stage.

[0017] According to one aspect of the present disclosure, a device for identifying a target object is provided, the device comprising: a classification unit, used to classify a target object to be identified in a target image, and determine a predicted category of the target object to be identified; a determination unit, used to determine whether the predicted category is correct based on hidden layer features of the target object to be identified; and a prompt unit, used to output a prompt message in response to the predicted category being incorrect.

[0018] In combination with any embodiment provided in the present disclosure, the device further includes an output unit for determining the predicted category as the final category of the target object to be identified in response to the predicted category being correct, and outputting the final category of the target object to be identified.

[0019] In combination with any embodiment provided by the present disclosure, the determination unit is used to: input the hidden layer features of the target object to be identified into the authenticity recognition model corresponding to the predicted category, so that the authenticity recognition model outputs a probability value; wherein the authenticity recognition model corresponding to the predicted category reflects the distribution law of the hidden layer features of the target object of the predicted category, and the probability value represents the probability that the final category of the target object to be identified is the predicted category; when the probability value is less than a probability threshold, it is determined that the predicted category is incorrect; when the probability value is greater than or equal to the probability threshold, it is determined that the predicted category is correct.

[0020] In combination with any embodiment provided in the present disclosure, the target image includes multiple superimposed target objects to be identified; the classification unit is used to: adjust the height of the target image to a preset height, wherein the target image is cropped from the acquired image according to the detection frames of the multiple superimposed target objects to be identified in the acquired image, and the height direction of the target image is the superimposed direction of the multiple superimposed target objects to be identified; classify the target objects to be identified in the adjusted target image to determine the predicted category of the target objects to be identified.

[0021] In combination with any embodiment provided in the present disclosure, the classification unit is used to: scale the height and width of the target image proportionally until the width of the target image reaches a preset width; when the width of the scaled target image reaches the preset width and the height of the scaled target image is greater than the preset height, reduce the height and width of the scaled target image proportionally until the height of the scaled target image is equal to the preset height.

[0022] In combination with any embodiment provided in the present disclosure, the classification unit is used to: scale the height and width of the target image proportionally until the width of the target image reaches the preset width; when the width of the scaled target image reaches the preset width and the height of the scaled target image is less than the preset height, fill the scaled target image with the first pixel so that the height of the filled target image is the preset height.

[0023] In combination with any embodiment provided by the present disclosure, the classification unit is used to: perform feature extraction on the adjusted target image to obtain a feature map, wherein the height dimension of the feature map corresponds to the height direction of the target image; perform average pooling on the feature map along the width dimension of the feature map to obtain a pooled feature map; segment the pooled feature map along the height dimension to obtain a preset number of features; and determine the predicted category of each of the multiple stacked target objects to be identified based on each feature.

[0024] In combination with any embodiment provided by the present disclosure, classifying the target objects to be identified in the adjusted target image to determine the predicted categories of the target objects to be identified is performed by a neural network, and the neural network includes a classification network; wherein the classification network includes K classifiers, K is the number of known categories when performing classification, and k is a positive integer; determining the predicted categories of each of the multiple target objects to be identified in the stacked target objects according to the each feature includes: respectively calculating the cosine similarity between each feature and the weight vector of each classifier; and determining the predicted categories of each of the multiple target objects to be identified in the stacked target objects according to the calculated cosine similarity.

[0025] In combination with any embodiment provided by the present disclosure, classifying the target object to be identified and determining the predicted category of the target object to be identified is performed by a neural network, wherein the neural network includes a feature extraction network, and the feature extraction network includes multiple convolutional layers. The last N convolutional layers of the multiple convolutional layers of the feature extraction network have a step size of 1 in the height dimension of the feature map, and N is a positive integer.

[0026] In combination with any embodiment provided by the present disclosure, the target objects to be identified in the target image are classified using a neural network, and the authenticity recognition model corresponding to the predicted category is established using the hidden layer features of the authentication target object of the predicted category, and the authentication target object is correctly predicted in the neural network training stage and / or testing stage.

[0027] According to one aspect of the present disclosure, an electronic device is provided, the device comprising a memory and a processor, the memory being used to store computer instructions executable on the processor, the processor being used to implement the target object recognition method described in any embodiment of the present disclosure when executing the computer instructions.

[0028] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the target object recognition method described in any embodiment of the present disclosure is implemented.

[0029] According to one aspect of the present disclosure, a computer program is provided. The computer program is stored on a computer-readable storage medium, wherein when the computer program is executed by a processor, the target object recognition method described in any embodiment of the present disclosure is implemented.

[0030] One or more embodiments of the present disclosure provide a target object recognition system, method, apparatus, device and storage medium, which classify target objects to be recognized in a target image, determine a predicted category of the target objects to be recognized, that is, determine to which of the known categories the target objects to be recognized belong; and determine whether the predicted category is correct based on the hidden features of the target objects to be recognized, and output prompt information if the predicted category is incorrect, so that target objects that do not belong to known categories, that is, target objects that do not belong to the current scene, can be recognized and prompts can be given.

[0031] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments of the present disclosure and are used to explain the technical solutions of the present disclosure together with the specification.

[0033] Figure 1 A flow chart of a method for identifying a target object provided by at least one embodiment of the present disclosure;

[0034] Figure 2 A flow chart of a method for classifying a target object to be identified in a target image provided by at least one embodiment of the present disclosure;

[0035] Figure 3 A schematic diagram of the structure of a target object recognition device provided by at least one embodiment of the present disclosure;

[0036] Figure 4 A schematic diagram of the structure of an electronic device provided for at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this disclosure.

[0038] The terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items. In addition, the term "at least one" herein means any combination of at least two of any one or more of a plurality of.

[0039] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0040] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present disclosure and to make the above-mentioned purposes, features and advantages of the embodiments of the present disclosure more obvious and understandable, the technical solutions in the embodiments of the present disclosure are further described in detail below in conjunction with the accompanying drawings.

[0041] Figure 1 is a flow chart of a target object identification method provided by at least one embodiment of the present disclosure. Figure 1 As shown, the method may include steps 101 to 103.

[0042] In step 101, a target object to be identified in a target image is classified to determine a predicted category of the target object to be identified.

[0043] In some embodiments, the target object to be identified may include sheet objects of various shapes, such as game coins. The target object to be identified may be a single target object, or one or more of a plurality of target objects stacked together. The thickness (height) of each target object stacked together is usually the same.

[0044] The multiple target objects to be identified contained in the target image are usually stacked along the thickness direction, the multiple game coins are stacked along the vertical direction (stand stack), the height direction (H) of the target image is the vertical direction, and the width direction (W) of the target image is the direction perpendicular to the height direction (H) of the target image; or, the multiple game coins are stacked along the horizontal direction (float stack), the height direction (H) of the target image is the horizontal direction, and the width direction (W) of the target image is the direction perpendicular to the height direction (H) of the target image.

[0045] In the disclosed embodiment, a classification network, such as a convolutional neural network (CNN), can be used to classify the target object to be identified to determine the predicted category of the target object to be identified. The classification network may include K classifiers, where K is the number of known categories when performing classification, and K is a positive integer. By classifying the target object to be identified, it can be determined which category of the known categories the target object to be identified belongs to. It should be noted that since the classification network judges the probability that the target object to be identified belongs to each known category based on the feature information (hidden layer features) of the target object to be identified, and determines the category with the highest probability as the predicted category to which the target object to be identified belongs, even if the target object to be identified does not belong to any known category, the classification network will always output one of the known categories as the classification result, i.e., the predicted category.

[0046] In step 102, it is determined whether the predicted category is correct based on the hidden layer features of the target object to be identified.

[0047] In specific implementation, the authenticity recognition model corresponding to the predicted category can be used to determine whether the predicted category is correct according to the hidden features of the target object to be identified; wherein the authenticity recognition model corresponding to a predicted category reflects the distribution law of the hidden features of the target object of the predicted category, and since the authenticity recognition model reflects the distribution law of the hidden features of the target objects of the same category, it is possible to determine whether the predicted category is correct. The authenticity recognition model can be a probability distribution model established according to the hidden features of the target objects of the same category.

[0048] In a specific implementation process, the authenticity identification model may include a Gaussian probability distribution model, or other models that can reflect the distribution law of hidden layer features of target objects of the same category.

[0049] For the hidden features input into a true or false identification model corresponding to a predicted category, the true or false identification model can output the probability value that the input hidden features belong to the hidden features of the target object of the predicted category, thereby determining whether the input hidden features belong to the hidden features of the target object of the predicted category. When the probability value is greater than or equal to the probability threshold, it is determined that the predicted category determined in step 101 is correct; when the probability value is less than the probability threshold, it is determined that the predicted category determined in step 101 is incorrect, that is, the true category of the target object to be identified does not belong to the known category during classification in step 101, but an unknown category. Among them, the hidden features of the target object refer to the features before being input into the classifier in the classification network when the target object is classified using the classification network.

[0050] In step 103, in response to the predicted category being incorrect, a prompt message is output.

[0051] In the disclosed embodiment, K true and false identification models can be established for K known categories. The K categories can be all categories of target objects in the current scene. For target objects outside these K categories, they can be considered as objects that do not belong to the current scene, or are called foreign objects, and their categories are unknown categories.

[0052] For a target object to be identified whose predicted category is incorrect, it indicates that the target object to be identified does not actually belong to any of the known categories but is an unknown category, that is, it can be determined that the target object to be identified does not belong to the current scene and is an external object.

[0053] In the example, in response to the predicted category being incorrect, that is, the target object to be identified is a foreign object, a prompt message of "unknown category" may be output.

[0054] In some embodiments, the target objects to be identified in the target image are classified to determine the predicted category of the target objects to be identified, that is, to determine which category of the known categories the target objects to be identified belong to; since the authenticity recognition model reflects the distribution law of hidden features of target objects of the same category, the authenticity recognition model corresponding to the predicted category can be used to determine whether the predicted category is correct based on the hidden features of the target objects to be identified. If the predicted category is incorrect, a prompt message is output to identify the target object that does not belong to any known category, that is, does not belong to the current scene, and give a prompt.

[0055] When the target image includes multiple target objects to be identified, when one of the target objects to be identified is a target object of unknown category, prompt information may be output to prompt relevant personnel that a target object of unknown category is mixed in the multiple target objects to be identified.

[0056] If the predicted category of the target object to be identified is correct, the predicted category may be determined as the final category of the target object to be identified, and the final category of the target object to be identified is output.

[0057] In some embodiments, whether the predicted category determined in step 101 is correct may be determined in the following manner.

[0058] The hidden features of the target object to be identified are input into the authenticity recognition model corresponding to the predicted category, so that the authenticity recognition model corresponding to the predicted category outputs a probability value, and the probability value represents the probability that the final category of the target object to be identified is the predicted category. When the probability value is less than a probability threshold, it is determined that the predicted category is incorrect; when the probability value is greater than or equal to the probability threshold, it is determined that the predicted category is correct.

[0059] Since the authenticity recognition model reflects the distribution law of hidden features of target objects of the same category, the authenticity recognition model corresponding to the predicted category is used to determine the probability that the hidden features of the input target object to be recognized belong to the hidden features of the target object of the predicted category. In the case where the probability value output by the authenticity recognition model is less than the probability threshold, it can be determined that the hidden features of the input target object to be recognized do not belong to the hidden features of the target object of the predicted category, so that it can be determined that the predicted category determined in step 101 is incorrect; on the contrary, in the case where the probability value output by the authenticity recognition model is greater than or equal to the probability threshold, it can be determined that the hidden features of the input target object to be recognized belong to the hidden features of the target object of the predicted category, so that it can be determined that the predicted category determined in step 101 is correct.

[0060] In some embodiments, the target object to be identified may be classified by the following method.

[0061] First, a target image is acquired. The target image is cropped from the acquired image according to the detection frames of multiple target objects stacked in the acquired image, and the height direction of the target image is the stacking direction of the multiple target objects. The target object to be identified may be one or more of the multiple target objects stacked together, for example, the target object to be identified is all of the multiple target objects stacked upright in the vertical direction, or one of the multiple target objects stacked sideways in the horizontal direction.

[0062] A target image including multiple target objects standing upright (called a side view image) can be captured by an image acquisition device arranged at the side of the target area, or a target image including multiple target objects standing sideways (called a top view image) can be captured by an image acquisition device arranged above the target area.

[0063] Next, the height of the target image is adjusted to a preset height, and the target object to be identified in the adjusted target image is classified to determine the predicted category of the target object to be identified.

[0064] In the embodiment of the present disclosure, by adjusting the height of the target image to a uniform height, it is more suitable for processing the hidden layer features, which is beneficial to improving the recognition accuracy of the target object.

[0065] In some embodiments, the height of the target image may be adjusted to a preset height in the following manner.

[0066] First, a preset height and a preset width corresponding to the target image are obtained to resize the target image, wherein the preset width can be set according to the average width of the target object, and the preset height can be set according to the average height of the target object and the maximum number of target objects to be identified.

[0067] In one example, the height and width of the target image may be scaled in proportion until the width of the target image reaches a preset width. Proportional scaling refers to enlarging or reducing the target image while maintaining the ratio between the height and width of the target image unchanged. The preset width and the preset height may be expressed in pixels or other units, which is not limited in the present disclosure.

[0068] When the width of the scaled target image reaches the preset width and the height of the scaled target image is greater than the preset height, the height and width of the scaled target image are proportionally reduced until the height of the scaled target image is equal to the preset height.

[0069] For example, assuming that the target object is a game coin, the preset width can be set to 224pix (pixels) based on the average width of the game coin; the preset height can be set to 1344pix based on the average height of the game coin and the maximum number of game coins to be identified, such as 72. First, the width of the target image can be adjusted to 224pix, and the height of the target image can be adjusted proportionally. In the case where the adjusted height is greater than 1344pix, the height of the adjusted target image can be adjusted again so that the height of the target image is 1344pix, and the width of the target image can be adjusted proportionally, thereby adjusting the height of the target image to the preset height of 1344pix. In the case where the adjusted height is equal to 1344pix, there is no need to adjust it again, that is, the height of the target image has been adjusted to the preset height of 1344pix.

[0070] In one example, the height and width of the target image are scaled proportionally until the width of the target image reaches a preset width; when the width of the scaled target image reaches the preset width and the height of the scaled target image is less than the preset height, the scaled target image is filled with the first pixel so that the height of the filled target image is the preset height.

[0071] The first pixel may be a pixel with a pixel value of (127, 127, 127), that is, a gray pixel. The first pixel may also be set to other pixel values, and the specific pixel value does not affect the effect of the embodiment of the present disclosure.

[0072] Still taking the target object as a game coin, the preset width is 224pix, the preset height is 1344pix, and the maximum number is 72 as an example, the width of the target image can be adjusted to 224pix first, and the height of the target image can be adjusted proportionally. If the adjusted height is less than 1344pix, the height part less than 1344pix is ​​filled with gray pixels, so that the height of the filled target image is 1344pix. If the adjusted height is equal to 1344pix, no filling is required, that is, the height of the target image has been adjusted to the preset height of 1344pix.

[0073] After the height of the target image is adjusted to a preset height, the target object to be identified in the adjusted target image can be classified.

[0074] Figure 2 A flow chart of a method for classifying a target object to be identified in a target image proposed by at least one embodiment of the present disclosure is shown. Figure 2 As shown, the method includes steps 301 to 304.

[0075] In step 301, feature extraction is performed on the adjusted target image to obtain a feature map.

[0076] In one example, the obtained feature map may include multiple dimensions, such as channel dimension, height dimension, width dimension, batch dimension, etc. The format of the feature map may be expressed as [BCHW], where B represents the batch dimension, C represents the channel dimension, H represents the height dimension, and W represents the width dimension. The height dimension of the feature map corresponds to the height direction of the target image, and the width dimension corresponds to the width direction of the target image.

[0077] In step 302, the feature map is average pooled along the width dimension of the feature map to obtain a pooled feature map.

[0078] By performing average pooling along the width dimension of the feature map, keeping the height dimension and channel dimension unchanged, a pooled feature map is obtained.

[0079] For example, when the feature map is 2048*72*8 (channel dimension is 2048, height is 72, width is 8), after average pooling in the width dimension, a feature map of 2048*72*1 is obtained.

[0080] In step 303, the pooled feature map is segmented along the height dimension to obtain a preset number of features.

[0081] By segmenting the pooled feature map in the height dimension, a preset number of features can be obtained, wherein each segment of features can be considered to correspond to a target object. The preset number is also the maximum number of target objects to be identified.

[0082] For example, the maximum number is 72. The pooled feature map in the above example is segmented in the height dimension, that is, the 2048*72*1 feature map is split along the height dimension to obtain 72 2048-dimensional vectors, each of which corresponds to the feature corresponding to the 1 / 72 area in the height direction of the target image. One feature can be represented by a 2048-dimensional vector.

[0083] In step 304, the predicted category of each target object to be identified is determined according to each feature.

[0084] In the disclosed embodiment, when the height of the adjusted target image is less than the preset height, the adjusted target image will be filled so that the height reaches the preset height; and when the height of the adjusted target image is greater than the preset height, the height of the adjusted target image is reduced to the preset height, and the width of the adjusted target image is proportionally reduced, so the feature map of the target image is obtained according to the target image of the preset height. In addition, since the preset height is set according to the maximum number of target objects to be identified, the feature map is segmented according to the maximum number, and each segmented feature (also referred to as a feature) obtained corresponds to a target object, and the target object is identified according to each segmented feature, which can reduce the influence of the number of target objects and improve the accuracy of identifying each target object. In addition, since the number of target objects contained in the target image may be different in different recognition processes, the height-to-width ratio of the target image may be very different. By adjusting the target image by maintaining the height-to-width ratio, the deformation of the image is reduced, and the recognition accuracy can be further improved.

[0085] In some embodiments, when the features corresponding to the portion filled by the first pixel, such as the gray pixel, in the filled target image are classified, the classification result is empty. The number of target objects contained in the target image can be determined based on the number of non-empty classification results obtained.

[0086] Assuming that the maximum number of target objects to be identified is 72, the feature map of the adjusted target image is divided into 72 segments, and the target object is identified according to the features of each segment, then 72 classification results can be obtained. In the case where the target image includes a gray pixel filled area, the classification result of the target object corresponding to the feature of the filled area is empty. For example, when 16 empty results are obtained, 56 non-empty classification results are obtained, so it can be determined that the target image contains 56 target objects.

[0087] Those skilled in the art should understand that the above preset width, preset height, and maximum number of target objects to be identified are examples, and the specific values ​​of these parameters can be set according to actual needs, which is not limited in the embodiments of the present disclosure.

[0088] In some embodiments, classifying the target object to be identified in the adjusted target image to determine the predicted category of the target object to be identified is performed by a neural network, and the neural network includes a classification network; the classification network includes K classifiers, where K is the number of known categories when performing classification, and k is a positive integer.

[0089] The neural network can determine the predicted category of each target object to be identified based on the various features obtained by segmenting the pooled feature map along the height dimension.

[0090] First, the cosine similarity between each feature and the weight vector of each classifier is calculated respectively.

[0091] In one example, before calculating the cosine similarity, the weight vectors of each classifier may be normalized, and each feature input to the classifier may be normalized to improve the classification accuracy of the neural network.

[0092] Next, the predicted category of each of the multiple target objects to be identified is determined according to the calculated cosine similarity.

[0093] For each feature, the cosine similarity between the feature and the weight vector of each classifier is calculated, and the category of the classifier with the largest cosine similarity is used as the predicted category of the target object to be identified corresponding to the feature.

[0094] By determining the predicted category of the target object to be identified corresponding to each feature according to the cosine similarity between each feature and the weight vector of each classifier, the classification effect of the classification network can be improved.

[0095] In some embodiments, the neural network includes a feature extraction network. The feature extraction network may include multiple convolutional layers, or multiple convolutional layers and multiple pooling layers, etc. After multiple layers of feature extraction, the underlying features can be gradually converted into middle-level or high-level features to improve the expressiveness of the target image, which is beneficial to subsequent processing.

[0096] In the example, the stride of the last N convolutional layers of the feature extraction network in the height dimension of the feature map is 1, so as to retain as many features in the height dimension as possible, where N is a positive integer.

[0097] Taking the feature extraction network as a residual network (ResNet, ResidualNetworks) including 4 residual units as an example, in the related art, the step size of the last convolutional layer in the 3rd and 4th residual units in the residual network is usually (2,2). In the embodiment disclosed in the present invention, the step size (2,2) can be changed to (1,2), so that the feature map is not downsampled in the height dimension, but is downsampled in the width dimension, thereby retaining the features in the height dimension as much as possible.

[0098] In some embodiments, other preprocessing may be performed on the target image, such as normalizing the pixel values ​​of the target image.

[0099] In an embodiment of the present disclosure, the method further includes training a neural network, wherein the neural network includes a feature extraction network for extracting features from the adjusted target image, and a classification network for classifying target objects to be identified in the target image.

[0100] In the training process of the neural network, the modules used include a preprocessing module 401 , an image enhancement module 402 , and a feature segmentation module 404 , and the neural network 403 includes a feature extraction network 4031 and a classification network 4032 .

[0101] In the disclosed embodiment, the neural network is trained using sample images and their annotation results.

[0102] In the example, the labeling result of the sample image includes the labeling category of each target object in the sample image. Taking game coins as an example, the category of each game coin is related to the denomination, and game coins of the same denomination belong to the same category. For a sample image including a plurality of game coins stacked upright, the denomination of each game coin is labeled in the sample image.

[0103] The processing process of the sample image 400 is taken as an example to describe the training process of the neural network, wherein the sample image 400 contains a plurality of stacked game coins, and the denomination of each game coin is marked in the sample image 400, that is, the real category of each game coin is marked.

[0104] First, the sample image 400 is preprocessed by the preprocessing module 401. The preprocessing includes adjusting the size of the sample image 400 while maintaining the aspect ratio, normalizing the pixel values ​​of the sample image 400, etc. The specific process of adjusting the size of the sample image 400 while maintaining the aspect ratio is as described above.

[0105] After preprocessing, the preprocessed sample image can also be enhanced by using the image enhancement module 402. Image enhancement of the preprocessed sample image includes: performing random flipping, random cropping, random fine-tuning of aspect ratio, random rotation and other operations on the preprocessed sample image to obtain an enhanced sample image, and the enhanced sample image can be used in the training stage of the neural network, thereby improving the robustness of the neural network.

[0106] For the enhanced sample image, the feature extraction network 4031 is used to obtain the feature maps of multiple target objects contained in the enhanced sample image. The specific structure of the feature extraction network 4031 is as described above.

[0107] Afterwards, the feature segmentation module 404 is used to segment the feature map in the height dimension to obtain a preset number of features.

[0108] Next, the classification network 4032 is used to determine the predicted category of each target object to be identified based on each feature.

[0109] According to the difference between the predicted category of the target object to be identified and the labeled category of the target object to be identified, the parameters of the neural network 403, including the parameters of the feature extraction network 4031 and the parameters of the classification network 4032, are adjusted.

[0110] In some embodiments, the loss function used to train the neural network includes a Connectionist Temporal Classification (CTC) loss function, that is, the parameters of the neural network can be updated by back propagation based on the CTC loss function.

[0111] In some embodiments, the trained neural network can also be tested using a test image and its annotation results, where the annotation results of the test image also include the annotation category of each target object to be identified in the test image. The testing process of the neural network is similar to the forward propagation process in the training process, except that image enhancement processing is no longer required. In the testing phase, the predicted category of the target object to be identified in the test image is predicted based on the input test image.

[0112] In some embodiments, a true-false identification model corresponding to a category is established using the hidden features of the authentication target object of the category. The authentication target object is correctly predicted during the training phase and / or the testing phase of the neural network. Wherein, correct prediction means that during the training phase and / or the testing phase, the predicted category of the authentication target object output by the neural network is the same as the labeled result.

[0113] For example, in the training and testing phases, n pieces of game coins of the i-th category are correctly predicted. According to the processing of the neural network, the hidden features corresponding to the n pieces of game coins can be obtained, and the authenticity recognition model corresponding to the category is established using the hidden features of the n pieces of game coins, such as a Gaussian probability distribution model. Where i = 1, 2, ... M, M is a positive integer. n is a positive integer.

[0114] For the obtained authenticity recognition model corresponding to the i-th category, the hidden features of the target object to be identified obtained by the neural network are input into the authenticity recognition model, and the probability value that the hidden features of the target object to be identified belong to the hidden features of the i-th category can be obtained. When the probability value is less than the probability threshold, the target object to be identified is identified as a foreign object.

[0115] In the disclosed embodiment, a true-false identification model corresponding to a category is established by utilizing the hidden features of an authentication target object of the category, thereby establishing a basis for determining whether the input hidden features are the hidden features of the target object of the category, that is, establishing a basis for determining whether the target object to be identified is a target object of an unknown category, thereby improving the recognition accuracy of the target object to be identified.

[0116] Figure 3 A schematic diagram of a target object recognition device provided by at least one embodiment of the present disclosure, such as Figure 3 As shown, the device includes: a classification unit 501, used to classify the target object to be identified in the target image and determine the predicted category of the target object to be identified; a determination unit 502, used to determine whether the predicted category is correct based on the hidden layer features of the target object to be identified; and a prompt unit 503, used to output prompt information in response to the predicted category being incorrect.

[0117] In some embodiments, the device further comprises an output unit for determining the predicted category as a final category of the target object to be identified in response to the predicted category being correct, and outputting the final category of the target object to be identified.

[0118] In some embodiments, the determination unit is specifically used to: input the hidden features of the target object to be identified into a true or false identification model corresponding to the predicted category, so that the true or false identification model outputs a probability value; wherein the true or false identification model corresponding to the predicted category reflects the distribution law of the hidden features of the target object of the predicted category, and the probability value represents the probability that the final category of the target object to be identified is the predicted category; when the probability value is less than a probability threshold, it is determined that the predicted category is incorrect; when the probability value is greater than or equal to the probability threshold, it is determined that the predicted category is correct.

[0119] In some embodiments, the target image includes multiple stacked target objects to be identified, and the classification unit is used to: adjust the height of the target image to a preset height, wherein the target image is cropped from the captured image according to the detection frames of the multiple stacked target objects to be identified in the captured image, and the height direction of the target image is the stacking direction of the multiple stacked target objects to be identified; classify the target objects to be identified in the adjusted target image to determine the predicted category of the target objects to be identified.

[0120] In some embodiments, the classification unit is specifically used to: scale the height and width of the target image proportionally until the width of the target image reaches a preset width; when the width of the scaled target image reaches the preset width and the height of the scaled target image is greater than the preset height, reduce the height and width of the scaled target image proportionally until the height of the scaled target image is equal to the preset height.

[0121] In some embodiments, the classification unit is used to: scale the height and width of the target image proportionally until the width of the target image reaches a preset width; when the width of the scaled target image reaches the preset width and the height of the scaled target image is less than the preset height, fill the scaled target image with the first pixel so that the height of the filled target image is the preset height.

[0122] In some embodiments, the classification unit is specifically used to: extract features from the adjusted target image to obtain a feature map, wherein the height dimension of the feature map corresponds to the height direction of the target image; perform average pooling on the feature map along the width dimension of the feature map to obtain a pooled feature map; segment the pooled feature map along the height dimension to obtain a preset number of features; and determine the predicted category of each of the multiple stacked target objects to be identified based on each feature.

[0123] In some embodiments, classifying the target objects to be identified in the adjusted target image to determine the predicted categories of the target objects to be identified is performed by a neural network, and the neural network includes a classification network; wherein the classification network includes K classifiers, K is the number of known categories when performing classification, and k is a positive integer; determining the predicted category of each of the multiple stacked target objects to be identified based on each feature, including: respectively calculating the cosine similarity between each feature and the weight vector of each classifier; determining the predicted category of each of the multiple stacked target objects to be identified based on the calculated cosine similarity.

[0124] In some embodiments, classifying the target object to be identified in the adjusted target image to determine the predicted category of the target object to be identified is performed by a neural network, wherein the neural network includes a feature extraction network, and the feature extraction network includes multiple convolutional layers. The last N convolutional layers of the multiple convolutional layers of the feature extraction network have a step size of 1 in the height dimension of the feature map, and N is a positive integer.

[0125] In some embodiments, the target objects to be identified in the target image are classified using a neural network, and the authenticity recognition model corresponding to the predicted category is established using the hidden layer features of the authentication target object of the predicted category, and the authentication target object is correctly predicted in the neural network training stage and / or testing stage.

[0126] The embodiments of the device disclosed herein can be applied to electronic devices, such as servers or terminal devices. The device embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by a processor reading the corresponding computer program instructions in the non-volatile memory into the memory and running them. From the hardware level, such as Figure 4 As shown, it is a hardware structure diagram of the electronic device where the target object recognition device is located. Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, the electronic device may also include other hardware according to the actual function of the electronic device, which will not be described in detail.

[0127] Accordingly, an embodiment of the present disclosure further provides a computer storage medium on which a computer program is stored, and when the program is executed by a processor, the method described in any embodiment is implemented.

[0128] Accordingly, an embodiment of the present disclosure further provides a computer program, which is stored on a computer-readable storage medium, wherein when the computer program is executed by a processor, the target object recognition method described in any embodiment of the present disclosure is implemented.

[0129] Accordingly, the present disclosure also provides an electronic device, such as Figure 4 As shown, the electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method described in any embodiment when executing the computer program.

[0130] The present disclosure may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable commands, data structures, program modules or other data. Examples of computer storage media include but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0131] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the specification disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0132] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

[0133] The above descriptions are only some embodiments of the present disclosure and are not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the scope of protection of the present disclosure.

[0134] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

Claims

1. A target object recognition method, characterized in that: The method comprises: Classifying the target object to be identified in the target image to determine the predicted category of the target object to be identified; Determining whether the predicted category is correct according to the hidden layer features of the target object to be identified; In response to the predicted category being incorrect, outputting a prompt message; Determining whether the predicted category is correct according to the hidden layer features of the target object to be identified includes: Input the hidden features of the target object to be identified into the authenticity recognition model corresponding to the predicted category, so that the authenticity recognition model outputs a probability value; wherein the hidden features of the target object refer to the features before being input into the classifier in the classification network when the target object is classified using the classification network; the authenticity recognition model corresponding to the predicted category is a Gaussian probability distribution model associated with multiple hidden features of the target object of the predicted category, and the probability value represents the probability that the final category of the target object to be identified is the predicted category; When the probability value is less than a probability threshold, determining that the predicted category is incorrect; When the probability value is greater than or equal to the probability threshold, it is determined that the predicted category is correct.

2. The method according to claim 1, characterized in that The method further comprises: In response to the predicted category being correct, the predicted category is determined as the final category of the target object to be identified, and the final category of the target object to be identified is output.

3. The method according to claim 1 or 2, characterized in that: The target image includes a plurality of target objects to be identified stacked together. Classifying the target object to be identified in the target image to determine the predicted category of the target object to be identified includes: Adjusting the height of the target image to a preset height by scaling, wherein the target image is cropped from the acquired image according to the detection frames of the multiple target objects to be identified stacked in the acquired image, and the height direction of the target image is the stacking direction of the multiple target objects to be identified stacked; The target object to be identified in the adjusted target image is classified to determine a predicted category of the target object to be identified.

4. The method according to claim 3, characterized in that Adjusting the height of the target image to the preset height includes: Scaling the height and width of the target image in equal proportion until the width of the target image reaches a preset width; When the width of the scaled target image reaches the preset width and the height of the scaled target image is greater than the preset height, the height and width of the scaled target image are proportionally reduced until the height of the scaled target image is equal to the preset height.

5. The method according to claim 3, characterized in that: Adjusting the height of the target image to the preset height includes: Scaling the height and width of the target image in equal proportion until the width of the target image reaches the preset width; When the width of the scaled target image reaches the preset width and the height of the scaled target image is less than the preset height, the scaled target image is padded with the first pixel so that the height of the padded target image is the preset height.

6. The method according to claim 3, characterized in that The preset height is set according to the maximum number of target objects to be identified; Classifying the target object to be identified in the adjusted target image to determine a predicted category of the target object to be identified includes: Performing feature extraction on the adjusted target image to obtain a feature map, wherein the height dimension of the feature map corresponds to the height direction of the target image; Performing average pooling on the feature map along the width dimension of the feature map to obtain a pooled feature map; According to the maximum number, segmenting the pooled feature map along the height dimension to obtain a preset number of features, wherein the preset number of features have a one-to-one correspondence with the stacked multiple target objects to be identified; The predicted category of each target object to be identified among the multiple stacked target objects to be identified is determined according to each feature.

7. The method according to claim 6, characterized in that Classifying the target object to be identified in the adjusted target image to determine the predicted category of the target object to be identified is performed by a neural network, wherein the neural network includes a classification network; wherein the classification network includes K classifiers, K is the number of known categories when performing classification, and k is a positive integer; Determining the predicted category of each of the plurality of stacked target objects to be identified according to the features includes: Calculating the cosine similarity between each feature and the weight vector of each classifier respectively; The predicted category of each of the plurality of stacked target objects to be identified is determined according to the calculated cosine similarity.

8. The method according to claim 6, characterized in that Classifying the target object to be identified in the adjusted target image to determine the predicted category of the target object to be identified is performed by a neural network, wherein the neural network includes a feature extraction network, and the feature extraction network includes multiple convolutional layers. The last N convolutional layers of the multiple convolutional layers of the feature extraction network have a step size of 1 on the height dimension of the feature map, and N is a positive integer.

9. The method according to claim 1, characterized in that: The target object to be identified in the target image is classified using a neural network, the authenticity identification model corresponding to the predicted category is established using the hidden layer features of the authentication target object of the predicted category, and the authentication target object is correctly predicted in the neural network training stage and / or testing stage.

10. A target object recognition device, characterized in that: The device comprises: A classification unit, used to classify the target object to be identified in the target image and determine the predicted category of the target object to be identified; A determination unit, used to determine whether the predicted category is correct according to the hidden layer features of the target object to be identified; a prompting unit, configured to output prompting information in response to the predicted category being incorrect; The determining unit is used for: Input the hidden features of the target object to be identified into the authenticity recognition model corresponding to the predicted category, so that the authenticity recognition model outputs a probability value; wherein the hidden features of the target object refer to the features before being input into the classifier in the classification network when the target object is classified using the classification network; the authenticity recognition model corresponding to the predicted category reflects a Gaussian probability distribution model of the predicted category related to multiple hidden features of the target object, and the probability value represents the probability that the final category of the target object to be identified is the predicted category; When the probability value is less than a probability threshold, determining that the predicted category is incorrect; When the probability value is greater than or equal to the probability threshold, it is determined that the predicted category is correct.

11. The device according to claim 10, characterized in that The device further includes an output unit, configured to, in response to the predicted category being correct, determine the predicted category as a final category of the target object to be identified, and output the final category of the target object to be identified.

12. The device according to claim 10 or 11, characterized in that The target image includes a plurality of target objects to be identified stacked together. The taxonomic units are used for: Adjusting the height of the target image to a preset height by scaling, wherein the target image is cropped from the acquired image according to the detection frames of the multiple target objects to be identified stacked in the acquired image, and the height direction of the target image is the stacking direction of the multiple target objects to be identified stacked; The target object to be identified in the adjusted target image is classified to determine a predicted category of the target object to be identified.

13. The device according to claim 12, characterized in that The taxonomic units are used for: Scaling the height and width of the target image in equal proportion until the width of the target image reaches a preset width; When the width of the scaled target image reaches the preset width and the height of the scaled target image is greater than the preset height, the height and width of the scaled target image are proportionally reduced until the height of the scaled target image is equal to the preset height.

14. The device according to claim 12, characterized in that The taxonomic units are used for: Scaling the height and width of the target image in equal proportion until the width of the target image reaches the preset width; When the width of the scaled target image reaches the preset width and the height of the scaled target image is less than the preset height, the scaled target image is padded with the first pixel so that the height of the padded target image is the preset height.

15. The device according to claim 12, characterized in that The preset height is set according to the maximum number of target objects to be identified; The taxonomic units are used for: Performing feature extraction on the adjusted target image to obtain a feature map, wherein the height dimension of the feature map corresponds to the height direction of the target image; Performing average pooling on the feature map along the width dimension of the feature map to obtain a pooled feature map; According to the maximum number, segmenting the pooled feature map along the height dimension to obtain a preset number of features, wherein the preset number of features have a one-to-one correspondence with the stacked multiple target objects to be identified; The predicted category of each target object to be identified among the multiple stacked target objects to be identified is determined according to each feature.

16. The device according to claim 15, characterized in that Classifying the target object to be identified in the adjusted target image to determine the predicted category of the target object to be identified is performed by a neural network, wherein the neural network includes a classification network; wherein the classification network includes K classifiers, K is the number of known categories when performing classification, and k is a positive integer; Determining the predicted category of each of the plurality of stacked target objects to be identified according to the features includes: Calculating the cosine similarity between each feature and the weight vector of each classifier respectively; The predicted category of each of the plurality of stacked target objects to be identified is determined according to the calculated cosine similarity.

17. An electronic device, characterized in that: The device comprises: processor; a memory for storing processor-executable instructions; The processor is configured to call the processor-executable instructions stored in the memory to execute the method according to any one of claims 1 to 9.

18. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images