Training method for object recognition model, object recognition method, and object recognition device

Abstract

The present disclosure relates to a training method for an object recognition model, an object recognition method, and an object recognition device. The training method includes: obtaining a training set, where the training set includes multiple groups of input samples for training and corresponding labeled results, each input sample includes an object image and metadata, and the metadata is configured to describe the corresponding object image; training the object recognition model using the training set based on a neural network; and ending the training when the training accuracy is greater than or equal to a preset accuracy, and obtaining the trained object recognition model.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more particularly, to a method for training an object recognition model, an object recognition method, and an object recognition device. Background Art

[0002] In order to recognize various objects such as plants, insects, etc., a user can provide a captured object image and use a pre-trained object recognition model to complete the recognition, thereby obtaining recognition results such as species information. However, the accuracy of current object recognition models is still relatively low, so there is a need for an improved object recognition model. Summary of the Invention

[0003] One object of the present disclosure is to provide a method for training an object recognition model, an object recognition method, and an object recognition device.

[0004] According to a first aspect of the present disclosure, there is provided a method for training an object recognition model, the training method comprising:

[0005] Obtaining a training set, wherein the training set includes multiple groups of input samples for training and corresponding labeled results, each input sample includes an object image and metadata, and the metadata is configured to describe the corresponding object image;

[0006] Based on a neural network, using the training set to train the object recognition model; and

[0007] Ending the training when the training accuracy is greater than or equal to a preset accuracy, and obtaining the trained object recognition model.

[0008] In some embodiments, before using the training set to train the object recognition model based on the neural network, the training method further includes:

[0009] Normalizing the metadata.

[0010] In some embodiments, the metadata includes at least one of the shooting geographical location, shooting time, shooting scene, object part, and object state of the object image.

[0011] In some embodiments, the neural network includes a classifier component and at least one block component, and the at least one block component includes a first block component;

[0012] Using the training set to train the object recognition model based on the neural network includes:

[0013] Generating a first image feature vector according to the object image through the first block component;

[0014] Generate a first embedding feature vector based on metadata;

[0015] Fuse the first image feature vector and the first embedding feature vector to generate a first fused feature vector; and

[0016] Generate a classification result according to the first fused feature vector by the classifier component to train the object recognition model.

[0017] In some embodiments, generating a first embedding feature vector based on metadata includes:

[0018] Generate a first embedding feature vector based on metadata through at least one fully connected layer.

[0019] In some embodiments, generating a first embedding feature vector based on metadata includes:

[0020] Generate a first embedding feature vector based on metadata through at least one fully connected layer and at least one rectified linear unit layer.

[0021] In some embodiments, the dimension of the first image feature vector is the same as the dimension of the first embedding feature vector.

[0022] In some embodiments, fusing the first image feature vector and the first embedding feature vector to generate a first fused feature vector includes:

[0023] Add each image vector component in the first image feature vector to the corresponding embedding vector component in the first embedding feature vector respectively to generate each fused vector component in the first fused feature vector.

[0024] In some embodiments, fusing the first image feature vector and the first embedding feature vector to generate a first fused feature vector includes:

[0025] Concatenate the first image feature vector and the first embedding feature vector in a preset dimension to generate a first fused feature vector.

[0026] In some embodiments, the at least one block component further includes a second block component connected in series with the first block component;

[0027] Generating a first image feature vector from an object image by the first block component includes:

[0028] Generate a second image feature vector from the object image by the second block component;

[0029] Generate a second embedding feature vector based on metadata;

[0030] Fuse the second image feature vector and the second embedding feature vector to generate a second fused feature vector; and

[0031] Generate a first image feature vector by the first block component according to the second fusion feature vector.

[0032] In some embodiments, the neural network further includes a preprocessing component;

[0033] Training the object recognition model using the training set based on the neural network includes:

[0034] Generate a preprocessed image by the preprocessing component according to the object image, and use the preprocessed image as the input of the block component adjacent to the preprocessing component in the at least one block component.

[0035] In some embodiments, the training method further includes:

[0036] Obtain a test set, where the test set includes multiple groups of input samples for testing and the labeled results corresponding to the input samples. Each input sample includes an object image and metadata, the metadata is configured to describe the corresponding object image, and the test set is different from the training set;

[0037] Use the test set to determine the model accuracy of the trained object recognition model; and

[0038] When the model accuracy is less than the preset accuracy, retrain the object recognition model.

[0039] In some embodiments, the neural network includes a residual neural network.

[0040] According to a second aspect of the present disclosure, an object recognition method is provided. The object recognition method includes:

[0041] Obtain an object image to be recognized and metadata, where the metadata is configured to describe the object image to be recognized;

[0042] Use the object recognition model to determine a recognition result according to the object image and the metadata, where the object recognition model is trained by using the training method as described above.

[0043] According to a third aspect of the present disclosure, an object recognition device is provided. The object recognition device includes a memory and a processor. Instructions are stored on the memory. When the instructions are executed by the processor, the above-described training method or object recognition method is implemented.

[0044] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided. Instructions are stored on the computer-readable storage medium. When the instructions are executed by a processor, the above-described training method or object recognition method is implemented.

[0045] According to a fifth aspect of the present disclosure, a computer program product is provided. The computer program product includes instructions that, when executed by a processor, implement the training method or object recognition method as described above.

[0046] Other features and advantages of the present disclosure will become clearer from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings forming a part of the specification depict embodiments of the present disclosure and, together with the description, are used to explain the principles of the present disclosure.

[0048] With reference to the accompanying drawings, the present disclosure can be more clearly understood from the following detailed description, where:

[0049] Figure 1 A schematic diagram of a network environment according to an exemplary embodiment of the present disclosure is shown;

[0050] Figure 2 A flowchart showing a training method for an object recognition model according to an exemplary embodiment of the present disclosure is shown;

[0051] Figure 3 A flowchart showing step S120 of the training method according to an exemplary embodiment of the present disclosure is shown;

[0052] Figure 4 A schematic diagram of a training method according to a specific example of the present disclosure is shown;

[0053] Figure 5 A flowchart showing step S121 of the training method according to an exemplary embodiment of the present disclosure is shown;

[0054] Figure 6 A schematic diagram of a training method according to another specific example of the present disclosure is shown;

[0055] Figure 7 A flowchart showing a training method for an object recognition model according to another exemplary embodiment of the present disclosure is shown;

[0056] Figure 8 A flowchart showing an object recognition method according to an exemplary embodiment of the present disclosure is shown;

[0057] Figure 9 A schematic diagram of an object recognition device according to an exemplary embodiment of the present disclosure is shown.

[0058] Note that in the embodiments described below, the same reference numerals are sometimes used commonly between different drawings to denote the same or functionally identical parts, and redundant descriptions thereof are omitted. In some cases, similar reference numerals and letters are used to denote similar items, and thus once an item is defined in one drawing, further discussion thereof is not required in subsequent drawings.

[0059] For ease of understanding, the positions, dimensions, ranges, etc. of the respective structures shown in the drawings and the like sometimes do not represent the actual positions, dimensions, ranges, etc. Accordingly, the present disclosure is not limited to the positions, dimensions, ranges, etc. disclosed in the drawings and the like. Detailed embodiments

[0060] Various exemplary embodiments of the present disclosure will be described in detail below with reference to the drawings. It should be noted that: unless otherwise specifically stated, the relative arrangements of components and steps, numerical expressions, and values set forth in these embodiments do not limit the scope of the present disclosure.

[0061] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way a limitation on the present disclosure, its application, or use. That is, the structures and methods herein are shown in an exemplary manner to illustrate different embodiments of the structures and methods in the present disclosure. However, those skilled in the art will understand that they merely illustrate exemplary ways in which the present disclosure can be implemented, rather than exhaustive ways. Additionally, the drawings need not be drawn to scale, and some features may be enlarged to show details of specific components.

[0062] Techniques, methods, and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, the techniques, methods, and devices should be regarded as part of the authorization specification.

[0063] In all the examples shown and discussed herein, any specific values should be construed as merely exemplary and not as a limitation. Thus, other examples of the exemplary embodiments may have different values.

[0064] Figure 1 A schematic diagram of a network environment 900 according to an exemplary embodiment of the present disclosure is shown. The network environment 900 may include a mobile device 902, a remote server 903, a training device 904, and a database 905, which are coupled to each other wired or wirelessly via a network 906. The network 906 may be embodied as a wide area network (such as a mobile phone network, a public switched telephone network, a satellite network, the Internet, etc.), a local area network (such as Wi-Fi, Wi-Max, ZigBeeTM, BluetoothTM, etc.), and / or other forms of networking functionality.

[0065] The mobile device 902 can be a mobile phone, a tablet computer, a laptop computer, a personal digital assistant, and / or other computing devices configured to collect, store, and / or transmit images such as digital photos. Thus, the mobile device 902 can include an image acquisition unit such as a digital camera and / or can be configured to receive images from other devices. The mobile device 902 can include a display. The display can be configured to provide one or more user interfaces to the user 901, and the user interfaces can include a plurality of interface elements with which the user 901 can interact, etc. For example, the user 901 can use the mobile device 902 to take pictures of objects such as plants and insects and upload or store the object images. The mobile device 902 can output species information and the like about the objects such as plants and insects to the user.

[0066] The remote server 903 can be configured to analyze object images and the like received from the mobile device 902 via the network 906 to determine the type of the object, etc., for example, for performing the object recognition method described below. The remote server 903 can also be configured to create and train an object recognition model as described below.

[0067] The training device 904 can be coupled to the network 906 to facilitate the training of the object recognition model, for example, for performing the object recognition model training method described below. The training device 904 can have multiple CPUs and / or GPUs to assist in training the object recognition model, and the specific training process will be elaborated in detail below.

[0068] The database 905 can be coupled to the network 906 and provide the data required for the remote server 903 to perform relevant calculations. The database 905 can be implemented using various database technologies known in the art. The remote server 903 can access the database 905 as needed for relevant operations.

[0069] It should be understood that the network environment herein is merely an example. Those skilled in the art can, as needed, add more devices or delete some devices, and can modify the functions and configurations of some devices.

[0070] In some examples, an object recognition model can be trained and established based on a neural network, and its training process is as follows:

[0071] Obtain a certain number of object images marked with marking results for each object category. The number of object images prepared for each object category can be equal or unequal. The marking results marked for each object image can include the object name in the object image (including scientific name, alias, category name of biological classification, etc.). The object images obtained for each object category can include as many images as possible of different shooting geographical locations, different shooting times, different shooting scenes, different object parts, different object states, etc. of the objects of this category.

[0072] Divide the object images processed through the above marking into a training set for training an object recognition model and a test set for testing the training results. Usually, the number of samples in the training set is significantly larger than the number of samples in the test set. For example, the number of samples in the test set accounts for 5% to 20% of the total number of samples, and correspondingly, the number of samples in the training set accounts for 80% to 95% of the total number of samples. Those skilled in the art should understand that the number of samples in the training set and the test set can be adjusted as needed.

[0073] Use the training set to train the neural network until the preset accuracy rate is reached. In some cases, the model accuracy rate of the trained neural network can also be tested using the test set as needed. If the model accuracy rate does not meet the requirements, increase the number of samples in the training set and use the updated training set to retrain the neural network until the model accuracy rate of the trained neural network meets the requirements.

[0074] The above neural network may include, for example, a convolutional neural network (CNN) or a residual neural network (Resnet), etc. Among them, the convolutional neural network is a deep feedforward neural network that uses a convolutional kernel to scan the object image, extracts the features to be recognized in the object image, and then recognizes the features to be recognized of the object. In addition, during the process of recognizing the object image, the original object image can be directly input into the convolutional neural network model without preprocessing the object image. Compared with other recognition models, the convolutional neural network model has higher recognition accuracy and recognition efficiency. The residual neural network model adds an identity mapping layer compared with the convolutional neural network model, which can avoid the phenomenon of accuracy saturation and even decline caused by the convolutional neural network as the network (the number of stacked layers in the network) increases. The identity mapping function of the identity mapping layer in the residual neural network model needs to satisfy that the sum of the identity mapping function and the input of the residual neural network model is equal to the output of the residual neural network model. After introducing the identity mapping, the change of the output of the residual neural network model is more obvious, so the accuracy and efficiency of object recognition can be greatly improved. In the following specific examples, the residual neural network will be taken as an example for specific elaboration. However, it can be understood that other types of neural networks can also be used for training without departing from the concept of the present disclosure.

[0075] According to the above description, when training an object recognition model, it is generally only trained based on the object image itself. For example, the RGB data of the object image is input into the neural network for training, which often results in low recognition accuracy. To improve the accuracy of the object recognition model, the present disclosure proposes a training method for the object recognition model, which is trained not only based on the object image itself but also based on the metadata configured to describe the object image to improve the recognition accuracy and recognition efficiency. As Figure 2 shown, in an exemplary embodiment of the present disclosure, the training method may include:

[0076] Step S110, obtaining a training set, where the training set includes multiple groups of input samples for training and the labeled results corresponding to the input samples. Each input sample includes an object image and metadata, and the metadata is configured to describe the corresponding object image.

[0077] Among them, the object image may include photos, videos, etc. obtained by a user taking pictures of objects such as plants and insects. The object image may be in, for example, RGB format. Each pixel in the object image may be composed of 3 color components: red (R), green (G), and blue (B). It can be understood that other data formats can also be used to represent the object image, such as RGBA format, HSV format, etc.

[0078] Meta data is data used to describe an object image, and each object image can have its corresponding meta data. In some embodiments, the meta data may include at least one of the shooting geographical location, shooting time, shooting scene, object part, and object state of the object image. Among them, the shooting scene can be used to describe the environment when shooting the object, such as whether it is indoor or outdoor, etc.; the object part can be used to describe the object part presented or mainly presented in the object image, such as the roots, stems, leaves, flowers, fruits, etc. of a plant, the head, abdomen, etc. of an insect; the object state can be used to describe the current stage of the object presented in the object image, such as the seedling stage, flowering stage, fruiting stage, etc. of a plant, the larval stage, adult stage, etc. of an insect. It can be understood that in some other embodiments, the meta data may also include other data used to describe the object image, which is not limited herein. During the process of training an object recognition model, all the meta data can be involved in the training process; or, a part of the meta data that can play a key role in the recognition of the object can be selected for training, so as to improve the accuracy of the model while maintaining a high training efficiency and avoiding waste of training resources.

[0079] In some embodiments, for the convenience of data processing during training, the training method may further include normalizing the meta data, so as to limit the value range of the meta data within a desired range, for example, making the absolute value of the meta data between 0 and 1 (including 0 and 1), to improve the training result. It can be understood that according to the magnitude of the role of the specific meta data in recognition and the relationship between the meta data and the object image, the value range of the meta data can be adjusted as needed, which is not limited herein. In addition, different normalization methods can be adopted according to different types of meta data.

[0080] For example, when the meta data is the shooting geographical location of the object image, longitude and latitude are usually used to represent the shooting geographical location. Then, during the normalization of the shooting geographical location, sine and cosine functions can be used to convert the longitude and latitude into values with absolute values between 0 and 1. In a specific example, if a shooting geographical location can be represented as (L1, L2), where L1 is the longitude of the shooting geographical location (expressed in degrees) and L2 is the latitude of the shooting geographical location (expressed in degrees), then the normalized shooting geographical location can be represented as (sin(πL1 / 180), cos(πL1 / 180), sin(πL2 / 180), cos(πL2 / 180)), that is, a vector with a dimension of 4 can be used to uniquely represent a specific shooting geographical location.

[0081] When the metadata is the shooting time of the object image, the shooting time can be divided into two parts: the shooting date and the shooting moment on that day, and the shooting moment is converted into the form of Coordinated Universal Time (UTC). Then, the shooting date and the shooting moment are respectively represented in a normalized manner. In a specific example, the shooting time can be expressed as (sin(2πd / Y), cos(2πd / Y), sin(2πt / D), cos(2πt / D)), where d represents that the current shooting date is the dth day of the year, Y represents the total number of days in a year (365 or 366), t represents that the current shooting moment is the tth moment of the day, and D represents the total number of moments in a day (for example, 24 when the moment is accurate to hours), that is, a specific shooting time can also be uniquely represented by a vector with a dimension of 4.

[0082] In summary, when the metadata is a quantity that can take continuous values, the sine function and the cosine function can be used to normalize the metadata. However, it can be understood that other functions can also be used to normalize the metadata that can take continuous values, and no limitation is imposed here.

[0083] When the metadata is one of the shooting scene, object part, and object state of the object image, or when the metadata is a certain discrete quantity, the method of "looking up a dictionary" can be used to determine the value of the metadata. For example, when the metadata represents the various parts of a plant, 1 can be used to represent the root part of the plant, 2 represents the stem part of the plant, 3 represents the leaf part of the plant, 4 represents the flower part of the plant, and 5 represents the fruit part of the plant. In this way, the object part can be encoded as a [5, x] matrix, where 5 corresponds to 5 classifications, that is, each classification corresponds to a vector with x values. After encoding the metadata such as the object part and the object state respectively, a vector of [n, x] can be obtained, and n is the number of such discrete features.

[0084] During the training process, various metadata can be combined together. For example, vectors are respectively formed according to various metadata, and then these vectors are added together for fusion. For example, the vectors of various metadata can be converted into vectors with a preset dimension (for example, 2048) through a full connection layer for several times (for example, 3 times), and then the corresponding components of these vectors are added respectively to obtain a vector representing all the metadata participating in the training.

[0085] Return Figure 2 , in the exemplary embodiment of the present disclosure, the training method may further include:

[0086] Step S120, based on a neural network, use a training set to train an object recognition model.

[0087] As described above, since the input samples of the training set include object images and their metadata, it helps to improve the accuracy of the trained object recognition model. For neural networks with different structures, different ways can be used to embed the metadata into the training process.

[0088] In some embodiments, as Figure 3 and Figure 4 shown, the neural network may include a classifier component 830 and at least one block component, where at least one block component is included in the image feature extraction network 810, and at least one block component may include a first block component. Accordingly, based on the neural network, training the object recognition model using the training set may include:

[0089] Step S121, generating a first image feature vector according to the object image through the first block component;

[0090] Step S122, generating a first embedded feature vector according to the metadata;

[0091] Step S123, fusing the first image feature vector and the first embedded feature vector to generate a first fused feature vector; and

[0092] Step S124, generating a classification result according to the first fused feature vector through the classifier component to train the object recognition model.

[0093] In a specific example, the first block component may be a block component adjacent to the classifier component 830 in the image feature extraction network 810. That is, during the training process, the object image is converted into a first image feature vector through the complete image feature extraction network 810 (for example, the backbone in a residual neural network), while the metadata is converted into a first embedded feature vector through the embedding network 820. The first image feature vector and the first embedded feature vector are fused to generate a first fused feature vector (not shown in the figure), and the generated first fused feature vector is input into the classifier component 830 to generate a classification result. In such an example, the image feature extraction network 810 and the embedding network 820 are independent of each other, and only before the final classification, the first image feature vector and the first embedded feature vector are fused.

[0094] In some embodiments, the embedding network 820 may include at least one fully connected layer. Accordingly, generating the first embedded feature vector according to the metadata may include generating the first embedded feature vector according to the metadata through at least one fully connected layer.

[0095] In some other embodiments, the embedding network 820 may further include at least one rectified linear layer. Accordingly, generating the first embedded feature vector according to the metadata may include generating the first embedded feature vector according to the metadata through at least one fully connected layer and at least one rectified linear layer.

[0096] In addition, various ways can be adopted to fuse the first image feature vector and the first embedded feature vector. In some embodiments, the dimension of the first image feature vector may be the same as the dimension of the first embedded feature vector. In this way, fusing the first image feature vector and the first embedded feature vector to generate the first fused feature vector may include adding each image vector component in the first image feature vector to the corresponding embedded vector component in the first embedded feature vector (add) respectively to generate each fused vector component in the first fused feature vector, and the dimension of the generated first fused feature vector remains the same as the dimension of the first image feature vector or the dimension of the first embedded feature vector. For example, if the dimension of the first image feature vector A is [ax, ay] and the dimension of the first embedded feature vector B is [bx, by], then, if A and B are added to obtain the first fused feature vector C, its dimension is [ax, ay], where it is required that ax = bx and ay = by. In this case, the dimension of the generated first fused feature vector remains unchanged, so the subsequent network structure does not need to be adjusted, and the main structure of the network will not be damaged.

[0097] In some other embodiments, fusing the first image feature vector and the first embedded feature vector to generate the first fused feature vector may include concatenating the first image feature vector and the first embedded feature vector in a preset dimension to generate the first fused feature vector. For example, if the dimension of the first image feature vector A is [ax, ay] and the dimension of the first embedded feature vector B is [bx, by], then, if A and B are concatenated in the first dimension, the first fused feature vector C will be obtained, whose dimension is [ax + bx, ay], where it is required that ay = by. In this case, some corresponding adjustments to the network structure may be involved in subsequent training because the dimension of the first fused feature vector has changed, but at this time, it is not required that the dimensions of the first image feature vector and the first embedded feature vector must be the same.

[0098] In some other embodiments, the image feature extraction network 810 and the embedding network 820 may not be completely independent. The input of a certain block component may be a fused feature vector generated by fusing the image feature vector generated by another block component above it with the corresponding embedding feature vector. In other words, the embedding network can be partially embedded in the image feature extraction network, inserting metadata at each stage to strengthen the extracted features, where each embedding network can be regarded as an independent embedding network. Specifically, as Figure 5 and Figure 6 shown, at least one block component may further include a second block component 812 connected in series with the first block component 811. Among them, Figure 6 each block component shown in is included in the image feature extraction network 810. Correspondingly, generating the first image feature vector by the first block component according to the object image may include:

[0099] Step S1211, generating a second image feature vector by the second block component according to the object image;

[0100] Step S1212, generating a second embedding feature vector according to the metadata;

[0101] Step S1213, fusing the second image feature vector with the second embedding feature vector to generate a second fused feature vector; and

[0102] Step S1214, generating the first image feature vector by the first block component according to the second fused feature vector.

[0103] Similarly, the way of forming the first embedding feature vector can be adopted to generate the second embedding feature vector according to the metadata, and the way of forming the first fused feature vector can be adopted to generate the second fused feature vector according to the second image feature vector and the second embedding feature vector. And so on, the fusion of the image feature vector and the embedding feature vector can also be performed at more block components, and the generated fused feature vector can be used as the input of the next-level block component to improve the training effect.

[0104] In addition, as Figure 6 shown, in some embodiments, the neural network may further include a preprocessing component 815 (stem), and this preprocessing component 815 is also included in the backbone. Correspondingly, based on the neural network, training the object recognition model using the training set may include: generating a preprocessed image by the preprocessing component according to the object image, and using the preprocessed image as the input of the block component adjacent to the preprocessing component in at least one block component. The specific setting methods of the preprocessing component and the block component can refer to the settings in the residual neural network, which will not be elaborated here.

[0105] According to the experimental results, as Figure 6As shown in the specific example, embedding in the same way at the four block components can strengthen the role of metadata, thereby obtaining a better training effect. Specifically, the object image is preprocessed by the preprocessing component 815 to generate a preprocessed image. Then, the preprocessed image is input into the fourth block component 814 and converted into a fourth image feature vector. And the metadata is input into the fourth embedding network 824 and converted into a fourth embedding feature vector. Then, the fourth image feature vector and the fourth embedding feature vector are fused to generate a fourth fused feature vector, which is input into the third block component 813 and converted into a third image feature vector. And the metadata is converted into a third embedding feature vector through the third embedding network 823. Then, the third image feature vector and the third embedding feature vector are fused to generate a third fused feature vector, which continues to be input into the second block component 812 and converted into a second image feature vector. And the metadata is converted into a second embedding feature vector through the second embedding network 822. Then, the second image feature vector and the second embedding feature vector are fused to generate a second fused feature vector, which continues to be input into the first block component 812 and converted into a first image feature vector. And the metadata is converted into a first embedding feature vector through the first embedding network 821. Among them, according to the characteristics of the residual neural network, the dimensions of the fourth embedding feature vector, the third embedding feature vector, the second embedding feature vector, and the first embedding feature vector output by the fourth embedding network 824, the third embedding network 823, the second embedding network 822, and the first embedding network 821 can be 256, 512, 1024, and 2048 respectively. In a specific example, the fourth embedding network 824, the third embedding network 823, and the second embedding network 822 can each include only one fully connected layer and one rectified linear unit layer, while the first embedding network 821 can include three fully connected layers and three rectified linear unit layers, where the last fully connected layer outputs a feature vector with a dimension of 2048. Finally, the first image feature vector and the first embedding feature vector are fused to generate a first fused feature vector, which is input into the classifier component 830 to generate a classification result.

[0106] It can be understood that in some other embodiments, there may also be other block components between two adjacent block components that will receive the fused feature vector. In addition, there may also be other block components between the preprocessing component and the fourth block component, and / or between the first block component and the classifier component.

[0107] Return Figure 2 , in the exemplary embodiment of the present disclosure, the training method may further include:

[0108] Step S130, end the training when the training accuracy is greater than or equal to the preset accuracy, and obtain the trained object recognition model.

[0109] Further, in order to verify the model accuracy of the object recognition model obtained by training, in some embodiments, as Figure 7 shown, the training method may further include:

[0110] Step S210, obtaining a test set, where the test set includes multiple groups of input samples for testing and the labeled results corresponding to the input samples. Each input sample includes an object image and metadata, and the metadata is configured to describe the corresponding object image, and the test set is different from the training set;

[0111] Step S220, using the test set to determine the model accuracy of the trained object recognition model; and

[0112] Step S230, when the model accuracy is less than the preset accuracy, retraining the object recognition model.

[0113] As described above, the input samples in the test set and the training set are not exactly the same. Therefore, the test set can be used to test whether the object recognition model also has a good recognition effect on object images outside the training set. During the test, by comparing the output results generated according to the object images and their metadata in the test set, the model accuracy of the object recognition model is calculated. In some examples, the calculation method of the model accuracy may be the same as the calculation method of the training accuracy. When the model accuracy obtained by the test is less than the preset accuracy, it indicates that the recognition effect of the object recognition model is not good enough. Therefore, the training set can be adjusted. Specifically, for example, the number of samples in the training set can be increased, or the object recognition model itself can be adjusted, or both of the above can be adjusted, and then the object recognition model is retrained to improve its recognition effect.

[0114] The present disclosure also proposes an object recognition method, as Figure 8 shown, the object recognition method may include:

[0115] Step S310, obtaining an object image to be recognized and metadata, where the metadata is configured to describe the object image to be recognized;

[0116] Step S320, using the object recognition model to determine the recognition result according to the object image and the metadata, where the object recognition model is obtained by training using the training method described above.

[0117] The present disclosure also proposes an object recognition device, as Figure 9 shown, the object recognition device may include a memory 720 and a processor 710. Instructions are stored on the memory 720, and when the instructions are executed by the processor 710, the above-described training method or object recognition method is implemented.

[0118] Among them, the processor 710 can perform various actions and processes according to the instructions stored in the memory 720. Specifically, the processor 710 can be an integrated circuit chip with the ability to process signals. The above-mentioned processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute various methods, steps and logic block diagrams disclosed in the embodiments of the present disclosure. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., and can be an X86 architecture or an ARM architecture, etc.

[0119] The memory 720 stores executable instructions, and these instructions are used to execute the above-mentioned training method or object recognition method of the object recognition model by the processor 710. The memory 720 can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM). It should be noted that the memory of the methods described herein is intended to include but not limited to these and any other suitable types of memory.

[0120] The present disclosure further provides a computer-readable storage medium, on which instructions are stored, and when the instructions are executed by a processor, the above-mentioned training method or object recognition method of the object recognition model is implemented.

[0121] Similarly, the computer-readable storage medium in the embodiments of the present disclosure can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. It should be noted that the computer-readable storage medium described herein is intended to include but not limited to these and any other suitable types of memory.

[0122] The present disclosure further provides a computer program product, which can include instructions, and when the instructions are executed by a processor, the above-mentioned training method or object recognition method of the object recognition model is implemented.

[0123] An instruction can be any instruction set that is directly executed by one or more processors, such as machine code, or any instruction set that is indirectly executed, such as a script. The terms "instruction", "application", "process", "step", and "program" in this article can be used interchangeably. Instructions can be stored in object code format for direct processing by one or more processors, or stored in any other computer language, including scripts or collections of independent source code modules that are interpreted on demand or compiled in advance. Instructions can include instructions that cause one or more processors to act as the various neural networks in this article. The functions, methods, and routines of instructions are explained in more detail in other parts of this article.

[0124] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0125] Generally speaking, various example embodiments of the present disclosure can be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while other aspects can be implemented in firmware or software that can be executed by a controller, a microprocessor, or other computing devices. When aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, devices, systems, technologies, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuits or logic, general hardware or a controller or other computing devices, or some combination thereof.

[0126] In the specification and claims, words such as "front", "rear", "top", "bottom", "above", "below", etc., if present, are used for descriptive purposes and not necessarily to describe a fixed relative position. It should be understood that such words are interchangeable under appropriate circumstances, so that the embodiments of the present disclosure described herein, for example, can operate in other orientations different from those shown or otherwise described herein.

[0127] As used herein, the word "exemplary" means "serving as an example, instance, or illustration", rather than a "model" to be precisely replicated. Any implementation described herein as exemplary is not necessarily to be construed as preferred or advantageous over other implementations. Moreover, the present disclosure is not limited by any theory expressed or implied in the foregoing technical field, background art, summary of the invention, or detailed description.

[0128] As used herein, the word "substantially" means including any minor variations caused by design or manufacturing defects, tolerances of devices or components, environmental effects, and / or other factors. The word "substantially" also allows for differences from a perfect or ideal situation due to parasitic effects, noise, and other practical considerations that may exist in an actual implementation.

[0129] Additionally, the previous description may have referred to elements or nodes or features that are "connected" or "coupled" together. As used herein, unless otherwise expressly stated, "connected" means that one element / node / feature is directly connected (or directly communicates) with another element / node / feature electrically, mechanically, logically, or otherwise. Similarly, unless otherwise expressly stated, "coupled" means that one element / node / feature can be connected to another element / node / feature directly or indirectly, mechanically, electrically, logically, or otherwise to allow interaction, even if the two features may not be directly connected. That is, "coupled" is intended to encompass both direct and indirect connections of elements or other features, including connections using one or more intermediate elements.

[0130] Additionally, for reference purposes only, terms such as "first", "second", etc. may also be used herein and are not intended to be limiting. For example, unless the context clearly indicates otherwise, words such as "first", "second", and other such numerical words referring to structures or elements do not imply an order or sequence.

[0131] It should also be understood that when the term "comprising / including" is used herein, it indicates the presence of the stated features, wholes, steps, operations, units, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, units, and / or components and / or combinations thereof.

[0132] In the present disclosure, the term "provide" is used in a broad sense to cover all ways of obtaining an object. Thus, "providing an object" includes, but is not limited to, "purchasing", "preparing / manufacturing", "arranging / setting", "installing / assembling", and / or "ordering" the object, etc.

[0133] Although some specific embodiments of the present disclosure have been described in detail by way of examples, those skilled in the art should understand that the above examples are for illustrative purposes only and not for limiting the scope of the present disclosure. The various embodiments disclosed herein can be combined arbitrarily without departing from the spirit and scope of the present disclosure. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A training method for an object recognition model, characterized in that The training method includes: Obtaining a training set, where the training set includes multiple groups of input samples for training and corresponding labeled results, and each input sample includes an object image and metadata, and the metadata is configured to describe the corresponding object image; Based on a neural network, using the training set to train the object recognition model, the neural network includes a classifier component and at least one block component, and the at least one block component includes a first block component and a second block component connected in series with the first block component; and Ending the training when the training accuracy is greater than or equal to a preset accuracy, and obtaining the trained object recognition model, where, based on a neural network, using the training set to train the object recognition model includes: Generating a second image feature vector according to the object image through the second block component, Generating a second embedding feature vector according to the metadata, Fusing the second image feature vector and the second embedding feature vector to generate a second fused feature vector, Generating a first image feature vector according to the second fused feature vector through the first block component, Generating a first embedding feature vector according to the metadata, Fusing the first image feature vector and the first embedding feature vector to generate a first fused feature vector, Generating a classification result according to the first fused feature vector through the classifier component to train the object recognition model.

2. The training method according to claim 1, wherein Before training the object recognition model based on a neural network using the training set, the training method further includes: Normalizing the metadata.

3. The training method according to claim 1, wherein The metadata includes at least one of the shooting geographical location, shooting time, shooting scene, object part, and object state of the object image.

4. The training method according to claim 1, wherein Generating a first embedding feature vector according to the metadata includes: Generating a first embedding feature vector according to the metadata through at least one fully connected layer.

5. The training method according to claim 1, characterized in that Generating a first embedding feature vector according to the metadata includes: Generating a first embedding feature vector according to the metadata through at least one fully connected layer and at least one rectified linear unit layer.

6. The training method according to claim 1, wherein The dimension of the first image feature vector is the same as the dimension of the first embedding feature vector.

7. The training method according to claim 6, wherein Fusing the first image feature vector and the first embedding feature vector to generate a first fused feature vector includes: Adding each image vector component in the first image feature vector to the corresponding embedding vector component in the first embedding feature vector respectively to generate each fused vector component in the first fused feature vector.

8. The training method according to claim 1, wherein Fusing the first image feature vector and the first embedding feature vector to generate a first fused feature vector includes: Connecting the first image feature vector and the first embedding feature vector in a preset dimension to generate a first fused feature vector.

9. The training method according to claim 1, wherein The neural network further includes a preprocessing component; Based on a neural network, using the training set to train the object recognition model includes: Generating a preprocessed image according to the object image through the preprocessing component, and using the preprocessed image as the input of the block component adjacent to the preprocessing component in the at least one block component.

10. The training method according to claim 1, wherein The training method further includes: Obtain a test set, where the test set includes multiple groups of input samples for testing and the labeled results corresponding to the input samples. Each input sample includes an object image and metadata, and the metadata is configured to describe the corresponding object image, and the test set is different from the training set; Use the test set to determine the model accuracy of the trained object recognition model; and When the model accuracy is less than the preset accuracy, retrain the object recognition model.

11. The training method according to claim 1, wherein The neural network includes a residual neural network.

12. An object recognition method, characterized in that, The object recognition method includes: Obtain an object image to be recognized and metadata, where the metadata is configured to describe the object image to be recognized; Use the object recognition model to determine the recognition result according to the object image and the metadata, where the object recognition model is trained by using the training method according to any one of claims 1 to 11.

13. An object recognition device, characterized in that, The object recognition device includes a memory and a processor. Instructions are stored on the memory, and when the instructions are executed by the processor, the training method according to any one of claims 1 to 11 or the object recognition method according to claim 12 is implemented.

14. A computer-readable storage medium, characterized in that, Instructions are stored on the computer-readable storage medium, and when the instructions are executed by the processor, the training method according to any one of claims 1 to 11 or the object recognition method according to claim 12 is implemented.

15. A computer program product, characterized in that, The computer program product includes instructions, and when the instructions are executed by the processor, the training method according to any one of claims 1 to 11 or the object recognition method according to claim 12 is implemented.

Images