Method for training feature extraction model, method for image classification and related device

Abstract

To provide a feature extraction model training method and an image classifying method which extract an effective image feature from an image to be classified or recognized and collate with an image feature of a known image category for recognition of an image category.SOLUTION: A method according to the present invention includes: a step 201 of extracting an image feature of each of sample images in a sample image set by a basic feature extracting module of an initial feature extraction model to obtain an initial feature vector set; a step 202 of obtaining each normalized feature vector by subjecting each of the initial feature vectors in the initial feature vector set to normalization processing by a normalization processing module of the initial feature extraction model; and a step 203 of carrying out training of the initial feature extraction model by using a preset highly discriminative loss function as a guide to obtain a target feature extraction model as a training result.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to the technical field of image processing, specifically to the technical field of artificial intelligence such as deep training, image recognition, etc., and in particular to a feature extraction model training method and an image classification method, as well as an apparatus, an electronic device, a computer-readable storage medium, and a computer program corresponding to these methods. [Background technology]

[0002] In order to improve the effectiveness of recognizing the category to which an image belongs, image recognition models or image classification models built based on machine learning and deep learning techniques have come to be used on a large scale.

[0003] As a key part of an image recognition model or image classification model, an important direction of research for those skilled in the art is how to extract effective image features from an image to be classified or recognized, and match them with image features of known image categories to recognize the image category. Summary of the Invention [Means for solving the problem]

[0004] Embodiments of the present disclosure provide a feature extraction model training method and an image classification method, as well as an apparatus, an electronic device, a computer-readable storage medium, and a computer program corresponding to these methods.

[0005] In a first aspect, an embodiment of the present disclosure provides a feature extraction model training method, including: extracting image features of each sample image in a sample image set by a basic feature extraction module of an initial feature extraction model to obtain an initial feature vector set, where the sample image set includes sample images of a plurality of image categories, and each sample image is marked with an image category to which it belongs; performing a normalization process on each initial feature vector in the initial feature vector set by a normalization process module of the initial feature extraction model to obtain each normalized feature vector; and training the initial feature extraction model using a predetermined high-discriminability loss function as a guide to obtain a target feature extraction model as a training result, where the high-discriminability loss function is abstracted based on a predetermined upper limit value and a predetermined lower limit value of vector similarity that are respectively set in advance for sample images of any two image categories depending on the presence or absence of similarity, where the vector similarity between the normalized feature vectors of two images that do not have similarity is equal to or less than the predetermined lower limit value, the vector similarity between the normalized feature vectors of two images that have similarity is equal to or more than the predetermined upper limit value, and the predetermined upper limit value is greater than the predetermined lower limit value.

[0006] In a second aspect, an embodiment of the present disclosure includes an image feature extraction unit configured to extract image features of each sample image in a sample image set by a basic feature extraction module of an initial feature extraction model to obtain an initial feature vector set, where the sample image set includes sample images of multiple image categories, and each sample image is marked with an image category to which it belongs; a normalization processing unit configured to perform a normalization processing on each initial feature vector in the initial feature vector set by a normalization processing module of the initial feature extraction model to obtain each normalized feature vector; and a normalization processing unit configured to perform a normalization processing on each initial feature vector in the initial feature vector set by a normalization processing module of the initial feature extraction model to obtain each normalized feature vector, where the normalization processing is performed based on a preset high discriminability loss function as a guide. and a highly distinctive loss function training unit configured to use the highly distinctive loss function to train an initial feature extraction model and obtain a target feature extraction model as a training result, wherein the highly distinctive loss function is abstracted based on predetermined upper and lower limit values ​​of vector similarity that are set in advance for sample images of any two types of image categories according to the presence or absence of similarity, and the vector similarity between the normalized feature vectors of two images that do not have similarity is equal to or less than the predetermined lower limit, and the vector similarity between the normalized feature vectors of two images that have similarity is equal to or more than the predetermined upper limit, and the predetermined upper limit is greater than the predetermined lower limit.

[0007] In a third aspect, an embodiment of the present disclosure provides an image classification method including the steps of: acquiring an image to be classified; inputting the image to be classified into a preset target feature extraction model obtained by the feature extraction model training method described in the first aspect to obtain an output real normalized feature vector; calculating vector similarities between the real normalized feature vector and standard normalized feature vectors of each type of image category in the feature vector library; and determining an image category to which the image to be classified belongs based on the magnitude of the vector similarities.

[0008] In a fourth aspect, an embodiment of the present disclosure provides an image classification device including: an image to be classified acquisition unit configured to acquire an image to be classified; an actual feature vector acquisition unit configured to obtain an actual normalized feature vector as an output by inputting the image to be classified into a preset target feature extraction model obtained by the feature extraction model training device described in the second aspect; a vector similarity calculation unit configured to calculate vector similarities between the actual normalized feature vector and standard normalized feature vectors of each type of image category in the feature vector library; and an image category determination unit configured to determine an image category to which the image to be classified belongs based on the magnitude of the vector similarity.

[0009] In a fifth aspect, an embodiment of the present disclosure provides an electronic device comprising at least one processor and a memory communicatively coupled to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions, when executed by the at least one processor, causing the at least one processor to perform the feature extraction model training method described in the first aspect and / or the image classification method described in the third aspect.

[0010] In a sixth aspect, an embodiment of the present disclosure provides a non-transitory computer readable storage medium having computer instructions stored thereon, the computer instructions being used to cause a computer to perform the feature extraction model training method of the first aspect and / or the image classification method of the third aspect.

[0011] In a seventh aspect, an embodiment of the present disclosure provides a computer program product which, when executed by a processor, realises the feature extraction model training method according to the first aspect and / or the image classification method according to the third aspect.

[0012] The feature extraction model training method and image classification method provided by the embodiments of the present disclosure add a normalization processing module, which not only normalizes the initial feature vector output from the basic feature extraction module, but also introduces a preset high distinctiveness loss function, which is obtained by abstracting sample images of any two image categories based on a predetermined upper limit value and a predetermined lower limit value of vector similarity that are preset depending on the presence or absence of similarity, and since the predetermined upper limit value is greater than the predetermined lower limit value, the feature extraction model can be guided to generate an image feature vector with clearer distinctiveness when calculating vector similarity, and more accurate image classification results for the category to which the image belongs can be obtained.

[0013] It should be noted that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present disclosure, and are not intended to limit the scope of the present disclosure. Other features of the present disclosure will become easier to understand from the following description.

[0014] Other features, objects and advantages of the present disclosure will become more apparent from a reading of the following detailed description of non-limiting embodiments, taken in conjunction with the drawings in which: [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 illustrates an exemplary system architecture to which the present disclosure can be applied. [Diagram 2] 1 is a flowchart of a method for training a feature extraction model according to an embodiment of the present disclosure. [Diagram 3] 1 is a flowchart of a method for training a feature extraction model using a highly discriminative loss function as a guide according to an embodiment of the present disclosure. [Figure 4A] FIG. 4 is a structural schematic diagram of a model corresponding to the solution of FIG. 3 according to an embodiment of the present disclosure. [Figure 4B] FIG. 4B is a structural schematic diagram of a model different from FIG. 4A according to an embodiment of the present disclosure. [Diagram 5]1 is a flowchart of an image classification method according to an embodiment of the present disclosure. [Figure 6] 1 is a flowchart of a method for determining an image category based on vector similarity according to an embodiment of the present disclosure. [Figure 7] FIG. 1 is a structural schematic diagram of a feature extraction model training apparatus according to an embodiment of the present disclosure; [Figure 8] FIG. 1 is a structural block diagram of an image classification device according to an embodiment of the present disclosure. [Figure 9] FIG. 2 is a structural schematic diagram of an electronic device suitable for performing the feature extraction model training method and / or the image classification method according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] The following describes exemplary embodiments of the present disclosure with reference to the drawings, and various details of the embodiments of the present disclosure are described to facilitate understanding, but it should be understood that these are merely illustrative. Therefore, it should be understood that those skilled in the art can make various changes and modifications to the embodiments of the present disclosure without departing from the scope and gist of the present disclosure. In the following description, for clarity and simplicity, descriptions of known functions and configurations are omitted. In addition, the embodiments of the present disclosure and features in the embodiments can be combined with each other as long as they do not cause contradictions.

[0017] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of relevant user personal information shall all comply with the provisions of relevant laws and regulations and shall not violate public order and morals.

[0018] FIG. 1 illustrates an exemplary system architecture 100 in which embodiments of the feature extraction model training method, image classification method, and related apparatus, electronic device, and computer-readable storage medium of the present application may be implemented.

[0019] 1, system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. Network 104 is used as a medium to provide a communications link between terminal devices 101, 102, 103 and server 105. Network 104 may include various types of connections, such as wires, wireless communications links, or fiber optic cables.

[0020] To send and receive messages, users can use terminal devices 101, 102, 103 to communicate with server 105 via network 104. Applications for realizing information communication between terminal devices 101, 102, 103 and server 105 can be installed, such as model training applications, image recognition applications, data transmission applications, and the like.

[0021] The terminal devices 101, 102, 103 and the server 105 may be hardware or software. If the terminal devices 101, 102, 103 are hardware, they may be various electronic devices equipped with a display screen, including but not limited to smartphones, tablet computers, laptop computers, desktop computers, etc. If the terminal devices 101, 102, 103 are software, they may be installed in the electronic devices listed above, or may be realized as multiple software or software modules, or may be realized as a single software or software module, but are not limited thereto. If the server 105 is hardware, it may be realized as a distributed server cluster consisting of multiple servers, or may be realized as a single server. If the server is software, it can be realized as multiple software or software modules, or as a single software or software module. No specific limitations are provided here.

[0022] The server 105 can provide various services by incorporating various applications. Take an image recognition application that can provide a user with a service of determining an image category of an image to be classified. When executing the image recognition application, the server 105 first receives the image to be classified input from the terminal devices 101, 102, and 103 via the network 104, then inputs the image to be classified into a preset target feature extraction model to obtain an actual normalized feature vector to be output, then calculates the vector similarity between the actual normalized feature vector and the standard normalized feature vector of each type of image category in the feature vector library, and finally determines the image category to which the image to be classified belongs based on the magnitude of the vector similarity. That is, the determined image category can be returned to the terminal devices 101, 102, and 103 via the network 104.

[0023] Among them, the target feature extraction model can be trained by a model training application built into the server 105 according to the following steps. the step of extracting image features of each sample image in the sample image set by a basic feature extraction module of the initial feature extraction model to obtain an initial feature vector set, the sample image set including sample images of a plurality of image categories, each of the sample images being marked with the image category to which it belongs; the step of performing normalization processing on each initial feature vector in the initial feature vector set by a normalization processing module of the initial feature extraction model to obtain each normalized feature vector; and the step of training the initial feature extraction model using a predetermined highly distinctive loss function as a guide to obtain a target feature extraction model as a training result, the step of the highly distinctive loss function being abstracted based on a predetermined upper limit value and a predetermined lower limit value of vector similarity that are respectively set in advance for sample images of any two types of image categories depending on the presence or absence of similarity, the vector similarity between the normalized feature vectors of two images that do not have similarity is equal to or less than the predetermined lower limit value, the vector similarity between the normalized feature vectors of two images that have similarity is equal to or more than the predetermined upper limit value, and the predetermined upper limit value is greater than the predetermined lower limit value.

[0024] Since the training of the target feature extraction model occupies a relatively large amount of computing resources and requires a relatively strong computing power, the feature extraction model training method provided by the following embodiments of the present application is generally executed by a server 105 having a relatively strong computing power and a relatively large amount of computing resources, and accordingly, the feature extraction model training device is also generally installed in the server 105. At the same time, it should be noted that if the terminal devices 101, 102, 103 also have the computing power and computing resources that meet the requirements, the terminal devices 101, 102, 103 can also complete the operations to be performed in the above server 105 through the model training application installed therein, and output the same results as the server 105. Accordingly, the feature extraction model training device may also be installed in the terminal devices 101, 102, 103. In this case, the exemplary system architecture 100 does not need to include the server 105 and the network 104.

[0025] Of course, the server for training and obtaining the target feature extraction model may be different from the server for calling the trained target feature extraction model. In particular, the target feature extraction model trained via the server 105 may use a lightweight target feature extraction model suitable for installation in the terminal devices 101, 102, and 103 obtained by model distillation. That is, the lightweight target feature extraction model in the terminal devices 101, 102, and 103 or the complex target feature extraction model in the server 105 can be flexibly selected according to the desired recognition accuracy.

[0026] It should be understood that the numbers of terminal devices, networks and servers in Figure 1 are merely exemplary, and the numbers of terminal devices, networks and servers may be arbitrarily increased or decreased according to the needs of implementation.

[0027] 2, which is a flow chart of a feature extraction model training method according to an embodiment of the present disclosure. The flow 200 includes the following steps.

[0028] In step 201, a basic feature extraction module of an initial feature extraction model extracts image features of each sample image in a sample image set to obtain an initial feature vector set.

[0029] The purpose of this step is for an entity executing the feature extraction model training method (e.g., the server 105 shown in FIG. 1) to input a sample image set including a plurality of sample images into a basic feature extraction module of an initial feature extraction model (i.e., an original model not trained with training samples), obtain initial feature vectors output by the basic feature extraction module for each sample image, and compile them to obtain an initial feature vector set.

[0030] Here, the sample image set includes sample images of multiple image categories, and each sample image is marked with the image category to which it belongs, and the mark may be expressed as a label added to the sample image or in other formats. The basic feature extraction module is a functional module for extracting image features from existing images, and typically includes a convolution layer, a pooling layer, and a possible upsampling layer, downsampling layer, etc., and the result of the feature extraction operation performed may not only be expressed in vector form, but also as a matrix or feature map. Here, only the feature vector is taken as a specific example, and in reality, other representation formats may be used, which does not affect the final result obtained in the later steps.

[0031] In step 202, a normalization module of the initial feature extraction model performs normalization on each initial feature vector in the initial feature vector set to obtain each normalized feature vector.

[0032] The purpose of this step is to, based on step 201, have the executing entity input the initial feature vector set output from the basic feature extraction module to a normalization processing module of the initial feature extraction model (i.e., the input terminal of the normalization processing module is connected to the output terminal of the basic feature processing module), and have the normalization processing module perform normalization processing on each initial feature vector in the initial feature vector set, thereby obtaining each normalized feature vector output from the normalization processing module.

[0033] The normalization process is intended to eliminate differences in expression between sample images of different origins after feature extraction processing by the basic feature extraction module, and by modifying each initial feature vector in the initial feature vector set to a smaller expression range, it is convenient for subsequent processing.

[0034] In step 203, an initial feature extraction model is trained using a pre-defined discriminative loss function as a guide, and a target feature extraction model is obtained as a training result.

[0035] Based on step 202, this step aims to train an initial feature extraction model using a preset high-discriminative loss function as a guide by the executing entity, and obtain a target feature extraction model as a training result, where the high-discriminative loss function is abstracted based on a predetermined upper limit value and a predetermined lower limit value of vector similarity that are preset for sample images of any two types of image categories, respectively, based on the presence or absence of similarity, that is, the vector similarity between the normalized feature vectors of two images that do not have similarity is equal to or less than the predetermined lower limit value, the vector similarity between the normalized feature vectors of two images that have similarity is equal to or more than the predetermined upper limit value, and the predetermined upper limit value is greater than the predetermined lower limit value.

[0036] For example, among three types of images, A, B, and C, an image in category A has similarity with an image in category B, and an image in category C has no similarity with an image in category B. In this case, if the image in category A has similarity with the image in category B, the vector similarity between the normalized feature vectors of the images is set to be greater than 80% (its predetermined upper limit value) in advance, and similarly, if the image in category C has no similarity with the image in category B, the vector similarity between the normalized feature vectors of the images is set to be smaller than 30% (its predetermined lower limit value) in advance, thereby obtaining a clear similarity or dissimilarity judgment result, and judging the similarity between the two types of images is easier than obtaining any judgment result that is evenly distributed between 0% and 100% when judging the similarity between the normalized feature vectors of two types of images. In summary, the purpose of setting this high-distinctness loss function is to make the image features corresponding to two types of images that should be similar have as high a vector similarity as possible, and the image features corresponding to two types of images that are not similar have as low a vector similarity as possible, so that the image category belonging judgment result becomes more straightforward. In addition, the feature vectors corresponding to the two images have similarity, and in subsequent practical applications, the image to be processed is considered to belong to that image category; otherwise, it is considered that the image cannot be determined to belong to that image category.

[0037] Specifically, in the present disclosure, whether two kinds of images have similarity can be defined as whether the image objects described in the two kinds of images have a small difference in at least one dimension. The dimension may include the size, shape, color, purpose, usage, etc. of the image object, as well as the similarity influence factors (such as color distribution, source, etc.) independently defined by the user in the actual application scene. For example, one blue apple image and one red apple image can be recognized as belonging to two kinds of images having similarity because they only have color difference and have almost no difference in the remaining dimensions; conversely, one pink peach image and one blue apple image can be recognized as two kinds of images not having similarity because they are both edible fruits in the dimension of usage and have obvious differences in other dimensions.

[0038] Preferably, the difference in similarity between the predetermined upper limit value and the predetermined lower limit value is equal to or greater than half of the entire range of similarity. For example, if the predetermined upper limit value is 83% and the predetermined lower limit value is 31%, the difference in similarity between the two is 52%, which is greater than half (50%) of the entire range of similarity (100%) and can provide sufficient distinction.

[0039] The feature extraction model training method provided by the embodiment of the present disclosure adds a normalization processing module, which, in addition to normalizing the initial feature vector output from the basic feature extraction module, further introduces a preset high distinctiveness loss function, which is obtained by abstracting the sample images of any two image categories based on a predetermined upper limit value and a predetermined lower limit value of vector similarity that are preset according to the presence or absence of similarity, and the predetermined upper limit value is greater than the predetermined lower limit value, so as to guide the feature extraction model to generate an image feature vector with clearer distinctiveness when calculating the vector similarity.

[0040] To better understand the process of how to guide the training of the initial feature extraction model through the preset highly discriminative loss function, the present disclosure also provides a specific implementation method in FIG. 3, whose flow 300 includes the following steps:

[0041] In step 301, a highly discriminative loss function is applied to guide the normalization process of the initial feature vector by the normalization module to obtain guide information.

[0042] As shown in FIG. 3A , the highly discriminative loss function is applied to the normalized feature vector output from the normalization processing module, so the training guide direction provided by the loss function is directly for the normalized feature vector. That is, by providing guide information directly to the output result of the normalization processing module, the normalization processing module can adjust the module parameters according to the correction direction provided by the guide information.

[0043] In step 302, the normalization processing module is controlled to inversely apply the guide information to guide the image feature extraction process by the basic feature extraction module.

[0044] Based on step 301, this step aims to control the normalization module to apply the guide information to guide the basic feature extraction module in the image feature extraction process by the execution entity, that is, when the high-discriminability loss function is directly applied to the normalization module, the normalization module needs to transmit the guide information to the upper-layer basic feature extraction module inversely, so that the basic feature extraction module can modify its own module parameters based on the guide information and output the initial feature vector according to the guide request to the normalization module.

[0045] Based on the methods shown in Figures 3 and 4A, when the highly distinctive loss function only acts directly on the normalized feature vector output from the normalization processing module, the guide information returned to the normalization processing module may not be fully understood by the basic feature extraction module in the upper layer. As a result, the effect of each update of the basic feature extraction module during training iterations may be reduced, more iterations may be required, and training may take longer.

[0046] To address this problem, when a basic feature extraction module includes multiple feature extraction sub-modules connected in series, a highly distinctive loss function is added to guide the feature extraction process of at least one of the feature extraction sub-modules. In other words, as shown in Figure 4B, in addition to the connection configuration for guiding the loss function in Figure 4A, a connection configuration for information feedback between the highly distinctive loss function and the second block (functional module) is added. This is equivalent to adding a function for causing the highly distinctive loss function to act on guiding the image feature extraction process of the basic feature extraction module. It is clear that direct guidance has a better effect than indirect guidance, which is expected to shorten training time and improve training efficiency.

[0047] In the above-mentioned embodiments, various aspects are described on how to train and obtain a target feature extraction model, but the present disclosure provides a method for solving a practical problem using a specifically trained target feature extraction model, so as to highlight the effect of the trained target feature extraction model from the actual usage scene as much as possible. The image classification method can be seen in the flow 500 shown in FIG. 5, which includes the following steps:

[0048] In step 501, an image to be classified is obtained.

[0049] The purpose of this step is for an executing entity that executes the image classification method (for example, the server 105 shown in FIG. 1 or another server different from the server 105) to acquire an image to be classified. For example, the image to be classified input by a user via a network may be received, and this image to be classified may be included in an image category determination request from the user.

[0050] In step 502, the classification target image is input to a preset target feature extraction model to obtain an actual normalized feature vector as an output.

[0051] The target feature extraction model called or used in this step is the target feature extraction model trained in the above-mentioned embodiment, and finally, the normalization processing module outputs the actual normalized feature vector corresponding to the image to be classified.

[0052] In step 503, the vector similarity between the real normalized feature vector and the standard normalized feature vector of each type of image category in the feature vector library is calculated.

[0053] Based on step 502, this step aims to calculate the vector similarity between the actual normalized feature vector and the standard normalized feature vector of each type of image category in the feature vector library by the executing subject.

[0054] In other words, the feature vector library is constructed in advance, and includes standard normalized feature vectors of multiple types of image categories to represent the feature vector of the image category, and the standard normalized feature vector may be obtained by inputting a standard image of the image category into the above-mentioned target feature extraction model and outputting it, or may be obtained by inputting multiple types of normal images under the image category into the above-mentioned target feature extraction model and integrating the output features, and there is no particular limitation here.

[0055] In step 504, an image category to which the classification target image belongs is determined based on the magnitude of the vector similarity.

[0056] Based on step 503, the purpose of this step is for the executing entity to determine the image category to which the classification target image belongs based on the magnitude of the vector similarity.

[0057] In practice, the image category to which the target standard normalized feature vector with the largest vector similarity belongs may usually be determined as the image category to which the image to be classified belongs, or several image categories with the highest vector similarities may be output together, and then the user's desired image category may be finally determined by direct selection.

[0058] The image classification method of this embodiment adds a normalization processing module, and in addition to normalizing the initial feature vector output from the basic feature extraction module, further introduces a preset high distinctiveness loss function. This high distinctiveness loss function is obtained by abstracting sample images of any two types of image categories based on a predetermined upper limit value and a predetermined lower limit value of vector similarity that are preset depending on the presence or absence of similarity, and since the predetermined upper limit value is greater than the predetermined lower limit value, the feature extraction model can be configured to generate an image feature vector with clearer distinctiveness when calculating vector similarity, thereby enabling to obtain more accurate image classification results for the category to which the image belongs.

[0059] Please refer to Fig. 6, which shows a flow chart of a method for determining an image category based on vector similarity. That is, a specific embodiment is provided for step 504 of flow 500 shown in Fig. 5, which replaces step 504 and does not adjust other steps of flow 500, resulting in a new complete embodiment. Flow 600 includes the following steps:

[0060] In step 601, a standard normalized feature vector that is a target having the greatest vector similarity to the actual normalized feature vector is determined.

[0061] That is, in this embodiment, a method is adopted in which the image category to which the standard normalized feature vector that is the target and has the largest vector similarity belongs is determined as the image category to which the classification target image belongs.

[0062] In step 602, it is determined whether the maximum vector similarity is smaller than a predetermined upper limit. If it is smaller, step 604 is executed; if not, step 506 is executed.

[0063] After determining the target reference normalized feature vector based on step 601, and before determining the image category to which the image to be classified belongs, this step has the execution body determine whether or not the largest vector similarity is smaller than a predetermined upper limit value, and the purpose of this step is to select a subsequent processing branch depending on the determination result.

[0064] In step 603, the image category to which the target standard normalized feature vector belongs is determined as the image category to which the classification target image belongs.

[0065] This step is based on the result of the determination in step 602 that the largest vector similarity is equal to or greater than a predetermined upper limit value, which indicates that the largest vector similarity exceeds the predetermined upper limit value, which is the minimum value indicating that two images belong to one category of images. Needless to say, if this minimum value is exceeded, the image category to which the target standard normalized feature vector belongs is determined to be the image category to which the image to be classified belongs.

[0066] In step 604, an image classification abnormality notification is returned, indicating that the image category to which the classification target image belongs could not be identified.

[0067] This step differs from step 603 in that the judgment result of step 602 is based on the fact that the largest vector similarity is smaller than a predetermined upper limit value, which means that the largest vector similarity does not exceed the predetermined upper limit value, which is the minimum value indicating that two images belong to one category of images.If the minimum value is not exceeded, it is naturally not possible to determine the image category to which the target standard normalized feature vector belongs as the image category to which the image to be classified belongs.Therefore, it is possible to return only an image classification anomaly notification that the image category to which the image to be classified belongs could not be identified, and this notification informs the user of the situation in which the image category could not be identified.

[0068] In step 605, the classification target image corresponding to the returned image classification abnormality notification is set as a new image category.

[0069] Based on step 604, this step is aimed at causing the executing entity to further determine the classification target image corresponding to the returned image classification anomaly notification as a new image category. Optionally, it is determined whether the largest vector similarity is less than the predetermined lower limit, and if it is less than the predetermined lower limit, it can be more reliably determined that the same image category as the classification target image does not exist in the feature vector library, thereby improving the accuracy of determining the new image category.

[0070] In step 606, the real normalized feature vector is added to the feature vector library as a standard normalized feature vector under the new image category.

[0071] Based on step 605, this step aims to add the real normalized feature vector to the feature vector library as a standard normalized feature vector under a new image category by the executing entity. That is, when there is only one image, the real normalized feature vector of the image to be classified can only be the standard normalized feature vector of the image of the category. After that, when multiple images of the category are received, the original standard normalized feature vector can be integrated and updated based on the real normalized feature vector of the subsequent image to obtain a more representative standard normalized feature vector.

[0072] 7 and 8, as implementations of the methods shown in the above figures, the present disclosure provides an embodiment of a feature extraction model training apparatus and an embodiment of an image classification apparatus, respectively, where the embodiment of the feature extraction model training apparatus corresponds to the embodiment of the feature extraction model training method shown in Fig. 2, and the embodiment of the image classification apparatus corresponds to the embodiment of the image classification method. The above apparatuses can be specifically applied to various electronic devices.

[0073] As shown in FIG. 7, the feature extraction model training device 700 of the present embodiment may include an image feature extraction unit 701, a normalization processing unit 702, and a highly discriminative loss function training unit 703. Among them, the image feature extraction unit 701 is configured to extract image features of each sample image in a sample image set by a basic feature extraction module of an initial feature extraction model to obtain an initial feature vector set. The sample image set includes sample images of multiple types of image categories, and each sample image is marked with the image category to which it belongs. The normalization processing unit 702 is configured to perform a normalization process on each initial feature vector in the initial feature vector set by a normalization processing module of the initial feature extraction model to obtain each normalized feature vector. The highly discriminative loss function training unit 703 is configured to train the initial feature extraction model using a preset highly discriminative loss function as a guide, and obtain a target feature extraction model as a training result. The high-distinctness loss function is abstracted based on a predetermined upper limit and a predetermined lower limit of vector similarity that are set in advance for sample images of any two image categories based on the presence or absence of similarity, such that the vector similarity between the normalized feature vectors of two images that do not have similarity is less than the predetermined lower limit, the vector similarity between the normalized feature vectors of two images that have similarity is greater than the predetermined upper limit, and the predetermined upper limit is greater than the predetermined lower limit.

[0074] In this embodiment, in the feature extraction model training apparatus 700, the specific processing of the image feature extraction unit 701, the normalization processing unit 702, and the highly discriminative loss function training unit 703 and the technical effects achieved thereby can refer to the relevant descriptions of steps 201 to 203 in the corresponding embodiment of FIG. 2, and the description thereof will be omitted here.

[0075] In some optional embodiments of this embodiment, the highly discriminative loss function training unit 703 further comprises: The highly discriminative loss function is applied to guide the normalization process of the initial feature vector by the normalization module to obtain guide information; The guide information may be configured to control the normalization processing module to apply the guide information in turn to guide the image feature extraction process by the base feature extraction module.

[0076] In some optional embodiments of this embodiment, the feature extraction model training apparatus 700 further comprises: The basic feature extraction module may further include an additional guiding unit configured to additionally apply a highly distinct loss function to guide the feature extraction process of at least one feature extraction sub-module in response to the basic feature extraction module including a plurality of feature extraction sub-modules connected in series.

[0077] In some optional embodiments of this embodiment, the difference between the similarity measure between the predetermined upper limit and the predetermined lower limit is equal to or greater than half of the entire interval of the similarity measure.

[0078] As shown in FIG. 8, the image classification device 800 of the present embodiment may include a classification target image acquisition unit 801, a real feature vector acquisition unit 802, a vector similarity calculation unit 803, and an image category determination unit 804. Among them, the classification target image acquisition unit 801 is configured to acquire a classification target image. The real feature vector acquisition unit 802 is configured to input the classification target image to a preset target feature extraction model obtained by the feature extraction model training device 700, thereby obtaining a real normalized feature vector as an output. The vector similarity calculation unit 803 is configured to calculate the vector similarity between the real normalized feature vector and the standard normalized feature vector of each type of image category in the feature vector library. The image category determination unit 804 is configured to determine the image category to which the classification target image belongs based on the magnitude of the vector similarity.

[0079] In this embodiment, the specific processing of the classification target image acquisition unit 801, the actual feature vector acquisition unit 802, the vector similarity calculation unit 803, and the image category determination unit 804 in the image classification device 800 and the technical effects achieved thereby may refer to the relevant descriptions in the method embodiments, and the description will be omitted here.

[0080] In some optional embodiments of this embodiment, the image category determination unit 804 further comprises: A target standard normalized feature vector having the greatest vector similarity to the real normalized feature vector is determined; In response to the largest vector similarity being equal to or greater than a predetermined upper limit, the image category to which the target standard normalized feature vector belongs may be set as the image category to which the classification target image belongs.

[0081] In some optional embodiments of this embodiment, the image classification device 800: The image classification anomaly notification returning unit may be configured to, in response to the largest vector similarity being less than a predetermined upper limit, return an image classification anomaly notification indicating that an image category to which the classification target image belongs could not be identified.

[0082] In some optional embodiments of this embodiment, the image classification device 800: a new image category determination unit configured to determine a new image category for the classification target image corresponding to the returned image classification anomaly notification; and a feature vector library adding unit configured to add the real normalized feature vector to the feature vector library as a standard normalized feature vector under the new image category.

[0083] The present embodiment is an embodiment of an apparatus corresponding to the embodiment of the above-mentioned method. The feature extraction model training apparatus and image classification apparatus provided by the present embodiment add a normalization processing module, so that in addition to normalizing the initial feature vector output from the basic feature extraction module, a preset high distinctiveness loss function is further introduced. This high distinctiveness loss function is obtained by abstracting sample images of any two types of image categories based on a predetermined upper limit value and a predetermined lower limit value of vector similarity that are preset depending on the presence or absence of similarity, and since the predetermined upper limit value is greater than the predetermined lower limit value, the feature extraction model can be configured to generate an image feature vector with clearer distinctiveness when calculating vector similarity, and more accurate image classification results for the category to which the image belongs can be obtained.

[0084] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device comprising at least one processor and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions, when executed by the at least one processor, causing the at least one processor to perform the feature extraction model training method and / or the image classification method described in any of the above embodiments.

[0085] According to an embodiment of the present disclosure, the present disclosure further provides a readable storage medium having stored thereon computer instructions for causing a computer to execute the feature extraction model training method and / or the image classification method described in any of the above embodiments.

[0086] An embodiment of the present disclosure provides a computer program capable of implementing the feature extraction model training method and / or the image classification method described in any of the above embodiments when executed by a processor.

[0087] 9 shows a schematic block diagram of an exemplary electronic device 900 that can be used to implement embodiments of the present disclosure. The electronic device may represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, mobile phones, smartphones, wearable devices, and other similar computing devices. It should be noted that the components shown, their connections, and their functions are merely exemplary and are not intended to limit the embodiments of the present disclosure described and / or claimed herein.

[0088] 9, the electronic device 900 includes a computing unit 901 that can execute various suitable operations and processes according to a computer program stored in a read-only memory (ROM) 902 or loaded from a storage unit 908 into a random access memory (RAM) 903. The RAM 903 can further store various programs and data necessary for the operation of the device 900. The computing unit 901, the ROM 902 and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0089] In the electronic device 900, several components including an input unit 906 such as a keyboard, a mouse, etc., an output unit 907 such as various types of displays, speakers, etc., a storage unit 908 such as a magnetic disk, an optical disk, etc., and a communication unit 909 such as a network plug-in, a modem, a wireless communication transceiver, etc. are connected to the I / O interface 905. The communication unit 909 enables the electronic device 900 to exchange information or data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0090] The computing unit 901 may be various general-purpose and / or special-purpose processing components having processing and computation capabilities. Some illustrative examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units that execute machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 executes various methods and processes, such as the feature extraction model training method and / or the image classification method described above. For example, in some embodiments, the feature extraction model training method and / or the image classification method may be realized as a computer software program tangibly included in a machine-readable medium, such as the storage unit 908. In some embodiments, some or all of the computer program may be loaded and / or installed in the electronic device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the feature extraction model training method and / or the image classification method described above may be executed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the feature extraction model training method and / or the image classification method in any other suitable manner (e.g., via firmware).

[0091] Various embodiments of the systems and techniques described herein may be realized in digital electronic circuitry systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. Each of these embodiments may be implemented in one or more computer programs that may be executed and / or interpreted in a programmable system that includes at least one programmable processor, which may be a special purpose or general purpose programmable processor, and may include receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0092] The program codes for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, a special purpose computer, or other programmable data processing apparatus, and when executed by the processor or controller, the functions or operations defined in the flowcharts and / or block diagrams are performed. The program codes may be executed entirely on the device, partially on the device, partially on the device as a stand-alone software package and partially on a remote device, or entirely on a remote device or server.

[0093] In the context of this disclosure, a machine-readable medium may be a tangible medium, and may include or store a program for use by or in combination with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination thereof. More specific examples of machine-readable storage media may include electrical connections based on one or more cables, portable computer disks, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), optical fibers, compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0094] To provide interaction with a user, the systems and techniques described herein may be implemented on a computer that includes a display device (e.g., a Cathode Ray Tube (CRT) or LCD (Liquid Crystal Display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) through which a user can provide input to the computer. Other types of devices may also be used to interact with a user. For example, feedback provided to the user may be any form of sensing feedback, e.g., visual feedback, auditory feedback, or haptic feedback, and may receive input from the user in any form, including audio input, speech input, or haptic input.

[0095] The systems and techniques described herein may be implemented in a computing system including a back-end component (e.g., a data server), a computing system including a middleware component (e.g., an application server), a computing system including a front-end component (e.g., a user computer having a graphical user interface or a web browser) through which a user may interact with the embodiments of the systems and techniques described herein, or a computing system including any combination of such back-end, middleware, or front-end components. The components of the system may be connected by digital data communication through any form or medium, such as a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0096] The computer system may include a client and a server. The client and the server are usually remote from each other and communicate with each other via a communication network. The relationship between the client and the server is generated by running computer programs having a client-server relationship with each other on the respective computers. The server may be a cloud server, also called a cloud computing server or cloud host, which is a host product in a cloud computing service system and solves the defects of the traditional physical host and virtual private server (VPS) services, such as high management difficulty and low business scalability.

[0097] The technical proposal of the embodiment of the present disclosure adds a normalization processing module, which not only normalizes the initial feature vector output from the basic feature extraction module, but also introduces a preset high distinctiveness loss function, which is obtained by abstracting sample images of any two image categories based on a predetermined upper limit value and a predetermined lower limit value of vector similarity that are preset according to the presence or absence of similarity, and since the predetermined upper limit value is greater than the predetermined lower limit value, the feature extraction model can be configured to generate an image feature vector with clearer distinctiveness when calculating vector similarity, and more accurate image classification results for the category to which the image belongs can be obtained.

[0098] It should be understood that the above-mentioned various forms of flow can be used to rearrange, add, or delete steps. For example, each step described in the present disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution disclosed in the present disclosure can be achieved. This specification is not limited thereto.

[0099] The above specific embodiments do not limit the scope of protection of the present disclosure. It should be understood that those skilled in the art can make various modifications, combinations, subcombinations, and substitutions in response to design requirements and other factors. Any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and principle of the present disclosure should be included within the scope of protection of the present disclosure.

Claims

1. A step of extracting image features of each sample image in a sample image set by a basic feature extraction module of an initial feature extraction model to obtain an initial feature vector set, the sample image set including sample images of multiple image categories, and each of the sample images is marked with the image category to which it belongs; performing a normalization process on each initial feature vector in the initial feature vector set by a normalization process module of the initial feature extraction model to obtain each normalized feature vector; a step of training the initial feature extraction model using a preset high-discriminability loss function as a guide to obtain a target feature extraction model as a training result, the high-discriminability loss function being abstracted based on a preset upper limit value and a preset lower limit value of vector similarity for sample images of any two types of image categories, depending on the presence or absence of similarity, the vector similarity between normalized feature vectors of two images that do not have similarity is equal to or less than the preset lower limit value, the vector similarity between normalized feature vectors of two images that have similarity is equal to or more than the preset upper limit value, and the preset upper limit value is greater than the preset lower limit value; In response to the basic feature extraction module including a plurality of feature extraction sub-modules connected in series, additionally applying the highly discriminative loss function to guide the feature extraction process of at least one of the feature extraction sub-modules; training the initial feature extraction model using a predefined highly discriminative loss function as a guide, Applying the highly discriminative loss function to guide the normalization process of the initial feature vector by the normalization module to obtain guide information; and controlling the normalization processing module to inversely apply the guide information to guide the basic feature extraction module to an image feature extraction process.

2. A step of extracting image features of each sample image in a sample image set by a basic feature extraction module of an initial feature extraction model to obtain an initial feature vector set, the sample image set including sample images of multiple image categories, and each of the sample images is marked with the image category to which it belongs; performing a normalization process on each initial feature vector in the initial feature vector set by a normalization process module of the initial feature extraction model to obtain each normalized feature vector; a step of training the initial feature extraction model using a preset high-discriminability loss function as a guide to obtain a target feature extraction model as a training result, the high-discriminability loss function being abstracted based on a preset upper limit value and a preset lower limit value of vector similarity for sample images of any two types of image categories, depending on the presence or absence of similarity, the vector similarity between normalized feature vectors of two images that do not have similarity is equal to or less than the preset lower limit value, the vector similarity between normalized feature vectors of two images that have similarity is equal to or more than the preset upper limit value, and the preset upper limit value is greater than the preset lower limit value; A method for training a feature extraction model, comprising: A feature extraction model training method, wherein a difference in similarity between the predetermined upper limit value and the predetermined lower limit value is equal to or greater than half of the entire range of similarity.

3. A step of extracting image features of each sample image in a sample image set by a basic feature extraction module of an initial feature extraction model to obtain an initial feature vector set, the sample image set including sample images of multiple image categories, and each of the sample images is marked with the image category to which it belongs; performing a normalization process on each initial feature vector in the initial feature vector set by a normalization process module of the initial feature extraction model to obtain each normalized feature vector; a step of training the initial feature extraction model using a preset high-discriminability loss function as a guide to obtain a target feature extraction model as a training result, the high-discriminability loss function being abstracted based on a preset upper limit value and a preset lower limit value of vector similarity for sample images of any two types of image categories, depending on the presence or absence of similarity, the vector similarity between normalized feature vectors of two images that do not have similarity is equal to or less than the preset lower limit value, the vector similarity between normalized feature vectors of two images that have similarity is equal to or more than the preset upper limit value, and the preset upper limit value is greater than the preset lower limit value; A feature extraction model training step including: obtaining an image to be classified; a step of inputting the classification target image into a preset target feature extraction model obtained by the feature extraction model training step, thereby obtaining an actual normalized feature vector as an output; Calculating vector similarity between the real normalized feature vector and the standard normalized feature vector of each type of image category in a feature vector library; and determining an image category to which the classification target image belongs based on the magnitude of the vector similarity.

4. The step of determining an image category to which the classification target image belongs based on the magnitude of the vector similarity includes: determining a target standard normalized feature vector having the greatest vector similarity to the real normalized feature vector; and in response to the largest vector similarity being equal to or greater than a predetermined upper limit, determining that the image category to which the target standard normalized feature vector belongs is the image category to which the classification target image belongs.

5. The image classification method according to claim 4 , further comprising the step of returning an image classification anomaly notification indicating that the image category to which the classification target image belongs cannot be identified in response to the largest vector similarity being less than the predetermined upper limit value.

6. A step of setting a classification target image corresponding to the returned image classification anomaly notification as a new image category; The method of claim 5 , further comprising adding the real normalized feature vector to the feature vector library as a standard normalized feature vector under the new image category.

7. an image feature extraction unit configured to extract image features of each sample image in a sample image set by a basic feature extraction module of an initial feature extraction model to obtain an initial feature vector set, the sample image set including sample images of multiple image categories, and each of the sample images is marked with the image category to which it belongs; a normalization processing unit configured to perform a normalization processing on each initial feature vector in the initial feature vector set by a normalization processing module of the initial feature extraction model to obtain each normalized feature vector; a highly distinctive loss function training unit configured to train the initial feature extraction model using a preset highly distinctive loss function as a guide to obtain a target feature extraction model as a training result, the highly distinctive loss function being abstracted based on a preset upper limit value and a preset lower limit value of vector similarity for sample images of any two types of image categories, depending on the presence or absence of similarity, the vector similarity between normalized feature vectors of two images that do not have similarity is equal to or less than the preset lower limit value, the vector similarity between normalized feature vectors of two images that have similarity is equal to or more than the preset upper limit value, and the preset upper limit value is greater than the preset lower limit value; an additional guiding unit configured to additionally apply the highly discriminative loss function to guide a feature extraction process of at least one of the feature extraction sub-modules in response to the basic feature extraction module including a plurality of feature extraction sub-modules connected in series; The highly discriminative loss function training unit further comprises: Applying the highly discriminative loss function to guide the normalization process of the initial feature vector by the normalization module to obtain guide information; A feature extraction model training device configured to control the normalization processing module to inversely apply the guide information to guide the basic feature extraction module to an image feature extraction process.

8. an image feature extraction unit configured to extract image features of each sample image in a sample image set by a basic feature extraction module of an initial feature extraction model to obtain an initial feature vector set, the sample image set including sample images of multiple image categories, and each of the sample images is marked with the image category to which it belongs; a normalization processing unit configured to perform a normalization processing on each initial feature vector in the initial feature vector set by a normalization processing module of the initial feature extraction model to obtain each normalized feature vector; a highly distinctive loss function training unit configured to train the initial feature extraction model using a preset highly distinctive loss function as a guide to obtain a target feature extraction model as a training result, the highly distinctive loss function being abstracted based on a preset upper limit value and a preset lower limit value of vector similarity for sample images of any two types of image categories, depending on the presence or absence of similarity, the vector similarity between normalized feature vectors of two images that do not have similarity is equal to or less than the preset lower limit value, the vector similarity between normalized feature vectors of two images that have similarity is equal to or more than the preset upper limit value, and the preset upper limit value is greater than the preset lower limit value; A feature extraction model training apparatus comprising: A feature extraction model training device, wherein a difference in similarity between the predetermined upper limit value and the predetermined lower limit value is equal to or greater than half of the entire range of similarity.

9. a classification target image acquisition unit configured to acquire an image to be classified; an actual feature vector acquisition unit configured to input the classification target image to a preset target feature extraction model obtained by a feature extraction model training device, thereby obtaining an actual normalized feature vector as an output; a vector similarity calculation unit configured to respectively calculate a vector similarity between the real normalized feature vector and a standard normalized feature vector of each type of image category in a feature vector library; an image category determination unit configured to determine an image category to which the classification target image belongs based on the magnitude of the vector similarity; An image classification device comprising: The feature extraction model training device comprises: an image feature extraction unit configured to extract image features of each sample image in a sample image set by a basic feature extraction module of an initial feature extraction model to obtain an initial feature vector set, the sample image set including sample images of multiple image categories, and each of the sample images is marked with the image category to which it belongs; a normalization processing unit configured to perform a normalization processing on each initial feature vector in the initial feature vector set by a normalization processing module of the initial feature extraction model to obtain each normalized feature vector; a highly distinctive loss function training unit configured to train the initial feature extraction model using a preset highly distinctive loss function as a guide to obtain a target feature extraction model as a training result, the highly distinctive loss function being abstracted based on a preset upper limit value and a preset lower limit value of vector similarity for sample images of any two types of image categories, depending on the presence or absence of similarity, the vector similarity between normalized feature vectors of two images that do not have similarity is equal to or less than the preset lower limit value, the vector similarity between normalized feature vectors of two images that have similarity is equal to or more than the preset upper limit value, and the preset upper limit value is greater than the preset lower limit value; An image classification device comprising:

10. The image category determining unit further comprises: determining a target standard normalized feature vector having the greatest vector similarity to the real normalized feature vector; 10. The image classification device of claim 9, configured to set the image category to which the target standard normalized feature vector belongs as the image category to which the classification target image belongs in response to the largest vector similarity being equal to or greater than a predetermined upper limit value.

11. The image classification device of claim 10 , further comprising an image classification anomaly notification returning unit configured to, in response to the largest vector similarity being less than the predetermined upper limit value, return an image classification anomaly notification indicating that an image category to which the image to be classified belongs cannot be identified.

12. a new image category determination unit configured to determine a new image category for a classification target image corresponding to the returned image classification anomaly notification; The image classification apparatus of claim 11 , further comprising: a feature vector library adding unit configured to add the real normalized feature vector to the feature vector library as a standard normalized feature vector under the new image category.

13. At least one processor; and a memory communicatively coupled to the at least one processor, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, cause the at least one processor to perform the feature extraction model training method of claim 1 or 2 and / or the image classification method of claim 3.

14. A non-transitory computer-readable storage medium having computer instructions stored thereon, comprising: A non-transitory computer-readable storage medium in which the computer instructions are used to cause a computer to perform the method for training a feature extraction model according to claim 1 or 2 and / or the method for classifying images according to claim 3.

15. A computer program product which, when executed by a processor, implements the steps of the method for training a feature extraction model according to claim 1 or 2 and / or the steps of the method for classifying images according to claim 3.

Images