Methods, devices, equipment, media and products for generating models and extracting features

Abstract

The present disclosure provides a method, device, equipment, medium and product for generating models and extracting features, which relates to the field of image processing technology, especially to the field of computer vision and deep learning technology. The specific implementation scheme is as follows: obtaining a sample image set; performing initial image feature extraction on the sample images in the sample image set to obtain initial image features; extracting global features and local features from the initial image features; determining the non-projected component of the first feature on the second feature; fusing the non-projected component and the second feature to obtain the target image features of the sample image; training the target image features corresponding to the sample images in the sample image set to generate an image feature extraction model. The image feature extraction model is used to extract both global features and local features to improve the accuracy while ensuring the recall rate of image matching.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to the field of computer vision and deep learning technology. Background Art

[0002] Image matching is a crucial step in the automated production of Points of Interest (POIs). Whether image features can accurately describe signboard images is crucial for image matching. Signboard image features can be described from two perspectives: global features and local features. Global features refer to the overall properties of an image. Common global features include color, texture, and shape. Local features are features extracted from local areas of an image. They include edges, corners, lines, curves, and areas with special attributes. From the perspective of a signboard, global features refer to the color and font of the signboard, while local features refer to the differences between individual characters on the signboard. Summary of the Invention

[0003] The present disclosure provides a method, apparatus, device, medium, and product for generating a model and extracting features.

[0004] According to one aspect of the present disclosure, a method for generating an image feature extraction model is provided, comprising: acquiring a sample image set; performing initial image feature extraction on sample images in the sample image set to obtain initial image features; extracting global features and local features from the initial image features; determining a non-projected component of a first feature on a second feature, wherein the first feature is the global feature and the second feature is the local feature, or the first feature is the local feature and the second feature is the global feature; fusing the non-projected component and the second feature to obtain target image features of the sample images; and training the target image features corresponding to the sample images in the sample image set to generate an image feature extraction model.

[0005] According to another aspect of the present disclosure, a method for extracting image features is provided, comprising: acquiring an image to be subjected to image feature extraction; inputting the image into an image feature extraction model generated by the method described above, and obtaining image features output by the image feature extraction model.

[0006] According to another aspect of the present disclosure, a device for generating an image feature extraction model is provided, comprising: a sample acquisition unit for acquiring a sample image set; a first extraction unit for performing initial image feature extraction on sample images in the sample image set to obtain initial image features; a second extraction unit for extracting global features and local features from the initial image features; a fusion unit for determining a non-projected component of a first feature on a second feature, wherein the first feature is the global feature and the second feature is the local feature, or the first feature is the local feature and the second feature is the global feature; and for fusing the non-projected component with the second feature to obtain target image features of the sample images; and a training unit for training target image features corresponding to the sample images in the sample image set to generate an image feature extraction model.

[0007] According to another aspect of the present disclosure, a device for extracting image features is provided, comprising: an image acquisition unit for acquiring an image to be subjected to image feature extraction; an extraction unit for inputting the image into an image feature extraction model generated by the method described above, to obtain image features output by the image feature extraction model.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described above.

[0009] According to yet another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above-described method.

[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the above-mentioned method when executed by a processor.

[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0013] Figure 1a This is a schematic diagram of the first type of signboard image;

[0014] Figure 1b This is a schematic diagram of the second type of signboard image;

[0015] Figure 2a This is a schematic diagram of the third type of signboard image;

[0016] Figure 2b This is a schematic diagram of the fourth type of signboard image;

[0017] Figure 3 is a flow chart of a method for generating an image feature extraction model according to an embodiment of the present disclosure;

[0018] Figure 4 is a flowchart of extracting initial image features according to an embodiment of the present disclosure;

[0019] Figure 5 is a flowchart of extracting local features according to an embodiment of the present disclosure;

[0020] Figure 6 is a flowchart of determining non-projection components according to an embodiment of the present disclosure;

[0021] Figure 7 is a flowchart of generating an image feature extraction model according to an embodiment of the present disclosure;

[0022] Figure 8 is a schematic diagram of an image feature extraction network provided according to an embodiment of the present disclosure;

[0023] Figure 9 is a flow chart of a method for extracting image features according to an embodiment of the present disclosure;

[0024] Figure 10 is a block diagram of a device for generating an image feature extraction model according to an exemplary embodiment;

[0025] Figure 11 is a block diagram of a device for extracting image features according to an exemplary embodiment;

[0026] Figure 12 It is a block diagram of an electronic device used to implement the method of generating an image feature extraction model or extracting image features according to the embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0028] The present disclosure is applicable to automated POI production scenarios. Specifically, it is applied to the process of attaching a sign image to a POI library. This can be understood as extracting image features from the sign image, matching these extracted image features with the image features of each POI in the POI library, and attaching the sign image to the POI with the highest similarity. It can be seen that whether the image features accurately describe the sign image is the key to image matching. To more clearly illustrate the differences in image features between different sign images, the following embodiments will be described with reference to the accompanying figures.

[0029] Figure 1a This is a schematic diagram of the first type of sign image. Figure 1b is a schematic diagram of the second type of signboard image. Figure 1a and Figure 1b As can be seen from the schematic diagram of the signboard image, Figure 1a and Figure 1b The signboard images are obviously different in terms of color, texture, etc. Figure 1a and Figure 1b The color, texture, and shape features extracted from the signboard images also exhibit distinct differences. These features, which reflect the overall properties of an image, are collectively referred to as global features. Generally, global features can accurately distinguish widely varying signboards. Figure 2a This is a schematic diagram of the third type of sign image. Figure 2b is a schematic diagram of the fourth type of signboard image. Figure 2a and Figure 2b As can be seen from the schematic diagram of the signboard image, Figure 2a and Figure 2b The signboard images are identical in color, texture, etc. Figure 2a The one in the picture is "AC Eye X", and Figure 2b The difference is only in the single characters. In this case, global features cannot accurately distinguish Figure 2a and Figure 2b Therefore, it is necessary to extract features from local regions of the image to highlight the differences. Local regions can include edges, corners, lines, curves, and areas with special attributes. Features extracted from local regions of the image are collectively referred to as local features.

[0030] In related technologies, most image feature extraction methods extract global features of images, while the POI production process based on signboard images requires that image features be Figure 2a and Figure 2b Therefore, it is necessary to optimize the image features from the perspective of local features, but the following problems may occur:

[0031] 1. When more attention is paid to local features, the recall rate of image matching drops sharply.

[0032] 2. During large-scale feature training, the data labeling accuracy is not high, which affects the feature expression of the model.

[0033] In summary, the flaws of the image feature extraction methods used in related technologies include: extracting only global features, which makes it impossible to distinguish different signs at a fine-grained level; focusing too much on local features, which leads to a rapid decrease in recall; and / or failing to address the impact of dirty data during training. Dirty data refers to data whose image features do not match the POIs. Dirty data is also called noisy data.

[0034] In light of this, this paper proposes an image feature extraction model for signboard images. This model simultaneously addresses both global and local features of signboard images, ensuring improved production accuracy without sacrificing recall. Furthermore, this model employs a multi-center clustering approach to train the image feature extraction model. Compared to related training methods, this model is more tolerant to dirty data during training, thereby reducing its impact on image features.

[0035] The following embodiments of the present disclosure will illustrate a method for generating an image feature extraction model in conjunction with the accompanying drawings.

[0036] Figure 3 is a flow chart of a method for generating an image feature extraction model according to an embodiment of the present disclosure; Figure 3 As shown, the method for generating an image feature extraction model in the embodiment of the present disclosure includes the following steps.

[0037] In step S301, a sample image set is obtained.

[0038] In the disclosed embodiment, a sample image set can be obtained from an existing sign image library. The sample image set includes multiple sample images. Each sample image is associated with a pre-labeled category. The labeled category refers to the POI to which it belongs. For example, POIs may include "KFC," "McDonald's," "XXX Dental Hospital," etc. If a sample image represents "KFC," then the labeled category of the sample image is "KFC."

[0039] In step S302, initial image features are extracted from the sample images in the sample image set to obtain initial image features.

[0040] The pre-trained neural network structure is used to extract the initial image features of the sample image.

[0041] In step S303, global features and local features are extracted from the initial image features.

[0042] Global features are extracted from the initial image features, and local features are extracted from the initial image features.

[0043] In one embodiment, a residual neural network model and an attention model may be used to extract local features of the initial image features. In another embodiment, a cascade of M shifted window codecs may be used to extract local features from the initial image features.

[0044] In step S304 , a non-projected component of the first feature on the second feature is determined.

[0045] In the disclosed embodiments, the local features obtained may include global features. Similarly, the global features obtained may include local features. In order to remove redundancy, reduce computational complexity, and improve accuracy, orthogonal fusion of local features and global features is performed.

[0046] In one embodiment, the specific process of orthogonally fusing local features and global features includes determining the projection component of the local feature on the global feature, and then subtracting the projection component from the local feature to obtain the non-projection component. In this embodiment, the first feature is the local feature and the second feature is the global feature.

[0047] In another embodiment, the specific process of orthogonally fusing the local features and the global features includes determining the projection component of the global feature on the local feature, and then subtracting the projection component from the global feature to obtain the non-projection component. In this embodiment, the first feature is the global feature and the second feature is the local feature.

[0048] In step S305 , the non-projected component and the second feature are fused to obtain the target image feature of the sample image.

[0049] In step S306, target image features corresponding to the sample images in the sample image set are trained to generate an image feature extraction model.

[0050] In the disclosed embodiments, training is performed using target image features corresponding to sample images in a sample image set, as well as annotated image features corresponding to the sample images. Training of the image feature extraction model is completed until convergence conditions are met between the target image features corresponding to the sample images and the annotated image features. The trained image feature extraction model is then applied to automated POI production.

[0051] The image feature extraction model provided by the embodiment of the present disclosure extracts global features and local features from an image, and then removes redundant features in the global features or local features by determining the non-projected components between the global features and the local features, so that the target image features obtained have both global features and local features, thereby improving the accuracy while ensuring the image matching recall rate.

[0052] On the basis of any of the above embodiments, the intermediate layer neural network of the Convolutional Neural Networks (CNN) is used as the network structure for extracting initial image features in the image feature extraction model.

[0053] Figure 4 is a flowchart of extracting initial image features according to an embodiment of the present disclosure; Figure 4 As shown, in the embodiment of the present disclosure, initial image feature extraction is performed on sample images in the sample image set to obtain initial image features, which includes the following steps.

[0054] In step S401, a pre-trained model is determined, and an intermediate layer neural network is extracted from the pre-trained model as an initial neural network for extracting initial image features.

[0055] In one embodiment, the pre-training model is a CNN. The process of pre-training the CNN includes obtaining a large number of sample signboard images, the sample image features carried by them, and the annotated image feature vectors. The CNN is trained using the obtained massive sample signboard images, the sample image features carried by them, and the annotated image feature vectors. Until the predicted image feature vector modeled by the CNN based on the sample signboard images and the sample image features carried by them meets the convergence condition with the annotated image feature vector, it can be determined that the CNN training is completed. In addition to CNN, the pre-training module can also be a VIT (Vision Transformer) network or a deep residual network (ResNet).

[0056] Extract the intermediate neural network layer from the pre-trained model. For example, if the CNN has 5 layers, use the 3rd layer as the intermediate neural network layer. If the CNN has 7 layers, use the 4th layer as the intermediate neural network layer. If the CNN has 8 layers, use the 4th layer as the intermediate neural network layer, and so on. Use the intermediate neural network layer as the initial neural network layer to extract the initial image features.

[0057] In step S402, the sample images in the sample image set are input into the initial neural network to obtain initial image features.

[0058] Extracting the initial image features of the sample image through the intermediate layer neural network of the pre-trained model can retain more local features in the sample image while reducing the amount of calculation and the complexity of the model compared to extracting the initial image features through the entire pre-trained model.

[0059] Based on any of the above embodiments, local features are extracted through a local feature branch in the image feature extraction model, wherein the local feature branch includes multiple dilated convolutions of different scales.

[0060] Figure 5 is a flowchart of extracting local features according to an embodiment of the present disclosure; Figure 5 As shown, extracting local features from initial image features in the embodiment of the present disclosure includes the following steps.

[0061] In step S501, multiple dilated convolutions of different scales are used to extract local features corresponding to different scales from the initial image features.

[0062] Through multiple dilated convolutions of different scales, we can focus on texts of different sizes in the same signboard image. Figure 2a As shown in the figure, the first-scale dilated convolution can extract "AC Eye X", and the second-scale dilated convolution can extract "Professional Optician Center". The first scale is larger than the second scale.

[0063] In the disclosed embodiments, the scale of the dilated convolution is a parameter that can be adjusted based on actual conditions. For example, if the font size of the signboard image to be processed is relatively large, a larger scale value can be set. In one embodiment, three different scales of dilated convolution are provided. Using these three different scales of dilated convolution, local features of three different scales are extracted from the initial image features.

[0064] In step S502, pooling processing is performed on local features of different scales.

[0065] Perform global average pooling on local features of different scales to obtain local features after pooling at the corresponding scale. In addition to average pooling, maximum pooling, minimum pooling, etc. can also be performed, which is not specifically limited in this disclosure.

[0066] In step S503, the local features of multiple scales obtained by the pooling process are convolved to obtain local features of the initial image features.

[0067] Using multiple dilated convolutions of different scales to extract local features, compared to using the scale-invariant feature transform (SIFT) algorithm and the image feature pyramid to obtain local features of the image, the local feature branch is essentially a global description of an image. Obtaining local features through SIFT is to describe the features of each point in the image, which can be understood as point features. Therefore, the local features extracted by SIFT cannot be trained in conjunction with the entire image feature extraction model. Therefore, the present disclosure simulates the image feature pyramid used in SIFT through multiple dilated convolutions of different scales, thereby solving the problem of font size changes in the same image and obtaining multi-scale local features.

[0068] Using multiple dilated convolutions of different scales to extract local features allows for the extraction of local features through training, increasing robustness. It also omits the point extraction step used in SIFT local feature extraction, avoiding the extraction of a large number of useless points. In other words, extracting local features using the local feature branch allows for direct localization and feature description of the points of interest. In other words, the local feature extraction step can be simplified, reducing computational effort.

[0069] In the disclosed embodiment, by utilizing dilated convolutions of different scales, a local feature branch for extracting local features is integrated into the image feature extraction model, which is beneficial for fusing local features with global features.

[0070] Using multiple dilated convolutions of different scales to extract local features from the initial image features can ensure the comprehensiveness of the extracted local features.

[0071] Based on any of the above embodiments, global features are extracted through a global feature branch in the image feature extraction model. The global feature branch includes convolution Conv and down sampling down sample.

[0072] In the disclosed embodiment, extracting global features from initial image features using the global feature branch includes: performing convolution and downsampling on the initial image features to extract global features from the initial image features. The global features are extracted from the initial image through convolution and downsampling.

[0073] Based on any of the above embodiments, the orthogonal fusion module in the image feature extraction model is used to fuse local and global features to obtain the target image features of the sample image. The specific process is as follows: determining the non-projected component of the first feature on the second feature, fusing the non-projected component with the second feature, and obtaining the target image features of the sample image. The first feature is a global feature and the second feature is a local feature, or the first feature is a local feature and the second feature is a global feature.

[0074] Figure 6 is a flowchart of determining non-projection components according to an embodiment of the present disclosure; Figure 6 As shown, in the embodiment of the present disclosure, determining the non-projection component of the first feature on the second feature includes the following steps.

[0075] In step S601 , based on the ratio of the first feature to the second feature, a projection component of the first feature on the second feature is determined.

[0076] In the embodiment of the present disclosure, the projection component of the first feature on the second feature is determined using the following formula.

[0077]

[0078] In the above formula, Represents the first feature F l The feature with height i and width j in the second feature F g The projection component on . Represents the first feature F l The feature with height i and width j. g Indicates the second feature.

[0079] In step S602 , based on the difference between the first feature and the projection component, a non-projection component of the first feature on the second feature is determined.

[0080] Subtract the projected component from the first feature to obtain the non-projected component.

[0081] In this embodiment, the non-projected component of the first feature on the second feature is determined by the following formula.

[0082]

[0083] In the above formula, F orth Represents the non-projected component of the first feature onto the second feature.

[0084] By subtracting the projection component from the first feature, the second feature component contained in the first feature is removed, so that the obtained first feature is a more discriminative orthogonal feature.

[0085] Based on any of the above embodiments, fusing the non-projected component and the second feature to obtain the target image feature of the sample image includes: concatenating the non-projected component and the second feature; and performing a pooling process on the concatenated feature to obtain the target image feature of the sample image. For example, the concatenated feature may be average pooled to obtain the target image feature of the sample image.

[0086] The non-projected component and the second feature are fused by splicing and pooling, so that the target image features include both local features and global features.

[0087] Based on any of the above embodiments, target image features are trained by multi-center clustering to generate an image feature extraction model.

[0088] Figure 7 is a flow chart of generating an image feature extraction model according to an embodiment of the present disclosure; Figure 7 As shown, in the embodiment of the present disclosure, target image features corresponding to sample images in a sample image set are trained to generate an image feature extraction model, which includes the following steps.

[0089] In step S701 , a plurality of cluster centers are set for each labeled category corresponding to the sample images in the sample image set.

[0090] Assume there are N labeled categories. For each labeled category, randomly initialize K cluster centers. Then use N*K cluster centers to perform classification training on the target image features. Generally, K can be 3. Both K and N are positive integers.

[0091] The purpose of setting K cluster centers is that for the same labeled category (i.e., POI), there may be multiple forms of expression. For example, KFC and McDonald's belong to two different categories. Taking the POI of KFC as an example, the input sign image may be the three letters "KFC", or the three Chinese characters "Kentucky", or the KFC trademark. Although "KFC", "Kentucky" and the KFC trademark belong to the same POI, the image features are completely different. Therefore, the present disclosure sets multiple cluster centers under one POI to meet the multiple different forms of expression of the same POI.

[0092] In step S702 , classification training is performed on the target image features corresponding to the sample images based on all cluster centers to determine the prediction category to which the target image features belong.

[0093] Continuing with the above example, if the cluster center of the target image feature is "KFC", then the predicted category of the target image feature is KFC. If the cluster center of the target image feature is "KFC", then the predicted category of the target image feature is also KFC.

[0094] In step S703 , an image feature extraction model is generated until the labeled category corresponding to the sample image and the predicted category meet a convergence condition.

[0095] The present disclosure uses a multi-cluster center training method to set multiple cluster centers for each labeled category, thereby relaxing the constraints of intra-class compactness and achieving the purpose of being robust to noisy data.

[0096] The image feature extraction network provided by the present disclosure can be understood as an improvement on CNN, so that the image features extracted by the improved CNN include both global features and local features. Figure 8 is a schematic diagram of an image feature extraction network provided according to an embodiment of the present disclosure; Figure 8 As shown in the figure, an intermediate neural network layer is extracted from the pretrained model and used as the initial neural network. The output of the initial neural network is connected to the input of the local feature branch and the input of the global feature branch, respectively. The output of the local feature branch and the output of the global feature branch are connected to the input of the orthogonal fusion module, which outputs the target image features. A sample image is input into the initial neural network, and after a series of processing, the target image features are obtained. During the model training phase, the target image features are classified and trained using multi-center clustering to generate an image feature extraction model.

[0097] Based on the same inventive concept, the present disclosure provides a method for extracting image features. Figure 9 FIG. 1 is a flow chart of a method for extracting image features according to an embodiment of the present disclosure. Figure 9 As shown, the method for extracting image features includes the following steps.

[0098] In step S901, an image to be subjected to image feature extraction is obtained.

[0099] In step S902, the image is input into the image feature extraction model to obtain the image features output by the image feature extraction model.

[0100] The image features obtained by the image feature extraction model contain both global and local information, which can improve accuracy while ensuring recall.

[0101] Based on the same concept, the embodiments of the present disclosure also provide a device for generating an image feature extraction model, and a device for extracting image features.

[0102] It is understandable that the device for generating an image feature extraction model and the device for extracting image features provided by the embodiments of the present disclosure include hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. In combination with the units and algorithm steps of the various examples disclosed in the embodiments of the present disclosure, the embodiments of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware-driven or computer software-driven hardware manner depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiments of the present disclosure.

[0103] Figure 10 FIG1 is a block diagram of a device for generating an image feature extraction model according to an exemplary embodiment. Figure 10 The device 1000 includes a sample acquisition unit 1001, a first extraction unit 1002, a second extraction unit 1003, a fusion unit 1004 and a training unit 1005.

[0104] A sample acquisition unit 1001 is used to acquire a sample image set; a first extraction unit 1002 is used to perform initial image feature extraction on sample images in the sample image set to obtain initial image features; a second extraction unit 1003 is used to extract global features and local features from the initial image features; a fusion unit 1004 is used to determine a non-projected component of a first feature on a second feature, where the first feature is a global feature and the second feature is a local feature, or the first feature is a local feature and the second feature is a global feature; and is also used to fuse the non-projected component and the second feature to obtain target image features of the sample image; a training unit 1005 is used to train target image features corresponding to sample images in the sample image set to generate an image feature extraction model.

[0105] In one embodiment, the second extraction unit 1003 is further used to use multiple hole convolutions of different scales to extract local features corresponding to different scales from the initial image features; perform pooling processing on the local features of different scales; and convolute the multiple local features of different scales obtained by the pooling processing to obtain local features of the initial image features.

[0106] In one embodiment, the second extraction unit 1003 is further configured to perform convolution and down-sampling on the initial image features to extract global features from the initial image features.

[0107] In one embodiment, the first extraction unit 1002 is also used to determine a pre-trained model and extract an intermediate layer neural network from the pre-trained model as an initial neural network for extracting initial image features; the sample image in the sample image set is input into the initial neural network to obtain the initial image features.

[0108] In one embodiment, the fusion unit 1004 is configured to determine a projection component of the first feature on the second feature based on a ratio of the first feature to the second feature; and determine a non-projection component of the first feature on the second feature based on a difference between the first feature and the projection component.

[0109] In one embodiment, the fusion unit 1004 is further configured to concatenate the non-projected component and the second feature; and perform pooling processing on the concatenated features to obtain target image features of the sample image.

[0110] In one embodiment, the training unit 1005 is further used to: set multiple cluster centers for each labeled category corresponding to the sample image in the sample image set; based on all the cluster centers, perform classification training on the target image features corresponding to the sample image to determine the predicted category to which the target image features belong; until the convergence condition is met between the labeled category corresponding to the sample image and the predicted category, thereby generating an image feature extraction model.

[0111] Figure 11 FIG. 1 is a block diagram of a device for extracting image features according to an exemplary embodiment. Figure 11 , the device 1100 includes an image acquisition unit 1101 and an extraction unit 1102.

[0112] The image acquisition unit 1101 is used to acquire the image to be subjected to image feature extraction; the extraction unit 1102 is used to input the image into the image feature extraction model to obtain the image features output by the image feature extraction model.

[0113] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0114] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0115] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0116] Figure 12 A schematic block diagram of an example electronic device 1200 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0117] like Figure 12As shown, the device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a random access memory (RAM) 1203. Various programs and data required for the operation of the device 1200 can also be stored in the RAM 1203. The computing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0118] Various components in device 1200 are connected to I / O interface 1205, including an input unit 1206, such as a keyboard and mouse; an output unit 1207, such as various types of displays and speakers; a storage unit 1208, such as a magnetic disk and optical disk; and a communication unit 1209, such as a network card, a modem, a wireless communication transceiver, etc. Communication unit 1209 allows device 1200 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0119] The computing unit 1201 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1201 performs the various methods and processes described above, such as the method for generating an image feature extraction model or the method for extracting image features. For example, in some embodiments, the method for generating an image feature extraction model or the method for extracting image features can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1208. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 1200 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded into the RAM 1203 and executed by the computing unit 1201, one or more steps of the method for generating an image feature extraction model or the method for extracting image features described above can be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured in any other appropriate manner (eg, by means of firmware) to execute the method for generating an image feature extraction model or the method for extracting image features.

[0120] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0121] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0122] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0123] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0124] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0125] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0126] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0127] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for generating an image feature extraction model, comprising: Get a sample image set; performing initial image feature extraction on the sample images in the sample image set to obtain initial image features; Extracting global features and local features from the initial image features; determining a non-projected component of a first feature on a second feature, where the first feature is the global feature and the second feature is the local feature, or the first feature is the local feature and the second feature is the global feature; fusing the non-projected component and the second feature to obtain a target image feature of the sample image; Training target image features corresponding to sample images in the sample image set to generate an image feature extraction model; The step of training target image features corresponding to sample images in the sample image set to generate an image feature extraction model includes: Setting a plurality of cluster centers for each labeled category corresponding to the sample images in the sample image set; Based on all cluster centers, classification training is performed on the target image features corresponding to the sample images to determine the prediction category to which the target image features belong; Until the convergence condition is met between the labeled category and the predicted category corresponding to the sample image, an image feature extraction model is generated.

2. The method according to claim 1, wherein The extracting local features from the initial image features comprises: Utilizing multiple dilated convolutions of different scales, respectively extracting local features corresponding to the different scales from the initial image features; Performing pooling processing on the local features of different scales; Convolution is performed on the local features of multiple different scales obtained by pooling to obtain local features of the initial image features.

3. The method according to claim 1, wherein The extracting global features from the initial image features comprises: Convolution and down-sampling are performed on the initial image features to extract global features from the initial image features.

4. The method according to claim 1, wherein The extracting initial image features from the sample images in the sample image set to obtain initial image features includes: Determining a pre-trained model, and extracting an intermediate layer neural network from the pre-trained model as an initial neural network for extracting initial image features; The sample images in the sample image set are input into the initial neural network to obtain initial image features.

5. The method according to any one of claims 1 to 4, wherein: The determining of the non-projected component of the first feature on the second feature includes: determining, based on a ratio of the first feature to the second feature, a projection component of the first feature on the second feature; A non-projection component of the first feature on the second feature is determined based on a difference between the first feature and the projection component.

6. The method according to any one of claims 1 to 4, wherein: The fusing the non-projected component and the second feature to obtain the target image feature of the sample image includes: concatenating the non-projected component and the second feature; Pooling is performed on the spliced features to obtain target image features of the sample image.

7. A method for extracting image features, comprising: Obtaining an image to be subjected to image feature extraction; The image is input into an image feature extraction model generated by the method according to any one of claims 1 to 6 above, to obtain image features output by the image feature extraction model.

8. A device for generating an image feature extraction model, comprising: A sample acquisition unit is used to acquire a sample image set; a first extraction unit, configured to extract initial image features from the sample images in the sample image set to obtain initial image features; A second extraction unit, configured to extract global features and local features from the initial image features; a fusion unit, configured to determine a non-projected component of a first feature on a second feature, where the first feature is the global feature and the second feature is the local feature, or the first feature is the local feature and the second feature is the global feature; and further configured to fuse the non-projected component with the second feature to obtain a target image feature of the sample image; A training unit, configured to train target image features corresponding to sample images in the sample image set to generate an image feature extraction model; Wherein, the training unit is further used for: Setting a plurality of cluster centers for each labeled category corresponding to the sample images in the sample image set; Based on all cluster centers, classification training is performed on the target image features corresponding to the sample images to determine the prediction category to which the target image features belong; Until the convergence condition is met between the labeled category and the predicted category corresponding to the sample image, an image feature extraction model is generated.

9. The device according to claim 8, wherein The second extraction unit is further used for: Utilizing multiple dilated convolutions of different scales, respectively extracting local features corresponding to the different scales from the initial image features; Performing pooling processing on the local features of different scales; Convolution is performed on the local features of multiple different scales obtained by pooling to obtain local features of the initial image features.

10. The device according to claim 8, wherein The second extraction unit is further used for: Convolution and down-sampling are performed on the initial image features to extract global features from the initial image features.

11. The device according to claim 8, wherein The first extraction unit is further configured to: Determining a pre-trained model, and extracting an intermediate layer neural network from the pre-trained model as an initial neural network for extracting initial image features; The sample images in the sample image set are input into the initial neural network to obtain initial image features.

12. The device according to any one of claims 8 to 11, wherein: The fusion unit is used for: determining, based on a ratio of the first feature to the second feature, a projection component of the first feature on the second feature; A non-projection component of the first feature on the second feature is determined based on a difference between the first feature and the projection component.

13. The device according to any one of claims 8 to 11, wherein: The fusion unit is further used for: concatenating the non-projected component and the second feature; Pooling is performed on the spliced features to obtain target image features of the sample image.

14. A device for extracting image features, comprising: An image acquisition unit, used for acquiring an image to be subjected to image feature extraction; An extraction unit is used to input the image into an image feature extraction model generated by the method according to any one of claims 1 to 6 above, to obtain image features output by the image feature extraction model.

15. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6, or the method according to claim 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 6, or to execute the method according to claim 7.

17. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6, or The method according to claim 7 is performed.

Images