Image comparison method, device, electronic device and storage medium
Through feature extraction and filtering driven by image characteristic prompt message, the problem of model diversity and low efficiency caused by various problem types in image processing is solved, and efficient image comparison is achieved.
Patent Information
- Application Number
- CN202411056302.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-08-02
AI Technical Summary
In image processing, there are multiple types of problems in the same application scenario, resulting in the need to train multiple models, which increases the problem of model training complexity and low processing efficiency.
By obtaining image characteristic prompt messages, extracting multiple types of features, combining feature prompt messages for feature filtering and weighting fusion, determining image data comparison results, avoiding the need to train multiple models and selecting between multiple models for different types of images The matching process.
The complexity of image processing is reduced and the processing efficiency is improved, and the appropriate features are automatically screened for comparison is realized, which improves the accuracy and efficiency of image processing.
Smart Images

Figure CN118587456B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an image comparison method, device, electronic device and storage medium. Background Art
[0002] With the rapid development of deep learning, methods for analyzing and processing images, such as target recognition and image retrieval, have become common application technologies. In the process of image processing, it is often necessary to compare the features of the image to be processed with the processed image. However, due to the diverse characteristics of various images, even in the same application scenario, it may be necessary to train multiple models for different images to extract features in order to better achieve feature comparison. For example, in the target classification scenario, when the sample images to be classified include natural images, cartoon images, animation images, and hand-drawn sketches with large style differences, it is necessary to train multiple models for inference for images of different styles; in the pedestrian re-identification scenario, when the sample images to be identified have multiple different problems such as incomplete human bodies, different imaging signals (such as infrared signals and visible light), different postures, changes in lighting, and differences in clarity, it is necessary to train multiple models for retrieval for each problem.
[0003] However, when there are too many types of problems in the same application scenario, so that a large number of models are required, not only is the model training complexity too high, but the selection and matching between the processing model and the image to be processed also requires manual judgment, resulting in low image processing efficiency. Summary of the invention
[0004] The purpose of the embodiments of the present application is to provide an image comparison method, device, electronic device and storage medium to improve the efficiency of image processing. The specific technical solution is as follows:
[0005] In a first aspect, an embodiment of the present application provides an image comparison method, comprising:
[0006] Acquire a first image and a second image;
[0007] Acquire a characteristic prompt message, wherein the characteristic prompt message indicates a target feature type for comparison in the first image and the second image;
[0008] Extracting multiple types of features from the first image to obtain a first feature set, and extracting multiple types of features from the second image to obtain a second feature set;
[0009] An image data comparison result is determined according to the first feature set, the second feature set, and the characteristic prompt message.
[0010] In one embodiment of the present application, extracting multiple types of features from the first image to obtain a first feature set, and extracting multiple types of features from the second image to obtain a second feature set, includes:
[0011] Input the first image into a first feature extraction model, perform feature extraction on the first image through multiple feature extraction layers of the first feature extraction model, and obtain multiple first image features of different granularities; perform feature extraction on each of the first image features using multiple granularity feature encoders of the first feature extraction model to obtain a first feature set; wherein one granularity feature encoder is connected to one feature extraction layer, and the granularity feature encoder is used to perform feature extraction on first image features output by the feature extraction layer connected to the encoder;
[0012] The second image is input into the first feature extraction model to obtain a second feature set.
[0013] In one embodiment of the present application, extracting multiple types of features from the first image to obtain a first feature set, and extracting multiple types of features from the second image to obtain a second feature set, includes:
[0014] Input the first image into a second feature extraction model, perform feature extraction on the first image through multiple feature extraction layers of the second feature extraction model, and obtain multiple first image features of different types; perform feature extraction on each of the first image features using multiple type feature encoders of the second feature extraction model to obtain a first feature set; wherein one type feature encoder is connected to one feature extraction layer, one feature extraction layer is connected to one or more type feature encoders, and the type feature encoder is used to extract features of the first image features output by the feature extraction layer connected to itself;
[0015] The second image is input into the second feature extraction model to obtain a second feature set.
[0016] In one embodiment of the present application, extracting multiple types of features from the first image to obtain a first feature set, and extracting multiple types of features from the second image to obtain a second feature set, includes:
[0017] Input the first image into a third feature extraction model, perform feature extraction on the first image through a feature extraction layer in a subnetwork of the third feature extraction model, and obtain a plurality of first image features of different types; perform feature extraction on each of the first image features using a plurality of type feature encoders in the subnetwork, and obtain a first feature set; wherein each subnetwork includes a connected feature extractor and a type feature encoder, and the type feature encoder is used to perform feature extraction on the first image features output by the feature extraction layer connected to itself;
[0018] The second image is input into the third feature extraction model to obtain a third feature set.
[0019] In one embodiment of the present application, extracting multiple types of features from the first image to obtain a first feature set, and extracting multiple types of features from the second image to obtain a second feature set, includes:
[0020] Inputting the first image and the second image into the visual big model respectively, extracting multiple types of features from the first image and the second image, and obtaining the first feature set and the second feature set;
[0021] The visual large model is trained in the following way:
[0022] Selecting a sample image having features of a first feature type and an image characteristic expert model for extracting features of the first feature type;
[0023] The currently selected image feature expert model is used as the teacher model, the visual big model is used as the student model, and the currently selected sample image is used as input to train the visual big model;
[0024] Select sample images with other feature types and use the image characteristic expert model for extracting the other feature types to continue training the large visual model until all preset feature types are trained.
[0025] In one embodiment of the present application, determining the image data comparison result according to the first feature set, the second feature set, and the characteristic prompt message includes:
[0026] Encoding the characteristic prompt message to obtain a first encoding feature;
[0027] splicing the first coding feature and the first feature set to obtain a first splicing feature, and splicing the first coding feature and the second feature set to obtain a second splicing feature;
[0028] Performing feature screening on the first splicing feature through the feature screening model to obtain a first screening feature; performing feature screening on the second splicing feature through the feature screening model to obtain a second screening feature;
[0029] The similarity between the first screening feature and the second screening feature is calculated to obtain the image data comparison result.
[0030] In one embodiment of the present application, determining the image data comparison result according to the first feature set, the second feature set, and the characteristic prompt message includes:
[0031] Based on the characteristic prompt message, calculating a first attention weight of each type of feature in the first feature set, and calculating a second attention weight of each type of feature in the second feature set;
[0032] Performing weighted fusion on each type of features in the first feature set according to the first attention weight to obtain a first target feature set;
[0033] Performing weighted fusion on each type of features in the second feature set according to the second attention weight to obtain a second target feature set;
[0034] The similarity between the first target feature set and the second target feature set is calculated to obtain an image data comparison result.
[0035] In a second aspect, an embodiment of the present application provides an image comparison method, comprising:
[0036] Acquire a first image and a second image;
[0037] Acquire a characteristic prompt message, wherein the characteristic prompt message indicates a target feature type for comparison in the first image and the second image;
[0038] Based on the characteristic prompt message, extracting target feature types from the first image to obtain a third feature set, and extracting target feature types from the second image to obtain a fourth feature set;
[0039] An image data comparison result is determined according to the third feature set and the fourth feature set.
[0040] In one embodiment of the present application, based on the characteristic prompt message, extracting the target feature type from the first image to obtain a third feature set, and extracting the target feature type from the second image to obtain a fourth feature set include:
[0041] Determining, according to the characteristic prompt message, each target type feature encoder in the second feature extraction model for extracting features of the target feature type;
[0042] Determine the target feature extraction layers to which each target type feature encoder is connected, and determine the number of layers N of the deepest target feature extraction layer;
[0043] Using the first N feature extraction layers of the second feature extraction model to extract features from the first image, to obtain third image features;
[0044] The target type feature encoder is used to extract features of the third image features to obtain a third feature set.
[0045] In one embodiment of the present application, based on the characteristic prompt message, extracting the target feature type from the first image to obtain a third feature set, and extracting the target feature type from the second image to obtain a fourth feature set include:
[0046] Determining, according to the characteristic prompt message, each target type feature encoder in a third feature extraction model for extracting features of the target feature type;
[0047] Determine the target feature extraction layers to which each target type feature encoder is connected, and determine the Nth subnetwork corresponding to the deepest target feature extraction layer;
[0048] Using the feature extraction layers in the first N sub-networks of the third feature extraction model to extract features from the first image, to obtain third image features;
[0049] The target type feature encoder is used to extract features of the third image features to obtain a third feature set.
[0050] In one embodiment of the present application, the second feature extraction model includes a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, and a fourth feature extraction layer; the type feature encoder includes a texture feature encoder, a contour feature encoder, a pattern feature encoder, an accessory feature encoder, a component feature encoder, a color feature encoder, and a global feature encoder; the first feature extraction layer is connected to the texture feature encoder and the contour feature encoder, the second feature extraction layer is connected to the pattern feature encoder and the accessory feature encoder, the third feature extraction layer is connected to the component feature encoder, and the fourth feature extraction layer is connected to the color feature encoder and the global feature encoder.
[0051] In one embodiment of the present application, the third feature extraction model includes a first subnetwork for extracting texture features, a second subnetwork for extracting contour features, a third subnetwork for extracting pattern features, a fourth subnetwork for extracting accessory features, a fifth subnetwork for extracting component features, a sixth subnetwork for extracting color features, and a seventh subnetwork for extracting global features.
[0052] In one embodiment of the present application, the obtaining of a characteristic prompt message includes:
[0053] Inputting the first image and the second image into an image characteristic classification model respectively, and extracting image type information of the first image and the second image;
[0054] Based on the preset correspondence between the image type information and the feature type, the common feature type of the first image and the second image is determined to obtain the feature prompt message.
[0055] In one embodiment of the present application, the image data comparison result is a target re-identification result or an image classification result.
[0056] In a third aspect, an embodiment of the present application provides an image comparison device, comprising:
[0057] An image acquisition module, used for acquiring a first image and a second image;
[0058] A message acquisition module, used to acquire a characteristic prompt message, wherein the characteristic prompt message indicates a target feature type for comparison in the first image and the second image;
[0059] a feature extraction module, configured to extract multiple types of features from the first image to obtain a first feature set, and to extract multiple types of features from the second image to obtain a second feature set;
[0060] An image comparison module is used to determine an image data comparison result based on the first feature set, the second feature set, and the characteristic prompt message.
[0061] In one embodiment of the present application, the feature extraction module is specifically used to:
[0062] Input the first image into a first feature extraction model, perform feature extraction on the first image through multiple feature extraction layers of the first feature extraction model, and obtain multiple first image features of different granularities; perform feature extraction on each of the first image features using multiple granularity feature encoders of the first feature extraction model to obtain a first feature set; wherein one granularity feature encoder is connected to one feature extraction layer, and the granularity feature encoder is used to perform feature extraction on first image features output by the feature extraction layer connected to the encoder;
[0063] The second image is input into the first feature extraction model to obtain a second feature set.
[0064] In one embodiment of the present application, the feature extraction module is specifically used to:
[0065] Input the first image into a second feature extraction model, perform feature extraction on the first image through multiple feature extraction layers of the second feature extraction model, and obtain multiple first image features of different types; perform feature extraction on each of the first image features using multiple type feature encoders of the second feature extraction model to obtain a first feature set; wherein one type feature encoder is connected to one feature extraction layer, one feature extraction layer is connected to one or more type feature encoders, and the type feature encoder is used to extract features of the first image features output by the feature extraction layer connected to itself;
[0066] The second image is input into the second feature extraction model to obtain a second feature set.
[0067] In one embodiment of the present application, the feature extraction module is specifically used to:
[0068] Input the first image into a third feature extraction model, perform feature extraction on the first image through a feature extraction layer in a subnetwork of the third feature extraction model, and obtain a plurality of first image features of different types; perform feature extraction on each of the first image features using a plurality of type feature encoders in the subnetwork, and obtain a first feature set; wherein each subnetwork includes a connected feature extractor and a type feature encoder, and the type feature encoder is used to perform feature extraction on the first image features output by the feature extraction layer connected to itself;
[0069] The second image is input into the third feature extraction model to obtain a third feature set.
[0070] In one embodiment of the present application, the feature extraction module is specifically used to:
[0071] Inputting the first image and the second image into the visual big model respectively, extracting multiple types of features from the first image and the second image, and obtaining the first feature set and the second feature set;
[0072] The visual large model is trained in the following way:
[0073] Selecting a sample image having features of a first feature type and an image characteristic expert model for extracting features of the first feature type;
[0074] The currently selected image feature expert model is used as the teacher model, the visual big model is used as the student model, and the currently selected sample image is used as input to train the visual big model;
[0075] Select sample images with other feature types and use the image characteristic expert model for extracting the other feature types to continue training the large visual model until all preset feature types are trained.
[0076] In one embodiment of the present application, the image comparison module is specifically used for:
[0077] Encoding the characteristic prompt message to obtain a first encoding feature;
[0078] splicing the first coding feature and the first feature set to obtain a first splicing feature, and splicing the first coding feature and the second feature set to obtain a second splicing feature;
[0079] Performing feature screening on the first splicing feature through the feature screening model to obtain a first screening feature; performing feature screening on the second splicing feature through the feature screening model to obtain a second screening feature;
[0080] The similarity between the first screening feature and the second screening feature is calculated to obtain the image data comparison result.
[0081] In one embodiment of the present application, the image comparison module is specifically used for:
[0082] Based on the characteristic prompt message, calculating a first attention weight of each type of feature in the first feature set, and calculating a second attention weight of each type of feature in the second feature set;
[0083] Performing weighted fusion on each type of features in the first feature set according to the first attention weight to obtain a first target feature set;
[0084] Performing weighted fusion on each type of features in the second feature set according to the second attention weight to obtain a second target feature set;
[0085] The similarity between the first target feature set and the second target feature set is calculated to obtain an image data comparison result.
[0086] In a fourth aspect, an embodiment of the present application provides an image comparison device, comprising:
[0087] An image acquisition module, used for acquiring a first image and a second image;
[0088] A message acquisition module, used to acquire a characteristic prompt message, wherein the characteristic prompt message indicates a target feature type for comparison in the first image and the second image;
[0089] a feature extraction module, configured to extract a target feature type from the first image to obtain a third feature set, and to extract a target feature type from the second image to obtain a fourth feature set based on the characteristic prompt message;
[0090] The image comparison module is used to determine the image data comparison result according to the third feature set and the fourth feature set.
[0091] In one embodiment of the present application, the feature extraction module is specifically used to:
[0092] Determining, according to the characteristic prompt message, each target type feature encoder in the second feature extraction model for extracting features of the target feature type;
[0093] Determine the target feature extraction layers to which each target type feature encoder is connected, and determine the number of layers N of the deepest target feature extraction layer;
[0094] Using the first N feature extraction layers of the second feature extraction model to extract features from the first image, to obtain third image features;
[0095] The target type feature encoder is used to extract features of the third image features to obtain a third feature set.
[0096] In one embodiment of the present application, the feature extraction module is specifically used to:
[0097] Determining, according to the characteristic prompt message, each target type feature encoder in a third feature extraction model for extracting features of the target feature type;
[0098] Determine the target feature extraction layers to which each target type feature encoder is connected, and determine the Nth subnetwork corresponding to the deepest target feature extraction layer;
[0099] Using the feature extraction layers in the first N sub-networks of the third feature extraction model to extract features from the first image, to obtain third image features;
[0100] The target type feature encoder is used to extract features of the third image features to obtain a third feature set.
[0101] In one embodiment of the present application, the second feature extraction model includes a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, and a fourth feature extraction layer; the type feature encoder includes a texture feature encoder, a contour feature encoder, a pattern feature encoder, an accessory feature encoder, a component feature encoder, a color feature encoder, and a global feature encoder; the first feature extraction layer is connected to the texture feature encoder and the contour feature encoder, the second feature extraction layer is connected to the pattern feature encoder and the accessory feature encoder, the third feature extraction layer is connected to the component feature encoder, and the fourth feature extraction layer is connected to the color feature encoder and the global feature encoder.
[0102] In one embodiment of the present application, the third feature extraction model includes a first subnetwork for extracting texture features, a second subnetwork for extracting contour features, a third subnetwork for extracting pattern features, a fourth subnetwork for extracting accessory features, a fifth subnetwork for extracting component features, a sixth subnetwork for extracting color features, and a seventh subnetwork for extracting global features.
[0103] In one embodiment of the present application, the message acquisition module is specifically used to:
[0104] Inputting the first image and the second image into an image characteristic classification model respectively, and extracting image type information of the first image and the second image;
[0105] Based on the preset correspondence between the image type information and the feature type, the common feature type of the first image and the second image is determined to obtain the feature prompt message.
[0106] In one embodiment of the present application, the image data comparison result is a target re-identification result or an image classification result.
[0107] The present application also provides an electronic device, including:
[0108] Memory, used to store computer programs;
[0109] The processor is used to implement any of the above-mentioned image comparison methods when executing the program stored in the memory.
[0110] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned image comparison methods is implemented.
[0111] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned image comparison methods.
[0112] Beneficial effects of the embodiments of the present application:
[0113] The image comparison method provided in the embodiment of the present application obtains a first image, a second image, and a characteristic prompt message indicating the target feature type for comparison in the first image and the second image, extracts multiple types of features from the first image to obtain a first feature set, extracts multiple types of features from the second image to obtain a second feature set, and determines the image data comparison result based on the first feature set, the second feature set, and the characteristic prompt message. In the image data comparison process, appropriate features are screened for comparison based on the characteristic prompt message, avoiding the need to train multiple models for different types of images and selecting matches between multiple models and the image to be processed for processing, reducing the complexity of image processing and improving the efficiency of image processing.
[0114] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS
[0115] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.
[0116] Figure 1-1 A schematic diagram of a flow chart of a first image comparison method provided in an embodiment of the present application;
[0117] Figure 1-2 This is a flowchart of the first image comparison method provided in the embodiment of the present application;
[0118] Figure 2-1 A possible implementation of step S102 provided in the embodiment of the present application;
[0119] Figure 2-2 This is a flowchart of a second image comparison method provided in an embodiment of the present application;
[0120] Figure 3-1The first possible implementation of step S103 provided in the embodiment of the present application;
[0121] Figure 3-2 A flowchart of a third image comparison method provided in an embodiment of the present application;
[0122] Figure 4-1 A second possible implementation of step S103 provided in the embodiment of the present application;
[0123] Figure 4-2 This is a flowchart of a fourth image comparison method provided in an embodiment of the present application;
[0124] Figure 5-1 A third possible implementation of step S103 provided in the embodiment of the present application;
[0125] Figure 5-2 A flowchart of a fifth image comparison method provided in an embodiment of the present application;
[0126] Figure 6 This is a flowchart of a sixth image comparison method provided in an embodiment of the present application;
[0127] Figure 7-1 The first possible implementation of step S104 provided in the embodiment of the present application;
[0128] Figure 7-2 A flowchart of a seventh image comparison method provided in an embodiment of the present application;
[0129] Figure 8-1 A second possible implementation of step S104 provided in the embodiment of the present application;
[0130] Figure 8-2 An example flowchart of the eighth image comparison method provided in the embodiment of the present application;
[0131] Fig. 9 A schematic diagram of a flow chart of a second image comparison method provided in an embodiment of the present application;
[0132] Figure 10-1 The first possible implementation of step S903 provided in the embodiment of the present application;
[0133] Figure 10-2 This is a flowchart of a ninth image comparison method provided in an embodiment of the present application;
[0134] Fig.11 A second possible implementation of step S903 provided in an embodiment of the present application;
[0135] Figure 12-1A flowchart of the tenth image comparison method provided in an embodiment of the present application;
[0136] Figure 12-2 This is a flowchart of an eleventh image comparison method provided in an embodiment of the present application;
[0137] Figure 12-3 This is a flowchart of a twelfth image comparison method provided in an embodiment of the present application;
[0138] Fig.13 A schematic diagram of the structure of a first image comparison device provided in an embodiment of the present application;
[0139] Fig.14 A schematic diagram of the structure of a second image comparison device provided in an embodiment of the present application;
[0140] Fig.15 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0141] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.
[0142] Because there are too many types of problems in the same application scenario in the related art, a large number of models are required, which not only makes the model training too complex, but also requires manual judgment in the selection and matching between the processing model and the image to be processed, resulting in large errors in the image processing results, thereby making the image processing accuracy low. In order to solve at least one of the above problems, an embodiment of the present application provides an image comparison method, which can be performed by an electronic device, specifically, the electronic device can be a personal terminal, a server, etc.
[0143] The following is a detailed description of the image comparison method provided in the embodiment of the present application. Figure 1-1 , Figure 1-1 A flowchart of a first image comparison method is provided for an embodiment of the present application, including:
[0144] Step S101, acquiring a first image and a second image.
[0145] The first image and the second image are image data to be compared. Specifically, the first image and the second image can be labeled or unlabeled images respectively. For example, in the application scenario of target re-identification, the first image is the query image of the target to be identified, and the second image is an image in the database of labeled targets. The first image is compared to see whether there is a target that is consistent with the labeled target in the second image; in the application scenario of image classification, the first image is an unlabeled image to be classified, and the second image is a registered image containing a known classification label. The first image is compared to see whether the categories of the second image are consistent; in the application scenario of image similarity matching, the first image and the second image can both be unlabeled images, and whether the two match is determined by comparing the similarity between the two.
[0146] In one example, when the second image is a labeled image in order to provide a sample control for the unlabeled first image, the second image can be a plurality of images in a pre-processed image database, which are sequentially compared with the first image until the comparison result is determined.
[0147] Step S102, obtaining a characteristic prompt message, wherein the characteristic prompt message indicates a target feature type used for comparison in the first image and the second image.
[0148] There are many types of features in an image. For example, in the application scenario of pedestrian re-identification, the image has the example of feature types shown in Table 1, and in the application scenario of image classification, the image has the example of feature types shown in Table 2. Different types of features have different degrees of difficulty in extracting in different images. Therefore, extracting different types of features as the basis for processing different images can improve the efficiency and accuracy of image processing. For example, it is difficult to extract color features in infrared images, but easy to extract texture features. Therefore, image processing should be performed on the texture features extracted from infrared images; when there is an incomplete target in the image, image processing should be performed on the target component features extracted from the image.
[0149] Table 1
[0150]
[0151] Table 2
[0152]
[0153] When the first image and the second image are different types of images, the type features that are relatively clear, easy to extract, and common to both the first image and the second image should be extracted as the basis for image comparison. For example, in the application scenario of pedestrian re-identification, when the first image is an infrared image and the second image is a visible light image, the appropriate features should be features that are not related to color and co-occur in different types of image spaces, such as contours and textures; when the first image and the second image are both complete visible light images of the human body, the appropriate features should be complete features (all types of features); when the first image is an incomplete human body and the second image is a complete visible light image of the human body, the appropriate features should be features of human body parts, pattern features, etc.; when the first image and the second image are visible light images of the complete human body facing forward and backward, respectively, the appropriate features should be global features, human contours, clothing colors, and other features that co-occur in different postures / viewing angles.
[0154] A characteristic prompt message is obtained for the first image and the second image, and the characteristic prompt message can indicate a common feature type in the first image and the second image, that is, a target feature type for comparison between the first image and the second image.
[0155] In an example, obtaining a feature prompt message can be achieved through a pre-configured algorithm model, or it can be manually selected and input based on the characteristics of the image.
[0156] Step S103: extract multiple types of features from the first image to obtain a first feature set, and extract multiple types of features from the second image to obtain a second feature set.
[0157] Step S104, determining the image data comparison result according to the first feature set, the second feature set, and the characteristic prompt message.
[0158] Multiple types of features may be extracted from the first image. All features in the first image, i.e., complete features, may be extracted. Only features of the target feature type may be extracted according to feature prompt messages. After the complete features of the first image are extracted, unclear and inappropriate features may be manually screened out to finally obtain a first feature set.
[0159] The second image is subjected to multiple types of feature extraction, which may be to extract all features in the second image, i.e., complete features, or to extract only features of the target feature type according to the feature prompt message, or to manually screen out unclear and inappropriate features after the complete features of the second image are extracted, and finally obtain the second feature set. When the second image is a labeled image in order to provide a sample control for the unlabeled first image, the second image may be a plurality of images in a pre-processed image database, and the image database stores the complete features of the second image, then there is no need to consider the feature prompt message, and the complete features can be used as the second feature set.
[0160] The first feature set and the second feature set are compared to determine the image data comparison result. Exemplarily, the feature set comparison can be achieved by a feature comparison model, a feature comparison algorithm, and the like.
[0161] In one embodiment of the present application, in the application scenario of target re-identification / pedestrian re-identification, the image data comparison result is the target re-identification result; in the application scenario of image classification, the image data comparison result is the image classification result; in the application scenario of image similarity matching, the image data comparison result is the image matching result. Figure 1-2 The example shown.
[0162] As can be seen from the above, the image comparison method provided in the embodiment of the present application obtains a first image, a second image, and a characteristic prompt message indicating the target feature type used for comparison in the first image and the second image, extracts multiple types of features from the first image to obtain a first feature set, extracts multiple types of features from the second image to obtain a second feature set, and determines the image data comparison result based on the first feature set, the second feature set, and the characteristic prompt message. In the image data comparison process, appropriate features are screened for comparison based on the characteristic prompt message, avoiding the need to train multiple models for different types of images and selecting matches between multiple models and the image to be processed for processing, reducing the complexity of image processing and improving the efficiency of image processing.
[0163] In one embodiment of the present application, Figure 2-1 As shown, the above step S102 obtains the characteristic prompt message, including:
[0164] Step S201, inputting the first image and the second image into an image characteristic classification model respectively, and extracting image type information of the first image and the second image;
[0165] Step S202: based on the preset correspondence between the image type information and the feature type, determine the common feature type of the first image and the second image, and obtain the feature prompt message.
[0166] The image characteristic classification model is used to identify the type of image and obtain image type information. Image type information can indicate the characteristics of the image, such as whether the image is an infrared / visible light image, whether there are significant patterns, attachments, etc.
[0167] Exemplarily, the image feature classification model can be trained using multiple labeled sample images, the labeled sample images are input into the image feature classification model, the image type information of the sample images is output by the image feature classification model, the obtained image type information is compared with the labeled image type information, the image feature classification model loss is calculated, and the image feature classification model is adjusted using the loss until the image feature classification model converges and the training is completed.
[0168] The first image and the second image are respectively input into the image characteristic classification model, the image type information of the first image and the second image is extracted, and based on the correspondence between the preset image type information and the feature type, the common feature type of the first image and the second image is determined, that is, the target feature type suitable for extraction for both the first image and the second image is determined, and a feature prompt message is obtained. Specifically, the correspondence between the image type information and the feature type is preset, and the feature type suitable for extraction of each type of image is determined for multiple image types, and a correspondence is established for them. For example, a correspondence can be established between the infrared image type and the texture feature and the contour feature; a correspondence can be established between the image type with incomplete targets and the component feature; and a correspondence can be established between the image type with visible light and a significant pattern and the significant pattern feature.
[0169] Exemplarily, the image type information includes at least one of subject target integrity, subject target posture, subject target color, image viewing angle, image light type, image clarity, lighting information, image resolution, image style, significant patterns in the image, and accessory category.
[0170] For example, the image comparison process is as follows: Figure 2-2 shown.
[0171] As can be seen from the above, the image comparison method provided in the embodiment of the present application determines the common feature type of the first image and the second image through the image feature classification model, obtains a feature prompt message, and realizes the analysis of the image characteristics. Based on this, feature guidance is provided for the comparison between the first image and the second image, and only the features of the target feature type need to be selected for comparison among multiple types of features, thereby avoiding the errors easily caused by inappropriate type feature comparisons and the manual intervention required when selecting and applying between multiple models, thereby improving the accuracy of image processing, and, in the case of having multiple models, being able to solve the processing problems of multiple types of images at the same time, further improving the efficiency of image processing.
[0172] In one embodiment of the present application, Figure 3-1 As shown, the above step S103 extracts multiple types of features from the first image to obtain a first feature set, and extracts multiple types of features from the second image to obtain a second feature set, including:
[0173] Step S301: input the first image into a first feature extraction model, perform feature extraction on the first image through multiple feature extraction layers of the first feature extraction model, and obtain multiple first image features of different granularities; use multiple granularity feature encoders of the first feature extraction model to perform feature extraction on each of the first image features to obtain a first feature set.
[0174] Wherein, a granularity feature encoder is connected to a feature extraction layer, and the granularity feature encoder is used to extract features of a first image feature output by the feature extraction layer connected to the granularity feature encoder;
[0175] Step S302: input the second image into the first feature extraction model to obtain a second feature set.
[0176] The first feature extraction model can extract feature maps of different granularities from feature maps of different depths. Different granularities indicate that the image range and clarity it processes are different. Specifically, different types of feature extraction require different extraction ranges and clarity. For example, global features require a larger extraction range but lower clarity requirements; texture features require a lower extraction range but higher clarity requirements.
[0177] The first image is input into the first feature extraction model, and multiple feature extraction layers of the first feature extraction model are used to perform feature extraction on the first image of different depths (process the first image), and first image features of multiple granularities are obtained at an appropriate image depth. The first image features are then extracted through multiple granular feature encoders connected thereto, and multiple types of features of the first image are obtained, thereby obtaining a first feature set, i.e., a complete feature.
[0178] The process of obtaining the second feature set is similar to that of the first image. The second image is input into the first feature extraction model, and multiple feature extraction layers are used to obtain second image features of multiple granularities. Multiple types of features are then extracted through multiple granular feature encoders to obtain the second feature set, i.e., the complete feature.
[0179] Exemplarily, after each granular feature encoder extracts multiple types of features, the obtained features can be simply spliced to obtain complete features, or various types of feature transformation small networks can be used to transform the features, such as indexing, discretization, data smoothing, normalization, digitization, regularization, etc., to obtain complete features according to the actual needs of the features obtained.
[0180] Exemplarily, both the first feature set and the second feature set may be a high-dimensional feature vector.
[0181] In an example, the formal expression of the complete features extracted by the first feature extraction model can be as follows:
[0182] ;
[0183] ;
[0184] ;
[0185] ;
[0186] ;
[0187] in, To complete the features, is the input image (first image, second image), The depth is The feature extraction layer For the input image go through The processed feature map, is the granularity 1 encoder, and the meanings of other symbols are similar. is the number of multi-granularity encoders.
[0188] The formal expression of feature transformation based on each feature is as follows:
[0189] ;
[0190] or
[0191] ;
[0192] or
[0193] ;
[0194] Or other combination transformation forms, which will not be listed here one by one. Among them, and They are all small feature transformation networks.
[0195] Exemplarily, the image comparison process of using the first feature extraction model to process the first image and the second image is as follows: Figure 3-2 As shown, the first feature extraction layer N1 Layers, the second feature extraction layer N2 Layers, and the third feature extraction layer N3 Layers represent three feature extraction layers respectively.
[0196] From the above, it can be seen that the image comparison method provided in the embodiment of the present application, through multiple feature extraction layers and granularity feature encoders in the first feature extraction model, extracts features of different granularities of the first image and the second image to obtain complete features, thereby avoiding omission errors in feature comparison and improving the accuracy of image processing.
[0197] In one embodiment of the present application, Figure 4-1 As shown, the above step S103 extracts multiple types of features from the first image to obtain a first feature set, and extracts multiple types of features from the second image to obtain a second feature set, including:
[0198] Step S401: input the first image into a second feature extraction model, perform feature extraction on the first image through multiple feature extraction layers of the second feature extraction model, and obtain multiple different types of first image features; use multiple types of feature encoders of the second feature extraction model to perform feature extraction on each of the first image features to obtain a first feature set.
[0199] Wherein, one type feature encoder is connected to one feature extraction layer, and one feature extraction layer is connected to one or more type feature encoders, and the type feature encoder is used to extract features of the first image features output by the feature extraction layer connected to itself;
[0200] Step S402: input the second image into the second feature extraction model to obtain a second feature set.
[0201] The multiple feature extraction layers of the second feature extraction model are respectively used to extract one type of feature. The first image is input into the second feature extraction model, and the multiple feature extraction layers of the second feature extraction model are used to extract multiple types of features of the first image, namely, first image features (process the first image). The first image features are then extracted through multiple type feature encoders connected thereto to obtain multiple types of features of the first image, thereby obtaining a first feature set, namely, a complete feature.
[0202] The process of obtaining the second feature set is similar to that of the first image. The second image is input into the second feature extraction model, and multiple feature extraction layers are used to obtain multiple types of second image features. Multiple types of features are then extracted through multiple type feature encoders to obtain the second feature set, i.e., the complete feature.
[0203] Exemplarily, after each type of feature encoder extracts multiple types of features, the obtained features can be simply spliced to obtain complete features, or various types of feature transformation small networks can be used to transform each feature, such as indexing, discretization, data smoothing, normalization, digitization, regularization, etc., which can be determined according to the actual needs of the features obtained to obtain complete features.
[0204] Exemplarily, both the first feature set and the second feature set may be a high-dimensional feature vector.
[0205] In an example, the formal expression of the complete features extracted by the second feature extraction model can be as follows:
[0206] ;
[0207] ;
[0208] ;
[0209] ;
[0210] ;
[0211] in, is the texture feature, is the contour feature, is the pattern feature, For the accessory characteristics, is the input image, The depth is The sub-network of For the input image go through The processed feature map, is the texture feature encoder, is the contour feature encoder, is the pattern feature encoder, It is the accessory feature encoder, and the meanings of other symbols are similar. is the number of multi-type encoders.
[0212] The formal expression of feature transformation based on each feature is as follows:
[0213] ;
[0214] or
[0215] ;
[0216] or
[0217] ;
[0218] Or other combination transformation forms, which will not be listed here one by one. Among them, and They are all small feature transformation networks.
[0219] Exemplarily, the image comparison process of using the second feature extraction model to process the first image and the second image is as follows: Figure 4-2 As shown, the first feature extraction layer N1 Layers, the second feature extraction layer N2 Layers, the third feature extraction layer N3 Layers, and the fourth feature extraction layer N4 Layers represent four feature extraction layers respectively.
[0220] From the above, it can be seen that the image comparison method provided in the embodiment of the present application, through multiple feature extraction layers and type feature encoders in the second feature extraction model, extracts different types of features of the first image and the second image to obtain complete features, thereby avoiding omission errors in feature comparison and improving the accuracy of image processing.
[0221] In one embodiment of the present application, Figure 5-1 As shown, the above step S103 extracts multiple types of features from the first image to obtain a first feature set, and extracts multiple types of features from the second image to obtain a second feature set, including:
[0222] Step S501: input the first image into the third feature extraction model, perform feature extraction on the first image through the feature extraction layer in the subnetwork of the third feature extraction model, and obtain multiple different types of first image features; use the multiple types of feature encoders in the subnetwork to perform feature extraction on each of the first image features to obtain a first feature set.
[0223] Each sub-network includes a connected feature extractor and a type feature encoder, and the type feature encoder is used to extract features of the first image features output by the feature extraction layer connected to itself;
[0224] Step S502: input the second image into the third feature extraction model to obtain a third feature set.
[0225] The multiple sub-networks of the third feature extraction model are respectively used to extract one type of feature. The first image is input into the second feature extraction model, and the feature extraction layers in the multiple sub-networks of the third feature extraction model are used to extract multiple types of features of the first image, namely, first image features (process the first image). Then, the first image features are extracted through multiple types of feature encoders connected to the feature extraction layers in the sub-networks to obtain multiple types of features of the first image, thereby obtaining a first feature set, namely, a complete feature.
[0226] The process of obtaining the second feature set is similar to that of the first image. The second image is input into the third feature extraction model, and the feature extraction layers in multiple sub-networks are used to obtain multiple types of second image features. Then, the type feature encoders in multiple sub-networks extract multiple types of features to obtain the second feature set, i.e., the complete features.
[0227] Exemplarily, after each type of feature encoder extracts multiple types of features, the obtained features can be simply spliced to obtain complete features, or various types of feature transformation small networks can be used to transform each feature, such as indexing, discretization, data smoothing, normalization, digitization, regularization, etc., which can be determined according to the actual needs of the features obtained to obtain complete features.
[0228] Exemplarily, both the first feature set and the second feature set may be a high-dimensional feature vector.
[0229] In an example, the formal expression of the complete features extracted by the third feature extraction model can be as follows:
[0230] ;
[0231] ;
[0232] ;
[0233] ;
[0234] ;
[0235] ;
[0236] ;
[0237] in, is the texture feature, is the contour feature, is the pattern feature, For the accessory characteristics, is the input image, is the texture feature encoder, is the contour feature encoder, is the pattern feature encoder, It is the accessory feature encoder, and the meanings of other symbols are similar. is the number of multi-type encoders.
[0238] The formal expression of feature transformation based on each feature is as follows:
[0239] ;
[0240] or
[0241] ;
[0242] or
[0243] ;
[0244] Or other combination transformation forms, which will not be listed here one by one. Among them, and They are all small feature transformation networks.
[0245] Exemplarily, the image comparison process of using the third feature extraction model to process the first image and the second image is as follows: Figure 5-2 As shown,
[0246] From the above, it can be seen that the image comparison method provided in the embodiment of the present application, through the feature extraction layers and type feature encoders in multiple sub-networks in the third feature extraction model, extracts different types of features of the first image and the second image to obtain complete features, thereby avoiding omission errors in feature comparison and improving the accuracy of image processing.
[0247] In one embodiment of the present application, the above step S103 extracts multiple types of features from the first image to obtain a first feature set, and extracts multiple types of features from the second image to obtain a second feature set, including:
[0248] Inputting the first image and the second image into the visual big model respectively, extracting multiple types of features from the first image and the second image, and obtaining the first feature set and the second feature set;
[0249] The visual large model is trained in the following way:
[0250] A sample image having a first feature type feature and an image characteristic expert model for extracting the first feature type feature are selected. The first feature type is selected according to actual needs.
[0251] The currently selected image feature expert model is used as the teacher model, the visual big model is used as the student model, and the currently selected sample image is used as input to train the visual big model.
[0252] Select sample images with other feature types and use the image characteristic expert model for extracting the other feature types to continue training the large visual model until all preset feature types are trained.
[0253] Exemplarily, the visual big model can be composed of a visual model obtained by cross-modal contrast learning training of images and texts (in image classification application scenarios, such as the EVA series visual model (a visual basic model), the EVA02 series visual model (a visual model)), or it can be obtained by using the ReID (person re-identification) feature decoupling training method (in person re-identification application scenarios, such as HumanBench (a human-centric generalist model), GreyReID (a visual big model), etc.). The sample image training set can be composed of images with various image characteristics. Image feature guidance can be added during the training process, and expert models of each image feature trained separately (such as a non-color feature extraction model trained using grayscale image samples, a feature extraction model with color information trained using RGB image samples, a component feature extraction model trained using target component area samples, a low-resolution feature extraction model trained using low-resolution samples, etc.) are used for guided training, so that the visual big model has the ability to extract complete features.
[0254] The first image and the second image are respectively input into the trained visual model, and various types of features are extracted from the first image and the second image to obtain the first feature set and the second feature set, that is, the complete features.
[0255] Exemplarily, the image comparison process of processing the first image and the second image using the visual macro model is as follows: Figure 6 shown.
[0256] From the above, it can be seen that the image comparison method provided in the embodiment of the present application extracts various types of features of the first image and the second image through a trained large visual model to obtain complete features, thereby avoiding omission errors in feature comparison and improving the accuracy of image processing.
[0257] In one embodiment of the present application, Figure 7-1 As shown, the above step S104 determines the image data comparison result according to the first feature set, the second feature set, and the characteristic prompt message, including:
[0258] Step S701, encoding the characteristic prompt message to obtain a first encoding feature;
[0259] Step S702, concatenating the first coding feature and the first feature set to obtain a first concatenated feature, and concatenating the first coding feature and the second feature set to obtain a second concatenated feature;
[0260] Step S703, feature screening is performed on the first splicing feature through the feature screening model to obtain a first screening feature; feature screening is performed on the second splicing feature through the feature screening model to obtain a second screening feature;
[0261] Step S704, calculating the similarity between the first screening feature and the second screening feature to obtain the image data comparison result.
[0262] The first coding feature represents the target feature type required for comparing the first image with the second image, and is used to provide guidance for feature screening. The first coding feature is spliced with the first feature set and the second feature set, respectively, and the spliced first spliced feature and the second spliced feature are respectively input into the feature screening model. The feature screening model performs feature screening on the first spliced feature and the second spliced feature under the guidance of the first coding feature, and obtains the first screening feature and the second screening feature of the target feature type.
[0263] Exemplarily, the feature screening model may be a graphic and text multimodal visual feature screening model, which is trained by using contrastive learning method through sample feature pairs obtained by pre-screening.
[0264] The similarity between the first screening feature and the second screening feature is calculated to obtain the image data comparison result. Specifically, it can be implemented by using similarity functions commonly used in retrieval algorithms such as Euclidean distance, cosine similarity, and Mahalanobis distance, or by using a similarity calculation model (the similarity calculation model is obtained by training pre-labeled sample feature pairs).
[0265] For example, the image comparison process is as follows: Figure 7-2 shown.
[0266] From the above, it can be seen that the image comparison method provided in the embodiment of the present application obtains the features of the target feature types in the first feature set and the second feature set by respectively splicing the first feature set, the second feature set and the encoded characteristic prompt message, and then calculates the similarity between the two to achieve comparison between the first image and the second image. It can automatically screen suitable features for comparison, reduce the complexity of image comparison, and improve the efficiency of image processing in a low-cost manner.
[0267] In one embodiment of the present application, Figure 8-1As shown, the above step S104 determines the image data comparison result according to the first feature set, the second feature set, and the characteristic prompt message, including:
[0268] Step S801, based on the characteristic prompt message, calculating a first attention weight of each type of feature in the first feature set, and calculating a second attention weight of each type of feature in the second feature set;
[0269] Step S802, performing weighted fusion on each type of features in the first feature set according to the first attention weight to obtain a first target feature set;
[0270] Step S803, performing weighted fusion on each type of features in the second feature set according to the second attention weight to obtain a second target feature set;
[0271] Step S804, calculating the similarity between the first target feature set and the second target feature set to obtain an image data comparison result.
[0272] The attention weight is calculated for each type of feature based on the feature prompt message. For example, a weight coefficient of 2 is assigned to the target feature type included in the feature prompt message, a weight coefficient of 1 is assigned to the feature type that is not included in the feature prompt message but is considered easy to compare based on historical experience, and a weight coefficient of 0 is assigned to the feature type that is not included in the feature prompt message and is considered difficult to compare based on historical experience. The attention weight is calculated for each type of feature in the first feature set and the second feature set. The specific calculation method can be set according to actual needs. For example, it can also be obtained through a pre-trained feature importance estimation model (trained by sample feature images that have been labeled with multiple feature attention weights).
[0273] According to the first attention weight and the second attention weight, each type of feature in the first feature set and the second feature set is weighted and fused to obtain a first target feature set and a second target feature set that only include the type of features required for comparison, and the similarity between the two is calculated to obtain the image data comparison result. Specifically, it can be implemented by using similarity functions commonly used in retrieval algorithms such as Euclidean distance, cosine similarity, and Mahalanobis distance, or it can be implemented by using a similarity calculation model (the similarity calculation model is obtained by training pre-labeled sample feature pairs).
[0274] In an image classification application scenario, if the second image is a plurality of registered images that have been labeled with classifications, the image data comparison result indicates that the classification of the first image is the category label of the second image with the highest similarity thereto.
[0275] For example, the image comparison process is as follows: Figure 8-2 shown.
[0276] From the above, it can be seen that the image comparison method provided in the embodiment of the present application calculates the attention weights of the first feature set and the second feature set, and performs weighted fusion on them respectively, to obtain the first target feature set and the second target feature set that only include the required feature types in the first feature set and the second feature set, and then calculates the similarity between the two to achieve comparison between the first image and the second image. It can automatically screen suitable features for comparison, reduce the complexity of image comparison, and improve the efficiency of image processing in a low-cost manner.
[0277] See also Fig. 9 The present application also provides a flowchart of an image comparison method, including:
[0278] Step S901, acquiring a first image and a second image;
[0279] Step S902, obtaining a characteristic prompt message, wherein the characteristic prompt message indicates a target feature type for comparison in the first image and the second image;
[0280] Step S903: based on the characteristic prompt message, extracting target feature types from the first image to obtain a third feature set, and extracting target feature types from the second image to obtain a fourth feature set;
[0281] Step S904: determining the image data comparison result according to the third feature set and the fourth feature set.
[0282] After receiving the characteristic prompt message, the target feature types are directly extracted from the first image and the second image to obtain the third feature set and the fourth feature set, based on which the image data comparison result is determined.
[0283] As can be seen from the above, the image comparison method provided in the embodiment of the present application obtains the first image, the second image, and the characteristic prompt message indicating the target feature type used for comparison in the first image and the second image. After obtaining the characteristic prompt message, the target feature type is extracted from the first image and the second image to obtain the third feature set and the fourth feature set, and the image data comparison result is determined based on this. In the image data comparison process, suitable features are screened for comparison based on the characteristic prompt message, which avoids the need to train multiple models for different types of images and select matches between multiple models and the image to be processed for processing, thereby reducing the complexity of image processing and improving the efficiency of image processing.
[0284] In one embodiment of the present application, Figure 10-1As shown, the above step S903 extracts the target feature type from the first image to obtain a third feature set based on the characteristic prompt message, and extracts the target feature type from the second image to obtain a fourth feature set, including:
[0285] Step S1001, determining, according to the characteristic prompt message, each target type feature encoder in a second feature extraction model for extracting features of the target feature type;
[0286] Step S1002, respectively determining the target feature extraction layers to which the target type feature encoders are connected, and determining the number of layers N of the deepest target feature extraction layer;
[0287] Step S1003, extracting features from the first image using the first N feature extraction layers of the second feature extraction model to obtain third image features;
[0288] Step S1004: extracting features of the third image features using the target type feature encoder to obtain a third feature set.
[0289] The multiple feature extraction layers of the second feature extraction model are used to extract one type of feature respectively. Since different types of feature encoders have independent structures and are input into different depth maps of the backbone network, when it is determined which type of feature needs to be used, the backbone network inference process can be stopped early after outputting the depth corresponding to the necessary feature map. Similarly, only the necessary type of target feature encoder needs to be executed.
[0290] Early stopping means that only necessary feature extraction and reasoning processes are executed, and the complete model reasoning process is stopped in advance. For example, if the feature prompt message indicates that only texture features and contour features need to be extracted, the feature extraction process can only execute the texture feature encoder and contour feature encoder corresponding to N1 layers; if the feature prompt message indicates that only pattern feature encoder and component feature encoder need to be extracted, the feature extraction process only needs to execute the pattern feature encoder and component feature encoder corresponding to N1 layers, N2 layers, and N3 layers.
[0291] According to the feature prompt message, each target type feature encoder used to extract features of the target feature type in the second feature extraction model is determined, as well as the target feature extraction layer connected to each target type feature encoder, and the number of layers N of the deepest target feature extraction layer is determined.
[0292] The first image is input into the second feature extraction model, and the target type features of the first image, i.e., the third image features, are respectively extracted using the first N feature extraction layers of the second feature extraction model (the first image is processed). The third image features are then extracted through a target type feature encoder connected thereto to obtain features of the target feature type of the first image, thereby obtaining a third feature set.
[0293] The process of obtaining the fourth feature set is similar to that of the third feature set.
[0294] Exemplarily, when the second image is a labeled image, its complete features have been pre-extracted and stored in the image database, then the fourth feature set may also be complete features obtained from the image database, and early stopping is performed only on feature extraction of the unlabeled first image.
[0295] In one embodiment of the present application, the above-mentioned second feature extraction model includes a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, and a fourth feature extraction layer; the type feature encoder includes a texture feature encoder, a contour feature encoder, a pattern feature encoder, an accessory feature encoder, a component feature encoder, a color feature encoder, and a global feature encoder; the first feature extraction layer is connected to the texture feature encoder and the contour feature encoder, the second feature extraction layer is connected to the pattern feature encoder and the accessory feature encoder, the third feature extraction layer is connected to the component feature encoder, and the fourth feature extraction layer is connected to the color feature encoder and the global feature encoder.
[0296] For example, the image comparison process is as follows: Figure 10-2 shown.
[0297] As can be seen from the above, the image comparison method provided in the embodiment of the present application determines each target type feature encoder in the second feature extraction model for extracting features of the target feature type based on the characteristic prompt message; determines the target feature extraction layers to which each target type feature encoder is connected respectively, and determines the number of layers N of the deepest target feature extraction layer; uses the first N layers of feature extraction layers of the second feature extraction model to extract features of the first image to obtain third image features. Through this strategy of early stopping on demand during the feature extraction process based on target type features, computational resources for feature extraction are saved, processing costs are reduced, and the efficiency of feature extraction and even image processing is improved.
[0298] In one embodiment of the present application, Fig.11 As shown, the above step S903 extracts the target feature type from the first image to obtain a third feature set based on the characteristic prompt message, and extracts the target feature type from the second image to obtain a fourth feature set, including:
[0299] Step S1101, determining, according to the characteristic prompt message, each target type feature encoder in a third feature extraction model for extracting features of the target feature type;
[0300] Step S1102, respectively determining the target feature extraction layers to which the target type feature encoders are connected, and determining the Nth sub-network corresponding to the target feature extraction layer with the deepest number of layers;
[0301] Step S1103, using the feature extraction layers in the first N sub-networks of the third feature extraction model to extract features from the first image to obtain third image features;
[0302] Step S1104: extract features of the third image features using the target type feature encoder to obtain a third feature set.
[0303] The multiple sub-networks of the third feature extraction model are respectively used to extract one type of feature. Since different types of sub-networks have independent structures and are input into different depth maps of the backbone network, when it is determined which type of feature needs to be used, the backbone network inference process can be stopped early after outputting the depth corresponding to the necessary feature map. Similarly, it is only necessary to execute the target feature encoder in the necessary type of sub-network.
[0304] Early stopping means that only necessary feature extraction and reasoning processes are executed, and the complete model reasoning process is stopped in advance. For example, if the feature prompt message indicates that only texture features need to be extracted, the feature extraction process can only execute the texture feature encoder in the N1 layers sub-network; if the feature prompt message indicates that only pattern feature encoders and component feature encoders need to be extracted, the feature extraction process only needs to execute the pattern feature encoders and component feature encoders in the N1 layers and N2 layers sub-networks.
[0305] According to the feature prompt message, determine each target type feature encoder in the third feature extraction model for extracting features of the target feature type, determine the sub-network corresponding to the target feature extraction layer connected to each target type feature encoder, and determine the Nth sub-network corresponding to the deepest target feature extraction layer.
[0306] The first image is input into the third feature extraction model, and the target feature extraction layers in the first N sub-networks of the third feature extraction model are used to respectively extract the target type features of the first image, namely, the third image features (process the first image), and then the third image features are extracted through the target type feature encoder to obtain the features of the target feature type of the first image, thereby obtaining a third feature set.
[0307] The process of obtaining the fourth feature set is similar to that of the third feature set.
[0308] Exemplarily, when the second image is a labeled image, its complete features have been pre-extracted and stored in the image database, then the fourth feature set may also be complete features obtained from the image database, and early stopping is performed only on feature extraction of the unlabeled first image.
[0309] In one embodiment of the present application, the third feature extraction model includes a first subnetwork for extracting texture features, a second subnetwork for extracting contour features, a third subnetwork for extracting pattern features, a fourth subnetwork for extracting accessory features, a fifth subnetwork for extracting component features, a sixth subnetwork for extracting color features, and a seventh subnetwork for extracting global features.
[0310] As can be seen from the above, the image comparison method provided in the embodiment of the present application determines each target type feature encoder in the second feature extraction model for extracting features of the target feature type based on the characteristic prompt message; determines the Nth subnetwork corresponding to the target feature extraction layer connected to each target type feature encoder, and determines the Nth subnetwork corresponding to the target feature extraction layer with the deepest number of layers; uses the target feature extraction layer in the first N subnetworks of the second feature extraction model to extract features of the first image to obtain third image features. Through this strategy of early stopping on demand during the feature extraction process based on target type features, computational resources for feature extraction are saved, processing costs are reduced, and the efficiency of feature extraction and even image processing is improved.
[0311] For example, in an image classification application scenario, an example flow chart of image comparison can be as follows: Figure 12-1 , Figure 12-2 , Figure 12-3 As shown, after obtaining the features, the classifier is used to obtain the final image classification result.
[0312] In one embodiment of the present application, Fig.13 As shown, a structural schematic diagram of a first image comparison device is provided, including:
[0313] An image acquisition module 1301 is used to acquire a first image and a second image;
[0314] The message acquisition module 1302 is used to acquire a characteristic prompt message, wherein the characteristic prompt message indicates a target characteristic type for comparison in the first image and the second image;
[0315] A feature extraction module 1303 is used to extract multiple types of features from the first image to obtain a first feature set, and to extract multiple types of features from the second image to obtain a second feature set;
[0316] The image comparison module 1304 is used to determine the image data comparison result according to the first feature set, the second feature set, and the characteristic prompt message.
[0317] As can be seen from the above, the image comparison device provided in the embodiment of the present application obtains the first image, the second image, and the characteristic prompt message indicating the target feature type used for comparison in the first image and the second image, extracts multiple types of features from the first image to obtain a first feature set, extracts multiple types of features from the second image to obtain a second feature set, and determines the image data comparison result based on the first feature set, the second feature set, and the characteristic prompt message. In the image data comparison process, suitable features are screened for comparison based on the characteristic prompt message, avoiding the need to train multiple models for different types of images and selecting matches between multiple models and the image to be processed for processing, reducing the complexity of image processing and improving the efficiency of image processing.
[0318] In one embodiment of the present application, the feature extraction module 1303 is specifically used to:
[0319] Input the first image into a first feature extraction model, perform feature extraction on the first image through multiple feature extraction layers of the first feature extraction model, and obtain multiple first image features of different granularities; perform feature extraction on each of the first image features using multiple granularity feature encoders of the first feature extraction model to obtain a first feature set; wherein one granularity feature encoder is connected to one feature extraction layer, and the granularity feature encoder is used to perform feature extraction on first image features output by the feature extraction layer connected to the encoder;
[0320] The second image is input into the first feature extraction model to obtain a second feature set.
[0321] From the above, it can be seen that the image comparison device provided in the embodiment of the present application, through multiple feature extraction layers and granularity feature encoders in the first feature extraction model, extracts features of different granularities of the first image and the second image to obtain complete features, thereby avoiding omission errors in feature comparison and improving the accuracy of image processing.
[0322] In one embodiment of the present application, the feature extraction module 1303 is specifically used to:
[0323] Input the first image into a second feature extraction model, perform feature extraction on the first image through multiple feature extraction layers of the second feature extraction model, and obtain multiple first image features of different types; perform feature extraction on each of the first image features using multiple type feature encoders of the second feature extraction model to obtain a first feature set; wherein one type feature encoder is connected to one feature extraction layer, one feature extraction layer is connected to one or more type feature encoders, and the type feature encoder is used to extract features of the first image features output by the feature extraction layer connected to itself;
[0324] The second image is input into the second feature extraction model to obtain a second feature set.
[0325] From the above, it can be seen that the image comparison device provided in the embodiment of the present application, through multiple feature extraction layers and type feature encoders in the second feature extraction model, extracts different types of features of the first image and the second image to obtain complete features, thereby avoiding omission errors in feature comparison and improving the accuracy of image processing.
[0326] In one embodiment of the present application, the feature extraction module 1303 is specifically used to:
[0327] Input the first image into a third feature extraction model, perform feature extraction on the first image through a feature extraction layer in a subnetwork of the third feature extraction model, and obtain a plurality of first image features of different types; perform feature extraction on each of the first image features using a plurality of type feature encoders in the subnetwork, and obtain a first feature set; wherein each subnetwork includes a connected feature extractor and a type feature encoder, and the type feature encoder is used to perform feature extraction on the first image features output by the feature extraction layer connected to itself;
[0328] The second image is input into the third feature extraction model to obtain a third feature set.
[0329] From the above, it can be seen that the image comparison device provided in the embodiment of the present application extracts different types of features of the first image and the second image through the feature extraction layers and type feature encoders in multiple sub-networks in the third feature extraction model to obtain complete features, thereby avoiding omission errors in feature comparison and improving the accuracy of image processing.
[0330] In one embodiment of the present application, the feature extraction module 1303 is specifically used to:
[0331] Inputting the first image and the second image into the visual big model respectively, extracting multiple types of features from the first image and the second image, and obtaining the first feature set and the second feature set;
[0332] The visual large model is trained in the following way:
[0333] Selecting a sample image having features of a first feature type and an image characteristic expert model for extracting features of the first feature type;
[0334] The currently selected image feature expert model is used as the teacher model, the visual big model is used as the student model, and the currently selected sample image is used as input to train the visual big model;
[0335] Select sample images with other feature types and use the image characteristic expert model for extracting the other feature types to continue training the large visual model until all preset feature types are trained.
[0336] From the above, it can be seen that the image comparison device provided in the embodiment of the present application extracts various types of features of the first image and the second image through a trained large visual model to obtain complete features, thereby avoiding omission errors in feature comparison and improving the accuracy of image processing.
[0337] In one embodiment of the present application, the image comparison module 1304 is specifically used to:
[0338] Encoding the characteristic prompt message to obtain a first encoding feature;
[0339] splicing the first coding feature and the first feature set to obtain a first splicing feature, and splicing the first coding feature and the second feature set to obtain a second splicing feature;
[0340] Performing feature screening on the first splicing feature through the feature screening model to obtain a first screening feature; performing feature screening on the second splicing feature through the feature screening model to obtain a second screening feature;
[0341] The similarity between the first screening feature and the second screening feature is calculated to obtain the image data comparison result.
[0342] From the above, it can be seen that the image comparison device provided in the embodiment of the present application obtains the features of the target feature types in the first feature set and the second feature set by respectively splicing the first feature set, the second feature set and the encoded characteristic prompt message, and then calculates the similarity between the two to achieve comparison between the first image and the second image. It can automatically screen suitable features for comparison, reduce the complexity of image comparison, and improve the efficiency of image processing in a low-cost manner.
[0343] In one embodiment of the present application, the image comparison module 1304 is specifically used to:
[0344] Based on the characteristic prompt message, calculating a first attention weight of each type of feature in the first feature set, and calculating a second attention weight of each type of feature in the second feature set;
[0345] Performing weighted fusion on each type of features in the first feature set according to the first attention weight to obtain a first target feature set;
[0346] Performing weighted fusion on each type of features in the second feature set according to the second attention weight to obtain a second target feature set;
[0347] The similarity between the first target feature set and the second target feature set is calculated to obtain an image data comparison result.
[0348] From the above, it can be seen that the image comparison device provided in the embodiment of the present application calculates the attention weights of the first feature set and the second feature set, and performs weighted fusion on them respectively, to obtain the first target feature set and the second target feature set that only include the required feature types in the first feature set and the second feature set, and then calculates the similarity between the two to achieve comparison between the first image and the second image. It can automatically screen suitable features for comparison, reduce the complexity of image comparison, and improve the efficiency of image processing in a low-cost manner.
[0349] In one embodiment of the present application, Fig.14 As shown, a structural schematic diagram of a second image comparison device is provided, including:
[0350] An image acquisition module 1401 is used to acquire a first image and a second image;
[0351] The message acquisition module 1402 is used to acquire a characteristic prompt message, wherein the characteristic prompt message indicates a target characteristic type for comparison in the first image and the second image;
[0352] A feature extraction module 1403 is used to extract target feature types from the first image to obtain a third feature set, and to extract target feature types from the second image to obtain a fourth feature set based on the characteristic prompt message;
[0353] The image comparison module 1404 is used to determine the image data comparison result according to the third feature set and the fourth feature set.
[0354] As can be seen from the above, the image comparison device provided in the embodiment of the present application obtains the first image, the second image, and the characteristic prompt message indicating the target feature type used for comparison in the first image and the second image. After obtaining the characteristic prompt message, the target feature type is extracted from the first image and the second image to obtain the third feature set and the fourth feature set, and the image data comparison result is determined based on this. In the image data comparison process, suitable features are screened for comparison based on the characteristic prompt message, which avoids the need to train multiple models for different types of images and select matches between multiple models and the image to be processed for processing, thereby reducing the complexity of image processing and improving the efficiency of image processing.
[0355] In one embodiment of the present application, the feature extraction module 1403 is specifically used to:
[0356] Determining, according to the characteristic prompt message, each target type feature encoder in the second feature extraction model for extracting features of the target feature type;
[0357] Determine the target feature extraction layers to which each target type feature encoder is connected, and determine the number of layers N of the deepest target feature extraction layer;
[0358] Using the first N feature extraction layers of the second feature extraction model to extract features from the first image, to obtain third image features;
[0359] The target type feature encoder is used to extract features of the third image features to obtain a third feature set.
[0360] As can be seen from the above, the image comparison device provided in the embodiment of the present application determines each target type feature encoder in the second feature extraction model for extracting features of the target feature type based on the characteristic prompt message; determines the target feature extraction layers to which each target type feature encoder is connected respectively, and determines the number of layers N of the deepest target feature extraction layer; uses the first N layers of feature extraction layers of the second feature extraction model to extract features of the first image to obtain third image features. Through this strategy of early stopping on demand during the feature extraction process based on target type features, computational resources for feature extraction are saved, processing costs are reduced, and the efficiency of feature extraction and even image processing is improved.
[0361] In one embodiment of the present application, the feature extraction module 1403 is specifically used to:
[0362] Determining, according to the characteristic prompt message, each target type feature encoder in a third feature extraction model for extracting features of the target feature type;
[0363] Determine the target feature extraction layers to which each target type feature encoder is connected, and determine the Nth subnetwork corresponding to the deepest target feature extraction layer;
[0364] Using the feature extraction layers in the first N sub-networks of the third feature extraction model to extract features from the first image, to obtain third image features;
[0365] The target type feature encoder is used to extract features of the third image features to obtain a third feature set.
[0366] As can be seen from the above, the image comparison device provided in the embodiment of the present application determines each target type feature encoder in the second feature extraction model for extracting features of the target feature type based on the characteristic prompt message; determines the Nth subnetwork corresponding to the target feature extraction layer connected to each target type feature encoder, and determines the Nth subnetwork corresponding to the target feature extraction layer with the deepest number of layers; uses the target feature extraction layer in the first N subnetworks of the second feature extraction model to extract features of the first image to obtain third image features. Through this strategy of early stopping on demand during the feature extraction process based on target type features, the computational resources for feature extraction are saved, the processing cost is reduced, and the efficiency of feature extraction and even image processing is improved.
[0367] In one embodiment of the present application, the second feature extraction model includes a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, and a fourth feature extraction layer; the type feature encoder includes a texture feature encoder, a contour feature encoder, a pattern feature encoder, an accessory feature encoder, a component feature encoder, a color feature encoder, and a global feature encoder; the first feature extraction layer is connected to the texture feature encoder and the contour feature encoder, the second feature extraction layer is connected to the pattern feature encoder and the accessory feature encoder, the third feature extraction layer is connected to the component feature encoder, and the fourth feature extraction layer is connected to the color feature encoder and the global feature encoder.
[0368] In one embodiment of the present application, the third feature extraction model includes a first subnetwork for extracting texture features, a second subnetwork for extracting contour features, a third subnetwork for extracting pattern features, a fourth subnetwork for extracting accessory features, a fifth subnetwork for extracting component features, a sixth subnetwork for extracting color features, and a seventh subnetwork for extracting global features.
[0369] In one embodiment of the present application, the message acquisition module 1402 is specifically used to:
[0370] Inputting the first image and the second image into an image characteristic classification model respectively, and extracting image type information of the first image and the second image;
[0371] Based on the preset correspondence between the image type information and the feature type, the common feature type of the first image and the second image is determined to obtain the feature prompt message.
[0372] As can be seen from the above, the image comparison device provided in the embodiment of the present application determines the common feature type of the first image and the second image through the image feature classification model, obtains a feature prompt message, and realizes the analysis of the image characteristics. Based on this, feature guidance is provided for the comparison between the first image and the second image, and only the features of the target feature type need to be selected for comparison among multiple types of features, thereby avoiding the errors easily caused by inappropriate type feature comparisons and the manual intervention required when selecting and applying between multiple models, thereby improving the accuracy of image processing, and, in the case of having multiple models, being able to solve the processing problems of multiple types of images at the same time, further improving the efficiency of image processing.
[0373] In one embodiment of the present application, the image data comparison result is a target re-identification result or an image classification result.
[0374] It should be noted that the human body model in this embodiment is not a human body model for a specific user and cannot reflect the personal information of a specific user.
[0375] The present application also provides an electronic device, such as Fig.15 As shown, including:
[0376] Memory 1501, used for storing computer programs;
[0377] The processor 1502 is used to implement any of the above-mentioned image comparison methods when executing the program stored in the memory 1501.
[0378] In addition, the electronic device may further include a communication bus and / or a communication interface, and the processor 1502, the communication interface, and the memory 1501 communicate with each other via the communication bus.
[0379] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0380] The communication interface is used for communication between the above electronic device and other devices.
[0381] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0382] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0383] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned image comparison methods are implemented.
[0384] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any image comparison method in the above embodiments.
[0385] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.
[0386] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0387] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0388] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.
Claims
1. An image comparison method, characterized in that: include: Acquire a first image and a second image; Acquiring a characteristic prompt message, wherein the characteristic prompt message indicates the target feature type used for comparison in the first image and the second image; the acquiring characteristic prompt message comprises: inputting the first image and the second image into an image feature classification model respectively, extracting the image type information of the first image and the second image; determining the common feature type of the first image and the second image based on a preset correspondence between the image type information and the feature type, and obtaining the characteristic prompt message, wherein the preset correspondence between the image type information and the feature type comprises: a correspondence between infrared image type and texture features and contour features, a correspondence between an incomplete target image type and component features, and a correspondence between an image type of visible light with a significant pattern and a significant pattern feature; Extracting multiple types of features from the first image to obtain a first feature set, and extracting multiple types of features from the second image to obtain a second feature set; An image data comparison result is determined according to the first feature set, the second feature set, and the characteristic prompt message.
2. The method according to claim 1, characterized in that The extracting multiple types of features from the first image to obtain a first feature set, and extracting multiple types of features from the second image to obtain a second feature set, includes: Input the first image into a first feature extraction model, perform feature extraction on the first image through multiple feature extraction layers of the first feature extraction model, and obtain multiple first image features of different granularities; perform feature extraction on each of the first image features using multiple granularity feature encoders of the first feature extraction model to obtain a first feature set; wherein one granularity feature encoder is connected to one feature extraction layer, and the granularity feature encoder is used to perform feature extraction on first image features output by the feature extraction layer connected to the encoder; The second image is input into the first feature extraction model to obtain a second feature set.
3. The method according to claim 1, characterized in that The extracting multiple types of features from the first image to obtain a first feature set, and extracting multiple types of features from the second image to obtain a second feature set, includes: Input the first image into a second feature extraction model, perform feature extraction on the first image through multiple feature extraction layers of the second feature extraction model, and obtain multiple first image features of different types; perform feature extraction on each of the first image features using multiple type feature encoders of the second feature extraction model to obtain a first feature set; wherein one type feature encoder is connected to one feature extraction layer, one feature extraction layer is connected to one or more type feature encoders, and the type feature encoder is used to extract features of the first image features output by the feature extraction layer connected to itself; The second image is input into the second feature extraction model to obtain a second feature set.
4. The method according to claim 1, characterized in that: The extracting multiple types of features from the first image to obtain a first feature set, and extracting multiple types of features from the second image to obtain a second feature set, includes: Input the first image into a third feature extraction model, perform feature extraction on the first image through a feature extraction layer in a subnetwork of the third feature extraction model, and obtain a plurality of first image features of different types; perform feature extraction on each of the first image features using a plurality of type feature encoders in the subnetwork, and obtain a first feature set; wherein each subnetwork includes a connected feature extractor and a type feature encoder, and the type feature encoder is used to perform feature extraction on the first image features output by the feature extraction layer connected to itself; The second image is input into the third feature extraction model to obtain a third feature set.
5. The method according to claim 1, characterized in that The extracting multiple types of features from the first image to obtain a first feature set, and extracting multiple types of features from the second image to obtain a second feature set, includes: Inputting the first image and the second image into the visual big model respectively, extracting multiple types of features from the first image and the second image, and obtaining the first feature set and the second feature set; The visual large model is trained in the following way: Selecting a sample image having features of a first feature type and an image characteristic expert model for extracting features of the first feature type; The currently selected image feature expert model is used as the teacher model, the visual big model is used as the student model, and the currently selected sample image is used as input to train the visual big model; Select sample images with other feature types and use the image characteristic expert model for extracting the other feature types to continue training the large visual model until all preset feature types are trained.
6. The method according to claim 1, characterized in that The determining the image data comparison result according to the first feature set, the second feature set, and the characteristic prompt message includes: Encoding the characteristic prompt message to obtain a first encoding feature; splicing the first coding feature and the first feature set to obtain a first splicing feature, and splicing the first coding feature and the second feature set to obtain a second splicing feature; Performing feature screening on the first splicing feature through the feature screening model to obtain a first screening feature; performing feature screening on the second splicing feature through the feature screening model to obtain a second screening feature; The similarity between the first screening feature and the second screening feature is calculated to obtain the image data comparison result.
7. The method according to claim 1, characterized in that The determining the image data comparison result according to the first feature set, the second feature set, and the characteristic prompt message includes: Based on the characteristic prompt message, calculating a first attention weight of each type of feature in the first feature set, and calculating a second attention weight of each type of feature in the second feature set; Performing weighted fusion on each type of features in the first feature set according to the first attention weight to obtain a first target feature set; Performing weighted fusion on each type of features in the second feature set according to the second attention weight to obtain a second target feature set; The similarity between the first target feature set and the second target feature set is calculated to obtain an image data comparison result.
8. An image comparison method, characterized in that: include: Acquire a first image and a second image; Acquiring a characteristic prompt message, wherein the characteristic prompt message indicates the target feature type used for comparison in the first image and the second image; the acquiring characteristic prompt message comprises: inputting the first image and the second image into an image feature classification model respectively, extracting the image type information of the first image and the second image; determining the common feature type of the first image and the second image based on a preset correspondence between the image type information and the feature type, and obtaining the characteristic prompt message, wherein the preset correspondence between the image type information and the feature type comprises: a correspondence between infrared image type and texture features and contour features, a correspondence between an incomplete target image type and component features, and a correspondence between an image type of visible light with a significant pattern and a significant pattern feature; Based on the characteristic prompt message, extracting target feature types from the first image to obtain a third feature set, and extracting target feature types from the second image to obtain a fourth feature set; An image data comparison result is determined according to the third feature set and the fourth feature set.
9. The method according to claim 8, characterized in that The step of extracting a target feature type from the first image to obtain a third feature set based on the characteristic prompt message and extracting a target feature type from the second image to obtain a fourth feature set includes: Determining, according to the characteristic prompt message, each target type feature encoder in the second feature extraction model for extracting features of the target feature type; Determine the target feature extraction layers to which each target type feature encoder is connected, and determine the number of layers N of the deepest target feature extraction layer; Using the first N feature extraction layers of the second feature extraction model to extract features from the first image, to obtain third image features; The target type feature encoder is used to extract features of the third image features to obtain a third feature set.
10. The method according to claim 8, characterized in that The step of extracting a target feature type from the first image to obtain a third feature set based on the characteristic prompt message and extracting a target feature type from the second image to obtain a fourth feature set includes: Determining, according to the characteristic prompt message, each target type feature encoder in a third feature extraction model for extracting features of the target feature type; Determine the target feature extraction layers to which each target type feature encoder is connected, and determine the Nth subnetwork corresponding to the deepest target feature extraction layer; Using the feature extraction layers in the first N sub-networks of the third feature extraction model to extract features from the first image, to obtain third image features; The target type feature encoder is used to extract features of the third image features to obtain a third feature set.
11. The method according to claim 3 or 9, characterized in that: The second feature extraction model includes a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, and a fourth feature extraction layer; the type feature encoder includes a texture feature encoder, a contour feature encoder, a pattern feature encoder, an accessory feature encoder, a component feature encoder, a color feature encoder, and a global feature encoder; the first feature extraction layer is connected to the texture feature encoder and the contour feature encoder, the second feature extraction layer is connected to the pattern feature encoder and the accessory feature encoder, the third feature extraction layer is connected to the component feature encoder, and the fourth feature extraction layer is connected to the color feature encoder and the global feature encoder.
12. The method according to claim 4 or 10, characterized in that: The third feature extraction model includes a first sub-network for extracting texture features, a second sub-network for extracting contour features, a third sub-network for extracting pattern features, a fourth sub-network for extracting appendage features, a fifth sub-network for extracting component features, a sixth sub-network for extracting color features, and a seventh sub-network for extracting global features.
13. An image comparison device, characterized in that: include: An image acquisition module, used for acquiring a first image and a second image; A message acquisition module, used for acquiring a characteristic prompt message, wherein the characteristic prompt message indicates the target feature type used for comparison in the first image and the second image; the acquiring characteristic prompt message comprises: inputting the first image and the second image into an image feature classification model respectively, extracting the image type information of the first image and the second image; determining the common feature type of the first image and the second image based on a preset correspondence between the image type information and the feature type, and obtaining the characteristic prompt message, wherein the preset correspondence between the image type information and the feature type comprises: a correspondence between infrared image type and texture features and contour features, a correspondence between an incomplete target image type and component features, and a correspondence between an image type of visible light with a significant pattern and a significant pattern feature; a feature extraction module, configured to extract multiple types of features from the first image to obtain a first feature set, and to extract multiple types of features from the second image to obtain a second feature set; An image comparison module is used to determine an image data comparison result based on the first feature set, the second feature set, and the characteristic prompt message.
14. The device according to claim 13, characterized in that The feature extraction module is specifically used for: Input the first image into a first feature extraction model, perform feature extraction on the first image through multiple feature extraction layers of the first feature extraction model, and obtain multiple first image features of different granularities; perform feature extraction on each of the first image features using multiple granularity feature encoders of the first feature extraction model to obtain a first feature set; wherein one granularity feature encoder is connected to one feature extraction layer, and the granularity feature encoder is used to perform feature extraction on first image features output by the feature extraction layer connected to the encoder; Inputting the second image into the first feature extraction model to obtain a second feature set; The feature extraction module is specifically used for: Input the first image into a second feature extraction model, perform feature extraction on the first image through multiple feature extraction layers of the second feature extraction model, and obtain multiple first image features of different types; perform feature extraction on each of the first image features using multiple type feature encoders of the second feature extraction model to obtain a first feature set; wherein one type feature encoder is connected to one feature extraction layer, one feature extraction layer is connected to one or more type feature encoders, and the type feature encoder is used to extract features of the first image features output by the feature extraction layer connected to itself; Inputting the second image into the second feature extraction model to obtain a second feature set; The feature extraction module is specifically used for: Input the first image into a third feature extraction model, perform feature extraction on the first image through a feature extraction layer in a subnetwork of the third feature extraction model, and obtain a plurality of first image features of different types; perform feature extraction on each of the first image features using a plurality of type feature encoders in the subnetwork, and obtain a first feature set; wherein each subnetwork includes a feature extractor and a type feature encoder connected thereto, and the type feature encoder is used to perform feature extraction on the first image features output by the feature extraction layer connected thereto; Inputting the second image into the third feature extraction model to obtain a third feature set; The feature extraction module is specifically used for: Inputting the first image and the second image into the visual big model respectively, extracting multiple types of features from the first image and the second image, and obtaining the first feature set and the second feature set; The visual large model is trained in the following way: Selecting a sample image having features of a first feature type and an image characteristic expert model for extracting features of the first feature type; The currently selected image feature expert model is used as the teacher model, the visual big model is used as the student model, and the currently selected sample image is used as input to train the visual big model; Selecting sample images with features of other feature types and an expert model for extracting image characteristics of the other feature types to continue training the large visual model until the training of the preset multiple feature types is completed; The image comparison module is specifically used for: Encoding the characteristic prompt message to obtain a first encoding feature; splicing the first coding feature and the first feature set to obtain a first splicing feature, and splicing the first coding feature and the second feature set to obtain a second splicing feature; Performing feature screening on the first splicing feature through the feature screening model to obtain a first screening feature; performing feature screening on the second splicing feature through the feature screening model to obtain a second screening feature; Calculating the similarity between the first screening feature and the second screening feature to obtain the image data comparison result; The image comparison module is specifically used for: Based on the characteristic prompt message, calculating a first attention weight of each type of feature in the first feature set, and calculating a second attention weight of each type of feature in the second feature set; Performing weighted fusion on each type of features in the first feature set according to the first attention weight to obtain a first target feature set; Performing weighted fusion on each type of features in the second feature set according to the second attention weight to obtain a second target feature set; The similarity between the first target feature set and the second target feature set is calculated to obtain an image data comparison result.
15. An image comparison device, characterized in that: include: An image acquisition module, used for acquiring a first image and a second image; A message acquisition module, used for acquiring a characteristic prompt message, wherein the characteristic prompt message indicates the target feature type used for comparison in the first image and the second image; the acquiring characteristic prompt message comprises: inputting the first image and the second image into an image feature classification model respectively, extracting the image type information of the first image and the second image; determining the common feature type of the first image and the second image based on a preset correspondence between the image type information and the feature type, and obtaining the characteristic prompt message, wherein the preset correspondence between the image type information and the feature type comprises: a correspondence between infrared image type and texture features and contour features, a correspondence between an incomplete target image type and component features, and a correspondence between an image type of visible light with a significant pattern and a significant pattern feature; a feature extraction module, configured to extract target feature types from the first image to obtain a third feature set, and to extract target feature types from the second image to obtain a fourth feature set based on the characteristic prompt message; The image comparison module is used to determine the image data comparison result according to the third feature set and the fourth feature set.
16. The device according to claim 15, characterized in that The feature extraction module is specifically used for: Determining, according to the characteristic prompt message, each target type feature encoder in the second feature extraction model for extracting features of the target feature type; Determine the target feature extraction layers to which each target type feature encoder is connected, and determine the number of layers N of the deepest target feature extraction layer; Using the first N feature extraction layers of the second feature extraction model to extract features from the first image, to obtain third image features; Using the target type feature encoder to extract features from the third image feature to obtain a third feature set; The feature extraction module is specifically used for: Determining, according to the characteristic prompt message, each target type feature encoder in a third feature extraction model for extracting features of the target feature type; Determine the target feature extraction layers to which each target type feature encoder is connected, and determine the Nth subnetwork corresponding to the deepest target feature extraction layer; Using the feature extraction layers in the first N sub-networks of the third feature extraction model to extract features from the first image, to obtain third image features; Using the target type feature encoder to extract features from the third image feature to obtain a third feature set; The second feature extraction model includes a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, and a fourth feature extraction layer; the type feature encoder includes a texture feature encoder, a contour feature encoder, a pattern feature encoder, an accessory feature encoder, a component feature encoder, a color feature encoder, and a global feature encoder; the first feature extraction layer is connected to the texture feature encoder and the contour feature encoder, the second feature extraction layer is connected to the pattern feature encoder and the accessory feature encoder, the third feature extraction layer is connected to the component feature encoder, and the fourth feature extraction layer is connected to the color feature encoder and the global feature encoder; The third feature extraction model includes a first sub-network for extracting texture features, a second sub-network for extracting contour features, a third sub-network for extracting pattern features, a fourth sub-network for extracting appendage features, a fifth sub-network for extracting component features, a sixth sub-network for extracting color features, and a seventh sub-network for extracting global features.
17. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-12 when executing a program stored in a memory.
18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.
