Image processing method and device, storage medium and chip

Abstract

The invention relates to an image processing method and device, a storage medium and a chip, and relates to the technical field of image processing, and the method comprises the steps: obtaining a to-be-detected image and a sample image which comprises a sample detection target, and carrying out the detection of the to-be-detected image according to the sample image and a mask image for the sample detection target; and determining a to-be-detected target in the to-be-detected image, and determining a detection result of the to-be-detected image according to the similarity between the to-be-detected target and the sample detection target. According to a sample image and a sample check target mask image corresponding to the sample image, a to-be-checked target is determined from a to-be-checked image in advance, and a check result of the to-be-checked image is determined according to the similarity between the to-be-checked target and the sample check target. Interference of background information in the to-be-checked image and other information irrelevant to checking can be reduced, and the accuracy of image checking is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, device, storage medium and chip. Background Art

[0002] Image classification is one of the most basic tasks in the field of computer vision. Its goal is to train a classification model to correctly predict the category of an image given an image. Common scenarios include animal classification, food classification, etc. Image classification has many forms, such as multi-category image classification, multi-label image classification, fine-grained image classification, and unsupervised image classification, which are adapted to different scenarios and needs.

[0003] In the related art, the method of searching images by images is used to check whether the images are qualified. However, when the image background is noisy or the object to be checked is small, the accuracy of this method is poor. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides an image processing method, device, storage medium and chip.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, including:

[0006] Acquire an image to be checked and a sample image, wherein the sample image includes a sample check target;

[0007] Determine the target to be checked in the image to be checked according to the sample image and the mask image for the sample check target;

[0008] The verification result of the image to be verified is determined according to the similarity between the object to be verified and the sample verification object.

[0009] Optionally, the determining the target to be checked in the image to be checked according to the sample image and the mask image for the sample check target includes:

[0010] The sample image and the nuclear image to be inspected are processed respectively by a basic classification model to obtain a first feature vector corresponding to the sample image and a second feature vector corresponding to the nuclear image to be inspected, wherein the basic classification model is a pre-trained model;

[0011] Determine a similarity matrix according to the first eigenvector and the second eigenvector;

[0012] According to the similarity matrix, pixel points in the mask image and the image to be inspected are matched to obtain target matching points, where the target matching points are a plurality of pixel points in the image to be inspected that are most similar to the mask image;

[0013] According to the target matching points, the nuclear target to be inspected is determined from the nuclear image to be inspected.

[0014] Optionally, matching pixel points in the mask image and the image to be inspected according to the similarity matrix to obtain target matching points includes:

[0015] Determine the similarity between the pixel points in the mask image and the pixel points in the to-be-checked core image according to the similarity matrix to obtain a forward similarity matrix;

[0016] According to the forward similarity matrix, a plurality of pixel points in the to-be-checked core image having the greatest similarity to the mask image are determined to obtain a first matching point;

[0017] Determine the similarity between the first matching point and the pixel points in the mask image according to the similarity matrix to obtain a reverse similarity matrix;

[0018] According to the inverse similarity matrix, a plurality of pixel points in the mask image having the greatest similarity to the image to be checked are determined to obtain second matching points;

[0019] The target matching point is determined according to the first matching point and the second matching point.

[0020] Optionally, the determining the nuclear target to be inspected from the nuclear image to be inspected according to the target matching point includes:

[0021] According to the target matching point, a target area is determined from the nuclear image to be inspected, wherein the target area is the smallest area including the target matching point;

[0022] The target to be inspected is determined according to the target area.

[0023] Optionally, determining the verification result of the image to be verified according to the similarity between the object to be verified and the sample verification object includes:

[0024] According to the target to be inspected, a target inspection image is determined from the image to be inspected;

[0025] Determining a sample verification image from the sample image according to the sample verification target;

[0026] Determining an actual similarity between the target verification image and the sample verification image;

[0027] The verification result of the image to be verified is determined according to the actual similarity.

[0028] Optionally, the determining the actual similarity between the target verification image and the sample verification image comprises:

[0029] The target verification image and the sample verification image are processed respectively by a basic classification model to obtain a third feature vector corresponding to the target verification image and a fourth feature vector corresponding to the sample verification image, wherein the basic classification model is a pre-trained model;

[0030] The actual similarity between the target verification image and the sample verification image is determined according to the third eigenvector and the fourth eigenvector.

[0031] Optionally, the sample image includes a plurality of sample sub-images, each of which includes a sample verification sub-target;

[0032] The step of determining the target to be checked in the image to be checked according to the sample image and the mask image for the sample check target comprises:

[0033] Determine a plurality of to-be-checked targets in the to-be-checked image according to the plurality of sample sub-images and mask images for the sample check sub-targets in the plurality of sample sub-images;

[0034] The step of determining the verification result of the image to be verified according to the similarity between the object to be verified and the sample verification object includes:

[0035] For any target to be checked, determine the similarity between the target to be checked and the corresponding sample check sub-target;

[0036] Determine the fusion similarity according to the similarity between the multiple to-be-checked targets and the corresponding sample check sub-targets;

[0037] The verification result of the image to be verified is determined according to the fusion similarity.

[0038] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, including:

[0039] An acquisition module is configured to acquire an image to be checked and a sample image, wherein the sample image includes a sample check target;

[0040] A first determination module is configured to determine the target to be checked in the image to be checked according to the sample image and the mask image for the sample check target;

[0041] The second determination module is configured to determine the verification result of the image to be verified according to the similarity between the object to be verified and the sample verification object.

[0042] Optionally, the first determining module includes:

[0043] A first acquisition submodule is configured to process the sample image and the nuclear image to be inspected respectively through a basic classification model to obtain a first feature vector corresponding to the sample image and a second feature vector corresponding to the nuclear image to be inspected, wherein the basic classification model is a pre-trained model;

[0044] A first determining submodule is configured to determine a similarity matrix according to the first eigenvector and the second eigenvector;

[0045] A second obtaining submodule is configured to match the pixel points in the mask image and the image to be inspected according to the similarity matrix to obtain target matching points, wherein the target matching points are a plurality of pixel points in the image to be inspected that are most similar to the mask image;

[0046] The second determination submodule is configured to determine the nuclear target to be inspected from the nuclear image to be inspected according to the target matching point.

[0047] Optionally, the second obtaining submodule includes:

[0048] A first obtaining subunit is configured to determine the similarity between the pixel points in the mask image and the pixel points in the to-be-checked nuclear image according to the similarity matrix, and obtain a forward similarity matrix;

[0049] A second obtaining subunit is configured to determine, according to the forward similarity matrix, a plurality of pixel points in the to-be-checked core image that have the greatest similarity to the mask image, to obtain a first matching point;

[0050] a third obtaining subunit, configured to determine the similarity between the first matching point and the pixel points in the mask image according to the similarity matrix, and obtain an inverse similarity matrix;

[0051] A fourth obtaining subunit is configured to determine, according to the inverse similarity matrix, a plurality of pixel points in the mask image that have the greatest similarity with the to-be-checked core image, to obtain second matching points;

[0052] The first determining subunit is configured to determine the target matching point according to the first matching point and the second matching point.

[0053] Optionally, the second determining submodule includes:

[0054] A second determining subunit is configured to determine a target area from the to-be-inspected nuclear image according to the target matching point, wherein the target area is a minimum area including the target matching point;

[0055] The third determining subunit is configured to determine the target to be inspected according to the target area.

[0056] Optionally, the second determining module includes:

[0057] A third determination submodule is configured to determine a target verification image from the to-be-inspected nuclear image according to the to-be-inspected nuclear target;

[0058] a fourth determination submodule, configured to determine a sample verification image from the sample image according to the sample verification target;

[0059] a fifth determination submodule, configured to determine an actual similarity between the target verification image and the sample verification image;

[0060] The sixth determination submodule is configured to determine a verification result of the image to be verified according to the actual similarity.

[0061] Optionally, the fifth determining submodule includes:

[0062] a fifth obtaining subunit, configured to process the target verification image and the sample verification image respectively through a basic classification model to obtain a third feature vector corresponding to the target verification image and a fourth feature vector corresponding to the sample verification image, wherein the basic classification model is a pre-trained model;

[0063] The fourth determining subunit is configured to determine the actual similarity between the target verification image and the sample verification image according to the third feature vector and the fourth feature vector.

[0064] Optionally, the sample image includes a plurality of sample sub-images, each of which includes a sample verification sub-target;

[0065] The first determining module includes:

[0066] a seventh determination submodule, configured to determine a plurality of to-be-checked targets in the to-be-checked target image according to the plurality of sample sub-images and mask images for the sample check sub-targets in the plurality of sample sub-images;

[0067] The second determining module includes:

[0068] An eighth determination submodule is configured to determine, for any to-be-checked target, the similarity between the to-be-checked target and the corresponding sample check subtarget;

[0069] A ninth determination submodule is configured to determine a fusion similarity according to similarities between the plurality of to-be-verified targets and the corresponding sample verification sub-targets;

[0070] The tenth determining submodule is configured to determine a verification result of the image to be verified according to the fusion similarity.

[0071] According to a third aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, including:

[0072] processor;

[0073] a memory for storing processor-executable instructions;

[0074] Among them, the processor is configured to implement the steps of the image processing method provided by the first aspect of the present disclosure when executed.

[0075] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the image processing method provided in the first aspect of the present disclosure are implemented.

[0076] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0077] By acquiring an image to be inspected and a sample image, the sample image includes a sample inspection target, and determining the target to be inspected in the image to be inspected according to the sample image and a mask image for the sample inspection target, and then determining the inspection result of the image to be inspected according to the similarity between the target to be inspected and the sample inspection target. The target to be inspected can be determined in advance from the image to be inspected according to the sample image and the mask image for the sample inspection target corresponding to the sample image, and the inspection result of the image to be inspected can be determined according to the similarity between the target to be inspected and the sample inspection target, so as to reduce the interference of background information in the image to be inspected and other information irrelevant to the inspection, and improve the accuracy of image inspection.

[0078] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0080] Figure 1 The figure is a schematic diagram of an application scenario of an image processing method according to an exemplary embodiment.

[0081] Figure 2 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0082] Figure 3 is a schematic diagram showing a sample image and a mask image according to an exemplary embodiment.

[0083] Figure 4 The present invention is a flowchart showing a method for determining a target to be inspected according to an exemplary embodiment.

[0084] Figure 5 The figure is a flowchart of a method for determining a target matching point according to an exemplary embodiment.

[0085] Figure 6 It is a schematic diagram showing a forward matching according to an exemplary embodiment.

[0086] Figure 7 is a schematic diagram showing a reverse matching according to an exemplary embodiment.

[0087] Figure 8 is a schematic diagram showing a target matching point according to an exemplary embodiment.

[0088] Fig. 9 The figure is a flow chart showing a method for determining a verification result according to an exemplary embodiment.

[0089] Fig.10 is a block diagram of an image processing apparatus according to an exemplary embodiment.

[0090] Fig.11 The invention is a block diagram of a device for image processing according to an exemplary embodiment. DETAILED DESCRIPTION

[0091] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0092] It should be noted that all actions of acquiring signals, information or data in the present disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the device is located and with the authorization given by the owner of the corresponding device.

[0093] Image classification is one of the most basic tasks in the field of computer vision. Its goal is to train a classification model to correctly predict the category of an image given an image. Common scenarios include animal classification, food classification, etc. Image classification has many forms, such as multi-category image classification, multi-label image classification, fine-grained image classification, and unsupervised image classification, which are adapted to different scenarios and needs.

[0094] The development of image classification has gone through several stages, from traditional machine learning methods to the rise of deep learning, and then to various innovations and optimizations. Among them, traditional machine learning methods such as SVM (Support Vector Machine) and logistic regression require manual extraction of image features, and then send the features into the model for classification training. The model structure is relatively simple and easy to implement. It performs well on small and medium-sized data sets, but when trained on large-scale data sets, dimensionality disaster or overfitting may occur. For the AlexNet network, this is the first model that uses deep CNN (Convolutional Neural Network) for image classification on the large-scale dataset ImageNet. On the one hand, in feature extraction, the CNN model can automatically extract low-order and high-order features of the image without human interference; on the other hand, the generalization and accuracy of the CNN model on large-scale datasets are also far better than machine learning methods. Therefore, structures such as VGG (Visual Geometry Group), ResNet, and DenseNet have emerged one after another, constantly refreshing their performance on the ImageNet dataset. With the excellent performance of the Transformer structure in NLP (Natural Language Processing) tasks, attempts have been made to apply it to image classification tasks in the hope of using its powerful global attention mechanism to extract high-level semantic features of images. The pioneering work is Vision Transformer (ViT), which divides the image into several patches of fixed size, and then regards each patch as a token. The input sequence is obtained through linear projection and position encoding, and then sent to a multi-layer Transformer encoder for feature extraction, and finally an additional category token is used for classification. ViT has shown excellent performance on large-scale pre-training datasets, and has gradually surpassed ResNet to become one of the most popular backbone networks. In practical applications, in order to save time and computing resources and improve the generalization ability of the model, the above network is usually used as an image feature extractor, and then a separate model structure is designed based on the extracted features to adapt to specific classification tasks and data.

[0095] For image classification, one possible implementation is to use ViT as the basic structure and then design a module suitable for the image verification task to complete the image classification. This method can use existing knowledge to guide the model's learning process and improve the model's accuracy and interpretability; however, it requires a large amount of data annotation, which consumes human resources and time, and the annotated data has certain noise and errors, which affects the performance of the model; in addition, the versatility of the model will also be limited by the data, that is, it performs well on the training data, but poorly on other data sets.

[0096] In another possible implementation, the image classification function is realized by the image search task; specifically, the goal of the image search task is to search the image database for the most similar image to the query image given a query image. This method is based on the deep learning ResNet50 network model. All the preprocessed images in the image library and the images to be retrieved are loaded into the ResNet50 network model to extract their corresponding feature vectors, and an image feature index database is established based on the feature vectors of the images in the image library. Then, the feature vector similarity between the preprocessed images to be retrieved and all the images in the image feature index database is calculated, and finally a set number of high similarity options are obtained to return the results, thereby realizing the function of searching for the same / similar images by image. In feature vector extraction, this method uses a pre-trained CNN module for feature extraction, and its feature receptive field is limited, which is not conducive to capturing global features. And for the image verification task, this method pays more attention to the information distribution of the entire image. Overall similarity means that the images are considered similar. The performance is poor when the image background is noisy or the target to be verified is small.

[0097] In view of the above technical problems, the present disclosure provides an image processing method, device, storage medium and chip, which obtains an image to be inspected and a sample image, wherein the sample image includes a sample inspection target, and determines the target to be inspected in the image to be inspected according to the sample image and a mask image for the sample inspection target, and then determines the inspection result of the image to be inspected according to the similarity between the target to be inspected and the sample inspection target. The target to be inspected can be determined in advance from the image to be inspected according to the sample image and the mask image for the sample inspection target corresponding to the sample image, and the inspection result of the image to be inspected can be determined according to the similarity between the target to be inspected and the sample inspection target, which can reduce the interference of background information in the image to be inspected and other information irrelevant to the inspection, and improve the accuracy of image inspection.

[0098] Figure 1 is a schematic diagram of an application scenario of an image processing method according to an exemplary embodiment. Figure 1As shown, it can be used for image verification in store dressing scenarios. For various events and festivals, the headquarters can issue an event dressing guide map, which can be a sample image. Stores can arrange event dress-ups based on the event dressing guide map and upload store dress-up images. Then, the image processing method provided in this embodiment can be used to verify the dress-up images uploaded by each store to determine whether they are qualified.

[0099] Figure 2 is a flowchart of an image processing method according to an exemplary embodiment. Figure 3 is a schematic diagram showing a sample image and a mask image according to an exemplary embodiment. Figure 2 and Figure 3 As shown, the method can be applied to a server or a terminal device. For example, the terminal device can be a mobile phone, a computer, a message sending and receiving device, a tablet device, etc., and includes the following steps.

[0100] In step S201, an image to be verified and a sample image are acquired, where the sample image includes a sample verification target.

[0101] In this embodiment, the sample image may be a standard image provided, which is used as a standard for judging whether the image to be checked is qualified. The image to be checked may be an image to be checked to determine whether it is qualified. Whether the image to be checked is qualified can be determined based on the sample image. For example, for the store dressing scene, the sample image may be an activity dressing guide map, and the image to be checked may be a dress-up image uploaded by each store. Among them, the sample image includes a sample verification target, which is a standard for image verification, so as to determine whether the image to be checked meets the verification standard through the sample verification target in the sample image. If the verification standard is met, the image to be checked is determined to be qualified. If the verification standard is not met, the image to be checked is determined to be unqualified. The sample verification target can be obtained by labeling in advance. For example, for the above-mentioned store dressing scene, the given activity dressing guide map includes at least one sample verification target, and the sample verification target can be a banner, a window sticker, a floor sticker, a DP point, an activity sticker, a column wrap, a hanging flag, and a pile head. Taking banners as an example, the inspection standards are: they are hung directly above the outside of the store, and the text on the banner is the corresponding event promotional slogan.

[0102] In step S202, the target to be checked in the image to be checked is determined according to the sample image and the mask image for the sample check target.

[0103] In this embodiment, if Figure 3As shown, the mask image is the mask image corresponding to the sample image, and the mask image is the mask image for the sample verification target in the sample image. That is, the size of the mask image is the same as the size of the sample image, and the pixel points corresponding to the sample verification target in the sample image are marked as 1, and the remaining pixel points are marked as 0, so that the mask image for the sample verification target corresponding to the sample image can be obtained. According to the mask image and the sample image, the features of the sample verification target can be determined, so that according to the features of the sample verification target, the target to be verified with features similar to the sample verification target can be found from the image to be verified.

[0104] In step S203, the verification result of the image to be verified is determined according to the similarity between the object to be verified and the sample verification object.

[0105] In this embodiment, the similarity between the target to be checked and the sample check target can be determined, and the check result of the image to be checked can be determined based on the similarity, and the check result can be a qualified check or a failed check. A similarity threshold can be set. If the similarity between the target to be checked and the sample check target is greater than or equal to the similarity threshold, the check result of the image to be checked can be determined to be a qualified check. If the similarity between the target to be checked and the sample check target is less than the similarity threshold, the check result of the image to be checked can be determined to be a failed check.

[0106] In this embodiment, by acquiring an image to be inspected and a sample image, the sample image includes a sample inspection target, and according to the sample image and a mask image for the sample inspection target, the target to be inspected in the image to be inspected is determined, and then according to the similarity between the target to be inspected and the sample inspection target, the inspection result of the image to be inspected is determined. According to the sample image and the mask image for the sample inspection target corresponding to the sample image, the target to be inspected can be determined in advance from the image to be inspected, and the inspection result of the image to be inspected can be determined according to the similarity between the target to be inspected and the sample inspection target, so that the interference of background information and other information irrelevant to the inspection in the image to be inspected can be reduced, and the accuracy of image inspection can be improved.

[0107] In a possible implementation, a method for determining a target to be inspected in an image to be inspected based on a sample image and a mask image for the sample inspection target may be as follows: obtaining target matching points based on the sample image, the mask image and the image to be inspected, the target matching points being a plurality of pixel points in the image to be inspected that are most similar to the points in the mask image; and determining the target to be inspected from the image to be inspected based on the target matching points.

[0108] Figure 4 is a flow chart showing a method for determining a target to be inspected according to an exemplary embodiment. Figure 4As shown, in a possible implementation manner, determining the target to be checked in the image to be checked according to the sample image and the mask image for the sample check target may include the following steps:

[0109] In step S401, the sample image and the to-be-checked core image are processed respectively by a basic classification model to obtain a first feature vector corresponding to the sample image and a second feature vector corresponding to the to-be-checked core image. The basic classification model is a pre-trained model.

[0110] In this embodiment, the basic classification model is a pre-trained ViT model, that is, the model CLIP-ViT pre-trained on a cross-modal dataset is used as a feature extractor to enrich the extracted feature information. Specifically, the CLIP-ViT pre-trained weights are loaded, and the feature vector output of the last patch-level layer of the Transformer structure in the ViT structure is used as the feature vector of the image. The dimension of the feature vector is HW*C, where H and W represent the number of patch-levels divided by ViT, and C is the dimension of the extracted feature vector. The sample image and the image to be inspected are respectively input into the basic classification model to obtain the first feature vector corresponding to the sample image and the second feature vector corresponding to the image to be inspected.

[0111] In step S402, a similarity matrix is ​​determined according to the first eigenvector and the second eigenvector.

[0112] In this embodiment, the similarity matrix of the sample image and the image to be checked can be obtained based on the similarity calculation formula for the vector. For example, the similarity calculation formula can be:

[0113]

[0114] Among them, the S matrix dimension is HW*HW, where Represents the first eigenvector z r The feature vector of the ith patch, Represents the second eigenvector z t The feature vector of the j-th patch, S ij Indicates the similarity between the two.

[0115] In step S403, pixel points in the mask image and the image to be inspected are matched according to the similarity matrix to obtain target matching points, which are a plurality of pixel points in the image to be inspected that are most similar to the mask image.

[0116] In this embodiment, the similarity between any two pixels in the mask image and the core image to be inspected can be determined based on the similarity matrix, and a plurality of pixels in the core image to be inspected with the greatest similarity can be determined as target matching points.

[0117] In step S404, the target to be inspected is determined from the image to be inspected according to the target matching points.

[0118] In this embodiment, the target to be inspected can be determined from the image to be inspected according to the multiple pixel points in the target matching point. The sample inspection target may include multiple targets, and the target to be inspected may also include multiple corresponding targets.

[0119] In this embodiment, the features of the sample image and the image to be inspected can be extracted in advance to obtain a similarity matrix, and then based on the similarity matrix, the mask image and the image to be inspected are combined to determine the target to be inspected that matches the sample target to be inspected from the image to be inspected, so as to eliminate the influence of the background content in the image to be inspected on the inspection result.

[0120] Figure 5 is a flowchart of a method for determining a target matching point according to an exemplary embodiment. Figure 6 is a schematic diagram of a forward matching according to an exemplary embodiment. Figure 7 is a schematic diagram showing a reverse matching according to an exemplary embodiment. Figure 8 is a schematic diagram showing a target matching point according to an exemplary embodiment. Figures 5 to 8 As shown, in a possible implementation manner, matching the pixel points in the mask image and the image to be inspected according to the similarity matrix to obtain the target matching points may include the following steps:

[0121] In step S501, the similarity between the pixels in the mask image and the pixels in the image to be checked is determined according to the similarity matrix to obtain a forward similarity matrix.

[0122] In this embodiment, the pixel points in the mask image and the image to be inspected are forward matched according to the similarity matrix to obtain the target matching points. Specifically, all the points in the mask image can be obtained based on the similarity matrix. With the first eigenvector z t The similarity score of , and the forward similarity matrix S is obtained f .

[0123] In step S502, a plurality of pixel points in the image to be checked having the greatest similarity to the mask image are determined according to the forward similarity matrix to obtain a first matching point.

[0124] In this embodiment, according to the similarity of each point in the forward similarity matrix, multiple pixel points in the to-be-checked core image with the greatest similarity to the mask image are determined to obtain the first matching point in the forward matching process. f, and then obtain the L points in the image to be tested that are most similar to the sample image based on the bipartite graph matching-Hungarian matching algorithm. The code matrix of the Hungarian matching algorithm is 1-S f , that is, maximize the similarity matrix and obtain the most matching L1 points in the image to be checked like Figure 6 As shown, the dot in the right figure is the first matching point.

[0125] In step S503, the similarity between the first matching point and the pixel points in the mask image is determined according to the similarity matrix to obtain a reverse similarity matrix.

[0126] In this embodiment, reverse matching is performed based on the first matching point, and the similarity between the first matching point and the pixel point in the mask image can be determined according to the similarity matrix to obtain a reverse similarity matrix. Specifically, reverse matching is performed based on L points in the image to be tested, and the similarity between the L points in the image to be tested and the first eigenvector z is calculated. r The similarity matrix between them is used to obtain the inverse similarity matrix.

[0127] In step S504, a plurality of pixel points in the mask image having the greatest similarity to the image to be checked are determined according to the inverse similarity matrix to obtain second matching points.

[0128] In this embodiment, according to the similarity between each point of the reverse similarity matrix, multiple pixel points in the mask image with the maximum similarity to the core image to be inspected are determined to obtain the second matching point in the reverse matching process. Specifically, based on the reverse similarity matrix, the reverse similarity matrix can be maximized based on the Hungarian matching algorithm to obtain the first matching point P in the mask image with the core image to be inspected. t The most similar L2 points like Figure 7 As shown, the dot in the left figure is the second matching point.

[0129] In step S505, a target matching point is determined according to the first matching point and the second matching point.

[0130] In this embodiment, interference points can be filtered based on the first matching point and the second matching point to obtain the target matching point. Optionally, the first matching point and the second matching point can be intersected, and the pixel points that do not fall on the mask image after reverse matching can be filtered to obtain the remaining multiple pixel points with the largest similarity as the target matching point. Figure 8 As shown, the dots in the figure are the target matching points.

[0131] In this embodiment, through bidirectional matching, the pixel points in the to-be-inspected image that are similar to the sample inspection target in the sample image can be determined more accurately, thereby obtaining a more accurate to-be-inspected target.

[0132] In a possible implementation, the method for determining the nuclear target to be inspected from the nuclear image to be inspected based on the target matching points may be: determining a target area from the nuclear image to be inspected based on the target matching points, the target area being the smallest area including the target matching points; and determining the nuclear target to be inspected based on the target area.

[0133] In this embodiment, the region can be delineated in the image to be checked according to the target matching points, and all the target matching points are delineated and the delineated area is minimized as much as possible to obtain the target region. If there is one sample check target, then there is also one target region; if there are multiple target check regions, then there are also multiple target regions, that is, the number of target regions corresponds to the number of sample check targets. And according to the target region, the region boundary coordinates are determined, so that the target to be checked in the image to be checked is determined based on the region boundary coordinates.

[0134] Fig. 9 is a flow chart showing a method for determining a verification result according to an exemplary embodiment. Fig. 9 As shown, in a possible implementation manner, determining the verification result of the image to be verified according to the similarity between the object to be verified and the sample verification object may include the following steps:

[0135] In step S901, a target verification image is determined from the images to be verified according to the target to be verified.

[0136] In this embodiment, the target verification image may be cropped from the image to be verified based on the region boundary coordinates corresponding to the target to be verified. The target verification image is the image corresponding to the sample verification target.

[0137] In step S902 , a sample verification image is determined from the sample images according to the sample verification target.

[0138] In this embodiment, the sample verification target is pre-labeled, and the sample verification image can be cropped from the sample image based on the sample verification frame corresponding to the pre-labeled sample verification target or the region boundary coordinates corresponding to the sample verification target.

[0139] In step S903 , the actual similarity between the target verification image and the sample verification image is determined.

[0140] In this embodiment, after obtaining the target verification image and the sample verification image, the actual similarity between the target verification image and the sample verification image can be determined. The actual similarity can be calculated by distance or cosine similarity.

[0141] In a possible implementation, a method for determining the actual similarity may be: processing the target verification image and the sample verification image respectively through a basic classification model to obtain a third eigenvector corresponding to the target verification image and a fourth eigenvector corresponding to the sample verification image, wherein the basic classification model is a pre-trained model; and determining the actual similarity between the target verification image and the sample verification image based on the third eigenvector and the fourth eigenvector.

[0142] In this embodiment, the target verification image and the sample verification image may be preprocessed in advance, which may include data enhancement operations such as image size change and horizontal flipping. Then the preprocessed target verification image and sample verification image are input into the basic classification model, and the third feature vector corresponding to the target verification image and the fourth feature vector corresponding to the sample verification image can be output. Among them, the basic classification model is a pre-trained ViT model, that is, the model CLIP-ViT pre-trained on a cross-modal dataset is used as a feature extractor to enrich the extracted feature information. Specifically, the CLIP-ViT pre-trained weights are loaded, and the last layer output of the ViT structure is used as the feature vector of the image. The calculation formula for the actual similarity between the target verification image and the sample verification image can be:

[0143]

[0144] Where A is the third eigenvector corresponding to the target verification image, B is the fourth eigenvector corresponding to the sample verification image, cos(A,B) is the actual similarity, |A| is the modulus of the third eigenvector, and |B| is the modulus of the fourth eigenvector.

[0145] In step S904, the verification result of the image to be verified is determined according to the actual similarity.

[0146] In this embodiment, the verification result can be either qualified or unqualified. A similarity threshold can be set. If the actual similarity is greater than or equal to the similarity threshold, the verification result of the image to be verified can be determined to be qualified. If the actual similarity is less than the similarity threshold, the verification result of the image to be verified can be determined to be unqualified.

[0147] By using the pre-trained ViT model as the basic classification model to extract the feature vector of the image in the process of determining the similarity matrix and determining the actual similarity between the target verification image and the sample verification image, the pre-trained basic classification model is an existing trained model and does not need to be trained again. The entire verification process of the image to be verified can be completed without training. Only a small number of sample images need to be labeled, avoiding a large amount of manual labeling and a large amount of model training process. Moreover, the pre-trained basic classification model can have strong versatility, is not limited to the training data set, and can complete classification on data sets of multiple scenes, which can improve the versatility of the basic classification model.

[0148] In a possible implementation, there may be multiple different verification standards, and the sample image may include multiple sample sub-images, each of which may correspond to a verification standard. For example, multiple styles of verification standards may be set for different stores, and one style of verification standard corresponds to one sample sub-image. By setting multiple verification standards, the verification pass rate can be improved, and the styles of the images to be verified can be more diverse.

[0149] The method for determining the target to be inspected in the image to be inspected according to the sample image and the mask image for the sample inspection target may be: determining the multiple targets to be inspected in the image to be inspected according to multiple sample sub-images and the mask images for the sample inspection sub-targets in the multiple sample sub-images.

[0150] In this embodiment, according to any sample sub-image and the mask image for the sample verification sub-target in the sample sub-image, a target to be verified can be determined from the target to be verified image, that is, the number of targets to be verified corresponds to the number of sample sub-images. The determination method can refer to the method for determining the target to be verified in the target to be verified image according to the sample image and the mask image for the sample verification target, and will not be repeated here.

[0151] In this case, the method for determining the verification result of the image to be verified based on the similarity between the target to be verified and the sample verification target can be: for any target to be verified, determine the similarity between the target to be verified and the corresponding sample verification sub-target; determine the fusion similarity based on the similarities between multiple targets to be verified and the corresponding sample verification sub-targets; and determine the verification result of the image to be verified based on the fusion similarity.

[0152] In this embodiment, when there are multiple targets to be inspected, the similarity between any target to be inspected and the sample inspection sub-target in the corresponding sample sub-image can be determined, so that multiple targets to be inspected can obtain multiple corresponding similarities, and the multiple similarities are fused to obtain fused similarities. Optionally, the fusion method can be to directly add multiple similarities to obtain fused similarities. Alternatively, multiple similarities are weighted and summed to obtain fused similarities. Finally, the inspection result of the image to be inspected is determined based on the obtained fused similarities.

[0153] In a feasible implementation, the method for determining the verification result of the image to be inspected based on the similarity between the target to be inspected and the sample verification target may be: for any target to be inspected, determining the similarity between the target to be inspected and the corresponding sample verification sub-target; determining the verification sub-result of the image to be inspected based on the similarity corresponding to any target to be inspected; determining the verification result of the image to be inspected based on the verification sub-results corresponding to multiple targets to be inspected.

[0154] In this embodiment, when there are multiple targets to be inspected, for any target to be inspected, its similarity with the sample inspection sub-target in the corresponding sample sub-image can be determined, and the inspection sub-result corresponding to the target to be inspected can be determined based on the similarity. Thus, multiple targets to be inspected can obtain multiple corresponding inspection sub-results. And the inspection result of the image to be inspected is determined based on the multiple inspection sub-results. Optionally, a voting method can be used to merge multiple inspection sub-results to obtain the final inspection result. The inspection result can also be determined based on the number of inspection sub-results that are qualified for inspection. For example, when the number of inspection sub-results that are qualified for inspection is greater than or equal to a preset number, the inspection result is determined to be qualified for inspection, and when the number of inspection sub-results that are qualified for inspection is less than a preset number, the inspection result is determined to be unqualified for inspection.

[0155] Fig.10 is a block diagram of an image processing device according to an exemplary embodiment. Fig.10 The image processing device 1000 includes an acquisition module 1001 , a first determination module 1002 and a second determination module 1003 .

[0156] The acquisition module 1001 is configured to acquire an image to be checked and a sample image, wherein the sample image includes a sample check target;

[0157] The first determination module 1002 is configured to determine the target to be checked in the image to be checked according to the sample image and the mask image for the sample check target;

[0158] The second determination module 1003 is configured to determine the verification result of the image to be verified according to the similarity between the object to be verified and the sample verification object.

[0159] Optionally, the first determining module 1002 includes:

[0160] A first acquisition submodule is configured to process the sample image and the nuclear image to be inspected respectively through a basic classification model to obtain a first feature vector corresponding to the sample image and a second feature vector corresponding to the nuclear image to be inspected, wherein the basic classification model is a pre-trained model;

[0161] A first determining submodule is configured to determine a similarity matrix according to the first eigenvector and the second eigenvector;

[0162] A second obtaining submodule is configured to match the pixel points in the mask image and the image to be inspected according to the similarity matrix to obtain target matching points, wherein the target matching points are a plurality of pixel points in the image to be inspected that are most similar to the mask image;

[0163] The second determination submodule is configured to determine the nuclear target to be inspected from the nuclear image to be inspected according to the target matching point.

[0164] Optionally, the second obtaining submodule includes:

[0165] A first obtaining subunit is configured to determine the similarity between the pixel points in the mask image and the pixel points in the to-be-checked nuclear image according to the similarity matrix, and obtain a forward similarity matrix;

[0166] A second obtaining subunit is configured to determine, according to the forward similarity matrix, a plurality of pixel points in the to-be-checked core image that have the greatest similarity to the mask image, to obtain a first matching point;

[0167] a third obtaining subunit, configured to determine the similarity between the first matching point and the pixel points in the mask image according to the similarity matrix, and obtain an inverse similarity matrix;

[0168] A fourth obtaining subunit is configured to determine, according to the inverse similarity matrix, a plurality of pixel points in the mask image that have the greatest similarity with the to-be-checked core image, to obtain second matching points;

[0169] The first determining subunit is configured to determine the target matching point according to the first matching point and the second matching point.

[0170] Optionally, the second determining submodule includes:

[0171] A second determining subunit is configured to determine a target area from the to-be-inspected nuclear image according to the target matching point, wherein the target area is a minimum area including the target matching point;

[0172] The third determining subunit is configured to determine the target to be inspected according to the target area.

[0173] Optionally, the second determining module 1003 includes:

[0174] A third determination submodule is configured to determine a target verification image from the to-be-inspected nuclear image according to the to-be-inspected nuclear target;

[0175] a fourth determination submodule, configured to determine a sample verification image from the sample image according to the sample verification target;

[0176] a fifth determination submodule, configured to determine an actual similarity between the target verification image and the sample verification image;

[0177] The sixth determination submodule is configured to determine a verification result of the image to be verified according to the actual similarity.

[0178] Optionally, the fifth determining submodule includes:

[0179] a fifth obtaining subunit, configured to process the target verification image and the sample verification image respectively through a basic classification model to obtain a third feature vector corresponding to the target verification image and a fourth feature vector corresponding to the sample verification image, wherein the basic classification model is a pre-trained model;

[0180] The fourth determining subunit is configured to determine the actual similarity between the target verification image and the sample verification image according to the third feature vector and the fourth feature vector.

[0181] Optionally, the sample image includes a plurality of sample sub-images, each of which includes a sample verification sub-target;

[0182] The first determining module 1002 includes:

[0183] a seventh determination submodule, configured to determine a plurality of to-be-checked targets in the to-be-checked target image according to the plurality of sample sub-images and mask images for the sample check sub-targets in the plurality of sample sub-images;

[0184] The second determining module 1003 includes:

[0185] An eighth determination submodule is configured to determine, for any to-be-checked target, the similarity between the to-be-checked target and the corresponding sample check subtarget;

[0186] A ninth determination submodule is configured to determine a fusion similarity according to similarities between the plurality of to-be-verified targets and the corresponding sample verification sub-targets;

[0187] The tenth determining submodule is configured to determine a verification result of the image to be verified according to the fusion similarity.

[0188] Regarding the image processing device 1000 in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0189] The present disclosure also provides a computer-readable storage medium on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the image processing method provided by the present disclosure are implemented.

[0190] Fig.11 1 is a block diagram of a device for image processing according to an exemplary embodiment. For example, the device 1100 may be provided as a server. Fig.11 The apparatus 1100 includes a processing component 1122, which further includes one or more processors, and a memory resource represented by a memory 1132 for storing instructions executable by the processing component 1122, such as an application. The application stored in the memory 1132 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1122 is configured to execute the instructions to perform the above-mentioned image processing method.

[0191] The device 1100 may also include a power supply component 1126 configured to perform power management of the device 1100, a wired or wireless network interface 1150 configured to connect the device 1100 to a network, and an input / output interface 1158. The device 1100 may operate based on an operating system stored in the memory 1132, such as Windows Server 2003. TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM or similar.

[0192] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0193] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: Acquire an image to be checked and a sample image, wherein the sample image includes a sample check target; Determine the target to be checked in the image to be checked according to the sample image and the mask image for the sample check target; The verification result of the image to be verified is determined according to the similarity between the object to be verified and the sample verification object.

2. The image processing method according to claim 1, characterized in that: The step of determining the target to be checked in the image to be checked according to the sample image and the mask image for the sample check target comprises: The sample image and the nuclear image to be inspected are processed respectively by a basic classification model to obtain a first feature vector corresponding to the sample image and a second feature vector corresponding to the nuclear image to be inspected, wherein the basic classification model is a pre-trained model; Determine a similarity matrix according to the first eigenvector and the second eigenvector; According to the similarity matrix, pixel points in the mask image and the image to be inspected are matched to obtain target matching points, where the target matching points are a plurality of pixel points in the image to be inspected that are most similar to the mask image; According to the target matching points, the nuclear target to be inspected is determined from the nuclear image to be inspected.

3. The image processing method according to claim 2, characterized in that: According to the similarity matrix, matching the pixel points in the mask image and the image to be checked to obtain target matching points includes: Determine the similarity between the pixel points in the mask image and the pixel points in the to-be-checked core image according to the similarity matrix to obtain a forward similarity matrix; According to the forward similarity matrix, a plurality of pixel points in the to-be-checked core image having the greatest similarity to the mask image are determined to obtain a first matching point; Determine the similarity between the first matching point and the pixel points in the mask image according to the similarity matrix to obtain a reverse similarity matrix; According to the inverse similarity matrix, a plurality of pixel points in the mask image having the greatest similarity to the image to be checked are determined to obtain second matching points; The target matching point is determined according to the first matching point and the second matching point.

4. The image processing method according to claim 2, characterized in that: The step of determining the nuclear target to be inspected from the nuclear image to be inspected according to the target matching point includes: According to the target matching point, a target area is determined from the nuclear image to be inspected, wherein the target area is the smallest area including the target matching point; The target to be inspected is determined according to the target area.

5. The image processing method according to claim 1, characterized in that: The step of determining the verification result of the image to be verified according to the similarity between the object to be verified and the sample verification object includes: According to the target to be inspected, a target inspection image is determined from the image to be inspected; Determining a sample verification image from the sample image according to the sample verification target; Determining an actual similarity between the target verification image and the sample verification image; The verification result of the image to be verified is determined according to the actual similarity.

6. The image processing method according to claim 5, characterized in that: The determining the actual similarity between the target verification image and the sample verification image comprises: The target verification image and the sample verification image are processed respectively by a basic classification model to obtain a third feature vector corresponding to the target verification image and a fourth feature vector corresponding to the sample verification image, wherein the basic classification model is a pre-trained model; The actual similarity between the target verification image and the sample verification image is determined according to the third eigenvector and the fourth eigenvector.

7. The image processing method according to any one of claims 1 to 6, characterized in that: The sample image includes a plurality of sample sub-images, each of which includes a sample verification sub-target; The step of determining the target to be checked in the image to be checked according to the sample image and the mask image for the sample check target comprises: Determine a plurality of to-be-checked targets in the to-be-checked image according to the plurality of sample sub-images and mask images for the sample check sub-targets in the plurality of sample sub-images; The step of determining the verification result of the image to be verified according to the similarity between the object to be verified and the sample verification object includes: For any target to be checked, determine the similarity between the target to be checked and the corresponding sample check sub-target; Determine the fusion similarity according to the similarity between the multiple to-be-checked targets and the corresponding sample check sub-targets; The verification result of the image to be verified is determined according to the fusion similarity.

8. An image processing device, characterized in that: include: An acquisition module is configured to acquire an image to be checked and a sample image, wherein the sample image includes a sample check target; A first determination module is configured to determine the target to be checked in the image to be checked according to the sample image and the mask image for the sample check target; The second determination module is configured to determine the verification result of the image to be verified according to the similarity between the object to be verified and the sample verification object.

9. An image processing device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the steps of the image processing method according to any one of claims 1 to 7 when executed.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the image processing method described in any one of claims 1 to 7 are implemented.

Images