Image matching method and device for protein immunoblot band images

Abstract

The present application relates to a kind of image matching method and device for protein immunoblotting strip image, the method includes the following steps: 1) obtains protein immunoblotting image, generates the gray image, binary image and edge graph of the image;2) respectively extracting the feature point of the gray image, binary image and edge graph, matching and removing redundancy;3) the multi-objective matching of the feature vector set corresponding to different images is carried out, and the image splicing of matching successful image is carried out.The present application can be accurately, quickly matched in large-scale data set to the WB image without texture, simple content.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image matching method and apparatus for protein immunoblot band images. Background Technology

[0002] Image matching is a common requirement, and the methods are relatively mature, divided into traditional machine learning matching and deep learning matching techniques. The basic process of both methods is consistent, requiring two steps: feature point detection and feature point extraction. Traditional machine learning methods are based on planning to find feature points, while deep learning methods are based on training to find feature points. Feature points are often points that differ significantly from surrounding pixels. These points, combined with their surrounding pixels, contain rich information and are generally unique and distinguishable. After finding feature points, feature vector extraction is performed, that is, encoding the information of the feature point and its surrounding pixels into a unique feature vector. Finally, matching is performed using all feature vectors in the image; therefore, a sufficient number of distinguishable feature points is crucial for image matching.

[0003] For images with simple content or lacking rich texture, such as WB images, it is difficult to extract enough distinguishable keypoints, or even if enough keypoints are found, the feature representation capability is insufficient. In WB images, the differences between bands are relatively subtle, and pure image matching cannot distinguish between extremely similar bands. These bands are different, but they may also be very similar, with only slight differences in shape, resulting in a small number of feature points and low distinguishability. Summary of the Invention

[0004] This invention addresses the technical problems existing in the prior art by providing an image matching method and apparatus for protein immunoblot images.

[0005] To address the aforementioned technical problems, in a first aspect, embodiments of the present invention provide an image matching method for protein immunoblot band images, comprising:

[0006] Step S1: Obtain protein immunoblot WB band images and generate grayscale, binary, and edge maps of the WB band images;

[0007] Step S2: Extract SIFT feature points, SURF feature points, and ORB feature points from the grayscale image, binary image, and edge image respectively, construct a feature vector set, and perform matching and deduplication on the SIFT feature points, SURF feature points, and ORB feature points in the feature set;

[0008] Step S3: Perform multi-object matching on the feature vector sets corresponding to different WB strip images. The multi-object matching includes region matching, feature matching, contour matching, and gray-scale matching; perform image stitching on the WB strip images with successful matching.

[0009] Preferably, the step S1 specifically includes:

[0010] Obtain the WB strip images that need to be image-matched, extract the gray-scale images of each of the WB strip images, extract the binary images of the WB strip images based on the otsu segmentation method, and extract the edge images of the WB strip images based on the canny edge detection method.

[0011] Preferably, the step S2 specifically includes:

[0012] Perform feature point detection on the gray-scale image, the binary image, and the edge image respectively. Find local extrema according to the differences of Gaussian blurred images at different scales to extract the sift feature points of the gray-scale image, the binary image, and the edge image respectively; detect the surf feature points of the gray-scale image, the binary image, and the edge image based on the determinant value of the Hessian matrix; extract the orb feature points of the gray-scale image, the binary image, and the edge image based on the FAST feature detection;

[0013] Based on the positions of the sift feature points, the surf feature points, and the orb feature points, perform feature point duplicate removal processing, and express the extracted sift feature points, surf feature points, and orb feature points as feature vectors based on the sift feature extraction method to construct a feature vector set.

[0014] Preferably, in the step S3, the region matching specifically includes:

[0015] Match the feature vector sets of the first WB strip image and the second WB strip image based on the ratio test method ratio test. For the first feature vector in the first WB strip image, find the two second feature vectors in the second WB strip image with the closest Euclidean distance to the first feature vector; d1 / d2 < ratio, where d1 and d2 are the Euclidean distances between the two second feature vectors and the first feature vector respectively, and d1 < d2;

[0016] Calculate the matching degree match ratio of the first WB strip image and the second WB strip image = M / min (the number of first feature vectors in the feature vector set of the first WB strip image, the number of second feature vectors in the feature vector set of the second WB strip image), where M is the number of feature vectors obtained by matching the feature vector sets of the first WB strip image and the second WB strip image based on the ratio test method ratio test;

[0017] The matched feature vectors are subjected to bidirectional deduplication, and the rectangular closure of the matched feature vectors is taken as the matching region.

[0018] The feature matching specifically includes:

[0019] The homography matrix of the matching regions of the first WB strip image and the second WB strip image is calculated based on the random sample consistency method to determine whether the actual regions of the feature matching of the first WB strip image and the second WB strip image match.

[0020] Preferably, in step S3, the contour matching specifically includes:

[0021] The first WB strip image after feature matching is segmented to obtain patch_A, and the second WB strip image is segmented to obtain patch_B;

[0022] Contours are obtained from the segmented patch_A and patch_B respectively; the absolute value of the difference between the number of contours in contour_A and the number of contours in contour_B is less than or equal to 2; contour matching is performed one by one between contours in contour_A and contour_B to obtain a contour matching matrix ms of size m*n.

[0023] The Hungarian algorithm based on minimum weight matching of bipartite graphs is used to calculate the contour matching matrix. The maximum cost matching value ms_max in the minimum cost matching results is taken as the contour matching degree. If the contour matching degree is less than the preset contour matching threshold, the matching is considered successful.

[0024] Preferably, in step S3, the grayscale matching specifically includes:

[0025] Find the intersection region patch_mask of patch_A_seg and patch_B_seg obtained by contour matching of the first WB strip image and the second WB strip image;

[0026] Based on patch_mask, calculate the absolute difference between the grayscale pixels at all corresponding positions in patch_A and patch_B, and then sum them to obtain diff_abs_sum;

[0027] If the diff_abs_sum is determined to be less than the preset pixel threshold, then the match is considered successful.

[0028] Preferably, step S3 involves image stitching of the successfully matched WB band images, specifically including:

[0029] The transformation matrix H from the first WB strip image to the second WB strip image is obtained using the findHomography function in OpenCV. A perspective transformation based on H is performed on the four corners of the first WB strip image, and the coordinates of the four corners of the second WB strip image are concatenated to obtain the translation transformation matrix Ht.

[0030] A' is obtained by performing a perspective transformation on the first WB strip image based on H. Then, a translation transformation based on Ht is performed on A' and the second WB strip image to ensure that the image coordinates are positive, resulting in A” and B’.

[0031] Find the similar image regions patch_A and patch_B in A” and B’, and then stitch the images together.

[0032] Secondly, embodiments of the present invention provide an image matching device for protein immunoblot band images, comprising:

[0033] The image preprocessing module acquires Western blot (WB) images of proteins and generates grayscale, binary, and edge maps of the WB images.

[0034] The feature point extraction module extracts SIFT, SURF, and ORB feature points from the grayscale image, binary image, and edge image respectively, constructs a feature vector set, and performs matching and deduplication on the SIFT, SURF, and ORB feature points in the feature set.

[0035] The image matching module performs multi-target matching on the feature vector sets corresponding to different WB strip images. The multi-target matching includes region matching, feature matching, contour matching, and grayscale matching. The successfully matched WB strip images are then stitched together.

[0036] Thirdly, embodiments of the present invention provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the image matching method for protein immunoblot images as described in the first aspect of the present invention.

[0037] Fourthly, embodiments of the present invention provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the image matching method for protein immunoblot images as described in the first aspect of the present invention.

[0038] The beneficial effects of this invention are as follows: First, a grayscale image, a binary image based on Otsu segmentation, and an edge image based on Canny edge detection are generated from the original image. Then, feature point detection is performed on these three generated images. Here, three different feature point detections are performed on each generated image. It can extract a sufficient number of distinguishable key points, find enough key points, and has sufficient feature representation capability. For WB strip images, it can extract subtle differences between strips and is highly sensitive to small differences while ensuring matching effect. Attached Figure Description

[0039] Figure 1 This is a flowchart of an image matching method for protein immunoblot images according to an embodiment of the present invention;

[0040] Figure 2 This is a flowchart of an image matching method for protein immunoblot images according to an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of the feature point feature extraction flow graph according to an embodiment of the present invention;

[0042] Figure 4 This is a schematic diagram of region matching according to an embodiment of the present invention;

[0043] Figure 5 This is a schematic diagram of the stitched image according to an embodiment of the present invention;

[0044] Figure 6 This is a schematic diagram of an electronic device according to an embodiment of the present invention;

[0045] Figure 7 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] Figures 1-2 This invention provides an image matching method for protein immunoblot images, comprising:

[0048] Step S1: Obtain protein immunoblot WB band images and generate grayscale, binary and edge maps of the WB band images;

[0049] Since WB (Wideband) images are simply black bands against a white background, their content is simple and lacks rich texture variations. Directly using common feature point detection methods fails to detect enough matching feature points, leading to the failure of subsequent feature matching. To generate sufficient expressive feature points, the number of feature points must be increased while ensuring their quality, i.e., their discriminability, is not too low. In this embodiment of the invention, since WB images are all black and white, the main distinguishing clue lies in the shape. First, a grayscale image, a binary image based on Otsu segmentation, and an edge image based on Canny edge detection are generated from the original image. Feature point detection is then performed on these three generated images.

[0050] Step S2: Extract SIFT feature points, SURF feature points, and ORB feature points from the grayscale image, binary image, and edge image respectively, construct a feature vector set, and perform matching and deduplication on the SIFT feature points, SURF feature points, and ORB feature points in the feature set;

[0051] For each generated image, three different feature point detection methods were applied: SIFT, SURF, and ORB. SIFT feature point detection works by finding local extrema based on the differences in Gaussian blurred images at different scales; SURF feature point detection uses the determinant of the Hessian matrix; and ORB feature point detection is based on the well-known FAST feature detection method. These three methods use different algorithms and are applied to images with three different processing methods, so the detected feature points are mostly concentrated in edge regions.

[0052] like Figure 3 As shown, a grayscale image img_g, an Otsu segmentation image img_o, and a Canny edge detection image img_c are generated from the original image. Then, feature point detection is performed on the generated images using OpenCV's built-in SIFT, SURF, and ORB functions, resulting in kp_sift_g, kp_surf_g, and kp_orb_g for the grayscale image img_g, kp_sift_o, kp_surf_o, and kp_orb_o for the Otsu segmentation image img_o, and kp_sift_c, kp_surf_c, and kp_orb_c for the Canny edge detection image img_c. Finally, all feature points are deduplicated based on their coordinate positions.

[0053] For the obtained feature points, the SIFT algorithm is used for feature extraction. Of course, SURF or ORB can also be used. To ensure good enough results, SIFT is used in this embodiment to transform the feature points into 128-dimensional feature vectors. The number of feature points in each image varies according to the image content, so the number of feature vectors also varies. SIFT, SURF, and ORB are all functions provided by OpenCV. The version of OpenCV used in this embodiment is 3.4.2.16.

[0054] Step S3: Perform multi-object matching on the feature vector sets corresponding to different WB stripe images. The multi-object matching includes region matching, feature matching, contour matching, and grayscale matching; perform image stitching on the WB stripe images with successful matching.

[0055] During the process of matching the feature vector sets corresponding to two images, the matching degree is used to filter feature points. The present invention adopts the ratio test to eliminate false positive matches. The region matching specifically includes:

[0056] Based on the ratio test method (ratio test), match the feature vector sets of the first WB stripe image and the second WB stripe image. For the first feature vector in the first WB stripe image, find the two second feature vectors in the second WB stripe image with the closest Euclidean distance to the first feature vector; d1 / d2 < ratio, where d1 and d2 are the Euclidean distances between the two second feature vectors and the first feature vector respectively, and d1 < d2; calculate the matching degree of the first WB stripe image and the second WB stripe image, match ratio = M / min (the number of first feature vectors in the feature vector set of the first WB stripe image, the number of second feature vectors in the feature vector set of the second WB stripe image), where M is the number of feature vectors obtained by matching the feature vector sets of the first WB stripe image and the second WB stripe image based on the ratio test method (ratio test); ratio is an empirical value. At the same time, calculate the matching degree for the filtered set of matching points. According to the matching degree, incorrect matches can be filtered. The matching degree threshold can be obtained by testing on the test set. Because there are still cases where, even though there are many feature point matches, there are still some differences in the shapes of some parts of the WB stripes, that is, using only the ratio test and match ratio cannot distinguish the subtle differences in the shapes of the WB stripes, because image matching is based on feature points rather than shapes. In view of this situation, while using the ratio test and match ratio to ensure the recall rate, the accuracy of the matching is improved based on post-processing. According to the pre-prepared test set, when ensuring the recall rate, the matching degree threshold is taken as 0.0808. If the matching degree is less than the threshold, the image pair is considered unmatched and the process ends directly; otherwise, it is necessary to enter the next judgment.

[0057] The matched feature vectors undergo bidirectional deduplication, and the rectangular closure of the matched feature vectors is used as the matching region. Bidirectional deduplication is also performed on the matched points; for points with one-to-many relationships, only one matching point is randomly retained. Then, the bounding rectangle of the deduplicated points is calculated, i.e., the rectangular closure of the matching points is used as the matching region.

[0058] Based on the above embodiments, the feature matching specifically includes:

[0059] The homography matrix of the matching regions of the first WB strip image and the second WB strip image is calculated based on the random sample consistency method to determine whether the actual regions of the feature matching of the first WB strip image and the second WB strip image match.

[0060] The feature matching process is similar to image feature point detection, feature point extraction, and feature matching, but it excludes a large number of interference regions, focusing only on regions that have been found to potentially match. Furthermore, the threshold for the match ratio is different, for example, set to 0.0070. Image pairs with a match ratio less than the threshold are considered mismatched and the process ends immediately; otherwise, further judgment is required.

[0061] If a match is found, the homography matrix is ​​calculated. Since two image blocks may have partially similar regions (e.g., the bands in a WB image are truncated), it's necessary to calculate the actual regions where features from the two images match. This requires calculating the homography matrix, also known as the perspective transformation matrix. Because the matching feature vectors after the ratio test may contain other errors, RANSAC (Random Sample Consensus) is used to find the optimal homography matrix. The transformation matrix H can be obtained using the `findHomography` function in OpenCV, which represents H*A→B, transforming image A (the first WB band image) into image B (the second WB band image).

[0062] Based on the above embodiments, to avoid errors in feature matching, further matching is performed on similar regions of the feature matching. For WB strip images, shape or contour is the most discriminative feature, so contour matching is used for post-processing. The contour matching specifically includes:

[0063] The first WB strip image after feature matching is segmented to obtain patch_A, and the second WB strip image is segmented to obtain patch_B. First, patch_A and patch_B are segmented. Traditional image segmentation will always make mistakes in some cases, so a WB strip segmentation model is trained using UNet, and then UNet is used to segment patch_A and patch_B.

[0064] Contours are extracted from the segmented patch_A and patch_B respectively to obtain contour_A and contour_B. The absolute value of the difference between the number of contours in contour_A and the number of contours in contour_B is constrained to be less than or equal to 2. Contour matching is performed one by one between the contours in contour_A and contour_B to obtain an m*n contour matching matrix ms. Contours are extracted from the segmented image to obtain contour_A and contour_B respectively, with the absolute value of the difference between the number of contours in contour_A and the number of contours in contour_B being constrained to be less than or equal to 2. This is because there may be slightly extra contours at the ends of stripes in patch_A or patch_B. After satisfying the constraint, contour matching is performed one by one between the contours in contour_A and contour_B to obtain an m*n contour matching matrix ms. The contour matching uses the matchShapes function in OpenCV.

[0065] The Hungarian algorithm based on minimum weight matching of bipartite graphs is used to calculate the contour matching matrix. The maximum cost matching value ms_max in the minimum cost matching results is taken as the contour matching degree. If the contour matching degree is less than a preset contour matching threshold, the match is considered successful. The contour matching matrix is ​​calculated using the Hungarian algorithm based on minimum weight matching of bipartite graphs, and the maximum cost matching value ms_max in the minimum cost matching results is taken as the contour matching degree. This minimum and maximum value is chosen for the stability of the contour matching results. In this embodiment, the contour matching threshold is selected as 100 based on the test set to ensure recall. If ms_max is less than 100, the match is considered reliable, and the process ends directly; otherwise, further post-processing is required.

[0066] Based on the above embodiments, for cases where contour matching is uncertain, it is necessary to analyze whether there are significant differences in grayscale distribution. This is because there may be cases where the WB bands are not simply different shades, but rather one has a grayscale gradient while the other does not. For cases with significant differences in grayscale distribution, pixel difference constraints can be used for differentiation. In step S3, the grayscale matching specifically includes:

[0067] Find the intersection region patch_mask of patch_A_seg and patch_B_seg obtained by contour matching of the first WB strip image and the second WB strip image;

[0068] The absolute difference between the grayscale pixels at all corresponding positions in patch_A and patch_B is calculated based on patch_mask, and then summed to obtain diff_abs_sum. The advantage of doing this is that it can accurately locate the comparison area, avoid interference from other pixels, take into account positional factors, and increase the stability of diff_abs_sum.

[0069] If the sum of diff_abs_sum is less than a preset pixel threshold, the match is considered successful. The threshold is also determined using a test set to ensure accuracy as much as possible. In this embodiment, the threshold for the sum of the absolute values ​​of pixel differences is the number of non-zero elements in patch_mask * 20, indicating that the minimum change for each pixel is 20. If the change is less than the minimum change, it is considered that there is no grayscale transformation.

[0070] Based on the above embodiments, with the transformation matrix H, Figure A and Figure B can be transformed. Step S3 involves image stitching of the successfully matched WB strip images, specifically including:

[0071] Translation transformation is performed first because the image coordinates are based on the top left corner of the image as the origin, the horizontal direction as the x-axis, and the vertical direction as the y-axis. Coordinates less than 0 cannot be displayed. This ensures that the x or y coordinates of all image pixels are greater than 0 after the transformation. In the above steps, the transformation matrix H for transforming the first WB strip image (image A) to the second WB strip image (image B) has been obtained based on the findHomography function in OpenCV. The perspective transformation based on H is performed on the four corners of the first WB strip image, and then the coordinates of the four corners of the first WB strip image are concatenated with those of the second WB strip image. Then, xmin, ymin, xmax, and ymax are calculated. The size of the concatenated image after transformation is (xmax-xmin)*(ymax-ymin). Thus, the translation transformation matrix Ht = [[1,0,-xmin],[0,1,-xmax],[0,0,1] can be constructed.

[0072] Image transformation: A' is obtained by performing a perspective transformation on the first WB strip image based on H. Then, a translation transformation based on Ht is performed on A' and the second WB strip image to ensure that the image coordinates are positive, resulting in A” and B’.

[0073] The masking process identifies similar image regions patch_A and patch_B in images A” and B’, and then stitches the images together. Assuming a mask image of the same size as A· / B is constructed, it undergoes the same transformations as in the image transformation steps for mask_A and mask_B, and the intersection region mask_and is calculated. Then, mask_and is transformed back to obtain raw_mask_A and raw_mask_B, where raw_mask_A represents the intersection region in image A, and raw_mask_B represents the intersection region in image B. The masked regions are then removed, i.e., patch_A = A[raw_mask_A], patch_B = B[raw_mask_B]. This process yields similar image regions patch_A and patch_B in images A and B, which are then used as input for post-processing.

[0074] If the feature matching based on the ratio test and match ratio is similar, then the optimal homography matrix of the matching images is calculated using the RANSAC method, and the images are then stitched together using the homography matrix. The image stitching process is as follows: Figure 4 , Figure 5 As shown, firstly, the boundingRect of the intersecting regions of the stripes in the image after homography matrix transformation is calculated as a mask. Then, the mask is transformed back to its corresponding original image region. The original image regions corresponding to the mask are used as input for contour matching. These two sub-image regions are segmented into binary images based on the same threshold, which is the average of the Otsu segmentation thresholds for the two sub-images. This avoids slight differences in boundary contours due to different thresholds. Since there may be more than one contour in an image, contour similarity is calculated by matching all contours in both images one by one. The similarity results are then calculated using the Hungarian algorithm based on minimum weight matching of bipartite images. The matching value with the highest cost among the minimum cost matching results is taken as the contour matching degree. This is done to ensure the stability of the contour matching results and increase its generalization ability. The contour matching degree threshold is also selected based on the test set while ensuring recall. Finally, for images that have already undergone contour matching, there may still be cases where the contours are the same, but not in different shades; one has a grayscale gradient while the other does not. In such cases, the sum of the absolute values ​​of the corresponding grayscale pixel differences is calculated based on the intersecting sub-image regions. The pixel difference threshold is determined based on the test set at the optimal accuracy. In summary, through the above post-processing, the system can be highly sensitive to minute differences while maintaining matching accuracy.

[0075] This invention also provides an image matching device for protein immunoblot images, based on the image matching method for protein immunoblot images described in the above embodiments, including:

[0076] The image preprocessing module acquires Western blot (WB) images of proteins and generates grayscale, binary, and edge maps of the WB images.

[0077] The feature point extraction module extracts SIFT, SURF, and ORB feature points from the grayscale image, binary image, and edge image respectively, constructs a feature vector set, and performs matching and deduplication on the SIFT, SURF, and ORB feature points in the feature set.

[0078] The image matching module performs multi-target matching on the feature vector sets corresponding to different WB strip images. The multi-target matching includes region matching, feature matching, contour matching, and grayscale matching. The successfully matched WB strip images are then stitched together.

[0079] Figure 6 This is a schematic diagram illustrating an embodiment of the electronic device provided in this invention. For example... Figure 6 As shown, this embodiment of the invention provides an electronic device 500, including a memory 510, a processor 520, and a computer program 511 stored in the memory 520 and executable on the processor 520. When the processor 520 executes the computer program 511, it performs the following steps:

[0080] Step S1: Obtain protein immunoblot WB band images and generate grayscale, binary, and edge maps of the WB band images;

[0081] Step S2: Extract SIFT feature points, SURF feature points, and ORB feature points from the grayscale image, binary image, and edge image respectively, construct a feature vector set, and perform matching and deduplication on the SIFT feature points, SURF feature points, and ORB feature points in the feature set;

[0082] Step S3: Perform multi-target matching on the feature vector sets corresponding to different WB strip images. The multi-target matching includes region matching, feature matching, contour matching, and grayscale matching. Then, stitch the successfully matched WB strip images together.

[0083] Please see Figure 7 , Figure 7 This is a schematic diagram illustrating an embodiment of a computer-readable storage medium provided by an embodiment of the present invention. For example... Figure 7 As shown, this embodiment provides a computer-readable storage medium 600, on which a computer program 611 is stored. When the computer program 611 is executed by a processor, it performs the following steps:

[0084] Step S1: Obtain protein immunoblot WB band images and generate grayscale, binary, and edge maps of the WB band images;

[0085] Step S2: Extract SIFT feature points, SURF feature points, and ORB feature points from the grayscale image, binary image, and edge image respectively, construct a feature vector set, and perform matching and deduplication on the SIFT feature points, SURF feature points, and ORB feature points in the feature set;

[0086] Step S3: Perform multi-target matching on the feature vector sets corresponding to different WB strip images. The multi-target matching includes region matching, feature matching, contour matching, and grayscale matching. Then, stitch the successfully matched WB strip images together.

[0087] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0088] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0089] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0090] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0091] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0092] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0093] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An image matching method for protein immunoblot band images, characterized in that, include: Step S1: Obtain protein immunoblot WB band images and generate grayscale, binary, and edge maps of the WB band images; Step S2: Extract the SIFT, SURF, and ORB feature points from the grayscale image, the binary image, and the edge image, respectively, to construct a feature vector set, and perform matching and deduplication on the SIFT, SURF, and ORB feature points in the feature vector set. Step S3: Perform multi-target matching on the feature vector sets corresponding to different WB strip images. The multi-target matching includes region matching, feature matching, contour matching, and grayscale matching. Then, stitch the successfully matched WB strip images together.

2. The image matching method for protein immunoblot images according to claim 1, characterized in that, Step S1 specifically includes: acquiring WB strip images that need to be matched, extracting grayscale images of each WB strip image, extracting binary images of the WB strip images based on the Otsu segmentation method, and extracting edge images of the WB strip images based on the Canny edge detection method.

3. The image matching method for protein immunoblot images according to claim 1, characterized in that, Step S2 specifically includes: performing feature point detection on the grayscale image, the binary image, and the edge image respectively; finding local extrema based on the differences in Gaussian blurred images at different scales to extract SIFT feature points of the grayscale image, the binary image, and the edge image respectively; detecting SURF feature points of the grayscale image, the binary image, and the edge image based on the determinant value of the Hessian matrix; and extracting ORB feature points of the grayscale image, the binary image, and the edge image based on FAST feature detection. Based on the positions of the SIFT feature points, the SURF feature points, and the ORB feature points, feature point deduplication is performed. Based on the SIFT feature extraction method, the extracted SIFT feature points, the SURF feature points, and the ORB feature points are expressed as feature vectors, and a feature vector set is constructed.

4. The image matching method for protein immunoblot band images according to claim 1, characterized in that, In the step S3, the region matching specifically includes: matching the feature vector sets of the first WB stripe image and the second WB stripe image based on the ratio test method (ratio test), for the first feature vector in the first WB stripe image, finding the two second feature vectors in the second WB stripe image with the closest Euclidean distance to the first feature vector; d1 / d2 < ratio, where d1 and d2 are the Euclidean distances between the two second feature vectors and the first feature vector respectively, and d1 < d2; calculating the matching degree of the first WB stripe image and the second WB stripe image, match ratio = M / min (the number of first feature vectors in the feature vector set of the first WB stripe image, the number of second feature vectors in the feature vector set of the second WB stripe image), where M is the number of feature vectors obtained by matching the feature vector sets of the first WB stripe image and the second WB stripe image based on the ratio test method (ratio test); performing a two-way de-duplication process on the matched feature vectors, and obtaining the rectangular closure of the matched feature vectors as the matching region; the feature matching specifically includes: calculating the homography matrix of the matching regions of the first WB stripe image and the second WB stripe image based on the random sample consensus method to determine whether the actual regions of the feature matching of the first WB stripe image and the second WB stripe image match.

5. The image matching method for protein immunoblot band images according to claim 4, characterized in that, In the step S3, the contour matching specifically includes: segmenting the first WB stripe image after feature matching to obtain patch_A, and segmenting the second WB stripe image to obtain patch_B; Contours are obtained from the segmented patch_A and patch_B respectively; the absolute value of the difference between the number of contours in contour_A and the number of contours in contour_B is less than or equal to 2; contour matching is performed one by one between the contours in contour_A and contour_B to obtain m. A contour matching matrix of size n, ms; Calculating the contour matching matrix based on the Hungarian algorithm of the minimum weight matching of the bipartite graph, and taking the maximum matching value ms_max in the minimum cost matching result as the contour matching degree; if the contour matching degree is less than the preset contour matching threshold, it is determined that the matching is successful.

6. The image matching method for protein immunoblot images according to claim 5, characterized in that, In the step S3, the gray-scale matching specifically includes: obtaining the intersection region patch_mask of patch_A_seg and patch_B_seg obtained by contour matching of the first WB stripe image and the second WB stripe image; Based on patch_mask, taking the absolute value of the difference between the gray-scale pixels at all corresponding positions in patch_A and patch_B, and then summing to obtain diff_abs_sum; If it is determined that the diff_abs_sum is less than the preset pixel threshold, it is determined that the matching is successful.

7. The image matching method for protein immunoblot images according to claim 6, characterized in that, In the step S3, for the WB stripe images with successful matching, image stitching specifically includes: obtaining the transformation matrix H from the first WB stripe image to the second WB stripe image based on the findHomography function in opencv, performing a perspective transformation based on H on the four corners of the first WB stripe image, and splicing with the four-corner coordinates of the second WB stripe image to obtain the translation transformation matrix Ht; A' is obtained by performing a perspective transformation on the first WB strip image based on H. Then, a translation transformation based on Ht is performed on A' and the second WB strip image to ensure that the image coordinates are positive, resulting in A” and B’. Find the similar image regions patch_A and patch_B in A” and B’, and then stitch the images together.

8. An image matching device for protein immunoblot band images, characterized in that, include: The image preprocessing module acquires Western blot (WB) images of proteins and generates grayscale, binary, and edge maps of the WB images. The feature point extraction module extracts SIFT, SURF, and ORB feature points from the grayscale image, the binary image, and the edge image, respectively, constructs a feature vector set, and performs matching and deduplication on the SIFT, SURF, and ORB feature points in the feature vector set. The image matching module performs multi-target matching on the feature vector sets corresponding to different WB strip images. The multi-target matching includes region matching, feature matching, contour matching, and grayscale matching. The successfully matched WB strip images are then stitched together.

9. An electronic device, characterized in that, include: Memory, used to store computer software programs; A processor is configured to read and execute the computer software program, thereby implementing claim 1. The image matching method for protein immunoblot images as described in any one of the seven claims.

10. A non-transitory computer-readable storage medium, characterized in that, The storage medium stores information for implementing claim 1.

7. A computer software program for the image matching method for protein immunoblot images as described in any one of the claims.

Images