A flag key point extraction method, system and device

Abstract

The application discloses a flag key point extraction method, system and device, and belongs to the technical field of image content auditing. The method comprises the following steps: 1) deformed flag generation: collecting various flag images, and performing image cleaning and screening; for each cleaned and screened image, a plurality of corresponding deformed flag images are generated through deformation, and respective deformation parameters are retained; 2) key point detection and matching: using a key point detection and matching method to perform key point detection and matching on the flag original image and the corresponding deformed image in step 1), so as to obtain point coordinates corresponding to the matching of the deformed image and the original image key points; 3) matching heat map generation; and 4) model training and optimization. The application can obtain key points of a flag which are obvious and do not change with deformation distortion, improve the accuracy of flag feature matching, and facilitate identification of the flag through feature matching.

Description

Technical Field

[0001] This invention relates to the field of image content review technology, specifically a method, system, and apparatus for extracting key points of a flag. Background Technology

[0002] Flag recognition technology is widely used in image content moderation. Major media platforms generate massive amounts of video and image data, which may contain sensitive or prohibited flags. Because the amount of target data to be detected is small, and flags are non-rigid targets that are prone to distortion, traditional feature detection and matching methods perform poorly on flags. Furthermore, traditional methods using flag keypoint detection and feature matching are not robust and significant for detecting keypoints on non-rigid targets; that is, the detected keypoints change considerably with distortion. Summary of the Invention

[0003] The technical objective of this invention is to address the above-mentioned shortcomings by providing a method, system, and apparatus for extracting key points of a flag, which can extract key points of the flag that are significant and do not change with deformation or distortion.

[0004] The technical solution adopted by this invention to solve its technical problem is:

[0005] A method for extracting key points of a flag, the implementation of which includes the following steps:

[0006] 1) Deformed Flag Generation: Collect various flag images, clean and filter the images, and for each cleaned and filtered image, deform it to generate multiple corresponding deformed flag images, and retain their respective deformation parameters;

[0007] 2) Key point detection and matching: Use the traditional key point detection and matching method to perform key point detection and matching on the original flag image and its corresponding deformed image described in step 1), and obtain the coordinates of the points that match the key points of the deformed image and the original image.

[0008] 3) Matching heatmap generation: First, initialize a mask image with all zeros and the same size as the corresponding image; then, refer to the deformation parameters described in step 1) to obtain the coordinates of the points in the deformed image that correspond to the key points in the original image, compare with the matching key point coordinates in step 2), and assign the value 1 to the correct matching key point positions in the same positions of the mask image; finally, combine the matching heatmaps of the original image and the deformed image to regenerate the matching heatmaps of each image.

[0009] 4) Model training and optimization: Select a semantic segmentation model, with the model input being an image and the expected output being the corresponding matching heatmap generated in step 3).

[0010] Using a convolutional neural network model, with a defined objective, the network learns to identify key points on the flag that are significant and do not change with deformation or distortion.

[0011] Preferably, random homography transformation and TPS deformation are used to achieve the image deformation, generating multiple images of the corresponding flag after deformation.

[0012] TPS, or Thin-Plate Spline Interpolation, is a method for image distortion.

[0013] Preferably, SIFT is used to achieve key point detection and matching.

[0014] SIFT, or Scale-invariant feature transform, is a traditional and commonly used keypoint detection and matching method. It's a descriptor used in image processing that possesses scale invariance and can detect keypoints in images; it's a type of local feature descriptor.

[0015] Preferably, the semantic segmentation model uses U-Net to achieve image segmentation.

[0016] Preferably, the specific implementation method for regenerating matching heatmaps for each image by combining the matching heatmaps of the original image and the deformed image is as follows:

[0017] Let I0 be the original image, and I1 and I2 be the deformed images of the original image. Let the deformation functions be T1 and T2, respectively. The heatmaps that correctly match I0 with I1 and I2 are M. 01 and M 02 The correct heatmap for matching I1 and I0 is M. 10 The correct thermogram for matching I2 and I0 is M. 20 Then the final matching heatmap of I0 is M0 = (M 01 +M 02 The final matching heatmap of I1 is M1 = (T1(M0) + M) / 2. 10 The final matching heatmap for I2 is M2 = (T2(M0) + M) / 2. 20 ) / 2.

[0018] This invention also claims a flag key point extraction system, including a deformable flag generation module, a key point detection and matching module, a matching heatmap generation module, and a model training and optimization module.

[0019] Deformed Flag Generation Module: Collects various flag images, cleans and filters the images, and for each cleaned and filtered image, deforms it to generate multiple corresponding deformed flag images, while retaining their respective deformation parameters;

[0020] Key point detection and matching module: Using key point detection and matching methods, key points are detected and matched in the original image and its corresponding deformed image in the deformed flag generation module to obtain the coordinates of the points that match the key points of the deformed image and the original image.

[0021] Matching heatmap generation module: First, initialize a mask image with all zeros, with the same size as the corresponding image; then, refer to the deformation parameters in the deformed flag generation module to obtain the true and correct matching point positions, compare with the matching results in the key point detection and matching module, and assign the value 1 to the correctly matched key point positions on the mask image; finally, combine the matching heatmaps of the original image and the deformed image to regenerate the matching heatmaps of each image.

[0022] Model Training and Optimization Module: Select a semantic segmentation model. The model input is an image, and the expected output is the corresponding matching heatmap generated in the matching heatmap generation module.

[0023] Preferably, the deformed flag generation module uses random homography transformation and TPS deformation to achieve image deformation, generating multiple images of the corresponding deformed flags.

[0024] Preferably, SIFT is used to achieve key point detection and matching.

[0025] Preferably, the semantic segmentation model adopts the U-Net model.

[0026] The present invention also claims a flag key point extraction device, comprising: at least one memory and at least one processor;

[0027] The at least one memory is used to store a machine-readable program;

[0028] The at least one processor is used to call the machine-readable program to implement the above-described flag key point extraction method.

[0029] Compared with the prior art, the flag key point extraction method, system, and apparatus of the present invention have the following advantages:

[0030] By using a convolutional neural network model and setting a target, the network learns to identify key points of the flag that are significant and do not change with deformation or distortion; this can improve the accuracy of flag feature matching and facilitate flag identification through feature matching. Attached Figure Description

[0031] Figure 1 This is a flowchart illustrating the implementation of the flag key point extraction method provided in this embodiment of the invention. Detailed Implementation

[0032] The present invention will be further described below with reference to specific embodiments.

[0033] This invention provides a method for extracting key points of a flag, the implementation of which includes the following steps:

[0034] 1. Deformed Flag Generation: Collect various flag images, clean and filter them. For each cleaned and filtered image, perform random homography transformation and TPS deformation to generate several corresponding deformed flag images, and retain their respective deformation parameters.

[0035] 2. Key point detection and matching: Using traditional key point detection and matching methods, such as SIFT, key point detection and matching are performed on the original flag image and its corresponding deformed image described in step 1 to obtain the coordinates of the points that match the key points of the deformed image and the original image.

[0036] 3. Matching heatmap generation:

[0037] First, initialize a mask image with all zeros, the size of which is the same as the corresponding image;

[0038] Then, referring to the deformation parameters described in step 1, obtain the coordinates of the points in the deformation diagram that correspond to the key points in the original diagram. Compare the matching key point coordinates in step 2, and assign the value 1 to the correct matching key point position in the same position in the mask diagram.

[0039] Finally, by combining the matching heatmaps of the original image and the deformed image, matching heatmaps for each image are regenerated.

[0040] The following example illustrates a method for regenerating matching heatmaps for each image by combining the original and deformed image's matching heatmaps:

[0041] Let I0 be the original image, and I1 and I2 be the deformed images of the original image. Let the deformation functions be T1 and T2, respectively. The heatmaps that correctly match I0 with I1 and I2 are M. 01 and M 02 ;

[0042] The correct heatmap for matching I1 and I0 is M. 10 The correct thermogram for matching I2 and I0 is M. 20 Then the final matching heatmap of I0 is M0 = (M 01 +M 02 The final matching heatmap of I1 is M1 = (T1(M0) + M) / 2. 10 The final matching heatmap for I2 is M2 = (T2(M0) + M) / 2. 20 ) / 2.

[0043] 4. Model Training and Optimization: Select a semantic segmentation model, such as U-Net. The model input is an image, and the expected output is the corresponding matching heatmap generated in step 3.

[0044] By using a convolutional neural network model and setting a target, the network learns to obtain key points of the flag that are significant and do not change with deformation and distortion, which can improve the accuracy of flag feature matching and facilitate flag identification through feature matching. This method is also applicable to key point extraction and matching of other non-rigid objects.

[0045] This invention also provides a flag key point extraction system, including a deformable flag generation module, a key point detection and matching module, a matching heatmap generation module, and a model training and optimization module.

[0046] The deformed flag generation module collects various flag images, cleans and filters the images, and performs random homography transformation and TPS deformation on each cleaned and filtered image to generate multiple corresponding deformed flag images, while retaining their respective deformation parameters.

[0047] Key point detection and matching module: Using traditional and commonly used key point detection and matching methods, key points are detected and matched between the original image and its corresponding deformed image in the deformed flag generation module, and the coordinates of the points that match the key points of the deformed image and the original image are obtained.

[0048] The matching heatmap generation module first initializes a mask image with all zeros, the same size as the corresponding image. Then, referring to the deformation parameters in the deformed flag generation module, it obtains the true and correct matching point positions. Comparing these with the matching results in the keypoint detection and matching modules, it assigns a value of 1 to the correctly matched keypoint positions on the mask image. Finally, it combines the matching heatmaps of the original image and the deformed image to regenerate the matching heatmaps for each image. The specific implementation is as follows:

[0049] Let I0 be the original image, and I1 and I2 be the deformed images of the original image. Let the deformation functions be T1 and T2, respectively. The heatmaps that correctly match I0 with I1 and I2 are M. 01 and M 02 ;

[0050] The correct heatmap for matching I1 and I0 is M. 10 The correct thermogram for matching I2 and I0 is M. 20 Then the final matching heatmap of I0 is M0 = (M 01 +M 02 The final matching heatmap of I1 is M1 = (T1(M0) + M) / 2. 10 The final matching heatmap for I2 is M2 = (T2(M0) + M) / 2. 20 ) / 2.

[0051] Model Training and Optimization Module: Select a semantic segmentation model. The model input is an image, and the expected output is the corresponding matching heatmap generated in the matching heatmap generation module.

[0052] This embodiment uses SIFT to achieve key point detection and matching; the semantic segmentation model uses the U-Net model.

[0053] The present invention also claims a flag key point extraction device, comprising: at least one memory and at least one processor;

[0054] The at least one memory is used to store a machine-readable program;

[0055] The at least one processor is used to call the machine-readable program to implement the above-described flag key point extraction method.

[0056] Through the above specific embodiments, those skilled in the art can easily implement the present invention. However, it should be understood that the present invention is not limited to the specific embodiments described above. Based on the disclosed embodiments, those skilled in the art can arbitrarily combine different technical features to achieve different technical solutions.

[0057] Except for the technical features described in the specification, all other technologies are known to those skilled in the art.

Claims

1. A method for extracting key points of a flag, characterized in that, The implementation of this method includes the following steps: 1) Deformed Flag Generation: Collect various original flag images, clean and filter the images, and for each cleaned and filtered image, deform it to generate multiple corresponding deformed flag images, while retaining their respective deformation parameters; 2) Key point detection and matching: Use the key point detection and matching method to perform key point detection and matching on the original flag image and its corresponding deformed image described in step 1), and obtain the coordinates of the points that match the key points of the deformed image and the original image. 3) Matching heatmap generation: First, initialize a mask image with all zeros and the same size as the corresponding image; then, refer to the deformation parameters described in step 1) to obtain the coordinates of the points in the deformed image that correspond to the key points in the original image, compare with the matching key point coordinates in step 2), and assign the value 1 to the correct matching key point positions in the same positions of the mask image; finally, combine the matching heatmaps of the original image and the deformed image to regenerate the matching heatmaps of each image. 4) Model Training and Optimization: Select a semantic segmentation model. The model input is an image, and the expected output is the corresponding matching heatmap generated in step 3). The semantic segmentation model used is U-Net. The specific implementation method for regenerating matching heatmaps for each image by combining the original image and the deformed image is as follows: Let I0 be the original image, and I1 and I2 be the deformed images of the original image. Let the deformation functions be T1 and T2, respectively. The heatmaps that correctly match I0 with I1 and I2 are M. 01 and M 02 The correct heatmap for matching I1 and I0 is M. 10 The correct thermogram for matching I2 and I0 is M. 20 Then the final matching heatmap of I0 is M0 = (M 01 +M 02 The final matching heatmap of I1 is M1 = (T1(M0) + M) / 2. 10 The final matching heatmap of I2 is M2 = (T2(M0) + M) / 2. 20 ) / 2.

2. The method for extracting key points of a flag according to claim 1, characterized in that, The image deformation is achieved by using random homography transformation and TPS deformation to generate multiple images of the corresponding flag after deformation.

3. A method for extracting key points of a flag according to claim 1 or 2, characterized in that, SIFT is used to achieve key point detection and matching.

4. A flag key point extraction system, characterized in that, It includes a deformable flag generation module, a key point detection and matching module, a matching heatmap generation module, and a model training and optimization module. Deformed Flag Generation Module: Collects various flag images, cleans and filters the images, and for each cleaned and filtered image, deforms it to generate multiple corresponding deformed flag images, while retaining their respective deformation parameters; Key point detection and matching module: Using key point detection and matching methods, key points are detected and matched in the original image and its corresponding deformed image in the deformed flag generation module to obtain the coordinates of the points that match the key points of the deformed image and the original image. Matching heatmap generation module: First, initialize a mask image with all zeros, the size of which is the same as the corresponding image; Then, referring to the deformation parameters in the deformed flag generation module, the true and correct matching point positions are obtained. Comparing these with the matching results in the keypoint detection and matching module, the correctly matched keypoint positions are assigned a value of 1 on the mask image. Finally, combining the matching heatmaps of the original image and the deformed image, a new matching heatmap for each image is generated. The specific implementation method is as follows: Let I0 be the original image, and I1 and I2 be the deformed images of the original image. Let the deformation functions be T1 and T2, respectively. The heatmaps that correctly match I0 with I1 and I2 are M. 01 and M 02 ; The correct heatmap for matching I1 and I0 is M. 10 The correct thermogram for matching I2 and I0 is M. 20 Then the final matching heatmap of I0 is M0 = (M 01 +M 02 The final matching heatmap of I1 is M1 = (T1(M0) + M) / 2. 10 The final matching heatmap of I2 is M2 = (T2(M0) + M) / 2. 20 ) / 2; Model training and optimization module: Select a semantic segmentation model, with the model input being an image, and the expected output being the corresponding matching heatmap generated in the matching heatmap generation module; the semantic segmentation model adopts the U-Net model.

5. A flag key point extraction system according to claim 4, characterized in that, The deformed flag generation module uses random homography transformation and TPS deformation to achieve image deformation and generate multiple images of the corresponding deformed flags.

6. A flag key point extraction system according to claim 4 or 5, characterized in that, SIFT is used to achieve key point detection and matching.

7. A flag key point extraction device, characterized in that, include: At least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to invoke the machine-readable program to implement the method according to any one of claims 1 to 3.

Images