An aircraft surface feature segmentation method based on contour constraint optimization

Abstract

The application belongs to the technical field of image processing, and particularly relates to an aircraft surface feature segmentation method based on contour constraint optimization, which comprises the following steps: collecting image data, learning feature information of the image by using a feature extraction backbone network, and outputting a feature information map; using a contour generation module CGM, first generating a target preselection frame, taking the preselection frame as a contour blueprint, and iteratively extracting target contour constraints multiple times; using a detection branch and a segmentation branch structure, respectively predicting a classification value, a positioning value and a segmentation mask of the target; using the positioning value of the predicted target and the generated contour constraint, gradually optimizing the segmentation mask of the target, and completing a target segmentation task. The target segmentation method has high efficiency and high segmentation precision, can significantly optimize the effect of the target segmentation result at the edge, improve the reliability of the segmentation result, further improve the precision of target geometric measurement, and has strong robustness in various complex target segmentation tasks.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, specifically relating to an aircraft surface feature segmentation method based on contour constraint optimization. Background Technology

[0002] Aircraft surface quality has a significant impact on its aerodynamic and safety performance. Aircraft surfaces contain numerous rivets, screws, and other connectors; the installation quality of these connectors, especially the degree of unevenness, is a crucial performance indicator of aircraft surface quality. Furthermore, after aircraft assembly, the assembly characteristics between the aircraft skin and connectors, such as seams and step differences between skin panels, directly affect the aircraft's assembly quality. In addition, during aircraft manufacturing, surface defects such as dents and scratches may occur. Accurately detecting and measuring these surface features and defects has become a major challenge in the aerospace manufacturing field.

[0003] Currently, the detection of the aforementioned target features mainly relies on manual detection supplemented by computer vision-based techniques.

[0004] For example, Chinese invention patent CN202210197322.4, entitled "A Deep Learning-Based Method and System for Detecting Damage to Aircraft Skin Surfaces," discloses a deep learning-based method and system for detecting damage to aircraft skin surfaces. This method, relating to image processing technology, mainly includes: acquiring a currently collected image of the aircraft skin surface; inputting the currently collected image into a trained aircraft skin surface damage detection model for damage category detection and damage region segmentation; the trained aircraft skin surface damage detection model is determined based on a deep learning neural network and a training dataset; the deep learning neural network includes a feature extraction network, an attention module, a multi-path multi-scale feature fusion module, a target detection network, and a fully convolutional segmentation network; the attention module is used to re-encode the original feature map output by the feature extraction network to determine the multi-scale feature map. The aforementioned patented visual technology has low versatility for various feature target data on aircraft surfaces, typically only satisfying the screening of a specific type of feature, and only able to classify and locate the feature type in a two-dimensional image, failing to meet the geometric measurement requirements for feature targets. Summary of the Invention

[0005] This application aims to solve the above-mentioned problems in the prior art, and proposes an aircraft surface feature segmentation method based on contour constraint optimization.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] An aircraft surface feature segmentation method based on contour constraint optimization includes the following steps:

[0008] Step 1: Collect two-dimensional image data containing the target to be segmented, use a feature extraction backbone network to learn the feature information of the image, and output a feature information map rich in image features;

[0009] Step 2: Using the feature information map as input, the contour generation module CGM is used to first generate a target preselection box, and then the target contour constraints are extracted iteratively multiple times using the preselection box as the contour blueprint.

[0010] Step 3: Using the detection branch and segmentation branch structure, with the generated target preselection box as input, predict the target's classification value, localization value, and segmentation mask respectively;

[0011] Step 4: Using the predicted target location value and the generated contour constraints, gradually optimize the target segmentation mask, and output the optimized segmentation mask and classification value to complete the target segmentation task.

[0012] Furthermore, in step 2, a contour iterative network (CIN) is used, with the target pre-selected box as the contour blueprint. After the second iteration, the outer contour constraints of the target to be segmented are extracted. Specifically, The decision is made by the individual.

[0013] Furthermore, in step 2, the initial shape of the target preselection box is a horizontal rectangular frame. For each side, the point that divides it into (ω+1) equal parts is taken as the contour feature corner point cp. i This constitutes the feature corner point set Cp = {cp} i}, and an initial contour constraint Ct0 of shape (4ω)-sided; where ω represents the number of equal divisions of the contour; then, the contour iterative network is used to predict the deviation of all feature corner points on the current contour constraint, and the deviation value is mapped back to the feature corner points to obtain the new contour feature corner point location, update the contour constraint fitting, and take the (ω+1) equal division point of each pair of adjacent feature corner points as the new contour feature corner point, update the feature corner point set, and complete one contour constraint iteration process. After the next iteration, the output contour constraints are...

[0014] Furthermore, the value i of the number of feature corner points in the feature corner point set should satisfy:

[0015]

[0016] Furthermore, the contour iterative network CIN uses recurrent periodic convolution to predict the deviation values ​​of each feature corner point:

[0017]

[0018] in:

[0019] d represents the contour constraint iteration index;

[0020] f represents the set of characteristic corner points;

[0021] K represents the circular convolution kernel; k is the kernel size; K t This represents the recurrent convolution kernel with a kernel size of t;

[0022] * indicates a convolution operation;

[0023] mod N represents a periodic operation, where N is the number of feature corner points.

[0024] Furthermore, in step 3, the detection branch takes the target pre-selected box as input and uses finer classification and regression sub-branches to map the feature information to a fixed dimension, predicting the classification value and localization value of the target. For P target pre-selected boxes, the classification sub-branch maps the feature information to a vector of dimension P×C, representing the probability that the target to be segmented in each pre-selected box is predicted to be one of C different categories, where C represents the number of classifiable features; the regression sub-branch maps the feature information to a vector of dimension P×4, representing the displacement deviation of each pre-selected box relative to the target to be segmented in the image.

[0025] Furthermore, in step 3, the segmentation branch takes the target pre-selection box as input and predicts the segmentation mask of the target. The segmentation branch first maps the target pre-selection box back to the original image, obtains an image block by cropping, and then predicts the segmentation mask of the target pixel by pixel on the image block.

[0026] Furthermore, step 4 specifically includes:

[0027] Based on the positioning information output by the regression sub-branch of the detection branch, the corrected bounding rectangle of the target to be segmented is obtained. The bounding rectangle is used to constrain the target segmentation mask, and the mask outside the bounding rectangle is clipped.

[0028] Based on the contour octagonal matching rule, the target segmentation mask is further optimized using the contour constraints. First, the center point of the contour constraints is aligned with the center point of the segmentation mask to construct the offset direction set Dt:

[0029]

[0030] Where (0, 0) represents the center point of alignment. After alignment, the contour constraint and the segmentation mask are in an initial matching state. Record the area In0 of the region currently within the contour constraint but not belonging to the segmentation mask, and the area Ex0 of the region currently belonging to the segmentation mask but outside the contour constraint. Then, calculate the matching degree mt0 between the current contour constraint and the segmentation mask, and add it to the matching degree set Mt={mt0}. Next, select any offset direction from the offset direction set, and translate the contour constraint by the number of pixels. Calculate the area In of the region within the contour constraint but not belonging to the segmentation mask when the contour constraint matches the segmentation mask after translation. dti The area Ex that belongs to the segmentation mask but extends beyond the contour constraints. dti and matching degree mt dti If mt dti >mt dti-1 Then update the matching degree set Mt={mt0...mt dti}, and continue to translate the contour constraint along the current offset direction, repeating the matching process; otherwise, end the current offset direction, start again from the alignment point, select the next offset direction, and perform the matching process of contour constraint and segmentation mask.

[0031] Furthermore, the formula for calculating the matching degree is:

[0032]

[0033] Where dt represents the direction of the current matching contour constraint offset, and i represents the offset index. After traversing the contour octagonal matching process, the matching degree set Mt is obtained. The contour constraint corresponding to the maximum value in the set is selected and matched with the segmentation mask as the optimal solution. The segmentation mask is then filled to obtain the final segmentation result.

[0034] The advantages of this application are:

[0035] 1. The target segmentation method of the present invention is highly efficient and accurate, significantly optimizes the effect of target segmentation results at the edges, improves the reliability of segmentation results, further improves the accuracy of target geometric measurement, and has strong robustness in various complex target segmentation tasks.

[0036] 2. This invention is based on deep learning algorithms and uses real-world two-dimensional images of aircraft surfaces as input to achieve automated segmentation of aircraft surface features (such as defects, rivets, screws, etc.). At the same time, in terms of segmentation accuracy, the contours of the features to be measured are used as constraints, which greatly improves the accuracy of feature segmentation, especially the segmentation effect at feature edges, significantly improving the accuracy of geometric measurement of feature targets. Attached Figure Description

[0037] Figure 1This is a flowchart of the aircraft surface feature segmentation method based on contour constraint optimization according to the present invention.

[0038] Figure 2 This is a network structure diagram of the aircraft surface feature segmentation method based on contour constraint optimization according to the present invention.

[0039] Figure 3 This is a schematic diagram illustrating the calculation of the contour constraint and segmentation mask matching degree of the present invention.

[0040] Figure 4 This is a network structure diagram of the second embodiment of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the invention clearer, the technical solutions of the embodiments of the invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the invention, not all embodiments. The components of the embodiments of the invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0042] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0043] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0044] In the description of this invention, it should be noted that the terms "upper," "vertical," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this invention is in use, or the orientation or positional relationship commonly understood by those skilled in the art. They are only used for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. In addition, the terms "first," "second," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0045] This invention presents an aircraft surface feature segmentation method based on contour constraint optimization, which uses a deep learning network to segment targets in an image. The method learns feature information from the image through a feature extraction backbone network, then fits the outer contour constraint of the target based on this feature information, initially segmenting the target in the image. Finally, the target contour constraint is used to optimize the segmentation result, achieving high-precision segmentation of each target instance in the image.

[0046] Example 1

[0047] An aircraft surface feature segmentation method based on contour constraint optimization includes the following steps:

[0048] Step 1: Collect two-dimensional image data containing the target to be segmented, use a feature extraction backbone network to learn the feature information of the image, and output a feature information map rich in image features;

[0049] Step 2: Using the feature information map as input, the Contour Generation Module (CGM) is used to first generate a target pre-selection box, and then the target contour constraints are extracted iteratively multiple times using the pre-selection box as the contour blueprint.

[0050] Step 3: Using the detection branch and segmentation branch structure, with the target preselection box generated in Step 2 as input, predict the target's classification value, localization value, and segmentation mask respectively;

[0051] Step 4: Using the predicted target location value and the generated contour constraints, gradually optimize the target segmentation mask, and output the optimized segmentation mask and classification value to complete the target segmentation task.

[0052] The target segmentation method of this invention is highly efficient and accurate, significantly optimizing the segmentation results at edges and improving the reliability of the segmentation results. Furthermore, it enhances the accuracy of geometric measurements of the target and exhibits strong robustness in various complex target segmentation tasks. Based on a deep learning algorithm, this invention uses real-world 2D images of the aircraft surface as input to automate the segmentation of aircraft surface features (e.g., defects, rivets, screws, etc.). Simultaneously, it uses the contours of the features to be measured as constraints to greatly improve the accuracy of feature segmentation, especially at feature edges, significantly enhancing the accuracy of geometric measurements of the target features.

[0053] Example 2

[0054] like Figure 1 As shown, the aircraft surface feature segmentation method based on contour constraint optimization of the present invention includes the following steps:

[0055] Step 1: Collect two-dimensional image data containing the target to be segmented, use a feature extraction backbone network to learn the feature information of the image, and output a feature information map rich in image features;

[0056] Step 2: Using the feature information map as input, the Contour Generation Module (CGM) is used to first generate a target preselection box, and then the target contour constraints are extracted iteratively multiple times using the preselection box as the contour blueprint.

[0057] Step 3: Using the detection branch and segmentation branch structure, with the target preselection box generated in Step 2 as input, predict the target's classification value, localization value, and segmentation mask respectively;

[0058] Step 4: Using the predicted target location value and the generated contour constraints, gradually optimize the target segmentation mask, and output the optimized segmentation mask and classification value to complete the target segmentation task.

[0059] Specifically, in step 2 above, the outer contour constraints of the target to be segmented are extracted using the target pre-selected box as a contour blueprint and the Contour Iteration Network (CIN) after four iterations. In particular, The decision is made by the individual.

[0060] The initial shape of the target preselection box is a horizontal rectangular box. For each side, the point that divides it into (ω+1) equal parts is taken as the contour feature corner point cp. i This forms a feature corner point set Cp = {cpi}, and an initial contour constraint Ct0 with a (4ω)-sided shape. Here, ω represents the number of equal divisions of the contour. Then, a contour iterative network is used to predict the deviation of all feature corner points on the current contour constraint, and this deviation value is mapped back to the feature corner points to obtain new contour feature corner point locations. The contour constraint fitting is updated, and the (ω+1) equal division points of each pair of adjacent feature corner points are taken as new contour feature corner points. The feature corner point set is updated, completing one contour constraint iteration process. After the next iteration, the output contour constraints are...

[0061] The number of feature corner points i in the feature corner point set should satisfy the following:

[0062]

[0063] The contour iterative network uses recurrent periodic convolution to predict the deviation values ​​of each feature corner point:

[0064]

[0065] in:

[0066] d represents the contour constraint iteration index;

[0067] f represents the set of characteristic corner points;

[0068] K represents the circular convolution kernel; k is the kernel size; K t This represents the recurrent convolution kernel with a kernel size of t;

[0069] * indicates a convolution operation;

[0070] mod N represents a periodic operation, where N is the number of feature corner points.

[0071] Specifically, in step 3 above, the detection branch takes the target pre-selected bounding boxes as input and uses finer classification and regression sub-branches to map the feature information to fixed dimensions, predicting the target's classification and localization values. For P target pre-selected bounding boxes, the classification sub-branch maps the feature information to a vector of dimension P×C, representing the probability that the target to be segmented in each pre-selected bounding box is predicted to be one of C different categories, where C represents the number of classifiable features; the regression sub-branch maps the feature information to a vector of dimension P×4, representing the displacement deviation of each pre-selected bounding box relative to the target to be segmented in the image.

[0072] Specifically, in step 3 above, the segmentation branch takes the target pre-selection box as input and predicts the segmentation mask of the target. The segmentation branch first maps the target pre-selection box back to the original image, obtains image blocks by cropping, and then predicts the target segmentation mask pixel by pixel on the image blocks.

[0073] Specifically, in step 4 above, the predicted target location value and the generated contour constraints are used to progressively optimize the target segmentation mask.

[0074] Based on the localization information output by the regression sub-branch of the detection branch, the corrected bounding box of the target to be segmented can be obtained. First, the bounding box is used to constrain the target segmentation mask, and the mask outside the bounding box is clipped.

[0075] Next, based on the contour octagonal matching rule, the target segmentation mask is further optimized using the contour constraints. First, the center of the contour constraints is aligned with the center of the segmentation mask to construct the offset direction set Dt:

[0076]

[0077] Here, (0, 0) represents the center point of alignment. After alignment, the contour constraint and the segmentation mask are in an initial matching state. The area In0 of the region currently within the contour constraint but not belonging to the segmentation mask, and the area Ex0 of the region currently belonging to the segmentation mask but outside the contour constraint, are recorded. Then, the matching degree mt0 between the current contour constraint and the segmentation mask is calculated and added to the matching degree set Mt = {mt0}. Specifically, In0 and Ex0 can be obtained by calculating the number of pixels in the corresponding region.

[0078] Next, select any offset direction from the set of offset directions, translate the contour constraint by pixel amount, and calculate the area In within the contour constraint but not belonging to the segmentation mask when the translated contour constraint matches the segmentation mask. dti The area Ex that belongs to the segmentation mask but extends beyond the contour constraints. dti and matching degree mt dti If mt dti >mt dti-1 Then update the matching degree set Mt={mt o ...mt dti The process continues by translating the contour constraint along the current offset direction and repeating the matching process; otherwise, the current offset direction is ended, and the process restarts from the alignment point, selecting the next offset direction and performing the matching process for the contour constraint and segmentation mask. The formula for calculating the matching degree is:

[0079]

[0080] Where dt represents the direction of the current matching contour constraint offset, and i represents the offset index. After traversing the eight-way contour matching process, a matching degree set Mt is obtained. The contour constraint corresponding to the maximum value in the set is selected as the optimal solution for matching with the segmentation mask. The segmentation mask is filled with the contour constraint as the target boundary to obtain the final segmentation result.

[0081] In the above formula dti Ex dti and mt ati The meanings of In0, Ex0, and In0 in the previous text are the same, except that they correspond to the matching positions when different offsets occur. The subscript 0 indicates the initial position, that is, no offset has occurred; the subscript dti indicates that an offset has occurred along a certain direction in the offset direction set DT.

[0082] Example 3

[0083] Examples 1 and 2 are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments; all technical solutions falling within the scope of the present invention's concept are within its protection. For example... Figure 4As shown, the technical solution of using the target location box adjusted for offset error as the contour blueprint, and then using the contour iteration network CIN to extract the contour constraints of the target to be segmented, should still be considered within the protection scope of this invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of this invention should be considered within the protection scope of this invention.

Claims

1. An aircraft surface feature segmentation method based on contour-constrained optimization, characterized in that: Includes the following steps: Step 1: Collect two-dimensional image data containing the target to be segmented, use a feature extraction backbone network to learn the feature information of the image, and output a feature information map rich in image features; Step 2: Using the feature information map as input, the contour generation module CGM is used to first generate a target preselection box, and then the target contour constraints are extracted iteratively multiple times using the preselection box as the contour blueprint. Step 3: Using the detection branch and segmentation branch structure, with the generated target preselection box as input, predict the target's classification value, localization value, and segmentation mask respectively; Step 4: Using the predicted target location value and the generated contour constraints, gradually optimize the target segmentation mask, and output the optimized segmentation mask and classification value to complete the target segmentation task; Step 4 specifically involves: Based on the positioning information output by the regression sub-branch of the detection branch, the corrected bounding rectangle of the target to be segmented is obtained. The bounding rectangle is used to constrain the target segmentation mask, and the mask outside the bounding rectangle is clipped. Based on the contour octagonal matching rule, the target segmentation mask is further optimized using the contour constraints. First, the center point of the contour constraints is aligned with the center point of the segmentation mask to construct a set of offset directions. : in, This indicates the center point of alignment. After alignment, the contour constraints and the segmentation mask are in an initial matching state. Record the area of ​​the region that is currently within the contour constraints but does not belong to the segmentation mask. And the area of ​​the region that currently belongs to the segmentation mask but is outside the contour constraints. Then, the matching degree between the current contour constraint and the segmentation mask is calculated. And add it to the matching set. Next, select any offset direction from the set of offset directions, translate the contour constraint by the number of pixels, and calculate the area of ​​the region that is within the contour constraint but does not belong to the segmentation mask when the translated contour constraint matches the segmentation mask. Area that belongs to the segmentation mask but extends beyond the contour constraints and matching degree ,like Then update the matching set. The process continues to translate the contour constraint along the current offset direction and repeats the matching process; otherwise, the current offset direction is ended, and the process restarts from the alignment point, selects the next offset direction, and performs the matching process of contour constraint and segmentation mask.

2. The aircraft surface feature segmentation method based on contour constraint optimization according to claim 1, characterized in that: In step 2, the target pre-selected box is used as the contour blueprint, and the contour iterative network CIN is used to process it... After the second iteration, the outer contour constraints of the target to be segmented are extracted.

3. The aircraft surface feature segmentation method based on contour constraint optimization according to claim 1, characterized in that: In step 2, the initial shape of the target pre-selection box is a horizontal rectangular frame. For each side, take its... The points that are divided equally are the feature corner points of the contour. This constitutes a set of characteristic corner points. and in the form of Initial contour constraints of the polygon ;in, This involves dividing the contour into equal parts; then, an iterative contour network is used to predict the deviation of all feature corner points on the current contour constraint, and this deviation value is mapped back to the feature corner points to obtain the new contour feature corner point locations. The contour constraint fitting is then updated, and each pair of adjacent feature corner points is taken. The dividing points are the new contour feature corner points. The feature corner point set is updated, completing one contour constraint iteration process. After the next iteration, the output contour constraints are... .

4. The aircraft surface feature segmentation method based on contour constraint optimization according to claim 3, characterized in that: The number of feature corner points in the feature corner point set is taken as follows: It should meet the following requirements: 。 5. The aircraft surface feature segmentation method based on contour constraint optimization according to claim 3, characterized in that: The contour iterative network CIN uses cyclic convolution to predict the deviation values ​​of each feature corner point: in: Indicates the contour constraint iteration index; Represents the set of characteristic corner points; Represents the circular convolution kernel; The kernel size; Indicates the core size is The corresponding recurrent convolution kernel; Indicates the convolution operation; Indicates periodic operation. This represents the number of characteristic corner points.

6. The aircraft surface feature segmentation method based on contour constraint optimization according to claim 1, characterized in that: In step 3, the detection branch takes the target pre-selected box as input and uses finer-grained classification and regression sub-branches to map the feature information to fixed dimensions, predicting the target's classification and localization values. A target preselection box, wherein the classification sub-branch maps feature information to a dimension of . The vector represents the predicted value of the target to be segmented in each preselected box. The probability of each different category This represents the number of classifiable features; the regression sub-branch maps feature information to a dimension of... The vector represents the displacement deviation of each preselected box relative to the target to be segmented in the image.

7. The aircraft surface feature segmentation method based on contour constraint optimization according to claim 1, characterized in that: In step 3, the segmentation branch takes the target pre-selection box as input and predicts the segmentation mask of the target. The segmentation branch first maps the target pre-selection box back to the original image, obtains an image block by cropping, and then predicts the segmentation mask of the target pixel by pixel on the image block.

8. The aircraft surface feature segmentation method based on contour constraint optimization according to claim 1, characterized in that: The formula for calculating the matching degree is: in, This represents the direction of the current matching contour constraint offset. This represents the offset index. After traversing the entire contour octagonal matching process, the matching degree set is obtained. The optimal solution is to select the contour constraint corresponding to the maximum value in the set and match it with the segmentation mask. The segmentation mask is then filled to obtain the final segmentation result.

Images