Visual detection method and system for quality of aerospace parachute based on deep learning

Abstract

The invention discloses an aerospace parachute quality visual detection method and system based on deep learning. Parachute canopy sewing stitch image data are collected through an equipment end, and training and optimization are performed through a deep learning and optimization module to obtain a visual detection model; and predicting the preprocessed image data through a model prediction module to obtain the type and defect information of the sewing thread. Through the secondary clustering algorithm combined with the DBSCAN algorithm, adaptive clustering is carried out on the detection frame, and accurate calculation of the stitch length and density is realized. An improved CSPDarknet53 backbone network and a PAFPN multi-scale feature fusion technology are adopted, and the design of a decoupling type detection head is combined, so that the precision of sewing stitch type identification and small defect detection is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of aerospace quality inspection technology, and in particular to an aerospace parachute quality visual inspection method and system based on deep learning. Background Art

[0002] In the field of aerospace, parachutes are key components for achieving aerodynamic deceleration and landing safety, and their quality is directly related to the recovery safety of the aircraft and the success of the mission. Therefore, the quality of the sewing process of the parachute greatly affects the safety and reliability of subsequent space operations. Traditional parachute quality inspection methods mainly rely on manual inspection and physical testing, which are not only time-consuming and labor-intensive, but also may be affected by human factors, resulting in limited accuracy and reliability of the inspection results.

[0003] Although the traditional parachute quality inspection method can meet the inspection needs to a certain extent, it still has some limitations. For example, inspectors obtain product quality parameters by visually inspecting product quality, measuring dimensions with measuring tools, and checking stitch density needle by needle. It is easy to have problems such as false detection, missed detection, and untimely detection. It is also limited by the inspector's subjective consciousness, personality, environment, cognition and other factors, resulting in the difficulty of consistent inspection results, large differences, poor reproducibility and consistency, backward inspection methods, and low inspection efficiency. Physical testing may require the destruction of parachute samples, increasing inspection costs and time. With the continuous development of aerospace operations, the requirements for parachute quality are getting higher and higher, and traditional inspection methods may not be able to meet higher inspection standards and needs. Summary of the invention

[0004] In order to solve the deficiencies in the prior art, the purpose of the present invention is to provide a method and system for visual inspection of aerospace parachute quality based on deep learning. The present invention is used for the inspection of the processing procedures and finished product inspection of parachute products, and is used to inspect the sewing quality of the product, including sewing form, sewing length, stitch density, sewing thread type and defect identification, etc. The purpose is to timely discover existing quality defects in the parachute processing process, and repair and improve them, so as to minimize the decline in the quality of parachute products caused by sewing quality defects. By replacing manual inspection with intelligent detection equipment, the existing inspection methods of parachute products are upgraded, the detection capabilities are improved, and the automatic real-time detection of the sewing quality of the parachute sewing process is achieved through cooperation with sewing equipment.

[0005] To achieve the purpose of the present invention, the technical solution adopted by the present invention is: A method for visual inspection of aerospace parachute quality based on deep learning, comprising: S101, collecting original parachute canopy sewing stitch image data; the original parachute canopy sewing stitch image data includes sample parachute canopy sewing stitch image data and test parachute canopy sewing stitch image data; S102, preprocessing the original parachute canopy sewing stitch image data to obtain preprocessed sample image data and preprocessed image data to be tested; S103, training the preprocessed sample image data through a modern convolutional neural network to obtain a visual detection model; S104, completing the detection of the target area of ​​the preprocessed image data to be tested through the visual detection model, and obtaining the target area detection frame coordinates, sewing type and defect information; S105, obtaining a data set of stitch length and stitch density distribution in the detection frame through the target area detection frame coordinates; S106, visually displaying the sewing type, stitch density distribution data set, stitch length, and defect information results.

[0006] Furthermore, in step S102, after removing noise and adjusting the size of the sample parachute canopy sewing stitch image data, the obtained sample image data is annotated with sewing type and defect information through labelimg software, and converted accordingly according to the requirements of the input storage format of modern convolutional neural networks.

[0007] Furthermore, step S103 is specifically as follows: inputting the preprocessed sample image data into the backbone network to obtain a feature map, the backbone network is composed of an improved version of CSPDarknet53, the network structure is based on Darknet53 and uses a Cross-Stage Partial Network structure, and is composed of five convolution modules, four C2f modules and one SPPF module; inputting the feature map into the feature enhancement network PAFPN for feature fusion to obtain an enhanced multi-scale feature map; the PAFPN is composed of two convolution modules, four C2f modules, four fusion modules and two upsampling modules; inputting the multi-scale feature map into the detector for target detection to obtain a visual detection model.

[0008] Furthermore, step S103 also includes optimizing the visual detection model, using the Adam optimizer to update parameters, and using the cosine annealing strategy to adjust the learning rate to optimize the visual detection model; the learning rate is calculated as follows: ; in, represents the learning rate at step t, and They represent the maximum learning rate and the minimum learning rate respectively, t represents the current step number, and T represents the total number of training rounds.

[0009] Furthermore, the detector includes three detection heads of different scales for detecting targets of different sizes; each detection head adopts a decoupled head structure and performs classification tasks and regression tasks respectively, the classification task is used to predict the sewing type, and the regression task is used to predict defect information.

[0010] Furthermore, the classification task is used to predict the sewing type, and the formula is: ; The regression task is used to predict the target bounding box, which can represent the defect information. The formula is: ; ; ; ; ; Among them, P is the probability that the target object contained in the bounding box belongs to each category, is the activation function, is the weight of the classification task, E is the feature map, , , and are the center point, width and height of the predicted bounding box respectively, conf is the confidence, which indicates the probability of the bounding box containing the target object and the accuracy of the bounding box. (t x ,t y ) is the offset predicted by the network, (c x ,c y ) is the coordinate of the upper left corner of the grid cell, (t w ,t h ) is the scale change of network prediction, (p w ,p h ) is the width and height of the prior box, is the probability that the bounding box contains the target object, It is the intersection-over-union ratio between the predicted bounding box and the true bounding box.

[0011] Furthermore, the target area detection frame coordinates are used to obtain a data set of stitch length and stitch density distribution in the detection frame, including: using the DBSCAN algorithm to perform a first clustering of the sewing stitches to obtain a segment set, taking each row of sewing segments in the segment set as a basis, and taking the first sewing stitch detection target frame and the last sewing stitch detection target frame of each row of sewing segments as retrieval targets; taking the center point of the last sewing stitch detection target frame of the initial retrieval segment as the origin, retrieving a fan-shaped angle area of ​​30° to the right of the origin and a fan radius of 3 times the average width Whether there is a first sewing stitch detection target frame of the target line segment within the range, if so, the target line segment and the initial search line segment are classified as the same line segment.

[0012] Furthermore, the original parachute canopy sewing stitch image data in step S101 is collected by a camera, a laser ranging sensor is vertically fixed at the end of the camera, and the camera captures the image of the parachute canopy with the laser ranging sensor turned on. Each pixel of the image represents the actual length for: ; Where, the focal length of the camera is f and the physical width of the sensor is S w , the pixel width of the image is W and the object distance provided by the laser ranging sensor is d.

[0013] Also provided is an aerospace parachute quality visual inspection system based on deep learning, which is used to implement a visual inspection method, including a device side, an intelligent detection edge side and a data processing cloud side; the device side is composed of a data acquisition device and a control device; the intelligent detection edge side is composed of a model prediction module, a data preprocessing module and a quality inspection module; the data processing cloud side is composed of a deep learning and optimization module, a data management and analysis module and a visualization module.

[0014] Furthermore, the data acquisition device is used to collect original parachute canopy sewing stitch image data; the control device drives the data acquisition device to perform operations according to the received control instructions; the model prediction module uses the trained visual inspection model to perform real-time prediction on the preprocessed data; the quality inspection module performs quality inspection on the prediction results to obtain the length, density and defects of the sewing stitches; the data preprocessing module preprocesses the collected original parachute canopy sewing stitch image data; the deep learning and optimization module uses big data and deep learning technology to train and optimize the model in the system; the data management and analysis module stores and manages the data uploaded by the device end and the intelligent detection edge end; the visualization module intuitively presents complex data and information.

[0015] Compared with the prior art, the present invention has the following advantages: Through the collaborative design of the device side, the intelligent detection edge side and the data processing cloud side, the edge side completes data preprocessing and real-time prediction, and the cloud side is responsible for model training and global optimization, which solves the problems of low efficiency and high latency of traditional manual detection. A secondary clustering algorithm combined with the DBSCAN algorithm is proposed to adaptively cluster the detection frame and realize the accurate calculation of the trace length and density.

[0016] The improved CSPDarknet53 backbone network and PAFPN multi-scale feature fusion technology, combined with the decoupled detection head design, effectively improve the accuracy of sewing stitch type recognition and small defect detection. The improved network structure optimizes the gradient flow through cross-stage local connection (CSP), enhances the feature expression ability, and cooperates with the cosine annealing strategy and Adam optimizer to solve the problem of missed detection caused by complex stitches and small defects in traditional methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of a method for visually inspecting the quality of an aerospace parachute based on deep learning in one embodiment of the present invention; Figure 2 is a flow chart of a method for visually inspecting the quality of an aerospace parachute based on deep learning in another embodiment of the present invention; Figure 3 It is a system structure block diagram of the aerospace parachute quality visual inspection system based on deep learning in the present invention; Figure 4 It is a function diagram of each module of the aerospace parachute quality visual inspection system based on deep learning in the present invention; Figure 5 It is an overall flow chart of the aerospace parachute quality visual inspection system based on deep learning of the present invention. DETAILED DESCRIPTION

[0018] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of this application.

[0019] like Figure 1 FIG. 1 is a flowchart of a method for visually inspecting the quality of an aerospace parachute based on deep learning in one embodiment of the present invention. A method for visually inspecting the quality of an aerospace parachute based on deep learning includes: Step S101: collecting original parachute canopy sewing stitch image data, including sample parachute canopy sewing stitch image data and test parachute canopy sewing stitch image data.

[0020] Specifically, in this embodiment, a high-precision camera is used as an image data acquisition device on the device side, which has high resolution and high-speed shooting capabilities to ensure that every detail is captured. The camera can be controlled by the control device to move according to a predetermined trajectory and speed, and at the same time collect the original parachute canopy sewing stitch image data in the current scene. The control device is usually connected to the intelligent detection edge end and the data processing cloud, receives its instructions and performs corresponding actions. This embodiment also emphasizes the use of advanced image stabilization technology on the device side to reduce jitter during image acquisition and ensure the quality of the acquired image data. In addition, for different lighting conditions, the camera is also equipped with an automatic light adjustment function to ensure that high-quality image data can still be captured under changing ambient light. These technologies significantly improve the availability of image data and lay the foundation for subsequent processing steps. Finally, the collected original parachute canopy sewing stitch image data is directly transmitted from the device side to the intelligent detection edge end via an Ethernet cable, and high-speed Ethernet technology is used to ensure high speed and low latency of data transmission.

[0021] Step S102: preprocessing the original sewing stitch image data to obtain preprocessed sample image data and preprocessed image data to be tested.

[0022] The original sewing stitch image data in step S1 is preprocessed by the data preprocessing module of the intelligent detection edge end to obtain preprocessed sample image data and preprocessed image data to be tested, and the data are sent to corresponding processing modules respectively.

[0023] The intelligent detection edge end preprocesses the original sewing stitch image data in step S1 according to the actual operation requirements through the data preprocessing module. As an optional solution, the preprocessing includes: noise removal, size adjustment and format conversion. The format conversion includes: converting the original parachute canopy sewing stitch image data into a data set format for training modern convolutional neural networks, such as VOC, COCO and YOLO. The noise removal method is non-local mean filtering. The formula of the non-local mean filtering algorithm is as follows: ; ; Among them, p( x ) represents the filtered image, I(y) Represents the original image I middle y The pixel value of the position, is a weight, indicating that in the original image I Medium, Pixel x and pixels y The similarity of h is the filtering parameter, which controls the decay speed of the weight function. N(x) and N(y) Respectively represent x and y The block area is centered. Z(x) is a normalization factor used to ensure that the distance calculation is not affected by different block sizes.

[0024] The resizing is performed using Lanczos interpolation, and the formula for Lanczos interpolation is as follows: ; ; in, Indicates that the target image is at position (x,y) The pixel value of Indicates that the source image is at position (i,j) The pixel value of Represents source pixel (i,j) For the target pixel (x,y) The total weight of and express The functions are calculated in x and y The interpolation weight in the direction.

[0025] The specific implementation of the format conversion includes: According to the requirements of the input storage format of modern convolutional neural networks, the corresponding conversion is performed, and this embodiment is converted into a VOC data set. The preprocessed sewing stitch image data is annotated with sewing type and defect information through labelimg software. After annotation, each image data forms a file with an XML suffix with the same name as the image, which contains the coordinate information and category of the detection box. The preprocessed sewing stitch image data and its corresponding XML suffix file are used to generate a VOC format data set through a script. The data set consists of an Annotations folder, an ImageSets folder, and a JPEGImages folder. The Annotations folder is used to store annotation files in XML format, the ImageSets folder is used to store division information such as training sets and verification sets, and the JPEGImages folder is used to store original image data.

[0026] Finally, the preprocessed sample image data and the preprocessed image data to be tested are obtained, and the preprocessed sewing stitch image data are sent to the model prediction module at the intelligent detection edge through the TCP / IP protocol, and the preprocessed sample image data are sent to the data management and analysis module at the data processing cloud through the MQTT protocol.

[0027] Step S103: training the preprocessed sample image data through a modern convolutional neural network to obtain a visual detection model.

[0028] The modern convolutional neural network is used to detect the types and defects of sewing stitches of parachute canopies, including but not limited to R-CNN, Fast R-CNN and YOLO.

[0029] The training by modern convolutional neural network is an optional solution, and the specific implementation includes: The preprocessed sample image data is input into the backbone network (Backbone) to obtain a feature map. The backbone network is composed of an improved version of CSPDarknet53, which is based on Darknet53 and uses a Cross-StagePartial Network (CSP) structure, mainly composed of five convolution modules, four C2f modules and one SPPF module. The preprocessed sample image data includes sewing type and defect information.

[0030] The feature map obtained by the backbone network is input into the feature enhancement network PAFPN (Path Aggregation Network for Instance Segmentation) to obtain an enhanced multi-scale feature map. The PAFPN consists of two convolution modules, four C2f modules, four fusion modules and two upsampling modules. In PAFPN, the scale of the feature map is adjusted by upsampling operation, so as to perform cross-scale feature fusion. After feature fusion, the convolution operation is used to further extract and integrate features, aiming to enhance the representation ability and robustness of the features.

[0031] The enhanced multi-scale feature map obtained by the feature enhancement network is input into the detector to obtain a visual detection model. The detector includes three detection heads of different scales for detecting objects of different sizes. Each detection head adopts a decoupled head structure to separate the classification and regression tasks. Among them, the classification task is used to predict the sewing type formula as follows: ; The regression task is used to predict the target bounding box, which can represent the defect information. The formula is as follows: ; ; ; ; ; Among them, P is the probability that the target object contained in the bounding box belongs to each category, is the activation function, is the weight of the classification task, E is the feature map, b x 、b y 、b w and b h are the center point, width and height of the predicted bounding box respectively, conf is the confidence, which indicates the probability of the bounding box containing the target object and the accuracy of the bounding box. (t x ,t y ) is the offset predicted by the network, (c x ,c y ) is the coordinate of the upper left corner of the grid cell, (t w ,t h ) is the scale change of network prediction, (p w ,p h ) is the width and height of the prior box, is the probability that the bounding box contains the target object, It is the intersection-over-union ratio between the predicted bounding box and the true bounding box.

[0032] The classification task of each detection head predicts the category of each grid cell in the input feature map. It uses convolution operations and activation functions to extract features, and outputs the predicted probability of each category through the fully connected layer. The classification loss is calculated based on these predicted probabilities to measure the difference between the category probability predicted by the model and the true label. The regression task of each detection head is responsible for predicting the bounding box of the target. It adopts the Anchor-Free detection method. It also uses convolution operations and activation functions to extract features, and outputs the coordinate information of the bounding box through the fully connected layer. The regression loss is calculated based on these predicted bounding boxes to measure the overlap between the predicted bounding box and the true bounding box. The calculation formula for the classification loss is as follows: ; in, C represents the total number of categories, t c Represents the one-hot encoding of the true label, if the target belongs to the category c ,but ,otherwise , p c The model predicts that the target belongs to the category c probability.

[0033] The regression loss is calculated as follows: ; ; ; in, is the total loss function of regression, and is the weight coefficient corresponding to the loss part, yes IoU Loss function, yes DFL Loss function, N represents the number of positive samples, is the predicted bounding box, is the predicted bounding box distance, is the corresponding ground-truth bounding box, The corresponding ground-truth bounding box distance, is the intersection-over-union ratio between the predicted box and the true box, is the difference between the predicted distance and the actual distance DFL loss.

[0034] The Adam optimizer is used to update the parameters, and the cosine annealing strategy is used to adjust the learning rate. The formula for parameter optimization by the Adam optimizer is: ; ; ; ; ; in, and Respectively T Momentum and variance at each moment, and denote the exponential decay rate of momentum and the exponential decay rate of variance, respectively. express T The gradient of time, and denote the deviation correction of momentum and the deviation correction of error, respectively. and Represent the parameters before and after the update, is the learning rate, is a small constant used to avoid dividing by 0 error.

[0035] The cosine annealing strategy adjusts the learning rate by simulating the change law of the cosine function and gradually reducing the learning rate so that the model can converge better. The calculation formula of this strategy is as follows: ; in, Indicatest Step learning rate, and Represent the maximum learning rate and the minimum learning rate respectively, t Indicates the current number of steps. T Indicates the total number of training rounds.

[0036] The trained model parameters are sent to the data management and analysis module via the TCP / IP protocol for subsequent analysis and optimization. The visual detection model is sent to the model prediction module deployed at the edge of the intelligent detection via the MQTT protocol.

[0037] Step S104: The visual inspection model in step S3 is used to complete the detection of the target area of ​​the preprocessed image data to be tested, and obtain the target area detection frame coordinates, sewing type and defect information.

[0038] The visual inspection model of the deep learning and optimization module from the data processing cloud is received through the MQTT protocol, and the obtained visual inspection model is loaded as the model of the prediction network. The preprocessed sewing stitch image data received from the data preprocessing module through the TCP / IP protocol is sent to the backbone network for feature extraction, and feature fusion is performed through the feature enhancement network. The fused feature map is sent to the detector for target detection, and the target detection results are post-processed, including coordinate transformation, NMS (Non-Maximum Suppression) and other operations to remove redundant detection frames and obtain the target area detection frame coordinates, sewing type and defect information.

[0039] The formula for the coordinate transformation is consistent with the formula used by the regression task to predict the target bounding box.

[0040] The NMS operation is mainly used to remove overlapping bounding boxes and retain the best one. The specific steps are as follows: For each detected bounding box , calculate its difference with the current highest confidence bounding box The intersection ratio . The calculation formula is as follows: ; in, is the intersection area of ​​the two bounding boxes, is the area of ​​the union of the two bounding boxes.

[0041] Set an NMS threshold. If If it is greater than the NMS threshold, it is considered and There are too many overlaps and should be suppressed, that is, removed from the candidate bounding box list. If the candidate bounding box list is not empty, the next highest confidence bounding box is selected as The above process is repeated until the candidate box list is empty, and the remaining bounding boxes are output as the final detection result.

[0042] The obtained sewing type and defect information are sent to the data management and analysis module deployed in the data processing cloud through the MQTT protocol, and the target area detection frame coordinates are sent to the quality inspection module through the TCP / IP protocol.

[0043] Step S105: Obtaining a data set of stitch length and stitch density distribution in the detection frame through the target area detection frame coordinates; sending the data set of stitch length and stitch density distribution to a data management and analysis module in the data processing cloud.

[0044] The data related to the target area detection frame include sewing type, sewing area detection frame coordinates, defect type and defect area detection frame coordinates. Sewing types include but are not limited to straight line type and sawtooth type. Defect types include but are not limited to broken thread, split thread, thread loop and missing thread.

[0045] The quality inspection module receives the data transmitted from the model prediction module, stores the corresponding data, and records the detection frame coordinates of the sewing type and sewing defect information of each category of objects (the upper left corner coordinate point of the target frame: x 1 , y 1 And the coordinates of the lower right corner: x 2 , y 2 ). In calculating the stitch length and stitch density distribution, the center point of each detection frame is (x,y,w, h) Indicates that (x,y) is the center point coordinate, w and h Represents the width and height of the detection box, and the calculation formula is as follows: ; ; When identifying the sewing type of a parachute canopy, since the sewing products of a parachute canopy have multiple lines of stitches, it is necessary not only to identify the type of sewing stitches, but also to cluster and group the detection frames on different lines. Then calculate the stitch length and stitch density distribution of each line. This method uses the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm to cluster the sewing stitches.

[0046] The specific steps of implementing the DBSCAN clustering algorithm include: Initialization parameters. Initialize two parameters to define the search radius and the range of the point. n is the minimum number of sample points, defining the minimum number of neighbors required to form a core point.

[0047] Determine the core point. For each p in the data set, Neighborhood Defined as: ; Where D is the data set, is the distance between p and q. If , then p is called the core point.

[0048] Cluster expansion. Assuming that the cluster is expanded from the core point p, the process is as follows: Initialize a cluster C and add p to C. For each point in C x ,examine x of Neighborhood .if There are points that have not been visited and meet the core point conditions. ,Will and All points in the neighborhood of are added to C and marked The above process is repeated until there are no new points in C that can be added.

[0049] Mark noise points. If a point is neither a core point nor any core point If the point is within the neighborhood of , the point is marked as a noise point.

[0050] Using DBSCAN clustering algorithm, the average width of sewing stitches 1.5 times of the search radius , set the minimum number of sample points n If the value is 1, each stitch can be grouped into one category. The average width calculation formula of the sewing stitch can be expressed as: ; in, M Represents the total number of detection boxes, Indicates i The width of the detection box.

[0051] After the first clustering, the clustering effect is not good due to the defects of sewing stitches, and the same line is divided into different line segments. A secondary clustering method is designed, based on each line of sewing line segments in the line segment set, and the first sewing stitch detection target frame and the last sewing stitch detection target frame of each line of sewing line segments are used as retrieval targets; the center point of the last sewing stitch detection target frame of the initial retrieval line segment is used as the origin, and the fan-shaped angle area to the right of the origin is 30°, and the fan radius is 3 times the average width. Whether there is a first sewing stitch detection target frame of the target line segment within the range, if so, the target line segment and the initial search line segment are classified as the same line segment.

[0052] Suppose the set of line segments obtained after clustering is , for each line segment , perform the following steps: Calculate the center point of the last sewing stitch detection target frame. The coordinates of the center point of the final frame are .

[0053] Sector area definition. As the origin, define the sector area. The angle range of the sector is , sector radius .

[0054] Detect the first frame of other line segments. (and ), detect its first frame Whether it is within the sector area. Use polar coordinates to determine and calculate Relative to Polar angle and distance , the formulas for the polar angle and distance are as follows: ; ; examine Is it in Inside, and .

[0055] Merge line segments. If If the first frame of and Update the segment collection for a new segment H .

[0056] The sewing stitch quality parameters are calculated for each segment of the clustered line. The sewing stitch quality parameters are mainly represented by stitch length, stitch density and sewing defects.

[0057] Option 1: An optional method for calculating stitch length and stitch density.

[0058] For the calculation of the trace length, an optional method is to calculate it by accumulating the Euclidean distances between the center points of adjacent detection frames. The calculation formula of the line length L is as follows: ; in, M is the total number of detection boxes, Indicates The difference between the coordinates of the center point of the i-th detection box and the coordinates of the center point of the i-th detection box.

[0059] For the trace density, one feasible method is to divide the obtained line length L into several small intervals evenly, and the length of each small interval is consistent. Traverse all the detection frames of the line, determine the position of each detection frame, classify it into the corresponding measurement interval, and count the number of detection frames in the measurement interval. For each measurement interval, calculate its trace density, the specific formula is as follows: ; in, N is the total number of measurement intervals, m is the number of detection boxes in the measurement interval, Indicates j The length of the measurement interval, Indicates j The stitch density of the measurement interval.

[0060] The serial number and stitch density of each measurement interval are recorded, and a stitch density distribution data set is constructed, and a stitch density distribution curve is drawn based on this data set.

[0061] The trace length and trace density distribution datasets are sent to the data management and analysis module of the data processing cloud through the MQTT protocol for subsequent data visualization and analysis.

[0062] Option 2: An optional ruler-based method for calculating stitch length and stitch density.

[0063] Place the ruler on the image, making sure it is parallel to the stitching but not overlapping so that both the stitching and the ruler markings can be seen. Use a camera or scanner to capture one image with the ruler in place and one image without the ruler, making sure the images are clear and high enough resolution to accurately measure the number of pixels in the stitching and the markings on the ruler.

[0064] The image processor performs grayscale processing on the image without a ruler, performs contour processing on the processed image without a ruler, fills the contour, and calculates the number of pixels of the sewing stitches. And the pixel length of the sewing seam in the measurement area The length of the measuring area of ​​the stitch can be determined from the image with the ruler. , thereby calculating the actual length represented by each pixel , the specific formula is as follows: ; Then calculate the stitch density , the stitch density can be defined as the actual length ratio occupied by the stitch per unit length. The specific formula is as follows: ; In the context of pursuing more efficient and automated measurement technology, an innovative solution is proposed to address the inconvenience and efficiency limitations caused by the manual placement of a ruler for measurement in Solution 2: a method for calculating the trace length and trace density based on a laser cross target. This method aims to achieve automatic recognition and accurate measurement of trace features by introducing a laser cross target as a reference object and combining image processing technology, thereby significantly improving measurement efficiency and freeing the operator's hands.

[0065] Solution 3: An optional method for calculating trace length and trace density based on a laser cross target.

[0066] The positions of the camera and the laser emitter are fixed. The camera and the laser emitter are fixedly mounted on the same bracket, and the relative positions between the two are fixed. When the system receives the command to start detection, it immediately sends a precise trigger signal to the laser emitter. After receiving the trigger signal, the laser emitter will respond quickly and emit a laser cross target with a color different from the parachute. In order to ensure that the laser cross target can accurately illuminate the two ends of the sewing stitch, the emission angle and position of the laser emitter need to be accurately calibrated. By adjusting the installation angle of the laser emitter, ensure that the intersection of the laser cross target is exactly at the two ends of the sewing stitch after emission. The camera captures an image with a laser cross target and an image without a laser cross target when the laser emitter is turned on and off. The image with the laser cross target contains two cross-shaped light spots and sewing stitches.

[0067] The image processor first grayscales the image with the laser cross target and the image without the laser cross target, and then performs contour processing on the processed image with the laser cross target and the image without the laser cross target to obtain the contour of the sewing trace with the laser cross target and the contour of the sewing trace without the laser cross target, respectively. The contour of the sewing trace with the laser cross target and the contour of the sewing trace without the laser cross target are subtracted to obtain a grayscale image containing only the contour with the laser cross target. The contour in the grayscale image is filled, and the Steger algorithm is used to extract the position of the center of the two crosses in the filled grayscale image, and then the pixel length between the two crosses is calculated by the Euclidean distance. , based on the length of the measured area between the two crosses , thereby calculating the actual length represented by each pixel , the specific formula is as described in Scheme 2.

[0068] The outline of the sewing seam without the laser cross target is filled in, and the image processor calculates the number of pixels of the sewing seam , the actual length of the sewing stitch is calculated by the number of pixels, and thus the density of the stitch is calculated , the specific formula is as described in Scheme 2.

[0069] When exploring more flexible and adaptable measurement methods, Schemes 2 and 3 usually require that the relative height between the camera and the parachute canopy to be measured remain fixed, which limits the diversity and practicality of the measurement scenarios to a certain extent. In order to overcome this limitation, an optional method for calculating the trace length and trace density based on a laser ranging sensor is introduced. This method not only retains the advantages of automated measurement, but also significantly enhances the adaptability of the measurement system to height changes.

[0070] Embodiment 4: An optional method for calculating trace length and trace density based on a laser ranging sensor.

[0071] Fix a laser distance sensor vertically at the end of the camera, make sure both the camera and the laser distance sensor are fixed, and use the laser distance sensor to measure the vertical distance from the camera to the parachute canopy d .

[0072] The camera's internal parameters are calibrated using standard camera calibration methods. The focal length of the camera is obtained by f , the physical width of the sensor S w and physical height S h , the pixel width of the image W and pixel height H And the object distance provided by the laser ranging sensord , calculate the actual length represented by each pixel The formula is as follows: ; The camera captures the image of the parachute canopy with the laser rangefinder sensor turned on, and the image is passed through the visual detection model to obtain the number of detection frames for the corresponding sewing stitches. M (The number of stitches). The image is preprocessed, and the image processor is used to identify the sewing stitches in the image and extract the contour information of the stitches.

[0073] For the extracted trace pixel coordinates, calculate the trace pixel length If the trace is a straight line, directly calculate the Euclidean distance from the start point to the end point. If the trace is a curve, add up the pixel distances between adjacent contour points in segments. The specific formula is as follows: ; Use the actual length represented by each pixel to convert the pixel length to the actual physical length L d , the specific formula is as follows: ; Count the number of detection frames of sewing stitches M , the trace density is calculated using the actual physical length of the trace and the number of detection frames. The trace density is expressed as the number of traces per unit length. The specific formula is as follows: .

[0074] Step S106: Output the sewing type, stitch density distribution data set, stitch length and defect information to the visualization module and visualize them on the data processing cloud. The defect information includes defect type and defect area.

[0075] The data of the data management and analysis module are sorted, and the results of sewing type, stitch density distribution data set, stitch length, defect type, and defect area are processed in JSON format. Among them, sewing types include straight and zigzag types, defect types include broken thread, split thread, thread loop and missing thread, stitch density distribution data set is output in the form of an array according to the sequence number and stitch density value of each measurement interval, stitch length includes the length value of each row of stitches, and defect area includes the pixel position of the four corners of each defect detection box.

[0076] The processed data is sent to the visualization module through the TCP / IP protocol, and the results are displayed and analyzed through the visualization module. The visualization interface displays the results of sewing type, stitch length, defect type, and defect area in the data processing cloud in the form of a table. The visualization interface constructs a stitch density distribution curve with the serial number of each measurement interval as the X-axis and the stitch density value as the Y-axis and displays it in the data processing cloud. By observing the stitch density distribution curve, if obvious valleys are found in the stitch density distribution curve, these positions may correspond to the existence of sewing defects (broken thread, split thread, and missing thread). By recording the serial number of the measurement interval where defects may exist in the stitch density distribution curve, the detection frame position information of the sewing defect is further obtained, and the relevant personnel are notified to deal with the sewing defect.

[0077] like Figure 2 FIG. 1 is a flowchart of a method for visually inspecting the quality of an aerospace parachute based on deep learning in another embodiment of the present invention. A system for visually inspecting the quality of an aerospace parachute based on deep learning comprises: Step S201: collecting original parachute canopy sewing stitch image data through the device end, and sending the original parachute canopy sewing stitch image data to the corresponding intelligent detection edge end. The original parachute canopy sewing stitch image data includes sample parachute canopy sewing stitch image data and parachute canopy sewing stitch image data to be tested.

[0078] Specifically, in this embodiment, the device side uses a high-precision camera as an image data acquisition device, which has high-resolution and high-speed shooting capabilities to ensure that every detail is captured. The camera can be controlled by the control device to move according to a predetermined trajectory and speed, and at the same time collect the original parachute canopy sewing stitch image data in the current scene. The control device is usually connected to the intelligent detection edge end and the data processing cloud, receives its instructions and performs corresponding actions. The collected original parachute canopy sewing stitch image data is directly transmitted from the device side to the data preprocessing module of the intelligent detection edge end via an Ethernet cable, and high-speed Ethernet technology is used to ensure high speed and low latency of data transmission.

[0079] Step S202: preprocessing the original sewing stitch image data in S201 through the data preprocessing module of the intelligent detection edge end to obtain preprocessed sample image data and preprocessed image data to be tested, and sending the corresponding data to the corresponding processing module.

[0080] The intelligent detection edge end preprocesses the original sewing stitch image data in S201 according to the actual operation requirements through the data preprocessing module. As an optional scheme, the preprocessing includes: noise removal, size adjustment and format conversion. The format conversion is mainly carried out according to the requirements of the input storage format of the modern convolutional neural network. The present embodiment converts it into a VOC data set. The preprocessed sewing stitch image data is annotated with the sewing type and defect information through the labelimg software. After the annotation, each image data forms a file with an XML suffix with the same name as the image, which contains the coordinate information and category of the detection frame. The preprocessed sewing stitch image data and its corresponding XML suffix file are generated into a VOC format data set through a script. The data set consists of an Annotations folder, an ImageSets folder and a JPEGImages folder. The Annotations folder is used to store annotation files in XML format, the ImageSets folder is used to store division information such as training sets and verification sets, and the JPEGImages folder is used to store original image data.

[0081] The preprocessed image data to be tested is sent to the model prediction module at the intelligent detection edge through the TCP / IP protocol, and the preprocessed sample image data is sent to the data management and analysis module at the data processing cloud through the MQTT protocol.

[0082] Step S203: The data in the data management and analysis module is transmitted to the deep learning and optimization module through the data processing cloud, and the initial visual detection model is obtained by training and optimizing the modern convolutional neural network.

[0083] The preprocessed sample image data is extracted from the data management and analysis module deployed on the data processing cloud, and the preprocessed sample image data is transmitted to the deep learning and optimization module of the data processing cloud via the TCP / IP protocol.

[0084] The initial visual detection model is obtained by training and optimizing modern convolutional neural networks, using the Adam optimizer for parameter update and the cosine annealing strategy to adjust the learning rate.

[0085] Step S204: Testing the initial visual inspection model using the verification set obtained by the data management and analysis module to determine whether the test result meets expectations.

[0086] Use the validation set to comprehensively evaluate the initial visual detection model, calculate the average precision mean, accuracy, recall and other indicators, and comprehensively evaluate the performance of the model. Determine whether the model has achieved the expected results, such as whether the average precision mean exceeds a certain threshold. If the expected results are not achieved, the data processing cloud sends instructions to the control device on the device side to control the data acquisition device to continue to collect data and repeat steps S201 to S204; if the expected results are achieved, the obtained visual detection model is sent to the model prediction module of the intelligent detection edge through the MQTT protocol and transmitted to the data management and analysis module through the TCP / IP protocol.

[0087] Step S205: The image data to be tested preprocessed in S202 is predicted by the visual inspection model obtained in S204, the detection of the target area is completed, the target area detection frame coordinates, sewing type and defect information are obtained, and the data is sent to the corresponding processing module.

[0088] The visual inspection model from the deep learning and optimization module of the data processing cloud is received through the MQTT protocol, and the obtained visual inspection model is loaded as the model of the prediction network. The preprocessed sewing stitch image data received from the data preprocessing module through the TCP / IP protocol is sent to the prediction network for target detection, and the target area detection frame coordinates, sewing type and defect information are obtained.

[0089] The obtained sewing type and defect information are sent to the data management and analysis module deployed in the data processing cloud through the MQTT protocol, and the coordinates of the marked area detection frame are sent to the quality inspection module through the TCP / IP protocol.

[0090] Step S206: Obtain the corresponding stitch length and stitch density distribution data set through the target area detection frame coordinate data obtained in S205, and send the stitch length and stitch density distribution data set to the data management and analysis module of the data processing cloud.

[0091] The trace length and trace density distribution datasets are sent to the data management and analysis module of the data processing cloud through the MQTT protocol for subsequent data visualization and analysis.

[0092] Step S207: Output the sewing type, stitch density distribution data set, stitch length and defect information to the visualization module and visualize them on the data processing cloud. The defect information includes defect type and defect area.

[0093] The data from the data management and analysis module are sorted, and the results of sewing type, stitch density distribution data set, stitch length, defect type, and defect area are processed in JSON format.

[0094] The processed data is sent to the visualization module through the TCP / IP protocol, the results are displayed and analyzed through the visualization module, and the relevant personnel are notified to deal with the sewing defects. At the same time, the data processing cloud sends instructions to the control device at the device end, controlling the data acquisition device to continue collecting data and repeat the above steps.

[0095] like Figure 3 As shown, a system structure block diagram of an aerospace parachute quality visual inspection system based on deep learning, including a device side, an intelligent detection edge side, and a data processing cloud side.

[0096] The device side consists of data acquisition equipment and control equipment, which are mainly responsible for data collection and perform corresponding operations according to control instructions. The intelligent detection edge side consists of model prediction module, data preprocessing module and quality detection module. A distributed structure is adopted. According to the processing requirements, edge devices with sufficient computing power, storage capacity and network connection capabilities are selected, and each module is deployed on a suitable edge device, which is mainly responsible for preprocessing and detecting data. The data processing cloud consists of deep learning and optimization module, data management and analysis module and visualization module, which are mainly responsible for global data storage, management, analysis and optimization.

[0097] The device end is mainly composed of a data acquisition device and a control device. The control device controls the data acquisition device to collect original parachute canopy sewing stitch image data, and sends the original parachute canopy sewing stitch image data to the corresponding intelligent detection edge end.

[0098] The intelligent detection edge end preprocesses the original sewing stitch image data collected by the device end through the data preprocessing module on the edge device 1, obtains the preprocessed sample image data and the preprocessed image data to be tested, uploads the preprocessed image data to the model prediction module on the edge device 2 for prediction, and sends the data to the data processing cloud.

[0099] The intelligent detection edge predicts the preprocessed image data to be tested through the model prediction module on the edge device 2 based on the visual inspection model sent by the data processing cloud, completes the detection of the target area, obtains the target area detection frame coordinates, sewing type and defect information, and sends the data to the data processing cloud, and transmits it to the quality inspection module on the edge device 3 for subsequent processing.

[0100] The quality detection module on the edge device 3 of the intelligent detection edge end calculates the stitch length and stitch density distribution data set according to the detection frame coordinate data output by the model prediction module. At the same time, the detected defects are further analyzed and evaluated, and the analysis results and defect information are uploaded to the data processing cloud for storage and analysis.

[0101] The data processing cloud receives preprocessed data from the intelligent detection edge, stores the preprocessed data in the data management and analysis module, and trains the preprocessed data through the deep learning and optimization module. After the training is completed, the obtained visual detection model is sent to the intelligent detection edge. At the same time, the model is continuously iterated and optimized based on the feedback data and results.

[0102] The data processing cloud receives data uploaded by the device and the intelligent detection edge, and stores and manages it. It also stores the results of deep learning and optimization module training. It queries, analyzes and mines the data as needed. It provides analysis results and decision support information to relevant personnel or systems.

[0103] The data processing cloud provides an intuitive data visualization interface through a visualization module, displaying the results of sewing stitch detection, stitch length and density distribution curves, sewing defect detection and other information, helping relevant personnel to better understand the data and detection results and make decisions.

[0104] The above-mentioned device end and the intelligent detection edge end are connected via Ethernet, and the intelligent detection edge end and the data processing cloud end are connected via the MQTT protocol.

[0105] like Figure 4 As shown in the figure, the function diagram of each module of an aerospace parachute quality visual inspection system based on deep learning includes: Data acquisition device 401, control device 402, model prediction module 403, quality detection module 404, data preprocessing module 405, deep learning and optimization module 406, data management and analysis module 407 and visualization module 408.

[0106] Among them, the data acquisition device 401 is used to collect the original parachute canopy sewing stitch image data; the control device 402 drives the data acquisition device to perform corresponding operations according to the received control instructions; the model prediction module 403 uses the trained visual detection model to make real-time predictions on the preprocessed data. These prediction results can be used for subsequent processing; the quality detection module 404 performs quality inspection on the prediction results to obtain the length, density and defects of the sewing stitches, thereby improving the reliability of the system; the data preprocessing module 405 preprocesses and converts the collected original parachute canopy sewing stitch image data to provide high-quality data input for subsequent model training and model prediction; the deep learning and optimization module 406 uses big data and deep learning technology to train and optimize the model in the system. Through continuous learning and iteration, the model can gradually improve the prediction accuracy and generalization ability; the data management and analysis module 407 is responsible for storing and managing the data uploaded by the device end and the intelligent detection edge end, and can provide data query, statistical analysis and data mining functions according to actual work requirements. These functions help to discover patterns and trends in data and provide support for decision-making; the visualization module 408 presents complex data and information in an intuitive and easy-to-understand way, making it easier for users to understand and analyze. Through the visualization interface, you can monitor the operating status of the system in real time, view data trends and analysis results, etc.

[0107] like Figure 5 As shown, the aerospace parachute quality visual inspection system based on deep learning of the present invention includes a data processing cloud, an intelligent detection edge end and a device end. Each end works together to realize digital real-time monitoring of the quality of the parachute processing process.

[0108] The beneficial effects of the present invention are that, through the collaborative work of the device end, the intelligent detection edge end and the data processing cloud end, the beneficial effects of efficient automation, high-precision prediction, real-time feedback, data management and visual output of the parachute canopy sewing stitch quality detection are achieved. These effects jointly improve the quality inspection level of parachutes and provide a strong guarantee for safe production in the aerospace field.

[0109] The applicant of the present invention has made a detailed explanation and description of the implementation examples of the present invention in conjunction with the drawings in the specification. However, those skilled in the art should understand that the above implementation examples are only preferred implementation schemes of the present invention, and the detailed description is only to help readers better understand the spirit of the present invention, but not to limit the scope of protection of the present invention. On the contrary, any improvements or modifications based on the inventive spirit of the present invention should fall within the scope of protection of the present invention.

Claims

1. A method for visual inspection of aerospace parachute quality based on deep learning, characterized in that: include: S101, collecting original parachute canopy sewing stitch image data; the original parachute canopy sewing stitch image data includes sample parachute canopy sewing stitch image data and test parachute canopy sewing stitch image data; S102, preprocessing the original parachute canopy sewing stitch image data to obtain preprocessed sample image data and preprocessed image data to be tested; S103, training the preprocessed sample image data through a modern convolutional neural network to obtain a visual detection model; S104, completing the detection of the target area of ​​the preprocessed image data to be tested through the visual detection model, and obtaining the target area detection frame coordinates, sewing type and defect information; S105, obtaining a data set of stitch length and stitch density distribution in the detection frame through the target area detection frame coordinates; S106, visually displaying the sewing type, stitch density distribution data set, stitch length, and defect information results.

2. The visual inspection method according to claim 1, characterized in that: In step S102, after removing noise and adjusting the size of the sample parachute canopy sewing stitch image data, the obtained sample image data is labeled with sewing type and defect information through labelimg software, and converted accordingly according to the requirements of the input storage format of modern convolutional neural networks.

3. The visual inspection method according to claim 2, characterized in that: Step S103 is specifically as follows: inputting the preprocessed sample image data into the backbone network to obtain a feature map, wherein the backbone network is composed of an improved version of CSPDarknet53, the network structure is based on Darknet53 and uses the Cross-Stage Partial Network structure, and is composed of five convolution modules, four C2f modules and one SPPF module; inputting the feature map into the feature enhancement network PAFPN for feature fusion to obtain an enhanced multi-scale feature map; the PAFPN is composed of two convolution modules, four C2f modules, four fusion modules and two upsampling modules; inputting the multi-scale feature map into the detector for target detection to obtain a visual detection model.

4. The visual inspection method according to claim 3, characterized in that: Step S103 also includes optimizing the visual detection model, using the Adam optimizer to update parameters, and using the cosine annealing strategy to adjust the learning rate to optimize the visual detection model; the learning rate is calculated as follows: ; in, represents the learning rate at step t, and They represent the maximum learning rate and the minimum learning rate respectively, t represents the current step number, and T represents the total number of training rounds.

5. The visual inspection method according to claim 4, characterized in that: The detector includes three detection heads of different scales for detecting targets of different sizes; each detection head adopts a decoupled head structure and performs a classification task and a regression task respectively, the classification task is used to predict the sewing type, and the regression task is used to predict defect information.

6. The visual inspection method according to claim 5, characterized in that: The classification task is used to predict the sewing type, and the formula is: ; The regression task is used to predict the target bounding box, which can represent the defect information. The formula is: ; ; ; ; ; Among them, P is the probability that the target object contained in the bounding box belongs to each category, is the activation function, is the weight of the classification task, E is the feature map, b x , b y , b w and b h are the center point, width and height of the predicted bounding box respectively, conf is the confidence, which indicates the probability of the bounding box containing the target object and the accuracy of the bounding box. (t x ,t y ) is the offset predicted by the network, (c x ,c y ) is the coordinate of the upper left corner of the grid cell, (t w ,t h ) is the scale change of network prediction, (p w ,p h ) is the width and height of the prior box, is the probability that the bounding box contains the target object, It is the intersection-over-union ratio between the predicted bounding box and the true bounding box.

7. The visual inspection method according to claim 6, characterized in that: The method of obtaining a data set of stitch length and stitch density distribution in the detection frame by using the target area detection frame coordinates includes: using the DBSCAN algorithm to perform the first clustering of the sewing stitches to obtain a line segment set, taking each row of sewing line segments in the line segment set as the basis, and taking the first sewing line stitch detection target frame and the last sewing line stitch detection target frame of each row of sewing line segments as the retrieval target; taking the center point of the last sewing line stitch detection target frame of the initial retrieval line segment as the origin, and retrieving a fan-shaped angle area of ​​30° to the right of the origin with a fan-shaped radius of 3 times the average width Whether there is a first sewing stitch detection target frame of the target line segment within the range, if so, the target line segment and the initial search line segment are classified as the same line segment.

8. The visual inspection method according to claim 7, characterized in that: The original parachute canopy sewing stitch image data in step S101 is collected by a camera, and a laser ranging sensor is vertically fixed at the end of the camera. The camera captures the image of the parachute canopy when the laser ranging sensor is turned on. Each pixel of the image represents the actual length for: ; Where, the focal length of the camera is f and the physical width of the sensor is S w , the pixel width of the image is W and the object distance provided by the laser ranging sensor is d.

9. A deep learning-based visual inspection system for aerospace parachute quality, which is used to implement the visual inspection method according to any one of claims 1 to 8, characterized in that: It includes the device side, the intelligent detection edge side and the data processing cloud side; the device side is composed of data acquisition equipment and control equipment; the intelligent detection edge side is composed of model prediction module, data preprocessing module and quality detection module; the data processing cloud side is composed of deep learning and optimization module, data management and analysis module and visualization module.

10. The visual inspection system according to claim 9, characterized in that: The data acquisition device is used to collect the original parachute canopy sewing stitch image data; the control device drives the data acquisition device to perform operations according to the received control instructions; the model prediction module uses the trained visual detection model to perform real-time prediction on the pre-processed data; The quality inspection module performs quality inspection on the prediction results to obtain the length, density and defects of the sewing stitches; the data preprocessing module preprocesses the collected original parachute canopy sewing stitch image data; the deep learning and optimization module uses big data and deep learning technology to train and optimize the model in the system; The data management and analysis module stores and manages the data uploaded by the device and the intelligent detection edge; The visualization module presents complex data and information intuitively.

Images