An aviation structure piece image preprocessing method based on a deep neural network model

Abstract

This invention discloses a preprocessing method for aerospace structural component images based on a deep neural network model, belonging to the field of aerospace structural component recognition. The specific steps are as follows: Training samples for a semantic segmentation network model are created, trained, and the model's weight parameters are obtained. A mask image of the location of the pallet containing the component is obtained using this model. Specific frames are extracted from the video, and the deep semantic segmentation model is used to calculate the mask image for each frame. The video is read frame by frame, and after grayscale conversion, a bitwise AND operation is performed with the specified mask image. The changes in the center point position and area of ​​the target component region between the current frame and the previous frame are calculated to determine if the frame has been correctly cropped. This invention can significantly improve the data quality of the acquired component images, providing highly reliable component image data for the subsequent training and application of component recognition models. Simultaneously, the efficiency of this model in processing component videos is significantly improved, saving manpower and resources in data cleaning.

Description

Technical Field

[0001] This invention relates to the field of aerospace computer technology, and in particular to a method for identifying aerospace structural components based on a deep neural network model. Background Technology

[0002] The use of deep neural networks for recognition of aerospace structural components presents challenges in data acquisition. To address this, video recordings were used to capture images of the parts and construct a dataset. However, the video recording process is marred by background noise and clutter, resulting in low data quality and hindering the subsequent construction and training of the recognition network. To capture images from multiple angles, the method employed involved rotating the part on a tray and recording it for one minute at a time, resulting in approximately 1500 images. However, the presence of background noise and clutter in the image dataset makes direct processing slow, leading to inefficient data processing. Summary of the Invention

[0003] To address the problem of excessive background clutter and slow data processing in the acquired part image dataset, this invention proposes a preprocessing method for aerospace structural parts images based on a deep neural network model.

[0004] To achieve the above-mentioned technical effects, the present invention is implemented through the following technical solution:

[0005] A method for image preprocessing of aerospace structural components based on a deep neural network model includes the following steps:

[0006] Step 1. Construct training and testing datasets for a deep convolutional neural network;

[0007] Step 2. Construct a deep convolutional neural network model based on ResNet-50 and DeeplabV3+ network structures;

[0008] Step 3. Train the model and select the network model that performs best in the test set;

[0009] Step 4. Crop the specified frame images of all videos, calculate the mask images of these video frames using a deep convolutional neural network model, and save the mask images;

[0010] Step 5. Read the video file frame by frame, copy one copy and save it in memory. After grayscale processing of the video frame, perform a bitwise AND operation with the mask image to find the location of the target part.

[0011] Step 6. Calculate the center point and area of ​​the current target part position, and compare them with the center point and position of the target part position in the previous frame;

[0012] Step 7. Determine whether the part cutting is correct by comparison. If it is incorrect, discard the image frame. If it is correct, continue reading the next frame.

[0013] Step 8. Transform the coordinates of the target part area to the original image, use the transformed coordinates to crop the copied original image, and save the cropped part image.

[0014] Furthermore, a deep convolutional neural network model was constructed. The model is based on the DeepLabV3+ semantic segmentation network of ResNet-50. The network model was modified to enhance the network's ability to fuse deep features of the image. The input of the model is a color image with a width and height of 512 pixels, and the output is a single-channel image with a width and height of 512 pixels.

[0015] Randomly select 200 videos, crop out the first frame of each video, and divide these images into 80% training set and 20% test set, with no overlap between the training and test sets. Label the training and test sets, with the tray area in white and the background area in black.

[0016] The model was trained using the training set for 100 epochs, with the cross-entropy function chosen as the loss function and the learning rate set to 0.001. After each epoch, the training results were saved, and the results were tested using the test set, with the loss function value recorded on the test set. After training was complete, the weight file corresponding to the minimum loss function was selected as the final weight file, and the remaining weight files were deleted.

[0017] Cropping frames 0, 350, 700, and 1050 of all captured videos and keeping them in the same folder, naming them with the video name plus the order of the frame in the entire video;

[0018] Call the saved parameter file and model, compress all frames into images with a width and height of 512 pixels, input them into the model for calculation, and store the results in another folder with the file name unchanged;

[0019] Read the video, copy each video frame, convert each frame to grayscale, and compress it to 512 pixels in both width and height. Read the mask image group corresponding to the video name, arrange them in ascending order of cropped frame position, and change the mask image every 350 frames. Perform a bitwise AND operation between this mask image and the grayscale video frames. Then, use thresholding to find the area enclosed by the maximum boundary. Obtain the minimum and maximum values ​​of the outline enclosing this maximum area in length and width.

[0020] Calculate the area and center point of the rectangular region enclosed by the maximum and minimum values ​​of the maximum outline in length and width. Use the area and center point of the previous frame as reference values ​​for the area and center point of the current frame. If the difference between the area and center point of the current frame and the previous frame is small, the cropping is considered correct; otherwise, it is considered incorrect. The first frame is considered correctly cropped by default.

[0021] If the cropping is correct, the maximum and minimum values ​​of the maximum outline in length and width will be varied to obtain the position of the target part area in the original image. This area will then be cropped from the copied video frame and saved in the specified folder.

[0022] Furthermore, the deep convolutional neural network model includes 5 EncoderBlocks and 3 DecoderBlocks. The outputs of EncoderBlock4 and EncoderBlock5 are further fused using a 1*1 convolutional kernel. The result is then fused with the output of the ASPP structure and transmitted to DecoderBlock1. DecoderBlock1 also receives the output from EncoderBlock3 as input. DecoderBlock2 receives the outputs from EncoderBlock2 and DecoderBlock1, while EncoderBlock3 only receives the output from DecoderBlock2. The final output is the target semantic segmentation image.

[0023] Furthermore, the decoding process of the deep convolutional neural network model is as follows: DecoderBlock1 receives the output of the deep fusion of feature maps and the output of EncoderBlock3 as input data. It first upsamples the feature map from the Aspp image spatial pyramid with a stride of 2, adjusts the size of the feature map to 64 pixels in both length and width, stacks it with the output of the EncoderBlock3 module, and then convolves the stacked data with 256 3*3 convolutional kernels to extract features and output them to DecoderBlock2.

[0024] DecoderBlock2 receives the output from EncoderBlock2 and the output from DecoderBlock1 as input. It upsamples the output of DecoderBlock1 with a stride of 2, adjusts the feature map to 128 pixels in both width and height, stacks it with the output of EncoderBlock2, decodes the feature map using 128 3*3 convolutional kernels, and outputs it to the next DecoderBlock3.

[0025] DecoderBlock3 only accepts the output from DecoderBlock2 as input. It upsamples the output of DecoderBlock2 with a stride of 4, adjusts the feature map size to 512 pixels in both width and height, and convolves the input with three 3*3 convolutional layers with 64, 32, and 1 kernels respectively. The activation function of the last convolutional layer is the softmax activation function. The final output is a single-channel image with a width and height of 512 pixels.

[0026] Further, in step 4, the video processing mask frame is obtained by extracting the 0th, 350th, 700th, and 1050th frames of all videos and naming them with the video name plus an underscore plus the frame number.

[0027] Furthermore, the video frame mask image is generated as follows: all cropped video frames are compressed using the saved parameter model, and the compressed image size is 512 pixels in length and 512 pixels in width; the compressed image is transmitted to the parameter model for calculation, and the output image is saved to another folder with the file name being the original name of the video frame.

[0028] Furthermore, the video is converted to grayscale: the video is acquired and the mask images corresponding to the 0th, 350th, 700th, and 1050th frames of the video are obtained sequentially from the folder where the mask images are stored, according to the video name, and stored in the same array mask_array in ascending order of video frames;

[0029] Read the acquired video file frame by frame from the beginning, and then read the video frames. frm Make a copy src_img , video frames frm Perform grayscale processing according to Formula 1;

[0030]

[0031] Where R, G, and B are the values ​​of the RGB three channels of the pixel, respectively;

[0032] grayscale processed video frames frm Compress it to an image with dimensions of 512 pixels each, and select a mask image;

[0033] The current compressed video frame frm With mask image mask Perform a bitwise AND operation on the result of the AND operation. frm The image is binarized according to Formula 2:

[0034]

[0035] Where x is the pixel value of the pixel.

[0036] Find all the boundaries and calculate the area of ​​the regions enclosed by each boundary. Find the boundary with the largest enclosed area and obtain the minimum width and height of that boundary. x_min , y_min and maximum value x_max , y_max Perform the following calculations:

[0037]

[0038]

[0039] Calculate the video frames respectively frm The center point of the target part area and enclosed area .

[0040] Furthermore, video frames frm The compression method is as follows:

[0041] The current video frame number num, and the current mask array index mask_num

[0042] If num%350==0 and num<1500:

[0043] mask = mask_array[mask_num]

[0044] mask_num = mask_num + 1

[0045] else:

[0046] mask = mask_1050

[0047] Where num is the video frame counter, which increments by 1 for each frame read; mask_num is the mask image array index counter, which changes the mask image every 350 frames read.

[0048] Furthermore, when processing video frames frm When the first frame is an image, record the current frame. This video frame frm of Set as the reference area and reference center point coordinates of the target part region in the next frame image, when the video frame... frm For frames after the first frame, calculate the area and center point coordinates of the current video frame's frm according to formulas 3 and 4. Calculate the area of ​​the current frame according to formulas 5 and 6. Distance from the center point The differences between them are used to determine whether the cutting area of ​​the target part is correct;

[0049]

[0050]

[0051] In formula 6 As a reference, the coordinate position of the center point on the width. For the current video frame frm At the coordinate position on the width, As a reference, the center point's coordinates on the altitude. Current video frame frm The coordinate position at the high altitude.

[0052] Furthermore, and If the cropping is correct, it is considered correct; otherwise, it is considered incorrect. Frames with incorrect cropping will be discarded, and the next frame will be read and processed. If the cropping is correct, the current video frame will be... frm of Set as reference area and center point coordinates for the next frame .

[0053] Furthermore, cropping the original image specifically involves: cropping the current video frame... frm Minimum width and height x_ min , y_min Maximum value x_max , y_max, Calculate the actual position of the target part region in the src_img image using formulas 7, 8, 9, and 10:

[0054]

[0055]

[0056]

[0057]

[0058] in For the video frames in step 7 frm The width of the image's src field at the beginning of the copy. For the video frames in step 7 frm The image copied at the beginning src_img of high; For the target part area in src_img The top-left and bottom-right corner coordinates of the rectangular region in the image are used to determine the coordinates in the original image. src_imgUpper crop rectangle area All pixels in between are saved as images of the target part. The parent directory of the storage folder is the part number corresponding to the video, and the current directory is the name of the video. The file name format is the current time followed by the image number. frm The video frame counter num.

[0059] The advantages of this application are:

[0060] This invention can significantly improve the data quality of acquired part images, providing highly reliable part image data for the subsequent training and application of part recognition models. Simultaneously, the model's efficiency in processing part videos is significantly improved, saving manpower and resources in data cleaning. Attached Figure Description

[0061] Figure 1 This is the overall flowchart of the present invention.

[0062] Figure 2 This is a structural diagram of the tray assembly.

[0063] Figure 3 This is a top view of the tray assembly.

[0064] Figure 4 This is a diagram showing the overall structure of a deep network model.

[0065] Figure 5 This is a diagram illustrating the feature fusion process between EncoderBlock4,5 and the Aspp module.

[0066] Figure 6 This is a diagram of the Aspp module structure.

[0067] Figure 7 This is the structure diagram of DecoderBlock1 and DecoderBlock2.

[0068] Figure 8 This is the structure diagram of DecoderBlock3. Detailed Implementation

[0069] To more clearly illustrate the technical solutions provided by this invention, the invention will be further described below in conjunction with the accompanying drawings and embodiments. It should be noted that the provided embodiments are merely some, not all, embodiments of this invention, and therefore should not be considered as limiting the scope of protection. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0070] Example 1

[0071] A method for image preprocessing of aerospace structural components based on a deep neural network model includes the following steps:

[0072] Step 1. Construct training and testing datasets for a deep convolutional neural network;

[0073] Step 2. Construct a deep convolutional neural network model based on ResNet-50 and DeeplabV3+ network structures;

[0074] Step 3. Train the model and select the network model that performs best in the test set;

[0075] Step 4. Crop the specified frame images of all videos, calculate the mask images of these video frames using a deep convolutional neural network model, and save the mask images;

[0076] Step 5. Read the video file frame by frame, copy one copy and save it in memory. After grayscale processing of the video frame, perform a bitwise AND operation with the mask image to find the location of the target part.

[0077] Step 6. Calculate the center point and area of ​​the current target part position, and compare them with the center point and position of the target part position in the previous frame;

[0078] Step 7. Determine whether the part cutting is correct by comparison. If it is incorrect, discard the image frame. If it is correct, continue reading the next frame.

[0079] Step 8. Transform the coordinates of the target part area to the original image, use the transformed coordinates to crop the copied original image, and save the cropped part image.

[0080] Furthermore, a deep convolutional neural network model was constructed. The model is based on the DeepLabV3+ semantic segmentation network of ResNet-50. The network model was modified to enhance the network's ability to fuse deep features of the image. The input of the model is a color image with a width and height of 512 pixels, and the output is a single-channel image with a width and height of 512 pixels.

[0081] Randomly select 200 videos, crop out the first frame of each video, and divide these images into 80% training set and 20% test set, with no overlap between the training and test sets. Label the training and test sets, with the tray area in white and the background area in black.

[0082] The model was trained using the training set for 100 epochs, with the cross-entropy function chosen as the loss function and the learning rate set to 0.001. After each epoch, the training results were saved, and the results were tested using the test set, with the loss function value recorded on the test set. After training was complete, the weight file corresponding to the minimum loss function was selected as the final weight file, and the remaining weight files were deleted.

[0083] Cropping frames 0, 350, 700, and 1050 of all captured videos and keeping them in the same folder, naming them with the video name plus the order of the frame in the entire video;

[0084] Call the saved parameter file and model, compress all frames into images with a width and height of 512 pixels, input them into the model for calculation, and store the results in another folder with the file name unchanged;

[0085] Read the video, copy each video frame, convert each frame to grayscale, and compress it to 512 pixels in both width and height. Read the mask image group corresponding to the video name, arrange them in ascending order of cropped frame position, and change the mask image every 350 frames. Perform a bitwise AND operation between this mask image and the grayscale video frames. Then, use thresholding to find the area enclosed by the maximum boundary. Obtain the minimum and maximum values ​​of the outline enclosing this maximum area in length and width.

[0086] Calculate the area and center point of the rectangular region enclosed by the maximum and minimum values ​​of the maximum outline in length and width. Use the area and center point of the previous frame as reference values ​​for the area and center point of the current frame. If the difference between the area and center point of the current frame and the previous frame is small, the cropping is considered correct; otherwise, it is considered incorrect. The first frame is considered correctly cropped by default.

[0087] If the cropping is correct, the maximum and minimum values ​​of the maximum outline in length and width will be varied to obtain the position of the target part area in the original image. This area will then be cropped from the copied video frame and saved in the specified folder.

[0088] Furthermore, the deep convolutional neural network model includes 5 EncoderBlocks and 3 DecoderBlocks. The outputs of EncoderBlock4 and EncoderBlock5 are further fused using a 1*1 convolutional kernel. The result is then fused with the output of the ASPP structure and transmitted to DecoderBlock1. DecoderBlock1 also receives the output from EncoderBlock3 as input. DecoderBlock2 receives the outputs from EncoderBlock2 and DecoderBlock1, while EncoderBlock3 only receives the output from DecoderBlock2. The final output is the target semantic segmentation image.

[0089] Furthermore, the decoding process of the deep convolutional neural network model is as follows: DecoderBlock1 receives the output of the feature map deep fusion in step 3 and the output of EncoderBlock3 as input data. It first upsamples the feature map from the Aspp image spatial pyramid with a stride of 2, adjusts the size of the feature map to 64 pixels in both length and width, stacks it with the output of the EncoderBlock3 module, and then convolves the stacked data with 256 3*3 convolutional kernels to extract features and output them to DecoderBlock2.

[0090] DecoderBlock2 receives the output from EncoderBlock2 and the output from DecoderBlock1 as input. It upsamples the output of DecoderBlock1 with a stride of 2, adjusts the feature map to 128 pixels in both width and height, stacks it with the output of EncoderBlock2, decodes the feature map using 128 3*3 convolutional kernels, and outputs it to the next DecoderBlock3.

[0091] DecoderBlock3 only accepts the output from DecoderBlock2 as input. It upsamples the output of DecoderBlock2 with a stride of 4, adjusts the feature map size to 512 pixels in both width and height, and convolves the input with three 3*3 convolutional layers with 64, 32, and 1 kernels respectively. The activation function of the last convolutional layer is the softmax activation function. The final output is a single-channel image with a width and height of 512 pixels.

[0092] Further, in step 4, the video processing mask frame acquisition involves extracting frames 0, 350, 700, and 1050 from all videos and naming them with the video name plus an underscore and the frame number, for example: 10.0.0.1_202001010000_0.jpg, where 10.0.0.1_202001010000_0.jpg is the video name, and the trailing 0 indicates that it is the 0th frame of the video.

[0093] Furthermore, the video frame mask image is generated as follows: all video frames cropped in step 6 are compressed using the parameter model saved in step 5. The compressed image size is 512 pixels in length and 512 pixels in width. The compressed image is then transmitted to the parameter model in step 5 for calculation. The output image is saved to another folder with the file name being the original name of the video frame.

[0094] Further, the video is converted to grayscale: the video is acquired and the mask images corresponding to the 0th, 350th, 700th, and 1050th frames of the video are sequentially obtained from the folder where the mask images are stored, according to the video name and the mask image naming method specified in step 6. The mask images are stored in the same array mask_array in ascending order of video frames.

[0095] Read the video file obtained in step 8 frame by frame, starting from the beginning, and read the video frames. frm Make a copy src_img , video frames frm Perform grayscale processing according to Formula 1;

[0096]

[0097] Where R, G, and B are the values ​​of the RGB three channels of the pixel, respectively;

[0098] grayscale processed video frames frm Compress it to an image with dimensions of 512 pixels each, and select a mask image;

[0099] The current compressed video frame frm With mask image mask Perform a bitwise AND operation on the result of the AND operation. frm The image is binarized according to Formula 2:

[0100]

[0101] Where x is the pixel value of the pixel.

[0102] Using OpenCV's findContours method, find all boundaries, calculate the area of ​​the region enclosed by each boundary, find the boundary with the largest area, and obtain the minimum width and height of that boundary. x_min , y_min and maximum value x_max , y_max Perform the following calculations:

[0103]

[0104]

[0105] Calculate the video frames respectively frm The center point of the target part area and enclosed area .

[0106] Furthermore, video frames frm The compression method is as follows:

[0107] The current video frame number num, and the current mask array index mask_num

[0108] If num%350==0 and num<1500:

[0109] mask = mask_array[mask_num]

[0110] mask_num = mask_num + 1

[0111] else:

[0112] mask = mask_1050

[0113] Where num is the video frame counter, which increments by 1 for each frame read; mask_num is the mask image array index counter, which changes the mask image every 350 frames read.

[0114] Furthermore, when the video frames processed in step 7... frm When the first frame is an image, record the current frame. This video frame frm of Set as the reference area and reference center point coordinates of the target part region in the next frame image, when the video frame... frm For frames after the first frame, calculate the area and center point coordinates of the current video frame's frm according to formulas 3 and 4. Calculate the area of ​​the current frame according to formulas 5 and 6. Distance from the center point The differences between them are used to determine whether the cutting area of ​​the target part is correct;

[0115]

[0116]

[0117] In formula 6 As a reference, the coordinate position of the center point on the width. For the current video frame frm At the coordinate position on the width, As a reference, the center point's coordinates on the altitude. Current video frame frm The coordinate position at the high altitude.

[0118] Furthermore, and If the cropping is correct, it is considered correct; otherwise, it is considered incorrect. Frames with incorrect cropping are discarded, and the next frame is read and processed starting from step 7. If the cropping is correct, the current video frame is... frm of Set as reference area and center point coordinates for the next frame .

[0119] Furthermore, the original image is cropped specifically by cropping the current video frame obtained in step 7. frm Minimum width and height x_min , y_min Maximum value x_max , y_max, Calculate the actual position of the target part region in the src_img image using formulas 7, 8, 9, and 10:

[0120]

[0121]

[0122]

[0123]

[0124] in For the video frames in step 7 frm The width of the image's src field at the beginning of the copy. For the video frames in step 7 frm The image copied at the beginning src_img of high; For the target part area in src_img The top-left and bottom-right corner coordinates of the rectangular region in the image are used to determine the coordinates in the original image. src_img Upper crop rectangle area All pixels in between are saved as images of the target part. The parent directory of the storage folder is the part number corresponding to the video, and the current directory is the name of the video. The file name format is the current time followed by the image number. frm The video frame counter num; the specific format is: part number / video name / year-month-day-hour-minute-second_num.jpg.

[0125] Example 2

[0126] This invention uses on-site images collected at different times to create training samples for a semantic segmentation network model, trains the model, and obtains its weight parameters. This model then generates a mask image of the location of the pallet containing the placed parts, with the pallet area in white and the background area in black. Specific frames are extracted from all collected videos, and the deep semantic segmentation model is used to calculate and save the mask image for each frame. The video is read frame by frame, with each frame copied and stored in memory. The video frame is then grayscaled and ANDed with the specified mask image to remove the background area. Thresholding and other methods are used to extract the target part area from the video frame. The changes in the center point position and area of ​​the target part area between the current and previous frames are calculated to determine if the frame has been correctly cropped. For correctly cropped frames, the coordinates of the bounding rectangle of the target part area are extracted, proportionally transformed to the coordinate space of the copied video frame, and the target part area is then cropped onto the copied video frame.

[0127] The parts were placed on a tray device for video recording. The tray device is shown in the attached image. Figure 2 Appendix Figure 3 As shown, the tray has a diameter of 1m, with black cloth on top and a rotating device with a diameter of 30cm at the bottom.

[0128] Specifically, this invention provides an image preprocessing method for aerospace structural components based on a deep neural network model. A deep convolutional neural network dataset is constructed, selecting 200 video samples of the acquired components. The first frame of each sample is extracted and labeled using LabelMe software. The tray area is labeled as pure white, and the remaining background is labeled as black. The labeled dataset is then divided into a training set and a test set, with a ratio of 80% for the training set and 20% for the test set, with no overlap between the two sets.

[0129] A deep convolutional neural network is constructed. The model is based on the ResNet-50 DeepLabV3+ semantic segmentation network. The input image is a color image with dimensions of 512 pixels (width and height), and the output is a single-channel image with a pure black background and a pure white tray area. The model is as follows... Figure 1As shown, except for the last layer of DecoderBlock3, all other layers use ReLU activation functions. Image is a color image with a length and width of 512 pixels, and outputImage is a single-channel image with a length and width of 512 pixels. EncoderBlock1, EncoderBlock2, EncoderBlock3, EncoderBlock4, and EncoderBlock5 are layers 1-3, 4-6, 7-10, 11-16, and 17-19 of the Resnet-50 network, respectively.

[0130] Deep semantic fusion of the model. For example... Figure 5 As shown, the outputs of EncoderBlock4 and EncoderBlock5 are stacked, and 768 1*1 convolutional kernels are used to adjust the output channels, resulting in 768 feature maps with dimensions of 32 pixels each. These feature maps are then used in conjunction with the Aspp module (…). Figure 6 As shown in the diagram, further fusion is performed, and then 1024 1*1 convolutional kernels are used to adjust the output channels before outputting to the next DecoderBlock1 module.

[0131] The model decoding process uses a three-block DecoderBlock structure. DecoderBlock1 receives the output from deep feature map fusion and the output from EncoderBlock3 as input. It first upsamples the feature map from the Aspp image spatial pyramid with a stride of 2, resizing it to 64 pixels in both width and height. This is then stacked with the output of EncoderBlock3. The stacked data is then convolved with 256 3x3 kernels to extract features, which are then output to DecoderBlock2. DecoderBlock2 receives the outputs from EncoderBlock2 and DecoderBlock1 as input. It upsamples the output of DecoderBlock1 with a stride of 2, resizing the feature map to 128 pixels in both width and height. This is then stacked with the output of EncoderBlock2, and the feature map is decoded using 128 3x3 kernels before being output to DecoderBlock3. DecoderBlock3 (…) Figure 8It only accepts the output from DecoderBlock2 as input, upsamples the output of DecoderBlock2 with a stride of 4, adjusts the size of the feature map to 512 pixels in both width and height, and convolves the input with three convolutional layers of size 3*3 with 64, 32 and 1 kernels respectively. The activation function of the last convolutional layer is the softmax activation function. The final output is a single-channel image with a size of 512 pixels in both width and height.

[0132] Model training uses cross-entropy as the loss function and consists of 100 training epochs. The model retains the training results and loss function values ​​from each epoch. After each training epoch, a test is performed, and the test loss function value is retained. After training is complete, the model with the smallest loss function on the test set is selected for the practical application. Model parameters from other epochs are deleted.

[0133] Video processing mask frame acquisition: Extract frames 0, 350, 700, and 1050 from all videos and name them as the video name plus an underscore plus the frame number, for example: 10.0.0.1_202001010000_0.jpgjpg, where 10.0.0.1_202001010000 is the video name, and the trailing 0 indicates that it is the 0th frame of the video.

[0134] The video frame mask image generation process uses a saved parameter model to compress all cropped video frames, resulting in a compressed image size of 512 pixels in length and width. The compressed image is then transmitted to the parameter model for calculation, and the output image is saved to a separate folder with the filename being the original name of the video frame.

[0135] The video is processed by acquiring the video and, according to the video name and the mask image naming method specified in step 6, sequentially acquiring the mask images corresponding to frames 0, 350, 700, and 1050 of the video, and storing them in the same array mask_array in ascending order of video frames.

[0136] Read video frames from the beginning, and then read the frames ( frm Make a copy as ( src_img ), to video frames frm Perform grayscale processing according to Formula 1;

[0137]

[0138] R, G, and B represent the RGB values ​​of a pixel, respectively.

[0139] grayscale processed video framesfrm Compression is performed to reduce the image size to 512 pixels in both width and height, using a mask image. The algorithm (1) is shown below:

[0140] Algorithm 1:

[0141] The current video frame number num, and the current mask array index mask_num

[0142] If num%350==0 and num<1500:

[0143] mask = mask_array[mask_num]

[0144] mask_num = mask_num + 1

[0145] else:

[0146] mask = mask_1050

[0147] Where num is the video frame counter, which increments by 1 for each frame read; mask_num is the mask image array index counter, which changes the mask image every 350 frames read.

[0148] The current compressed video frame frm With mask image mask Perform a bitwise AND operation on the result of the AND operation. frm The image is binarized according to Formula 2.

[0149]

[0150] Where x is the pixel value of the pixel.

[0151] Use OpenCV's `findContours` method to find all boundaries. Calculate the area of ​​the region enclosed by each of the found boundaries, and find the boundary with the largest enclosed area. Then, obtain the minimum width and height of this boundary. x_min , y_min and maximum value x_max , y_max Calculate the area and center point location of the two target part regions according to Formulas 3 and 4 respectively.

[0152]

[0153]

[0154] Calculate the video frames respectively frm The center point and enclosed area of ​​the target part region .

[0155] When processing frames frm When the first frame is an image, record the current frame. , this frame frm of Set as the reference area and reference center point coordinates of the target part region in the next frame image, when the video frame... frm For frames after the first frame, calculate the area and center point coordinates of the current frame's frm according to formulas 3 and 4. Formulas 5 and 6 are used to calculate the difference between the area and the distance to the center point in the current frame to determine whether the target part's cutting area is correct.

[0156]

[0157]

[0158] In formula 6 As a reference, the coordinate position of the center point on the width. For the current frame frm At the coordinate position on the width, As a reference, the center point's coordinates on the altitude. Current frame frm The coordinate position at the high altitude.

[0159] if and If the cropping is correct, it is considered correct; otherwise, it is considered incorrect. Frames with incorrect cropping are discarded, and the next frame is read and processed starting from step 9. If the cropping is correct, the current frame is... frm of Set as reference area and center point coordinates for the next frame .

[0160] Crop the video from the original image. Use the current frame obtained in step 9. frm Minimum width and height x_ min , y_min Maximum value x_max , y_max Calculate the actual position of the target part region in the src_img image using formulas 7, 8, 9, and 10:

[0161]

[0162]

[0163]

[0164]

[0165] in For step 9 frm The width of the image's src field at the beginning of the copy. For step 9 frm The image copied at the beginning src_img of high. For the target part area in src_img The top-left and bottom-right corner coordinates of the rectangular region in the image are used to determine the coordinates in the original image. src_img Upper crop rectangle area All pixels in between are saved as images of the target part. The parent directory of the storage folder is the part number corresponding to the video, and the current directory is the name of the video. The file name format is the current time followed by the image number. frm The video frame counter is num. The specific format is: Part number / Video name / Year-Month-Day-Hour-Minute-Second_num.jpg.

[0166] The pseudocode for a single video processing algorithm is shown in Algorithm 2 below:

[0167] Algorithm 2: Single Video Processing Algorithm

[0168] Acquire the video and retrieve the mask image generated from the video using the naming convention in step 6: mask_array=[mask_0,mask_350,mask_700,mask_1050]; counter num=0,mask_num=0; area and center point of the target part image in the previous frame: area',center'

[0169] Loop:

[0170] Read a single video frame (frm). If reading fails, exit the loop and terminate the program; if reading succeeds, proceed to the next step.

[0171] Copy the frm image and save it as src_img.

[0172] The frm is grayscaled according to formula (1).

[0173] Compress the frm image to a size of 512 pixels in both width and height.

[0174] If num%350==0 and num<1500:

[0175] mask = mask_arrya[mask_num]

[0176] mask_num = mask_num + 1

[0177] else:

[0178] mask = mask_1050

[0179] num = num + 1

[0180] frm = bitwise AND operation between mask and frm

[0181] Using formula (2) for frm, the grayscale values ​​greater than 40 in frm are set to 255, and the grayscale values ​​less than or equal to 40 are set to 0.

[0182] Use OpenCV's `findcouours` function to find boundaries and retrieve all boundaries in the `frm` file.

[0183] Use OpenCV's countArea function to calculate the area enclosed by all boundaries and find the contour corresponding to the largest area.

[0184] Find the boundary that encloses the largest area, and obtain the bounding rectangle (x_min, y_min, x_max, y_max). (x_min, y_min) represents the coordinates of the top-left corner of the rectangle, and (x_max, y_max) represents the coordinates of the bottom-right corner.

[0185] According to formulas 3 and 4, calculate the area and center point of the target region in the current image.

[0186] If num==0:

[0187] area'=area

[0188] center'=center

[0189] Else:

[0190] According to Formulas 5 and 6, calculate the difference between the area and center point of the target part's image region in the current frame and the area and center point of the target part's image region in the previous frame, detaArea, distance.

[0191] if detaArea < 0.3 and distance < 100:

[0192] The program continues to execute.

[0193] Else:

[0194] Break out of the current loop and return to the beginning of the loop.

[0195] As described in step 11, x_min, y_min, x_max, y_max are converted to the actual (x_min, y_min) and (x_max, y_max) in the original image, and the copied image src_img is cropped.

[0196] Save the cropped image to the parent directory containing the image number corresponding to the video, and the current directory containing the video name. The filename should be a JPG image with the video time plus the specified number.

[0197] End loop

[0198] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific embodiments of the present invention are limited to these descriptions. For those skilled in the art, other embodiments derived without departing from the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for image preprocessing of aerospace structural components based on a deep neural network model, characterized in that, Includes the following steps: Step 1. Construct training and testing datasets for a deep convolutional neural network; Step 2. Construct a deep convolutional neural network model based on ResNet-50 and DeeplabV3+ network structures; Step 3. Train the model and select the network model that performs best in the test set; Step 4. Crop the specified frame images of all videos, calculate the mask images of these video frames using a deep convolutional neural network model, and save the mask images; Step 5. Read the video file frame by frame, copy one copy and save it in memory. After grayscale processing of the video frame, perform a bitwise AND operation with the mask image to find the location of the target part. Step 6. Calculate the center point and area of ​​the current target part position, and compare them with the center point and position of the target part position in the previous frame; Step 7. Determine whether the part cutting is correct by comparison. If it is incorrect, discard the image frame. If it is correct, continue reading the next frame. Step 8. Transform the coordinates of the target part area to the original image, use the transformed coordinates to crop the copied original image, and save the cropped part image. The deep convolutional neural network model includes 5 EncoderBlocks and 3 DecoderBlocks. The outputs of EncoderBlock4 and EncoderBlock5 are further fused using a 1*1 convolutional kernel. The result is then fused with the output of the ASPP structure and transmitted to DecoderBlock1. DecoderBlock1 also receives the output from EncoderBlock3 as input. DecoderBlock2 receives the outputs from both EncoderBlock2 and DecoderBlock1. EncoderBlock3 only receives the output from DecoderBlock2. The final output is the target semantic segmentation image.

2. The image preprocessing method for aerospace structural components based on a deep neural network model according to claim 1, characterized in that, A deep convolutional neural network model was constructed. The model is based on the DeepLabV3+ semantic segmentation network of ResNet-50. The network model was modified to enhance the fusion of deep features of the image. The input of the model is a color image with a width and height of 512 pixels, and the output is a single-channel image with a width and height of 512 pixels. Randomly select 200 videos, crop out the first frame of each video, and divide these images into 80% training set and 20% test set, with no overlap between the training and test sets. Label the training and test sets, with the tray area in white and the background area in black. The model was trained using the training set for 100 training epochs, with the cross-entropy function chosen as the loss function and the learning rate set to 0.

001. After each training epoch, the training results of the model were saved, and the training results were tested using the test set, with the loss function value recorded on the test set. After training is complete, select the weight file corresponding to the minimum loss function as the final weight file, and delete the rest of the weight files. Cropping frames 0, 350, 700, and 1050 of all captured videos and keeping them in the same folder, naming them with the video name plus the order of the frame in the entire video; Call the saved parameter file and model, compress all frames into images with a width and height of 512 pixels, input them into the model for calculation, and store the results in another folder with the file name unchanged; Read the video, copy each video frame, convert each frame to grayscale, and compress it to 512 pixels in both width and height; read the mask image group corresponding to the video name, arrange them in ascending order according to the cropped frame position, and change the mask image every 350 frames; perform a bitwise AND operation between this mask image and the grayscale video frame; then use threshold segmentation to find the area enclosed by the largest boundary. Obtain the minimum and maximum values ​​of the area outline enclosed by the maximum boundary in length and width; The area and center point of the rectangular region enclosed by the maximum and minimum values ​​of the maximum outline in length and width are calculated. The area and center point of the previous frame are used as reference values ​​for the area and center point of the current frame. If the difference between the area and center point of the current frame and the area and center point of the previous frame is small, the cropping is considered correct; otherwise, it is considered incorrect. The first frame is considered to be cropped correctly by default. If the cropping is correct, the maximum and minimum values ​​of the maximum outline in length and width are transformed to obtain the position of the target part area in the original image. The target part area is then cropped from the copied video frame and saved in the specified folder.

3. The image preprocessing method for aerospace structural components based on a deep neural network model according to claim 1, characterized in that, The decoding process of the deep convolutional neural network model is as follows: DecoderBlock1 receives the output of the deep fusion of feature maps and the output of EncoderBlock3 as input data. It first upsamples the feature map from the Aspp image spatial pyramid with a stride of 2, adjusts the size of the feature map to 64 pixels in both length and width, stacks it with the output of the EncoderBlock3 module, and then convolves the stacked data with 256 3*3 convolutional kernels to extract features and output them to DecoderBlock2. DecoderBlock2 receives the output from EncoderBlock2 and the output from DecoderBlock1 as input. It upsamples the output of DecoderBlock1 with a stride of 2, adjusts the feature map to 128 pixels in both width and height, stacks it with the output of EncoderBlock2, decodes the feature map using 128 3*3 convolutional kernels, and outputs it to the next DecoderBlock3. DecoderBlock3 only accepts the output from DecoderBlock2 as input. It upsamples the output of DecoderBlock2 with a stride of 4, adjusts the feature map size to 512 pixels in both width and height, and convolves the input with three 3*3 convolutional layers with 64, 32, and 1 kernels respectively. The activation function of the last convolutional layer is the softmax activation function. The final output is a single-channel image with a width and height of 512 pixels.

4. The image preprocessing method for aerospace structural components based on a deep neural network model according to claim 1, characterized in that, In step 4, the video processing mask frame is obtained by extracting the 0th, 350th, 700th, and 1050th frames of all videos and naming them with the video name plus an underscore and the frame number.

5. The image preprocessing method for aerospace structural components based on a deep neural network model according to claim 1, characterized in that, Video frame mask image generation: Compress all cropped video frames using the saved parameter model. The compressed image size is 512 pixels long and 512 pixels wide. The compressed image is then transmitted to the parameter model for calculation. The output image is saved to another folder with the file name being the original name of the video frame.

6. The image preprocessing method for aerospace structural components based on a deep neural network model according to claim 1, characterized in that, To perform grayscale processing on the video: Obtain the video and, based on the video name, retrieve the mask images corresponding to frames 0, 350, 700, and 1050 of the video from the folder where the mask images are stored. Store these images in the same array, mask_array, in ascending order of video frames. Read the acquired video file frame by frame from the beginning, and then read the video frames. frm Make a copy src_img , video frames frm Perform grayscale processing according to Formula 1; Where R, G, and B are the values ​​of the RGB three channels of the pixel, respectively; grayscale processed video frames frm Compress it to an image with dimensions of 512 pixels each, and select a mask image; The current compressed video frame frm With mask image mask Perform a bitwise AND operation on the result of the AND operation. frm The image is binarized according to Formula 2: Where x is the pixel value of the pixel; Find all the boundaries and calculate the area of ​​the regions enclosed by each boundary. Find the boundary with the largest enclosed area and obtain the minimum width and height of that boundary. x_min , y_min and maximum value x_max , y_max Perform the following calculations: Calculate the video frames respectively frm The center point of the target part area and enclosed area .

7. A method for image preprocessing of aerospace structural components based on a deep neural network model according to claim 6, characterized in that, video frames frm The compression method is as follows: The current video frame number num, the current mask array index mask_num If num%350==0 and num<1500: mask = mask_array[mask_num] mask_num = mask_num + 1 else: mask = mask_1050 Where num is the video frame counter, which increments by 1 for each frame read; mask_num is the mask image array index counter, which changes the mask image every 350 frames read.

8. The image preprocessing method for aerospace structural components based on a deep neural network model according to claim 6, characterized in that, When processing video frames frm When the first frame is an image, record the current frame. This video frame frm of Set as the reference area and reference center point coordinates of the target part region in the next frame image, when the video frame... frm For frames after the first frame, calculate the area and center point coordinates of the current video frame's frm according to formulas 3 and 4. Calculate the area of ​​the current frame according to formulas 5 and 6. Distance from the center point The differences between them are used to determine whether the cutting area of ​​the target part is correct; In formula 6 As a reference, the coordinate position of the center point on the width. For the current video frame frm At the coordinate position on the width, As a reference, the center point's coordinates on the altitude. Current video frame frm The coordinate position at the high altitude.

9. The image preprocessing method for aerospace structural components based on a deep neural network model according to claim 8, characterized in that, and If the cut is correct, it is considered correct; otherwise, it is considered incorrect. Frames with incorrect cropping will be discarded, and the next frame will be read and processed; if the cropping is correct, the current video frame will be processed. frm of Set as reference area and center point coordinates for the next frame .

10. The image preprocessing method for aerospace structural components based on a deep neural network model according to claim 1, characterized in that, Cropping the original image specifically involves: cropping the current video frame obtained. frm Minimum width and height x_min , y_ min Maximum value x_max , y_max, Calculate the actual position of the target part region in the src_img image using formulas 7, 8, 9, and 10: in For the video frames in step 7 frm The width of the image's src field at the beginning of the copy. For the video frames in step 7 frm The image copied at the beginning src_img of high; For the target part area in src_img The coordinates of the top left and bottom right corners of the rectangular region in the image are in the original image. src_img Upper clipping rectangle area All pixels within the specified range are saved as images of the target part. The parent directory of the storage folder is the part number corresponding to this video, and the current directory is the name of this video. The file name format is the current time followed by the image number. frm The video frame counter num.

Images