A method and device for detecting sizing reed collision based on image processing

Abstract

The present invention discloses a method and device for detecting sizing yarn hitting the reed based on image processing, which belongs to the field of sizing yarn, and includes: pre-processing the sizing yarn hitting the reed image to be tested by main direction texture filtering, setting irrelevant pixels to zero, and obtaining the region of interest; extracting the deep feature map by multi-scale fusion of the reed coil features by feature extraction neural network, and obtaining the reed coil structure enhanced feature map by the reed coil structure information attention mechanism; inputting it into the detection frame generation neural network to generate an elliptical detection frame, and obtaining the detection result according to the elliptical detection frame. The main direction texture filtering is used to effectively remove the background noise of the sizing yarn hitting the reed image to be tested, and the feature extraction neural network and the reed coil structure information attention mechanism are combined to extract features, and then an elliptical detection frame is generated to perform sizing yarn hitting the reed detection, which effectively avoids noise interference, improves the detection accuracy of sizing yarn hitting the reed, and can stably perform detection in complex environments, with strong robustness.

Description

Technical Field

[0001] The invention relates to a method and a device for detecting sizing yarn collision with a reed based on image processing, and belongs to the technical field of sizing yarn. Background Art

[0002] Sizing is a pre-weaving process, and its quality is directly related to the quality of the finished product and the overall production efficiency of the subsequent weaving process. During the sizing process, reed collision may occur, which refers to the yarn breaking due to uneven tension, improper operation or equipment failure when passing through the reed teeth. The broken yarn is then entangled on the reed teeth to form a ball or a lump. This not only affects the normal passage of the yarn, but may also cause the loom to operate poorly, and in severe cases may cause shutdown.

[0003] The traditional method uses manual visual monitoring of the reed teeth. However, this manual inspection method relies on the operator's concentration and experience, and cannot ensure timely detection of reed collisions. Secondly, the manual inspection method consumes a lot of manpower costs, which is not conducive to enterprise cost control.

[0004] At present, in the field of automatic reed image detection, there is a machine vision-based frame difference method for detecting sizing yarn reed. For example, Chinese patent No. CN113516628A discloses a machine vision-based sizing yarn reed detection method. However, since the frame difference method used in this method has a poor detection effect on slow or stationary objects and is more sensitive to noise, this technology still has certain limitations in detection accuracy. The prior art also has detection technology based on image preprocessing. For example, Chinese patent No. CN114897788A discloses a yarn package hairiness detection method based on guided filtering and discrete difference. However, this technology is difficult to completely remove small irrelevant information or noise when processing sizing yarn reed images, and these methods may face the risk of failure when the collected image is unstable due to environmental interference.

[0005] Deep learning can process complex nonlinear data more effectively and has excellent adaptive capabilities, which has attracted widespread attention in the field of fault detection. There is a Chinese patent CN118781128A, a knitwear defect detection method and system based on image processing, which applies deep learning technology to the field of sizing and reed detection. However, the neural network module used by this type of technology cannot effectively detect reed balls with unique shapes and different sizes, and the detection effect is poor. Another paper "Fabric Defect Detection Based on YOLOv5-DCN" also uses deep learning technology, but when this technology is used for the reed ball target detection task, there is an imbalance problem between positive and negative samples. In the reed image to be tested, the reed ball occupies a small proportion of the original image, and most of the area is the background area. When the positive and negative samples are unbalanced, the optimization performance of the loss function of the model training will decrease. In addition, the square detection frame used by this technology has poor adaptability to the sizing and reed, resulting in poor detection effect of the sizing and reed. Summary of the invention

[0006] In order to solve at least one of the above problems, the present invention provides a method and device for detecting sizing yarn hitting a reed based on image processing.

[0007] As one aspect of the present invention, a method for detecting sizing yarn hitting a reed based on image processing is provided, comprising:

[0008] The collected image of the sizing yarn hitting the reed to be tested is preprocessed by main direction texture filtering, and irrelevant pixel points outside the sizing yarn reed tooth table area in the image of the sizing yarn hitting the reed to be tested are set to zero to obtain the area of ​​interest;

[0009] The multi-scale feature extraction neural network is used to fuse the reed coil features in the region of interest, and the deep feature map of the region of interest is extracted;

[0010] Through the attention mechanism of the reed ball structure information, the structural information in the deep feature map is enhanced to obtain the reed ball structure enhanced feature map;

[0011] The enhanced feature map of the reed coil structure is input into the detection frame generation neural network to generate an elliptical detection frame, and the sizing yarn reed detection result of the sizing yarn reed image to be tested is obtained according to the elliptical detection frame, wherein the elliptical detection frame represents the area where the reed may exist.

[0012] Furthermore, the collected image of the sizing yarn hitting the reed to be tested is preprocessed by main direction texture filtering, and irrelevant pixels outside the sizing yarn reed tooth table area in the image of the sizing yarn hitting the reed to be tested are set to zero to obtain the area of ​​interest, including:

[0013] The texture features of the image of the sizing yarn hitting the reed to be tested are analyzed by main direction texture filtering, and the main direction of the sizing yarn reed tooth table area in the image of the sizing yarn hitting the reed to be tested is calculated;

[0014] The texture features related to the sizing reed tooth table are extracted according to the main direction to obtain the texture feature image. The calculation formula of the texture feature image is:

[0015]

[0016] in, x Indicates the horizontal coordinate value. y Indicates the vertical coordinate value. f represents the filter frequency, σ represents the Gaussian standard deviation, represents the phase shift, Indicates the main direction;

[0017] Perform threshold processing on the texture feature image to remove noise in the texture feature image;

[0018] The morphological opening operation is performed on the texture feature image after threshold processing to eliminate the area with an area less than 20 pixels in the image and obtain the mask of the sizing reed tooth table area;

[0019] The pixel values ​​of the image of the sizing yarn hitting the reed to be tested, except for the mask of the sizing yarn reed tooth table area, are set to zero to obtain the area of ​​interest.

[0020] Furthermore, the texture features of the image of the sizing yarn hitting the reed to be tested are analyzed by main direction texture filtering, and the main direction of the sizing yarn reed tooth table area in the image of the sizing yarn hitting the reed to be tested is calculated, including:

[0021] Convert the image of the sizing yarn hitting the reed to be tested into a grayscale image;

[0022] Calculate the horizontal and vertical gradients of each pixel in the grayscale image;

[0023] According to the horizontal gradient and vertical gradient of each pixel, the gradient direction of the corresponding pixel is calculated by the inverse tangent function;

[0024] According to the gradient direction of each pixel, the frequency of occurrence of each gradient direction is counted, and the gradient direction corresponding to the maximum frequency is taken as the main direction.

[0025] Furthermore, the reed coil features in the region of interest are multi-scale fused by a feature extraction neural network for multi-scale feature fusion to extract a deep feature map of the region of interest, including:

[0026] The first feature map is extracted through the first feature extraction convolution layer, and the expression of the first feature map is:

[0027]

[0028] in, X 1 represents the first feature map, Maxpoolingrepresents the maximum pooling operation, ELRelu represents the activation function of the learnable parameters, Conv 3×3 represents a convolution operation with a convolution kernel size of 3×3. Conv 5d×5d represents a dilated convolution operation with a kernel size of 5 and an interval of 1. Conv 7d×7d represents a dilated convolution operation with a kernel size of 7 and an interval of 2. Conv 9d×9d It represents a dilated convolution operation with a kernel size of 9 and an interval of 3. X 0 represents the input image, and + represents the element-by-element addition of the matrix;

[0029] The second feature map is extracted through the second feature extraction convolution layer, and the expression of the second feature map is:

[0030]

[0031] in, X 2 represents the second feature map, Maxpooling represents the maximum pooling operation, ELRelu represents the activation function of the learnable parameters, Conv 3×3 represents a convolution operation with a convolution kernel size of 3×3. Conv 5d×5d represents a dilated convolution operation with a kernel size of 5 and an interval of 1. Conv 7d×7d represents a dilated convolution operation with a kernel size of 7 and an interval of 2. X 0 represents the input image, and + represents the element-by-element addition of the matrix;

[0032] The third feature map is extracted through the third feature extraction convolution layer, and the expression of the third feature map is:

[0033]

[0034] in, X 3 represents the third feature map, Maxpooling represents the maximum pooling operation, ELRelu represents the activation function of the learnable parameters, Conv 3×3 represents a convolution operation with a convolution kernel size of 3×3. Conv 5d×5d represents a dilated convolution operation with a kernel size of 5 and an interval of 1. X 0 represents the input image, and + represents the element-by-element addition of the matrix;

[0035] The first feature map, the second feature map and the third feature map are fused by element-by-element addition to obtain a deep feature map of the region of interest.

[0036] Furthermore, the activation function layer is an activation function of a learnable parameter, and the formula of the activation function of the learnable parameter is:

[0037]

[0038] in, Represents the output data of the activation function, x Represents input data, represents the learnable parameters, e Represents the natural logarithm.

[0039] Furthermore, the structural information in the deep feature map is enhanced through the attention mechanism of the reed ball structure information, and the reed ball structure enhanced feature map is obtained, including:

[0040] The deep feature map is filtered by a flexible low-pass filter to obtain a low-pass filtered feature map that retains structural information. The expression of the flexible low-pass filter is:

[0041]

[0042] in, Y represents the low-pass filter characteristic graph, M represents the width of the deep feature map, N Represents the height of the deep feature map, Indicates that in the preset local window of the deep feature map ( x , y ) is the local mean calculated at the center, represents the global mean of the deep feature map, represents the standard deviation within the preset local window, Represents the global standard deviation of the deep feature map;

[0043] The low-pass filter feature map and the deep feature map are processed pixel by pixel to obtain a composite feature map;

[0044] Through the attention layer, attention weight is added to each channel of the composite feature map to obtain the reed coil structure enhanced feature map.

[0045] Furthermore, the enhanced feature map of the reed yarn coil structure is input into the detection frame generation neural network to generate an elliptical detection frame, and the sizing yarn reed detection result of the sizing yarn reed image to be tested is obtained according to the elliptical detection frame, including:

[0046] Obtaining the major axis radius of the ellipse feature in the reed coil structure enhancement feature map through the ellipse parameter extraction network iand the minor axis radius j ;

[0047] According to the major axis radius and the minor axis radius, an elliptical detection frame is generated through the elliptical detection frame, and the position information of the elliptical detection frame is obtained at the corresponding position of the elliptical feature in the reed coil structure enhancement feature map. ,in, c x ' The horizontal coordinate of the center coordinate of the predicted elliptical detection box, c y ' The ordinate represents the center coordinate of the predicted elliptical detection box. r 0 represents the base scale of the ellipse feature corresponding to the predicted ellipse detection box. β Indicates the angle of the major axis of the predicted ellipse detection box relative to the horizontal axis.

[0048] Furthermore, the ellipse parameter extraction network is trained using a parameter loss function, and the parameter loss function is used to adjust the ratio of the major axis radius to the minor axis radius of the ellipse feature to be less than 2;

[0049] The ellipse detection frame generation head is trained using the ellipse detection frame loss function, and the ellipse detection frame loss function is used to optimize the error of the ellipse detection frame generation head.

[0050] Furthermore, the expression of the parameter loss function is:

[0051]

[0052] in, Lc Represents the parameter loss result corresponding to the ellipse feature, i represents the major axis radius of the ellipse feature, j Represents the minor axis radius of the ellipse feature;

[0053] The formula for the ellipse detection box loss function is:

[0054]

[0055] in, L r Represents the loss result of the ellipse detection box, c x The horizontal coordinate represents the center coordinate of the real elliptical detection frame. c y The ordinate represents the center coordinate of the true elliptical detection frame. c x ' The horizontal coordinate of the center coordinate of the predicted elliptical detection box, c y' The ordinate represents the center coordinate of the predicted elliptical detection box. Represents the base scale of the ellipse feature corresponding to the true ellipse detection frame, Represents the base scale of the ellipse feature corresponding to the predicted ellipse detection box, represents the intersection-over-union ratio of the true elliptical detection box, Represents the intersection-over-union ratio of the predicted elliptical detection box.

[0056] As another aspect of the present invention, a sizing yarn collision reed detection device based on image processing is provided, which is used to implement the above-mentioned sizing yarn collision reed detection method based on image processing, comprising:

[0057] A preprocessing module is used to preprocess the collected sizing yarn hitting the reed image to be tested by main direction texture filtering, and to set irrelevant pixel points outside the sizing yarn reed tooth table area in the sizing yarn hitting the reed image to be tested to zero, so as to obtain an area of ​​interest;

[0058] A deep feature map extraction module, used to perform multi-scale fusion of the reed coil features in the region of interest through a feature extraction neural network of multi-scale feature fusion, and extract a deep feature map of the region of interest;

[0059] The structure enhancement module is used to enhance the structural information in the deep feature map through the attention mechanism of the reed ball structure information to obtain the reed ball structure enhanced feature map;

[0060] The detection module is used to input the enhanced feature map of the reed coil structure into the detection frame generation neural network to generate an elliptical detection frame, and obtain the sizing yarn reed detection result of the sizing yarn reed image to be tested according to the elliptical detection frame, wherein the elliptical detection frame represents the area where the reed may exist.

[0061] The beneficial effects of the present invention are:

[0062] After effectively removing the background, noise and other interference information in the sizing yarn hitting the reed image to be tested through main direction texture filtering, feature extraction is performed by combining a feature extraction neural network with four layers of different variable sizes and an attention mechanism for the structural information of the reed coil. Then, an elliptical detection frame is generated by a detection frame generation neural network for sizing yarn hitting the reed detection. This can effectively avoid background and noise interference, greatly improve the detection accuracy of sizing yarn hitting the reed, and can also stably and reliably perform sizing yarn hitting the reed inspection in complex environments, with strong robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0064] Figure 1 It is a brief flow chart of the method for detecting sizing yarn collision with reed based on image processing provided in Example 1 of the present invention.

[0065] Figure 2 It is a brief structural diagram of a sizing yarn collision reed detection device based on image processing provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0066] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0067] Embodiment 1:

[0068] The embodiment of the present invention provides a method for detecting sizing yarn hitting a reed based on image processing, such as Figure 1 As shown, including:

[0069] S100, preprocessing the collected image of the sizing yarn hitting the reed to be tested by main direction texture filtering, setting irrelevant pixel points outside the sizing yarn reed tooth table area in the image of the sizing yarn hitting the reed to be tested to zero, and obtaining an area of ​​interest;

[0070] Specifically, since the gradient directions of the reed tooth surface area and the irrelevant background area in the sizing yarn hitting the reed image are significantly different, the gradient direction of the reed tooth surface area is consistent, and the number of pixels whose gradient direction in the sizing yarn hitting the reed image to be tested is the largest, the embodiment of the present invention first calculates the main direction of the sizing yarn hitting the reed image to be tested, and then pre-processes the collected sizing yarn hitting the reed image to be tested according to the main direction texture filtering, retains the image information consistent with the main direction, removes irrelevant areas such as the background, and extracts the reed tooth surface area.

[0071] S200, performing multi-scale fusion of the reed wire features in the region of interest through a feature extraction neural network for multi-scale feature fusion, and extracting a deep feature map of the region of interest;

[0072] Specifically, the feature extraction neural network in the embodiment of the present invention is a hybrid convolutional neural network with three layers of different variable sizes. Since the yarn balls formed by the sizing and reeding are of different sizes, the feature extraction neural network with three layers of different variable sizes can better adapt to the sizes of various yarn balls. The feature maps obtained from the three layers are then fused to effectively extract the deep feature maps of the area of ​​interest.

[0073] S300, enhancing the structural information in the deep feature map through the reed ball structure information attention mechanism to obtain the reed ball structure enhanced feature map;

[0074] Specifically, an embodiment of the present invention extracts structural information from a deep feature map through an attention mechanism for the structural information of the reed yarn ball. The structural information here mainly refers to the structural information of the sizing reed teeth, the reed yarn ball, the residual background, the residual noise, etc. Structural information with relatively low importance, such as the structural information of the residual background and the residual noise, corresponds to a smaller weight, and structural information with relatively high importance, such as the structural information of the reed yarn ball, corresponds to a larger weight. Based on the above structural information and corresponding weights, the deep feature map of the sizing reed image to be tested is enhanced with an emphasis on the structural information, and a structural enhanced feature map of the reed yarn ball is obtained, which helps to understand the structural information of the reed yarn ball, and thus makes the detection result more accurate.

[0075] S400, inputting the reed yarn coil structure enhancement feature map into the detection frame generation neural network to generate an elliptical detection frame, and obtaining the sizing yarn reed collision detection result of the sizing yarn reed collision image to be tested according to the elliptical detection frame, wherein the elliptical detection frame indicates the area where the reed collision may exist.

[0076] Since the yarn balls formed by the sizing reed are generally circular, the embodiment of the present invention adopts an elliptical detection frame, which is more suitable for target detection of the sizing reed than a conventional square detection frame.

[0077] In summary, after the embodiment of the present invention effectively removes the background, noise and other interference information in the image of the sizing yarn hitting the reed to be tested through the main direction texture filtering, it combines the feature extraction neural network with four layers of different variable sizes and the attention mechanism of the reed coil structure information to perform feature extraction, and then generates an elliptical detection frame through the detection frame generation neural network to perform sizing yarn hitting the reed detection. It can effectively avoid background and noise interference, greatly improve the detection accuracy of sizing yarn hitting the reed, and can also stably and reliably perform sizing yarn hitting the reed inspection in complex environments, with strong robustness.

[0078] Furthermore, the collected image of the sizing yarn hitting the reed to be tested is preprocessed by main direction texture filtering, and irrelevant pixels outside the sizing yarn reed tooth table area in the image of the sizing yarn hitting the reed to be tested are set to zero to obtain the area of ​​interest, including:

[0079] S110, analyzing the texture features of the image of the sizing yarn hitting the reed to be tested by main direction texture filtering, and calculating the main direction of the sizing yarn reed tooth table area in the image of the sizing yarn hitting the reed to be tested;

[0080] Specifically, the calculation method of the main direction includes:

[0081] S111, converting the image of the sizing yarn to be tested hitting the reed into a grayscale image;

[0082] S112, calculating the horizontal gradient and vertical gradient of each pixel in the grayscale image;

[0083] S113, according to the horizontal gradient and vertical gradient of each pixel, the gradient direction of the corresponding pixel is calculated by an inverse tangent function; the specific calculation formula of the gradient direction is shown in formula (1):

[0084] (1)

[0085] in, θ represents the gradient direction (in degrees), v represents the vertical gradient, h represents the horizontal gradient;

[0086] S114. According to the gradient direction of each pixel, count the frequency of occurrence of each gradient direction, and take the gradient direction corresponding to the maximum frequency as the main direction.

[0087] Specifically, the embodiment of the present invention uses a histogram to count the frequency of occurrence of each gradient direction, and takes the gradient direction corresponding to the maximum frequency as the main direction θ max ;

[0088] S120, extracting texture features related to the sizing reed tooth table according to the main direction to obtain a texture feature image;

[0089] Specifically, the calculation method of the texture feature image is shown in formula (2):

[0090] (2)

[0091] in, x Indicates the horizontal coordinate value. y Indicates the vertical coordinate value. f represents the filter frequency, σ represents the standard deviation, Indicates phase offset.

[0092] S130, performing threshold processing on the texture feature image to remove noise in the texture feature image;

[0093] S140, performing a morphological opening operation on the texture feature image after threshold processing, eliminating the area with an area less than 20 pixels in the image, and obtaining a mask of the sizing reed tooth table area;

[0094] S150, setting the pixel values ​​of the image of the sizing yarn hitting the reed to be tested except for the mask of the sizing yarn reed tooth table area to zero, and obtaining the region of interest.

[0095] Furthermore, the reed coil features in the region of interest are multi-scale fused by a feature extraction neural network for multi-scale feature fusion to extract a deep feature map of the region of interest, including:

[0096] The first feature map is extracted through the first feature extraction convolution layer. The expression of the first feature map is shown in formula (3):

[0097] (3)

[0098] in, X 1 represents the first feature map, Maxpooling represents the maximum pooling operation, ELRelu represents the activation function of the learnable parameters, Conv 3×3 represents a convolution operation with a convolution kernel size of 3×3. Conv 5d×5d represents a dilated convolution operation with a kernel size of 5 and an interval of 1. Conv 7d×7d represents a dilated convolution operation with a kernel size of 7 and an interval of 2. Conv 9d×9d It represents a dilated convolution operation with a kernel size of 9 and an interval of 3. X 0 represents the input image, i.e., the region of interest, and + represents the element-by-element addition of the matrix;

[0099] The second feature map is extracted through the second feature extraction convolution layer. The expression of the second feature map is shown in formula (4):

[0100] (4)

[0101] in, X 2 represents the second feature map, Maxpooling represents the maximum pooling operation, ELRelu represents the activation function of the learnable parameters, Conv 3×3 represents a convolution operation with a convolution kernel size of 3×3. Conv 5d×5d represents a dilated convolution operation with a kernel size of 5 and an interval of 1. Conv 7d×7d represents a dilated convolution operation with a kernel size of 7 and an interval of 2. X 0 represents the input image, and + represents the element-by-element addition of the matrix;

[0102] The third feature map is extracted through the third feature extraction convolution layer. The expression of the third feature map is shown in formula (5):

[0103] (5)

[0104] in, X 3 represents the third feature map, Maxpooling represents the maximum pooling operation, ELRelu represents the activation function of the learnable parameters, Conv 3×3 represents a convolution operation with a convolution kernel size of 3×3. Conv 5d×5d represents a dilated convolution operation with a kernel size of 5 and an interval of 1. X 0 represents the input image, and + represents the element-by-element addition of the matrix;

[0105] By adding element by element, the first feature map, the second feature map and the third feature map are fused to obtain the deep feature map of the region of interest. X 4.

[0106] Since the reed teeth are objects with large-scale features, the embodiment of the present invention increases the operator size to 5×5, 7×7 and 9×9 to increase the receptive field, that is, to increase the range of feature extraction neural network processing, so as to more effectively extract features from the region of interest; in addition, since the yarn balls produced by the reed have different sizes, the embodiment of the present invention adopts multi-scale feature extraction. In different neural network layers, convolution kernels of various sizes are used to adapt to reed yarn balls of various sizes. Small convolution kernels are mainly used to capture local information, and large convolution kernels are mainly used to capture global information. Multi-scale feature fusion is then performed, which is more suitable for reed feature extraction of yarn balls of various sizes, greatly improving the robustness of the extracted deep feature map, and can make the detection of reed yarn balls of various sizes more accurate.

[0107] Furthermore, the activation function of the learnable parameters is shown in formula (6):

[0108] (6)

[0109] in, Represents the output data of the activation function, x Represents input data, represents the learnable parameters, e Represents the natural logarithm, approximately equal to 2.71828.

[0110] Since the grayscale value distribution of the reed tooth area and the reed ball area in the region of interest is relatively close, the value distribution in the feature map calculation is relatively compact, and the feature discrimination is reduced. Therefore, the embodiment of the present invention adopts an activation function with learnable parameters to regulate the mapping method of the negative value area, so as to remove the features of the reed tooth area while retaining the features of the reed ball area. Specifically, the embodiment of the present invention trains different learnable parameters by training different neurons. , the learnable parameters It is possible to learn different mapping methods for negative regions and dynamically adjust the output amplitude of negative features. Specifically, if the negative features are more important for the representation of the reed area during training, Can be adjusted to a larger positive value to make the output in the negative region grow faster. If the negative features are relatively minor, can be adjusted to a smaller value to reduce the contribution of negative areas by introducing learnable parameters , which can enhance the adaptability of the feature extraction neural network constructed based on the activation function of the learnable parameters to different eigenvalue distributions, making the extraction of deep features more flexible, and thus being able to more completely capture the characteristics of the reed ball area.

[0111] Furthermore, the structural information in the deep feature map is enhanced through the attention mechanism of the reed ball structure information, and the reed ball structure enhanced feature map is obtained, including:

[0112] S310, filtering the deep feature map through a flexible low-pass filter to obtain a low-pass filtered feature map that retains structural information. The expression of the flexible low-pass filter is shown in formula (7):

[0113] (7)

[0114] in, Y represents the low-pass filter characteristic graph, M represents the width of the deep feature map, N Represents the height of the deep feature map, Indicates that in the preset local window of the deep feature map ( x , y ) is the local mean calculated at the center, represents the global mean of the deep feature map, represents the standard deviation within the preset local window, Represents the global standard deviation of the deep feature map;

[0115] Since the values ​​calculated by filtering under each filtering window vary according to the situation in the window, strong signals are relatively enhanced and weak signals are relatively weakened. The signal of the reed ball is stronger than the signal of noise. Compared with ordinary mean filtering, it can enhance the signal of the reed ball.

[0116] S320, performing pixel-by-pixel addition processing on the low-pass filter feature map and the deep feature map to obtain a composite feature map;

[0117] Specifically, as shown in formula (8):

[0118] (8)

[0119] in, Y2 represents a composite feature map, Y 2 channels are used C express, Y 2 height H express, Y 2 width with W express;

[0120] S330. Add attention weight to each channel of the composite feature map through the attention layer to obtain a reed coil structure enhanced feature map.

[0121] Specifically, the composite feature map Y 2 Input attention layer, which can adaptively calculate the attention weight of each channel of the composite feature map A , which contains information about the importance of each channel in the composite feature map. Y 2 is multiplied element-by-element by A to organize the information of the composite feature map channel by channel, add attention weight to each channel, and output the enhanced feature map of the reed coil structure, as shown in formula (9):

[0122] (9)

[0123] in, Y 3 represents the enhanced characteristic diagram of the reed coil structure. A represents the attention weight of each channel of the composite feature map, and ⊙ represents the element-wise multiplication operation.

[0124] In the traditional attention mechanism, the output of the previous step of the attention layer is the input of the attention layer. The training of attention has no emphasis, which makes the detection network insufficiently understand the structural information of the reed wire ball, and thus leads to inaccurate detection of the reed wire ball. In contrast, the attention mechanism for the structural information of the reed wire ball provided by the embodiment of the present invention adds the calculation of the extracted structure to the traditional attention mechanism. The attention mechanism for the structural information of the reed wire ball can calculate the attention weights of different structures through training. The weight can represent the importance of each structure or each channel. The obtained composite feature map containing structural information is input into the attention layer. The attention layer can assign values ​​to each channel of the composite feature map according to the corresponding weights of each channel. Specifically, a greater weight is given to the structural information of the reed wire ball, and a smaller weight is given to the relatively less important information such as the structural information of the residual background and residual noise, so as to achieve attention enhancement with a focus on structural information, highlight the importance of structural information, and thus enable the overall detection network to fully understand the structural information of the reed wire ball and improve the accuracy of the detection of the reed wire ball.

[0125] Furthermore, the enhanced feature map of the reed yarn coil structure is input into the detection frame generation neural network to generate an elliptical detection frame, and the sizing yarn reed detection result of the sizing yarn reed image to be tested is obtained according to the elliptical detection frame, including:

[0126] S410, obtaining the major axis radius of the elliptical feature in the reed coil structure enhancement feature map through the elliptical parameter extraction network i and the minor axis radius j ;

[0127] S420, according to the major axis radius and the minor axis radius, an elliptical detection frame is generated through an elliptical detection frame generation head, an elliptical detection frame is generated at a position corresponding to the elliptical feature in the reed coil structure enhancement feature map, and position information of the elliptical detection frame is obtained. The position information of the elliptical detection frame is ,in, c x ' The horizontal coordinate of the center coordinate of the predicted elliptical detection box, c y ' The ordinate represents the center coordinate of the predicted elliptical detection box. r 0 represents the base scale of the ellipse feature corresponding to the predicted ellipse detection box. β Indicates the angle of the major axis of the predicted ellipse detection box relative to the horizontal axis.

[0128] Furthermore, the ellipse parameter extraction network is trained using a parameter loss function, and the parameter loss function is used to adjust the ratio of the major axis radius to the minor axis radius of the ellipse feature to be less than 2;

[0129] The ellipse detection frame generation head is trained using the ellipse detection frame loss function, and the ellipse detection frame loss function is used to optimize the error of the ellipse detection frame generation head.

[0130] Since the reed wire ball is generally not formed into a regular circle, the embodiment of the present invention adopts a more adaptable elliptical detection frame to match the reed wire ball structure enhancement feature map with the elliptical parameter extraction network to obtain the major axis radius of the elliptical feature in the reed wire ball structure enhancement feature map. i and the minor axis radius j , and an independent parameter loss function is used to limit the major-minor axis ratio to less than 2, and the major axis and minor axis are trained so that the parameter training will not deviate from the shape of the reed coil, and the obtained parameters are more compatible with the reed coil, thereby improving the accuracy of the generation result of the elliptical detection frame generation head, and thus improving the detection accuracy of the sizing reed. On this basis, the elliptical detection frame generation head is optimized separately through the elliptical detection frame loss function to reduce the error of the elliptical detection frame and improve the detection accuracy of the sizing reed.

[0131] Furthermore, the expression of the parameter loss function is shown in formula (10):

[0132] (10)

[0133] in, Lc Represents the parameter loss result corresponding to the ellipse feature, i represents the major axis radius of the ellipse feature, j Represents the minor axis radius of the ellipse feature;

[0134] The expression of the ellipse detection box loss function is shown in formula (11):

[0135] (11)

[0136] in, L r Represents the loss result of the ellipse detection box, c x The horizontal coordinate represents the center coordinate of the real elliptical detection frame. c y The ordinate represents the center coordinate of the true elliptical detection frame. c x ' The horizontal coordinate of the center coordinate of the predicted elliptical detection box, c y ' The ordinate represents the center coordinate of the predicted elliptical detection box. Represents the base scale of the ellipse feature corresponding to the true ellipse detection frame, Represents the base scale of the ellipse feature corresponding to the predicted ellipse detection box, represents the intersection-over-union ratio of the true elliptical detection box, Represents the intersection-over-union ratio of the predicted elliptical detection box.

[0137] The ellipse detection box loss function adopts a piecewise function design, so that when the horizontal coordinate or vertical coordinate of the ellipse detection box loss function deviates from the true value by more than 1, the loss is greatly increased, thereby making the center coordinate of the detection box more accurate.

[0138] Embodiment 2:

[0139] Another embodiment of the present invention provides a sizing yarn collision reed detection device 010 based on image processing, which is used to implement the above-mentioned sizing yarn collision reed detection method based on image processing, such as Figure 2 As shown, including:

[0140] The preprocessing module 011 is used to preprocess the collected image of the sizing yarn hitting the reed to be tested by main direction texture filtering, and set irrelevant pixel points outside the sizing yarn reed tooth table area in the image of the sizing yarn hitting the reed to be tested to zero, so as to obtain the area of ​​interest;

[0141] Specifically, since in the sizing yarn hitting the reed image, the gradient directions of the reed tooth surface area and the irrelevant background area are significantly different, the gradient direction of the reed tooth surface area is consistent, and the number of pixels whose gradient direction in the sizing yarn hitting the reed image to be tested is the largest, the preprocessing module of the embodiment of the present invention first calculates the main direction of the sizing yarn hitting the reed image to be tested, and then preprocesses the collected sizing yarn hitting the reed image to be tested according to the main direction texture filtering, retains the image information consistent with the main direction, removes irrelevant areas such as the background, and extracts the reed tooth surface area.

[0142] A deep feature map extraction module 012 is used to perform multi-scale fusion of the reed coil features in the region of interest through a feature extraction neural network for multi-scale feature fusion to extract a deep feature map of the region of interest;

[0143] Specifically, the feature extraction neural network in the deep feature map extraction module provided in the embodiment of the present invention is a hybrid convolutional neural network with three layers of different variable sizes. Since the yarn balls formed by the sizing and reeding are of different sizes, the feature extraction neural network with three layers of different variable sizes can better adapt to the sizes of various yarn balls of sizing and reeding, and then the feature maps obtained from the three layers are fused to effectively extract the deep feature maps of the area of ​​interest.

[0144] The structure enhancement module 013 is used to enhance the structural information in the deep feature map through the reed ball structure information attention mechanism to obtain the reed ball structure enhanced feature map;

[0145] Specifically, the structure enhancement module 013 in the embodiment of the present invention extracts the structure information in the deep feature map through the attention mechanism of the reed yarn structure information. The structure information here mainly refers to the structure information of the sizing reed teeth, the reed yarn, the residual background, the residual noise, etc. The structure information with relatively low importance, such as the structure information of the residual background and the residual noise, corresponds to a smaller weight, and the structure information with relatively high importance, such as the structure information of the reed yarn, corresponds to a larger weight. Based on the above structure information and corresponding weights, the deep feature map of the sizing reed image to be tested is enhanced with attention focused on the structure information to obtain the reed yarn structure enhancement feature map, which can enable the overall detection network to more effectively understand the structure information of the reed yarn, and thus make the detection result more accurate.

[0146] The detection module 014 is used to input the enhanced feature map of the reed coil structure into the detection frame generation neural network to generate an elliptical detection frame, and obtain the sizing yarn reed detection result of the sizing yarn reed image to be tested according to the elliptical detection frame, wherein the elliptical detection frame represents the area where the reed may exist.

[0147] Since the yarn ball formed by the sizing reed is usually irregular and its overall shape is close to an ellipse, the detection module of the embodiment of the present invention adopts an elliptical detection frame, which is more suitable for the target detection of the sizing reed than the conventional square detection frame.

[0148] After the preprocessing module of the embodiment of the present invention effectively removes the background, noise and other interference information in the sizing yarn hitting the reed image to be tested through the main direction texture filtering, it combines the feature extraction neural network with three layers of different layers of variable sizes in the deep feature map extraction module and the reed coil structure information attention mechanism in the structure enhancement module to perform feature extraction, and then generates an elliptical detection frame through the detection frame generation neural network in the detection module to perform sizing yarn hitting the reed detection. It can effectively avoid background and noise interference, greatly improve the detection accuracy of sizing yarn hitting the reed, and can also stably and reliably perform sizing yarn hitting the reed inspection in complex environments, with strong robustness.

[0149] The specific working principle of the sizing yarn collision reed detection device based on image processing provided by an embodiment of the present invention can refer to the principle description of the above-mentioned sizing yarn collision reed detection method based on image processing, which will not be repeated here.

[0150] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.

[0151] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting sizing reed collision based on image processing, characterized in that: include: Preprocessing the collected image of the sizing yarn hitting the reed to be tested by main direction texture filtering, setting irrelevant pixel points outside the sizing yarn reed tooth table area in the image of the sizing yarn hitting the reed to be tested to zero, and obtaining an area of ​​interest; Perform multi-scale fusion of the reed coil features in the region of interest through a feature extraction neural network for multi-scale feature fusion to extract a deep feature map of the region of interest; The structural information in the deep feature map is enhanced by using the attention mechanism of the reed ball structure information to obtain the reed ball structure enhanced feature map; Inputting the reed-collision coil structure enhancement feature map into a detection frame generation neural network to generate an elliptical detection frame, and obtaining a sizing yarn reed collision detection result of the sizing yarn reed collision image to be tested according to the elliptical detection frame, wherein the elliptical detection frame indicates an area where the reed collision exists; The method preprocesses the collected image of the sizing yarn hitting the reed to be tested by main direction texture filtering, sets irrelevant pixel points outside the sizing yarn reed tooth table area in the image of the sizing yarn hitting the reed to be tested to zero, and obtains the area of ​​interest, including: Analyzing the texture features of the image of the sizing yarn hitting the reed to be tested by the main direction texture filtering, and calculating the main direction of the sizing yarn reed tooth table area in the image of the sizing yarn hitting the reed to be tested; The texture features related to the sizing reed tooth table are extracted according to the main direction to obtain a texture feature image. The calculation formula of the texture feature image is: in, x Indicates the horizontal coordinate value. y Indicates the vertical coordinate value. f represents the filter frequency, σ represents the Gaussian standard deviation, represents the phase shift, Indicates the main direction; Performing threshold processing on the texture feature image to remove noise in the texture feature image; Performing a morphological opening operation on the texture feature image after threshold processing, eliminating the area with an area less than 20 pixels in the image, and obtaining a sizing reed tooth table area mask; The pixel values ​​of the image of the sizing yarn hitting the reed to be tested are set to zero except for the mask of the sizing yarn reed tooth table area, so as to obtain the region of interest; The method of analyzing the texture features of the image of the sizing yarn hitting the reed to be tested by the main direction texture filtering and calculating the main direction of the sizing yarn reed tooth table area in the image of the sizing yarn hitting the reed to be tested comprises: Converting the image of the sizing yarn to be tested hitting the reed into a grayscale image; Calculate the horizontal gradient and vertical gradient of each pixel in the grayscale image; According to the horizontal gradient and vertical gradient of each pixel, the gradient direction of the corresponding pixel is calculated by the inverse tangent function; According to the gradient direction of each pixel, the frequency of occurrence of each gradient direction is counted, and the gradient direction corresponding to the maximum frequency is taken as the main direction.

2. The method according to claim 1, characterized in that The feature extraction neural network using multi-scale feature fusion performs multi-scale fusion on the reed coil features in the region of interest to extract a deep feature map of the region of interest, including: A first feature map is extracted through a first feature extraction convolution layer, and the expression of the first feature map is: in, X 1 represents the first feature map, Maxpooling represents the maximum pooling operation, ELRelu represents the activation function of the learnable parameters, Conv 3×3 represents a convolution operation with a convolution kernel size of 3×3. Conv 5d×5d represents a dilated convolution operation with a kernel size of 5 and an interval of 1. Conv 7d×7d represents a dilated convolution operation with a kernel size of 7 and an interval of 2. Conv 9d×9d It represents a dilated convolution operation with a kernel size of 9 and an interval of 3. X 0 represents the input image, and + represents the element-by-element addition of the matrix; The second feature map is extracted by the second feature extraction convolution layer, and the expression of the second feature map is: in, X 2 represents the second feature map, Maxpooling represents the maximum pooling operation, ELRelu represents the activation function of the learnable parameters, Conv 3×3 represents a convolution operation with a convolution kernel size of 3×3. Conv 5d×5d represents a dilated convolution operation with a kernel size of 5 and an interval of 1. Conv 7d×7d represents a dilated convolution operation with a kernel size of 7 and an interval of 2. X 0 represents the input image, and + represents the element-by-element addition of the matrix; The third feature map is extracted by the third feature extraction convolution layer, and the expression of the third feature map is: in, X 3 represents the third characteristic graph, Maxpooling represents the maximum pooling operation, ELRelu represents the activation function of the learnable parameters, Conv 3×3 represents a convolution operation with a convolution kernel size of 3×3. Conv 5d×5d represents a dilated convolution operation with a kernel size of 5 and an interval of 1. X 0 represents the input image, and + represents the element-by-element addition of the matrix; The first feature map, the second feature map and the third feature map are fused by element-by-element addition to obtain a deep feature map of the region of interest.

3. The method according to claim 2, characterized in that The activation function layer is an activation function of a learnable parameter, and the formula of the activation function of the learnable parameter is: in, Represents the output data of the activation function, x Represents input data, represents the learnable parameters, e Represents the natural logarithm.

4. The method according to claim 1, characterized in that: The method enhances the structural information in the deep feature map by using the attention mechanism of the reed ball structure information to obtain the reed ball structure enhanced feature map, including: The deep feature map is filtered by a flexible low-pass filter to obtain a low-pass filtered feature map retaining structural information. The expression of the flexible low-pass filter is: in, Y represents the low-pass filtering characteristic graph, M represents the width of the deep feature map, N represents the height of the deep feature map, It means that in the preset local window of the deep feature map, ( x , y ) is the local mean calculated at the center, represents the global mean of the deep feature map, represents the standard deviation within the preset local window, represents the global standard deviation of the deep feature map; Perform pixel-by-pixel addition processing on the low-pass filter feature map and the deep feature map to obtain a composite feature map; Through the attention layer, attention weight is added to each channel of the composite feature map to obtain a reed coil structure enhanced feature map.

5. The method according to claim 1, characterized in that The step of inputting the reed yarn coil structure enhancement feature map into a detection frame generation neural network to generate an elliptical detection frame, and obtaining a sizing yarn reed detection result of the sizing yarn reed image to be tested according to the elliptical detection frame comprises: The major axis radius of the elliptical feature in the reed coil structure enhancement feature map is obtained through the elliptical parameter extraction network i and the minor axis radius j ; According to the major axis radius and the minor axis radius, an elliptical detection frame generation head is used to generate an elliptical detection frame at a position corresponding to the elliptical feature in the reed coil structure enhancement feature map and obtain position information of the elliptical detection frame. ,in, c x ' The horizontal coordinate of the center coordinate of the predicted elliptical detection box, c y ' The ordinate represents the center coordinate of the predicted elliptical detection box. r 0 represents the base scale of the ellipse feature corresponding to the predicted ellipse detection box. β Indicates the angle of the major axis of the predicted ellipse detection box relative to the horizontal axis.

6. The method according to claim 5, characterized in that The ellipse parameter extraction network is trained using a parameter loss function, and the parameter loss function is used to adjust the ratio of the major axis radius to the minor axis radius of the ellipse feature to be less than 2; The ellipse detection frame generation head is trained using an ellipse detection frame loss function, and the ellipse detection frame loss function is used to optimize the error of the ellipse detection frame generation head.

7. The method according to claim 6, characterized in that The expression of the parameter loss function is: in, Lc Represents the parameter loss result corresponding to the ellipse feature, i represents the major axis radius of the ellipse feature, j represents the minor axis radius of the ellipse feature; The formula of the ellipse detection box loss function is: in, L r Represents the loss result of the ellipse detection box, c x The horizontal coordinate represents the center coordinate of the real elliptical detection frame. c y The ordinate represents the center coordinate of the true elliptical detection frame. c x ' The horizontal coordinate of the center coordinate of the predicted elliptical detection box, c y ' The ordinate represents the center coordinate of the predicted elliptical detection box. Represents the base scale of the ellipse feature corresponding to the true ellipse detection frame, Represents the base scale of the ellipse feature corresponding to the predicted ellipse detection box, represents the intersection-over-union ratio of the true elliptical detection box, Represents the intersection-over-union ratio of the predicted elliptical detection box.

8. A sizing yarn collision reed detection device based on image processing, used to implement the sizing yarn collision reed detection method based on image processing according to any one of claims 1 to 7, characterized in that: include: A preprocessing module is used to preprocess the collected sizing yarn hitting the reed image to be tested by main direction texture filtering, and to set irrelevant pixel points outside the sizing yarn reed tooth table area in the sizing yarn hitting the reed image to be tested to zero, so as to obtain an area of ​​interest; A deep feature map extraction module, used for performing multi-scale fusion of the reed coil features in the region of interest through a feature extraction neural network of multi-scale feature fusion to extract a deep feature map of the region of interest; A structure enhancement module, used to enhance the structural information in the deep feature map through the reed ball structure information attention mechanism to obtain the reed ball structure enhanced feature map; A detection module is used to input the reed coil structure enhancement feature map into a detection frame generation neural network to generate an elliptical detection frame, and obtain the sizing yarn reed detection result of the sizing yarn reed image to be tested according to the elliptical detection frame, wherein the elliptical detection frame indicates an area where the reed may exist.

Images