A real-time monitoring method based on machine vision and deep learning coupling algorithm

Abstract

The present invention discloses a real-time monitoring method based on a coupling algorithm of machine vision and deep learning; the method comprises the following steps: 1. A laser emits a laser beam, leaving a light spot on the end face of the yarn bobbin. At the same time, the RGB‑D camera rotates to collect RGB images and depth maps of the yarn bobbin at different pitch angles. 2. Image processing is performed on the RGB images and depth images captured at different pitch angles to obtain the depth values ​​of the center of the light spot and the center of each yarn bobbin in the image, and the contours of each yarn bobbin and its bobbin are extracted. 3. Calculate the yarn bobbin margin measured by vision; 4. Calculate the yarn bobbin margin by deep learning; 5. Fusion the yarn bobbin margin measured by vision with the yarn bobbin margin measured by vision. The present invention collects RGB images and depth images at different viewing angles; then, computer vision methods are used to measure the end face margins of each yarn bobbin on these collected images, and the average value is taken, which can effectively reduce the runout error of the coil margin.

Description

Technical Field

[0001] The present invention belongs to the technical field of textile parameter detection, and specifically relates to a real-time monitoring method based on a machine vision and deep learning coupling algorithm. Background Art

[0002] Automatic detection of bobbin end margins is a core issue in the textile industry. In textile factories, yarn bobbins are typically arranged in 20 to 30 rows. Workers place the bobbins on the brackets and insert the thread ends through a guide tube into the loom. During the weaving process, workers inspect the bobbins and replace any used bobbins. This method is not only inefficient but also creates problems such as tension-induced thread breaks, which are difficult to detect in a timely manner and can lead to more serious consequences. A common solution to this problem is to use sensor-based tension detection. However, a factory can have thousands of bobbins, requiring thousands of sensors. Maintaining each sensor also requires significant labor costs, making this approach relatively uneconomical.

[0003] In view of the shortcomings of traditional line tension-based detection methods such as high cost and low reliability, the present invention proposes a mobile machine vision detection device installed on a guide rail, which integrates computer vision detection of the end face margin of the bobbin yarn and deep learning detection of the bobbin yarn margin, and uses a Kalman filter to eliminate the false detections caused by the two margin measurement methods, thereby achieving accurate detection of the end face margin of the bobbin yarn. A mobile platform equipped with an RGB-D camera moves and cruises along the guide rail, and moves to the corresponding position according to the corresponding detection control instructions to perform image acquisition of the bobbin yarn and online detection of the end face margin of the bobbin yarn. The method proposed in the present invention can be applied to the margin measurement of various bobbins, greatly improving the production efficiency of textile enterprises and reducing production costs. Summary of the Invention

[0004] To address the high cost, low efficiency, and poor reliability of traditional manual yarn thread identification, and to achieve rapid and accurate detection of yarn end margins, thereby improving the production efficiency and reducing production costs of textile enterprises, this paper proposes a real-time monitoring method based on a coupled machine vision and deep learning algorithm.

[0005] The real-time monitoring method based on the coupling algorithm of machine vision and deep learning includes the following steps:

[0006] Step 1: A downward-facing laser and an RGB-D camera, whose pitch angle is controlled by a power element, are positioned above a support holding the yarn package. The RGB-D camera's lens axis and the laser's axis are aligned in the same vertical plane. The laser emits a beam, leaving a spot on the end face of the yarn package. Simultaneously, the RGB-D camera rotates, capturing RGB images and depth maps of the yarn package at different pitch angles.

[0007] Step 2: Perform image processing on the RGB images and depth images captured at different pitch angles to obtain the depth values ​​of the center of the light spot and the center of each bobbin in the image, and extract the outline of each bobbin and its spool.

[0008] Step 3: Calculate the distance L from the light spot to the laser AP and the distance L from the light spot to the RGB-D camera PB as follows:

[0009]

[0010]

[0011] Among them, L AB The distance from the laser and RGB-D camera; L PC is the depth value of the center of the spot, θ is the pitch angle of the laser;

[0012] Calculate the distance L between the center of the RGB image and the center of the light spot OP as follows:

[0013]

[0014] Among them, L OB is the depth value of the RGB image center; β is the angle between the line connecting the center of the light spot to the optical center of the RGB imaging lens and the line connecting the laser to the optical center of the RGB imaging lens;

[0015] θ and β satisfy the following relationship:

[0016]

[0017] Construct the package edge l i The expression is as follows:

[0018]

[0019] Among them, l O'P' is the pixel distance between the center of the RGB image and the center of the light spot; d1 is the pixel diameter of the spool; d2 is the pixel diameter of the bobbin.

[0020] Take the margin l calculated from the images taken at different pitch angles i The average value of the package edge distance l is measured visually. s According to the package yarn margin l s The changes in the yarn consumption during the textile process can be detected in real time.

[0021] Step 4: Use the trained deep convolutional neural network to identify the bobbin margin and obtain the deep learning bobbin margin l n.

[0022] Step 5: Integrate visual measurement to measure the package yarn margin l s Compared with the visual measurement of package yarn edge l s , construct the fused package margin l r as follows:

[0023] l r =kl s +l n (1-k)

[0024] Among them, k is the weight coefficient, and its expression is:

[0025]

[0026] Among them, l max Indicates the maximum edge distance of the package yarn.

[0027] Preferably, the image processing in step 2 includes the following steps:

[0028] 2-1. Align the RGB image and the depth image.

[0029] 2-2. Collect the pixel coordinates of the center of the light spot in the RGB image and the pixel coordinates of the center of the bobbin of the yarn.

[0030] 2-3. Read the depth values ​​of the corresponding coordinates of the center of the light spot and the center of the bobbin of the yarn on the depth map.

[0031] 2-4. Extract the image of each bobbin from the RGB image to obtain multiple sub-images; extract the contours of the bobbin end face and the spool end face in each sub-image through edge detection, and fit the obtained contours into an ellipse.

[0032] 2-5. Correct the elliptical contour obtained by fitting in step 2 into a circle.

[0033] As a preference, the bobbin margin l obtained by deep learning in step 4 n The acquisition process is as follows:

[0034] 4-1. Collect multiple images with non-zero margins, zero margins, and no bobbins as training samples.

[0035] 4-2. Expand the training samples through image enhancement algorithm.

[0036] 4-3. Convert the samples into grayscale images and normalize them as input to the deep convolutional neural network. The deep convolutional neural network consists of a bobbin recognition module and a margin measurement module. The bobbin recognition module distinguishes whether the bobbin margin is zero; the margin measurement module detects the overall diameter of each bobbin and the spool diameter.

[0037] 4-4. Extract sub-images containing a single bobbin from the RGB image, grayscale and normalize each sub-image, and input the processed sub-images into a deep convolutional neural network to obtain the overall bobbin radius and the spool radius.

[0038] 4-5. Constructing the edge distance l' of the package yarn i The expression is as follows:

[0039]

[0040] Among them, l O'P' is the pixel distance between the center of the RGB image and the center of the light spot; r1 is the spool radius obtained in step 4-3; r2 is the overall radius obtained in step 4-3.

[0041] 4-6. Calculate the margin l' from images taken at different pitch angles i The average value of the package yarn margin l is used as the deep learning n .

[0042] As a preference, after step 5 is executed, the bobbin margin is estimated by the Kalman filter to obtain the final vector consisting of the bobbin margin and thread speed. The specific process is as follows:

[0043]

[0044]

[0045]

[0046]

[0047]

[0048]

[0049] in, It is the predicted value composed of the package end face distance and thread consumption speed; k is the currently measured package yarn margin; l k-1 v is the package yarn edge distance measured in the previous measurement cycle; k The current thread consumption speed; v k-1 is the thread consumption speed of the previous measurement cycle; t is the measurement cycle; u is the state control vector; A is the state transfer matrix; B is the input gain matrix; is the current prior estimated covariance; P k-1 is the posterior estimated covariance of the previous measurement period; Q is the process noise covariance; K kis the filter gain matrix; R is the measurement noise covariance; Z k The bobbin margin l obtained in step 6 r and the thread consumption speed; H is the observation matrix; P k Estimated covariance for the current posterior.

[0050] Preferably, the frame rate of the RGB-D camera when shooting the bobbin is 30FPS.

[0051] Preferably, the RGB image and depth image of the bobbin captured by the RGB-D camera have the same resolution.

[0052] Preferably, the real-time monitoring method based on a coupled machine vision and deep learning algorithm utilizes a measurement device comprising a guide rail, a mobile platform, and an RGB-D camera and laser mounted on the mobile platform. The mobile platform slides on the guide rail and is driven by a power element. The mobile platform is capable of moving directly above multiple supports holding bobbins. The RGB-D camera is capable of flipping about a horizontal axis driven by the power element. The RGB-D camera and the laser are positioned at the same height. The laser is positioned with an angle facing downward.

[0053] Preferably, the mobile platform is driven by a motor through a rack and pinion.

[0054] Preferably, the pitch angle of the laser is adjustable. The rotation axes of the RGB-D camera and the laser are parallel to each other.

[0055] Preferably, the flip range of the RGB-D camera is -30° to +30° centered vertically downward.

[0056] The beneficial effects of the present invention are:

[0057] 1. The present invention proposes a package yarn margin measurement algorithm based on computer vision, which can realize the measurement of package yarn margin without prior knowledge.

[0058] 2. During the acquisition of RGB and depth images, the present invention obtains RGB and depth images from different viewing angles by rotating the RGB-D camera at a constant speed and small angle within a relatively short period of time. Computer vision methods are then used to measure the end face margins of each bobbin using these acquired images. The measurement results are averaged, effectively reducing the runout error of the bobbin margin and improving the measurement accuracy of the bobbin margin.

[0059] 3. The present invention proposes a deep convolutional neural network with a two-level model, which measures the bobbin yarn margins on the collected images respectively and takes the arithmetic mean of the measurement results, thereby realizing fast and accurate measurement of the bobbin yarn margins.

[0060] 4. This invention proposes a fusion algorithm that effectively integrates the package end margins measured by computer vision and neural network. This method overcomes the low accuracy of package end margins measured by computer vision when the package end margins are small, thereby improving the accuracy of package margin measurement.

[0061] 5. The present invention proposes to use Kalman filtering to effectively estimate the fusion results of the bobbin yarn margin measured by computer vision and the bobbin yarn margin measured by neural network, eliminating the false detection situation that exists in the two bobbin yarn margin measurement processes, and further improving the accuracy and robustness of bobbin yarn margin measurement.

[0062] 6. The mobile platform designed in the present invention is moved to the approximate position within the guide rail area corresponding to the bracket to be tested using conventional control methods. The RGB-D camera is then used as the image sensor for image visual servo control. The mobile platform switches to image visual servo control mode and is controlled to move to the optimal bobbin image capture position. By combining conventional control with image visual servo control, the mobile platform can be quickly and accurately moved to the optimal bobbin image capture position. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 Schematic diagram of the device for measuring the end face margin of bobbin yarn according to the present invention.

[0064] Figure 2 Schematic diagram of the end margin of the bobbin yarn.

[0065] Figure 3 This is a flow chart of measuring the end face margin of the bobbin yarn of the present invention.

[0066] Figure 4 This is a flow chart of the precise parking of the mobile platform of the present invention.

[0067] Figure 5 This is a diagram of the image processing process of the bobbin yarn.

[0068] Figure 6 Schematic diagram of the light spot image of the laser irradiating the yarn package.

[0069] Figure 7 Schematic diagram of the imaging process of bobbin yarn.

[0070] Figure 8 Schematic diagram of the comparison of the bobbin image before and after correction.

[0071] Figure 9 This is a schematic diagram of auxiliary calculation of the end face margin of the package yarn. DETAILED DESCRIPTION

[0072] In order to make the technical means, creative features, objectives and effects of the invention easier to understand, the present invention is further described below with reference to specific illustrations.

[0073] like Figure 1 As shown, a real-time monitoring method based on a coupled machine vision and deep learning algorithm is employed. The measuring device comprises a guide rail 6, a mobile platform 5, and an RGB-D camera 4 and a laser 3 mounted on the mobile platform 5. The mobile platform 5 slides on the guide rail 6 and is driven by a motor via a rack and pinion. The RGB-D camera 4 can flip around a horizontal axis driven by a power element. The RGB-D camera 4 and the laser 3 are positioned at the same height. The power element utilizes a servo motor. The pitch angle of the laser 3 is adjustable. The rotation axes of the RGB-D camera 4 and the laser 3 are parallel to each other and perpendicular to the line connecting the two. The tilt range of the RGB-D camera 4 is -30° to +30°, centered vertically downward. The guide rail 6 is located directly above the bracket 1 on which the measured bobbin 2 is mounted.

[0074] like Figure 2 As shown, the end face margin of the bobbin 2 mentioned in the present invention refers to the radial distance from the outer circumferential surface of the bobbin 2 to the outer circumferential surface of the bobbin 2.

[0075] like Figure 3 As shown, the real-time monitoring method based on the coupling algorithm of machine vision and deep learning includes the following steps:

[0076] Step 1: Figure 4 As shown in the figure, at fixed time intervals, the mobile platform periodically inspects the bobbins on each bobbin holder on the guide rail. Upon receiving a command to measure the bobbin end face margin, the mobile platform begins moving to the guide rail area corresponding to the holder to be tested. Within this area, the RGB-D camera's field of view fully covers the entire holder. The mobile platform then switches to visual servoing control, stopping on the guide rail at the optimal position for capturing bobbin images.

[0077] Step 2: After mobile platform 5 reaches the detection position, the laser adjusts its angle to emit a laser beam, leaving a light spot on the end face of yarn cone 2. Simultaneously, RGB-D camera 4 begins capturing the yarn cone's RGB image and depth map. The frame rate of RGB-D camera 4 is 30 FPS. During image acquisition, RGB-D camera 4 rotates from one end to the other within its tiltable range, continuously capturing RGB and depth images of yarn cone 2. This entire process lasts two seconds, capturing a total of 60 pairs of RGB and depth images. The captured RGB and depth images of yarn cone 2 have the same resolution.

[0078] Step 3: Perform the same image processing method on the RGB images and depth images of the 60 pairs of yarns 2 collected, such as Figure 5 The specific process is as follows:

[0079] (1) Reading the collected RGB image and depth image of the bobbin 2, and aligning the RGB image with the depth image;

[0080] (2) Processing the RGB image of the aligned bobbin 2 to obtain the pixel coordinates of the bobbin center of bobbin 2 and the pixel coordinates of the light spot center;

[0081] (3) Read the depth value of the corresponding coordinates on the depth map according to the pixel coordinates of the center of the line axis and the pixel coordinates of the center of the light spot on the obtained RGB image;

[0082] (4) Process the aligned RGB images to obtain a sub-image of each bobbin 2, where the center of each sub-image is the pixel coordinate of the bobbin 2. Use edge detection to extract the contour features of each bobbin 2 end face and the bobbin end face, and fit them with elliptical graphic features. In addition, Figure 6 As shown in FIG, the center of the light spot is connected to the center of the RGB image with a pixel line as a line feature of the light spot on the RGB image.

[0083] (5) Figure 7 As shown, according to the imaging principle, the image captured by the RGB-D camera 4 will be deformed, and the circular bobbin 2 will become an ellipse in the image. Therefore, it is necessary to correct the elliptical bobbin shape in the bobbin image to restore it to a circular bobbin shape image, as shown in FIG. Figure 8 In the same way, the pixel line from the center of the light spot drawn in step (4) to the center of the collected RGB image is corrected.

[0084] Step 4: Calculate the end face margin of bobbin 2 in the RGB image. Figure 9 As shown, according to the design of the computer vision measurement device of the present invention, the laser 3 and the RGB-D camera 4 are placed at the same height, and the distance between them is a known quantity L AB .

[0085] According to the pixel coordinates of the light spot on the RGB image, the corresponding depth value is read to obtain the depth value L of the light spot position read by the RGB-D camera 4. PC , the laser's pitch angle θ is known, then

[0086]

[0087]

[0088] Among them, L APis the distance from the light spot to the laser 3; L PB is the distance from the light spot to the RGB-D camera 4.

[0089] In ΔAPB, there are

[0090]

[0091] According to the pixel coordinates corresponding to the image principal point O', read the depth value L of the image center position in the corresponding depth map OB , β is the angle between the line from the spot center P to the optical center B of the RGB imaging lens and the line from the laser to the optical center B of the RGB imaging lens; in ΔPOB,

[0092]

[0093] According to the rectified image, the rectified pixel distance of line segment OP on the RGB image of bobbin 2 is l O'P' Taking the measurement of the margin of a bobbin sub-image as an example, the pixel diameter of the bobbin bobbin in the rectified image is d1, and the pixel diameter of the end face of bobbin 2 in the rectified image is d2. The actual value of the margin of bobbin 2 is defined as l, then

[0094]

[0095] The above steps are to calculate the end face margin of a bobbin 2 in one of the 60 bobbin RGB images collected within one image acquisition cycle. For the same bobbin in the 60 bobbin RGB images, the same margin calculation algorithm is used. The end face margin of the bobbin 2 calculated from the 60 images is recorded as l i (i=1,2…60). Then the average value of the margins calculated for the 60 sub-images of the same bobbin 2 is taken as the actual measured value l of the end face margin of the bobbin 2. s :

[0096]

[0097] Step 5. Use deep learning-based measurement of the end face margin of bobbin 2. A large number of images with non-zero end face margins of bobbin 2, images with zero end face margins of bobbin 2 (i.e., images of unwound bobbins), and some background images are collected as training samples. In order to further expand the number of training samples, an image enhancement algorithm is used to expand the samples. Then, the images are converted into grayscale images, and finally normalized as the input of a deep convolutional neural network dedicated to bobbin end face margin measurement. The deep convolutional neural network is divided into a bobbin recognition module and a margin measurement module. The bobbin recognition module is used to distinguish whether the margin of bobbin 2 is 0; the margin measurement module is used to detect the margin of each bobbin 2, and its output is the spool pixel radius and the bobbin 2 end face pixel radius. After the model is trained, the trained model is used to measure the margin of bobbin 2. The RGB sub-image of a single bobbin 2 is grayscaled and normalized, and the processed image data to be inspected is input into a deep convolutional neural network dedicated to bobbin end face margin measurement. The deep convolutional neural network first inputs the data into the bobbin identification module, which makes a judgment on the input data. If the margin of bobbin 2 is zero, the end face margin result of bobbin 2 is directly output. If the end face margin of bobbin 2 is not zero, the processed bobbin 2 sub-image is further input into the margin measurement module. The margin measurement module performs calculations based on the input RGB sub-image of bobbin 2 and outputs the image results after calculation. The output results are the pixel radius r1 of the spool and the pixel radius r2 of bobbin 2. The end face margin calculation for an RGB sub-image of bobbin 2 is as follows:

[0098]

[0099] Similarly, the above steps are used to calculate the edge distance of a bobbin 2 in one of the 60 RGB images of bobbins collected during one image acquisition cycle. For the same bobbin 2 in the 60 RGB images, the same method of calculating the edge distance of bobbin 2 is used, and the calculated edge distance of bobbin 2 is recorded as l' i , (i=1,2…60). Then, the average value of the margins calculated for the 60 bobbin sub-images corresponding to the same bobbin 2 is taken as the actual measurement value l obtained by the deep learning measurement method for the margin of the bobbin 2. n .

[0100]

[0101] Step 6: As the thread consumption gradually decreases, the results of computer vision measurement may be inaccurate, and the method for measuring the end margin of yarn 2 based on machine learning may also have false detection. Therefore, the present invention combines the end margin of yarn 2 measured by computer vision and the end margin of yarn 2 measured by deep learning to reduce the false detection caused by the two margin measurement methods. The fusion process is as follows:

[0102] (1) First, determine whether the end face margins of the package 2 measured by computer vision and the end face margins of the package 2 measured by deep learning are zero. If the end face margins of the package 2 measured by computer vision and the end face margins of the package 2 measured by deep learning are both zero, then directly output the margin results.

[0103] (2) If one or both of the end face margins of the package yarn 2 measured by computer vision and the end face margins of the package yarn 2 measured by deep learning are not zero, the measurement results of the two measurement methods are fused. The fusion algorithm is as follows:

[0104]

[0105] l r =kl s +l n (1-k)

[0106] Where, l r represents the result of fusion calculation, l s It indicates the end face margin of a single bobbin 2 measured by computer vision, l n represents the end face margin of a single bobbin 2 measured in a deep learning-based measurement method, l max Indicates the maximum margin of bobbin 2. The calculation method of the weight k in the fusion algorithm provided by the present invention can reduce the weight k as the bobbin end face margin becomes smaller, thereby reducing l s The weight in the fusion algorithm can make up for the low accuracy of the measurement results when the package yarn margin is small using computer vision measurement.

[0107] Step 7: To further avoid the occurrence of false detection in the two measurement methods mentioned above and to further improve the measurement accuracy, the present invention uses a Kalman filter to estimate the end face margin of the fused bobbin 2. The theoretical thread margin can be calculated based on the thread consumption speed and time interval of the bobbin 2. The result is used as an estimated value to calculate the end face margin of the fused bobbin 2 in the present invention. r As the actual measured value, the Kalman filter is used to estimate the end face margin of package yarn 2. The calculation process is as follows:

[0108]

[0109]

[0110]

[0111]

[0112]

[0113]

[0114] Where, t is the measurement period; l k is the currently measured package yarn margin; l k-1 v is the package yarn edge distance measured in the previous measurement cycle; k The current thread consumption speed; v k-1 is the thread consumption speed of the previous measurement cycle; u is the state control vector; A is the state transfer matrix; B is the input gain matrix; is the current prior estimated covariance; P k-1 is the posterior estimated covariance of the previous measurement period; Q is the process noise covariance; K k is the filter gain matrix; R is the measurement noise covariance; Z k The bobbin margin l obtained in step 6 r A vector consisting of the thread consumption speed; vector is the predicted value composed of the end face distance of bobbin 2 and thread consumption speed; H is the observation matrix; P k is the posterior estimated covariance of the current cycle. By using the Kalman filter to comprehensively estimate the actual measured value and theoretical calculated value of the end face margin of the yarn 2, the final calculation result vector is The calculated value of the package yarn end face margin distance is taken as the final measurement result of the package yarn end face margin distance.

[0115] The method for measuring the end face margin of a bobbin 2 by using computer vision used in the present invention can be used not only for measuring circular targets, but also for measuring the size of targets with other irregular shapes.

Claims

1. A real-time monitoring method based on a coupled machine vision and deep learning algorithm, characterized by: The following steps are involved: Step 1: Arrange a laser tilted downward and an RGB-D camera (4) whose pitch angle is adjusted by a power element above a bracket on which a bobbin is mounted; the lens axis of the RGB-D camera (4) and the axis of the laser (3) are in the same vertical plane; the laser emits a laser beam, leaving a light spot on the end face of the bobbin (2); and simultaneously, the RGB-D camera (4) rotates to collect RGB images and depth maps of the bobbin at different pitch angles; Step 2: performing image processing on the RGB images and depth images obtained at different pitch angles to obtain the depth values ​​of the center of the light spot and the center of each bobbin (2) in the image, and extracting the outline of each bobbin (2) and its spool; Step 3: Calculate the distance L from the light spot to the laser (3) AP and the distance L from the light spot to the RGB-D camera (4) PB as follows: Among them, L AB is the distance between the laser (3) and the RGB-D camera (4); L PC is the depth value of the center of the spot, θ is the pitch angle of the laser; Calculate the distance L between the center of the RGB image and the center of the light spot OP as follows: Among them, L OB is the depth value of the RGB image center; β is the angle between the line connecting the center of the light spot to the optical center of the RGB imaging lens and the line connecting the laser to the optical center of the RGB imaging lens; θ and β satisfy the following relationship: Construct the margin l of the bobbin (2) i The expression is as follows: Among them, l O'P' is the pixel distance between the center of the RGB image and the center of the light spot; d1 is the pixel diameter of the spool; d2 is the pixel diameter of the bobbin (2); Take the margin l calculated from the images taken at different pitch angles i The average value of the package edge distance l is measured visually. s ; Step 4: Use the trained deep convolutional neural network to identify the bobbin margin and obtain the deep learning bobbin margin l n ; Step 5: Integrate visual measurement to measure the package yarn margin l s Package margin l with deep learning n , construct the fused package margin l r as follows: l r =kl s +l n (1-k) Among them, k is the weight coefficient, and its expression is: Among them, l max Indicates the maximum edge distance of the package yarn.

2. The real-time monitoring method based on a machine vision and deep learning coupling algorithm according to claim 1, characterized in that: The image processing in step 2 includes the following steps: 2-1. Align the RGB image and the depth image; 2-2. Collect the pixel coordinates of the center of the light spot in the RGB image and the pixel coordinates of the center of the bobbin of the bobbin (2); 2-3. Read the depth values ​​of the corresponding coordinates of the center of the light spot and the center of the bobbin of the yarn (2) on the depth map; 2-4. Extracting an image of each bobbin (2) from the RGB image to obtain a plurality of sub-images; extracting the contours of the end face of the bobbin (2) and the end face of the spool in each sub-image by edge detection, and fitting the obtained contours into an ellipse; 2-5. Correct the elliptical contour obtained by fitting in step (4) into a circle.

3. The real-time monitoring method based on the machine vision and deep learning coupling algorithm according to claim 1, characterized in that: The bobbin margin l obtained through deep learning in step 4 n The acquisition process is as follows: 4-1. Collect multiple images with non-zero margins, zero margins, and no bobbins as training samples; 4-2. Expand the training samples through image enhancement algorithm; 4-3. The samples are converted into grayscale images and normalized as inputs of a deep convolutional neural network; the deep convolutional neural network is divided into a bobbin identification module and a margin measurement module; the bobbin identification module is used to distinguish whether the margin of the bobbin (2) is 0; the margin measurement module is used to detect the overall diameter and the spool diameter of each bobbin (2); 4-4. Extracting a sub-image containing a single bobbin (2) from the RGB image, and gray-scaling and normalizing each sub-image; inputting the processed sub-image into a deep convolutional neural network to obtain the overall radius of the bobbin (2) and the bobbin radius; 4-5. Constructing the margin l' of the bobbin (2) i The expression is as follows: Among them, l O'P' is the pixel distance between the center of the RGB image and the center of the light spot; r1 is the spool radius obtained in step 4-3; r2 is the overall radius obtained in step 4-3; 4-6. Calculate the margin l' from images taken at different pitch angles i The average value of the package yarn margin l is used as the deep learning n .

4. The real-time monitoring method based on a machine vision and deep learning coupling algorithm according to claim 1, characterized in that: After step 5 is executed, the bobbin margin is estimated through the Kalman filter to obtain the final vector consisting of the bobbin margin and thread speed. The specific process is as follows: in, is the predicted value composed of the end face distance of the bobbin (2) and the thread consumption speed; l k is the currently measured package yarn margin; l k-1 v is the package yarn edge distance measured in the previous measurement cycle; k The current thread consumption speed; v k-1 is the thread consumption speed of the previous measurement cycle; t is the measurement cycle; u is the state control vector; A is the state transfer matrix; B is the input gain matrix; is the current prior estimated covariance; P k-1 is the posterior estimated covariance of the previous measurement period; Q is the process noise covariance; K k is the filter gain matrix; R is the measurement noise covariance; Z k The bobbin margin l obtained in step 6 r and the thread consumption speed; H is the observation matrix; P k Estimated covariance for the current posterior.

5. The real-time monitoring method based on the coupling algorithm of machine vision and deep learning according to claim 1, characterized in that: The frame rate of the RGB-D camera (4) when photographing the bobbin (2) is 30 FPS.

6. The real-time monitoring method based on a machine vision and deep learning coupling algorithm according to claim 1, characterized in that: The RGB image and depth image of the bobbin (2) captured by the RGB-D camera have the same resolution.

7. The real-time monitoring method based on a machine vision and deep learning coupling algorithm according to claim 1, characterized in that: The measuring device used includes a guide rail (6), a mobile platform (5), and an RGB-D camera (4) and a laser (3) mounted on the mobile platform (5); the mobile platform (5) slides on the guide rail (6) and is driven by a power element; the mobile platform (5) can move directly above a plurality of brackets equipped with bobbins; the RGB-D camera (4) can be flipped around a horizontal axis under the drive of the power element; the RGB-D camera (4) and the laser (3) are arranged at the same height; and the laser (3) is arranged tilted downward.

8. The real-time monitoring method based on the machine vision and deep learning coupling algorithm according to claim 7, characterized in that: The mobile platform (5) is driven by a motor through a gear rack.

9. The real-time monitoring method based on the machine vision and deep learning coupling algorithm according to claim 7, characterized in that: The pitch angle of the laser (3) can be adjusted; the rotation axes of the RGB-D camera (4) and the laser (3) are parallel to each other.

10. The real-time monitoring method based on the coupling algorithm of machine vision and deep learning according to claim 7, characterized in that: The flip range of the RGB-D camera (4) is -30° to +30° centered vertically downward.

Images