Method, system and equipment for measuring track width of excavator based on monocular camera

Abstract

The present invention discloses a method, system and equipment for measuring the track width of an excavator based on a monocular camera. The measuring method comprises: using a monocular camera to collect track images and preprocessing them; using a track plate detection model to identify the track images, using a target frame to frame the track plates to obtain a target frame set R; screening out a target frame with the highest matching degree with the track plates from the target frame set R, and obtaining the pixel width of the track plates according to the target frame; using a bolt detection model to identify the bolts in the image area framed by the target frame and outputting the bolt target frame, and obtaining the pixel width of the bolts according to the image area framed by the bolt target frame; the physical width of the bolts is known to be, and the physical width of the track plates is. The present invention uses a monocular camera that can shoot at a long distance to collect track images, and converts the track width by analyzing the pixel ratio relationship between the bolts and the track plates, so as to achieve low-cost, fast and accurate measurement of the track width.

Description

Technical Field

[0001] The present invention relates to the technical field of excavator track size measurement, and in particular to a method, system and device for measuring the width of an excavator track based on a monocular camera. Background Art

[0002] The track is an important part of the excavator chassis. In order to meet different application scenarios and customized needs, the track usually has a variety of width specifications (usually between 400 mm and 800 mm). During the assembly process of the excavator, it is common for the track width specifications to not match the order configuration. Errors in the assembly of the track will not only increase the time for returning to the factory for replacement and cause economic losses, but may also cause safety hazards during use. Therefore, the track width specification detection is an important part of the factory quality inspection of the excavator. Traditionally, the track width specification detection usually relies on manual measurement and confirmation of the track width with tools such as tape measures, which is not only time-consuming and labor-intensive, but also prone to human errors. Therefore, how to intelligently, efficiently and accurately detect the track width has become a problem that needs to be solved urgently.

[0003] At present, the existing track width measurement methods usually use 3D cameras such as binocular or ToF to obtain the three-dimensional coordinate information of the track, and then use Euclidean distance to calculate the track width. Although this method has high measurement accuracy, the optimal imaging distance of ordinary 3D cameras is limited, and the 3D camera needs to be installed at a position close to the track. For safety reasons, 3D cameras are not allowed to be installed close to the excavator quality inspection area to avoid affecting the normal movement and debugging of the excavator. However, 3D cameras that can achieve long-distance accurate measurement are expensive. Summary of the invention

[0004] In order to solve the problems existing in the prior art, the present invention provides a method, system and equipment for measuring the track width of an excavator based on a monocular camera. The monocular camera can collect track images at a long distance, and combined with an innovative calculation and analysis method, it can realize automatic measurement of the track width.

[0005] The technical solution adopted by the present invention to solve the technical problem is: a method for measuring the width of an excavator track based on a monocular camera, comprising: S1, using a monocular camera to shoot the excavator track from top to bottom to collect a track image; S2, preprocessing the track image to filter out noise; S3, using a trained track shoe detection model to recognize the track image, using a target frame to frame the track shoe in the track image, and obtaining a target frame set ; S4, from the target frame set Filter out the target frame with the highest matching degree with the track plate , according to the target frame Get the pixel width of the track shoe ; S5, using the trained bolt detection model to detect the bolts in the target frame The bolt is identified in the framed image area and a bolt target frame is output, and the pixel width of the bolt is obtained according to the image area framed by the bolt target frame ; S6. The physical width of the known bolt is , the physical width of the track shoe is .

[0006] Furthermore, the target frame The screening process includes:

[0007] S4.1, take the short side of the target frame as the center line and expand 30 pixels to both sides to obtain two rectangular edge detection areas;

[0008] S4.2. Identify a set of n candidate straight line segments in the edge detection area , selecting the straight line segment with the highest fitting degree from the n candidate straight line segments as the vertical edge of the track shoe;

[0009] S4.3, repeat steps S4.1-S4.2 to obtain the vertical edges of the track shoes corresponding to all target frames;

[0010] S4.4. Calculate the average of the fit of the two vertical sides of the track shoe corresponding to each target frame, and take the target frame with the highest average value as the target frame .

[0011] Furthermore, in step S4.2, the process of selecting the straight line segment with the highest fitting degree includes:

[0012] Calculate the length of each straight line segment , the angle between the straight line segment and the short side of the target box , and the distance between the midpoint of the straight line segment and the vertical side of the target box ;

[0013] The length And the angle Straight line segments with angles > 5° are deleted from the set. Indicates the height of the edge detection area; if the straight line segment set is an empty set after deleting the straight line segments that do not meet the requirements, it means that the positioning error of the target frame is too large, and the target frame is deleted from the target frame set R;

[0014] If the set of straight line segments is not an empty set after deleting the straight line segments that do not meet the requirements, the fitting degree of the remaining straight line segments in the set is calculated. , , , is the adjustment factor, with values ​​of 1, 0.8, and 0.2;

[0015] The straight line segment with the highest fitting degree is used as the vertical side of the track shoe of the target frame.

[0016] Furthermore, step S5 specifically includes:

[0017] Taking the four boundaries of the bolt target frame as the reference, expand outward by 10 pixels to obtain the rectangular area X;

[0018] Extract six straight line segments of the bolt in the rectangular area X , calculate the angles between each of the six straight line segments ;

[0019] If the angle <5°, then the two straight line segments are judged to be parallel to each other, and a total of three pairs of parallel straight line segments are obtained;

[0020] Calculate the distance between the midpoints of each pair of parallel line segments , , , the corresponding pixel width of the bolt is ;

[0021] The final width of the bolt in pixels is the average of the pixel widths of the two bolts on the track shoe.

[0022] Furthermore, the preprocessing of the crawler image includes bilateral filtering, and the formula of bilateral filtering is:

[0023] ,

[0024] ,

[0025] in, Indicates the coordinates of the pixel currently being convolved, Indicates the range pixel coordinates, Represents the value calculated by two Gaussian functions, , represents the smoothing parameter.

[0026] Furthermore, the track shoe detection model is a yolov8 model, a Faster-RCNN model, and an SSD model.

[0027] The present invention also provides a system for measuring the track width of an excavator based on a monocular camera, characterized in that the system adopts the method for measuring the track width of an excavator based on a monocular camera, and comprises:

[0028] An acquisition module, including a monocular camera and a light source, is used to capture images of the excavator tracks from top to bottom;

[0029] An image denoising module is used to denoise the collected track images;

[0030] The analysis and calculation module is used to identify the track image using the trained track shoe detection model, and to frame the track shoe in the track image with a target frame to obtain a target frame set. ; From the target frame set Filter out the target frame with the highest matching degree with the track plate , according to the target frame Get the pixel width of the track shoe ; Use the trained bolt detection model to detect the bolts in the target frame The bolt is identified in the framed image area and a bolt target frame is output, and the pixel width of the bolt is obtained according to the image area framed by the bolt target frame ; The physical width of the known bolt is , the physical width of the track shoe is .

[0031] The present invention also provides a computer device, comprising: a processor; a memory for storing executable instructions; wherein the processor is used to read the executable instructions from the memory and execute the executable instructions to implement the method for measuring the track width of an excavator based on a monocular camera.

[0032] The beneficial effect of the present invention is that the method, system and device for measuring the track width of an excavator based on a monocular camera of the present invention uses a monocular camera that can shoot at a long distance to collect track images, and converts the track width by analyzing the pixel ratio relationship between the bolt and the track shoe, so as to achieve low-cost, fast and accurate measurement of the track width. In addition, multiple screening analyses are performed in the process of obtaining the pixel width of the track shoe to improve the accuracy of the pixel width of the track shoe. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0034] Figure 1 It is a flow chart of the method for measuring the track width of an excavator based on a monocular camera of the present invention.

[0035] Figure 2 It is a structural schematic diagram of the crawler of the present invention.

[0036] Figure 3 It is a schematic diagram of a target frame of the present invention framing a track shoe area on a track image.

[0037] Figure 4 It is a schematic diagram of the target frame expansion of the present invention.

[0038] Figure 5 It is a schematic diagram of the straight line segment fitting of the present invention.

[0039] Figure 6 It is a schematic diagram of bolt identification of the present invention. DETAILED DESCRIPTION

[0040] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0041] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0042] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0043] like Figures 1 to 6 As shown, the method for measuring the track width of an excavator based on a monocular camera of the present invention comprises: S1, using a monocular camera to photograph the excavator track from top to bottom to collect a track image. S2, preprocessing the track image to filter out noise. S3, using a trained track shoe detection model to identify the track image, using a target frame to frame the track shoe in the track image, and obtaining a target frame set. S4. From the target box set Filter out the target frame with the highest matching degree with the track plate , according to the target box Get the pixel width of the track shoe S5. Use the trained bolt detection model to detect the bolts in the target frame. The bolt is identified in the framed image area and the bolt target frame is output. The pixel width of the bolt is obtained according to the image area framed by the bolt target frame. S6. The physical width of the known bolt is , the physical width of the track shoe is .

[0044] The present invention collects crawler images through a monocular camera. The monocular camera can be installed at a long distance and will not affect the movement and debugging of the excavator. The monocular camera shoots from top to bottom facing the crawler. Affected by the light and environment on site, the collected crawler images have certain noise interference. Therefore, it is necessary to denoise the crawler images. Denoising can be done by bilateral filtering. While denoising, bilateral filtering can retain the edge characteristics of the image. The formula for bilateral filtering is:

[0045] ,

[0046] ,

[0047] in, Indicates the coordinates of the pixel currently being convolved, Indicates the range pixel coordinates, Represents the value calculated by two Gaussian functions, , Represents a smoothing parameter. Then, the trained track shoe detection model is used to output multiple track shoe target frames, and then the target frames are calculated and analyzed to obtain the pixel width of the track shoe. The trained bolt detection model is used to obtain the pixel width of the bolt. The same specification of hexagonal head bolts are used on tracks of different width specifications. The physical width of the track shoe can be converted based on the pixel width and physical size of the bolt. The measurement method of the present invention uses a low-cost camera, high calculation efficiency, and higher cost performance.

[0048] It should be noted that in order to capture the entire track boundary, the monocular camera will capture part of the ground image in the track image. Therefore, analysis and processing are required when identifying the track plate boundary. The track is composed of multiple track plates of the same specifications, so the track width can be obtained by measuring the width of the track plate. The track plate is roughly rectangular, and the distance between the two short sides is the track plate width.

[0049] The preprocessed track image is input into the newly connected track shoe detection model. The track shoe detection model can frame the track shoe area in the image. Since a track image contains more than one track shoe, there will be multiple target frames in an image. Affected by the model's own detection performance and training effect, there may be deviations between the areas framed by these target frames and the track shoe. As shown in the figure, the dotted frame is the target frame. Some of the target frames output by the model will shift to the left side of the track shoe, some to the right side of the track shoe, and some areas are much larger than the track shoe, etc. Therefore, the target frames should be screened to select the target frame with the highest matching degree with the track shoe.

[0050] Specifically, the target frame The screening process includes: S4.1, taking the short side of the target frame as the center line, expanding 30 pixels to both sides to obtain two rectangular edge detection regions. S4.2, identifying a set of n candidate straight line segments in the edge detection region. , select the straight line segment with the highest fitting degree from the n candidate straight line segments as the vertical edge of the track shoe. S4.3, repeat steps S4.1-S4.2 to obtain the vertical edges of the track shoe corresponding to all target frames. S4.4, calculate the average fitting degree of the two vertical edges of the track shoe corresponding to each target frame, and take the target frame with the highest average fitting degree as the target frame. .

[0051] like Figure 4 As shown, the two short sides of the target frame are respectively short side a and short side b. With short side a as the center line, 30 pixels are added to the left and right sides to form an edge detection area, that is, an edge detection area contains 60 pixels, and the height of the edge detection area is equal to the length of short side a. Then, the vertical edge of the track shoe is identified in the edge detection area. Theoretically, the two short sides of the optimal target frame should basically coincide with the two vertical sides of the track shoe, but the target frame output by the detection model may have a certain deviation (but this deviation will not be too outrageous). Therefore, the present invention sets an edge detection area to identify the vertical edge of the track shoe in the edge detection area. For example, the Canny operator is used to extract the edges of all features in the edge detection area, and then the Hough transform is used to detect straight lines, and the detected straight line segments are combined into a set It should be noted that in addition to the vertical edge of the track shoe, other straight line segments may be detected in the edge detection area. These straight line segments may be scratches, gaps, etc. on the ground. Therefore, the detected straight line segments need to be screened.

[0052] The process of screening straight line segments includes: calculating the length of each straight line segment , the angle between the straight line segment and the short side of the target box , and the distance between the midpoint of the straight line segment and the vertical side of the target box . Change the length And the angle Straight line segments with angles > 5° are deleted from the set. Indicates the height of the edge detection area. If the straight line segment set is an empty set after deleting the straight line segments that do not meet the requirements, it means that the positioning error of the target frame is too large, and the target frame is deleted from the target frame set R. If the straight line segment set is not an empty set after deleting the straight line segments that do not meet the requirements, the fitting degree of the remaining straight line segments in the set is calculated. , , , is the adjustment factor, and its values ​​are 1, 0.8, and 0.2 respectively. The straight line segment with the highest fitting degree is used as the vertical side of the track shoe of the target frame. The fitting degree calculation formula in this embodiment is proposed based on the application scenario of track shoe detection and has particularity.

[0053] That is to say, the lengths and inclination angles of the straight line segments detected in the edge detection area are different, while the vertical side of the track shoe is vertical and its length is close to the length of the short side of the target frame. Therefore, by judging the length and angle of the straight line segment, some interference items can be deleted. The remaining straight line segment is likely to be the vertical side of the track shoe. The fitting degree of each remaining straight line segment is calculated, and the length information is included in the calculation parameters of the fitting degree. , Angle information And distance information , and take the straight line segment with the highest fitting degree as the vertical edge of the track shoe. Thus, the pixel coordinates of the two vertical edges of the track shoe can be determined. Repeating the above screening process can obtain the corresponding vertical edges of the remaining target frames in the set R. Then calculate the average fitting degree of the two vertical edges of each target frame, and select the target frame with the highest average value as the optimal target frame. Optimal target box The distance between the midpoints of the two corresponding vertical edges is the pixel width of the track shoe. The reason for choosing the center of the vertical side for distance calculation here is that the vertical side is obtained by fitting, and using the center for distance calculation can improve the accuracy of the width value.

[0054] Since track shoes have many width specifications, the physical width cannot be calculated based on the pixel width of the track shoe alone. However, the bolt specifications used for tracks of different specifications are the same, so the physical width of the track shoe can be obtained based on the bolt size. The framed image area is input into the trained bolt detection model, and the bolt detection model can use the target frame to frame the bolt area in the image area. Then, the four boundaries of the bolt target frame are expanded outward by 10 pixels to obtain a rectangular area X. The six straight line segments of the bolt are extracted in the rectangular area X. , calculate the angles between each of the six straight line segments If the angle <5°, the two straight line segments are considered parallel to each other, and a total of three pairs of parallel straight line segments are obtained. Calculate the distance between the midpoints of each pair of parallel straight line segments , , , the corresponding pixel width of the bolt is The final pixel width of the bolt. is the average of the pixel widths of the two bolts on the track shoe.

[0055] It should be noted that the hexagonal bolt is a regular hexagon, and the bolt target box is theoretically a square. Due to the influence of model performance and training effect, the area framed by the bolt target box may have deviations, resulting in the bolt boundary not being completely framed in the bolt target box (such as Figure 6 Therefore, the present invention takes the four boundaries of the bolt target frame as the reference and expands them outward by 10 pixels each (if the expansion is too small, the bolt boundary may be incomplete, and if the expansion is too large, unnecessary image processing calculation will be increased) to form a larger rectangular area X (such as Figure 6 The solid line frame in the figure is used to make the six sides of the bolt appear completely in the rectangular area X, so as to ensure that the pixel width of the bolt can be accurately calculated later. The six straight line segments are detected in the rectangular area X using the Canny operator and Hough transform. , in the rectangular area X, there are usually no other straight line features except the bolt boundary. After analysis, we can mark which two straight line segments are parallel to each other, and get three pairs of parallel straight line segments in total. The distance between the two parallel straight line segments is the pixel width of the bolt. In order to improve accuracy, the average value of the distance between the three pairs of straight line segments is used as the final pixel width of the bolt.

[0056] Knowing the physical width and pixel width of the bolt, according to the formula The physical width of the track shoe can be obtained. Track shoe detection models include yolov8 model, Faster-RCNN model, SSD model, etc.

[0057] The present invention also provides a system for measuring the width of an excavator track based on a monocular camera, and a method for measuring the width of an excavator track based on a monocular camera. The system comprises: an acquisition module, comprising a monocular camera and a light source, for photographing an image of the excavator track from top to bottom; an image denoising module, for denoising the acquired track image; an analysis and calculation module, for recognizing the track image using a trained track shoe detection model, and framing the track shoe in the track image with a target frame to obtain a target frame set. ; From the target box collection Filter out the target frame with the highest matching degree with the track plate , according to the target box Get the pixel width of the track shoe ; Use the trained bolt detection model to detect the target The bolt is identified in the framed image area and the bolt target frame is output. The pixel width of the bolt is obtained according to the image area framed by the bolt target frame. ; The physical width of the known bolt is , the physical width of the track shoe is .

[0058] The present invention also provides a computer device, comprising: a processor; a memory for storing executable instructions; wherein the processor is used to read the executable instructions from the memory and execute the executable instructions to implement the method for measuring the track width of an excavator based on a monocular camera.

[0059] In summary, the monocular camera-based excavator track width measurement method, system and equipment of the present invention use a monocular camera that can shoot at a long distance to collect track images, and convert the track width by analyzing the pixel ratio relationship between the bolts and the track shoes, so as to achieve low-cost, fast and accurate measurement of the track width. In addition, multiple screening analyses are performed in the process of obtaining the pixel width of the track shoes to improve the accuracy of the pixel width of the track shoes. The measurement method of the present invention has been tested and verified, and the maximum error of the track width measurement is 5mm, while the minimum difference in width between crawlers of different width specifications is 5cm. Therefore, the measuring method of the present invention can be used to automatically measure the width of the crawler, so as to technically find out whether the assembly specifications are wrong.

[0060] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A method for measuring the track width of an excavator based on a monocular camera, characterized in that: include: S1. Use a monocular camera to shoot the crawler of the excavator from top to bottom to collect crawler images; S2, preprocessing the crawler image to filter out noise; S3, using the trained track shoe detection model to identify the track shoe in the track shoe image, using a target frame to frame the track shoe in the track shoe image, and obtaining a target frame set ; S4, from the target frame set Filter out the target frame with the highest matching degree with the track plate , according to the target frame Get the pixel width of the track shoe ; S5, using the trained bolt detection model to detect the bolts in the target frame The bolt is identified in the framed image area and a bolt target frame is output, and the pixel width of the bolt is obtained according to the image area framed by the bolt target frame ; S6. The physical width of the known bolt is , the physical width of the track shoe is ; The target frame The screening process includes: S4.1, take the short side of the target frame as the center line and expand 30 pixels to both sides to obtain two rectangular edge detection areas; S4.

2. Identify a set of n candidate straight line segments in the edge detection area , selecting the straight line segment with the highest fitting degree from the n candidate straight line segments as the vertical edge of the track shoe; S4.3, repeat steps S4.1-S4.2 to obtain the vertical edges of the track shoes corresponding to all target frames; S4.

4. Calculate the average value of the fit of the two vertical sides of the track shoe corresponding to each target frame, and take the target frame with the highest average value as the target frame .

2. The method for measuring the track width of an excavator based on a monocular camera according to claim 1, characterized in that: In step S4.2, the process of selecting the straight line segment with the highest fitting degree includes: Calculate the length of each straight line segment , the angle between the straight line segment and the short side of the target box , and the distance between the midpoint of the straight line segment and the vertical side of the target box ; The length And the angle Straight line segments with angles > 5° are deleted from the set. Indicates the height of the edge detection area; if the straight line segment set is an empty set after deleting the straight line segments that do not meet the requirements, it means that the positioning error of the target frame is too large, and the target frame is deleted from the target frame set R; If the set of straight line segments is not an empty set after deleting the straight line segments that do not meet the requirements, the fitting degree of the remaining straight line segments in the set is calculated. , , , is the adjustment factor, with values ​​of 1, 0.8, and 0.2; The straight line segment with the highest fitting degree is used as the vertical side of the track shoe of the target frame.

3. The method for measuring the track width of an excavator based on a monocular camera according to claim 1, characterized in that: Step S5 specifically includes: Taking the four boundaries of the bolt target frame as the reference, expand outward by 10 pixels to obtain the rectangular area X; Extract six straight line segments of the bolt in the rectangular area X , calculate the angles between each of the six straight line segments ; If the angle <5°, then the two straight line segments are judged to be parallel to each other, and a total of three pairs of parallel straight line segments are obtained; Calculate the distance between the midpoints of each pair of parallel line segments , , , the corresponding pixel width of the bolt is ; The final pixel width of the bolt is the average of the pixel widths of the two bolts on the track shoe.

4. The method for measuring the track width of an excavator based on a monocular camera according to claim 1, characterized in that: The preprocessing of the crawler image includes bilateral filtering, and the formula of bilateral filtering is: ， ， in, Indicates the coordinates of the pixel currently being convolved, Represents the range pixel coordinates, function represents the original pixel value of the pixel point, and function g represents the pixel value output after filtering; Represents the value calculated by two Gaussian functions, , represents the smoothing parameter.

5. The method for measuring the track width of an excavator based on a monocular camera according to claim 1, characterized in that: The track shoe detection models are yolov8 model, Faster-RCNN model and SSD model.

6. A system for measuring the track width of an excavator based on a monocular camera, characterized in that: The method for measuring the track width of an excavator based on a monocular camera as described in any one of claims 1 to 5 is adopted, and the measuring system comprises: An acquisition module, including a monocular camera and a light source, is used to capture images of the excavator tracks from top to bottom; An image denoising module is used to denoise the collected track images; The analysis and calculation module is used to identify the track image using the trained track shoe detection model, and to frame the track shoe in the track image with a target frame to obtain a target frame set. ; From the target frame set Filter out the target frame with the highest matching degree with the track plate , according to the target frame Get the pixel width of the track shoe ; Use the trained bolt detection model to detect the bolts in the target frame The bolt is identified in the framed image area and a bolt target frame is output, and the pixel width of the bolt is obtained according to the image area framed by the bolt target frame ; The physical width of the known bolt is , the physical width of the track shoe is .

7. A computer device, characterized in that: include: processor; A memory for storing executable instructions; The processor is used to read the executable instructions from the memory and execute the executable instructions to implement the method for measuring the track width of an excavator based on a monocular camera as described in any one of claims 1 to 5.

Images