Machine vision line patrol method, device and equipment based on spot detection and medium

Abstract

The embodiment of the invention discloses a machine vision line patrol method and device based on spot detection, equipment and a medium. The method comprises the following steps: acquiring a machine vision track image, and setting a plurality of preset one-dimensional coordinate values in the machine vision track image; according to each preset one-dimensional coordinate value, carrying out image extraction on the machine vision track image to obtain machine vision track sub-images corresponding to each preset one-dimensional coordinate value; performing spot detection on each machine vision track sub-image to obtain key point coordinates of a plurality of spots; determining a real-time steering angle of a track in the machine vision track image according to the coordinates of the key points; and controlling the machine to move according to the track in the machine vision track image according to the real-time steering angle. According to the machine vision line patrol method based on spot detection, the trajectory identification calculation amount can be reduced, the identification real-time performance is improved, and the applicability is high.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a machine vision line patrol method, device, equipment and medium based on spot detection. Background Art

[0002] With the improvement of intelligent algorithms, the application scenarios of machine vision technology are becoming more and more extensive. For example, in unmanned driving, machine vision line inspection is used to control the machine to drive along a predetermined track.

[0003] In the prior art, Hough Lines Transform (HoughLines / HoughLinesP) is usually used to detect lane lines. The Hough Lines Transform algorithm has the strongest anti-interference ability in machine straight line detection. It can detect solid lines, dotted lines, and incomplete lines, and has no special requirements for the uniformity of curves. However, the Hough Transform needs to convert all points on the lane lines in the image from the spatial rectangular coordinate system to the parameter space, resulting in a very large amount of calculation. In the case of a single detection environment and insufficient computing power, using the Hough Transform algorithm for trajectory detection will result in a very large delay. Summary of the invention

[0004] The present invention provides a machine vision line patrol method, device, equipment and medium based on spot detection, so as to reduce the amount of trajectory recognition calculation and improve the real-time recognition.

[0005] According to one aspect of the present invention, a machine vision line inspection method based on spot detection is provided, the method comprising:

[0006] Acquire a machine vision trajectory image, and set a plurality of preset one-dimensional coordinate values ​​in the machine vision trajectory image;

[0007] According to each of the preset one-dimensional coordinate values, image extraction is performed on the machine vision trajectory image to obtain machine vision trajectory sub-images corresponding to each of the preset one-dimensional coordinate values;

[0008] Performing spot detection on each of the machine vision trajectory sub-images to obtain key point coordinates of a plurality of spots;

[0009] Determining the real-time steering angle of the track in the machine vision track image according to the coordinates of each of the key points;

[0010] The machine is controlled to move according to the real-time steering angle along the trajectory in the machine vision trajectory image.

[0011] According to another aspect of the present invention, a machine vision line patrol device based on spot detection is provided, the device comprising:

[0012] A preset one-dimensional coordinate value setting module is used to obtain a machine vision trajectory image and set a plurality of preset one-dimensional coordinate values ​​in the machine vision trajectory image;

[0013] An image extraction module, used to perform image extraction on the machine vision trajectory image according to each of the preset one-dimensional coordinate values, to obtain machine vision trajectory sub-images corresponding to each of the preset one-dimensional coordinate values;

[0014] A spot detection module, used to perform spot detection on each of the machine vision trajectory sub-images to obtain key point coordinates of multiple spots;

[0015] A steering angle determination module, used to determine the real-time steering angle of the track in the machine vision track image according to the coordinates of each key point;

[0016] A motion control module is used to control the machine to move along the trajectory in the machine vision trajectory image according to the real-time steering angle.

[0017] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0018] at least one processor; and

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the machine vision line patrol method based on spot detection described in any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the machine vision line patrol method based on spot detection described in any embodiment of the present invention when executed.

[0022] According to another aspect of the present invention, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the machine vision line patrol method based on spot detection according to any embodiment of the present invention is implemented.

[0023] The technical solution of the embodiment of the present invention obtains a machine vision trajectory image and sets multiple preset one-dimensional coordinate values ​​in the machine vision trajectory image; performs image extraction on the machine vision trajectory image according to each preset one-dimensional coordinate value to obtain machine vision trajectory sub-images corresponding to each preset one-dimensional coordinate value; performs spot detection on each machine vision trajectory sub-image to obtain key point coordinates of multiple spots; determines the real-time steering angle of the trajectory in the machine vision trajectory image according to each key point coordinate; controls the machine to move according to the trajectory in the machine vision trajectory image according to the real-time steering angle, thereby solving the problem of track line recognition in machine vision line patrol, reducing the amount of trajectory recognition calculation, improving recognition real-time performance, and having strong applicability.

[0024] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 is a flow chart of a machine vision line patrol method based on spot detection provided according to an embodiment of the present invention;

[0027] Figure 2 A machine vision trajectory image provided according to an embodiment of the present invention;

[0028] Figure 3 is a schematic diagram of image extraction provided according to an embodiment of the present invention;

[0029] Figure 4 is a schematic diagram of spot detection provided according to an embodiment of the present invention;

[0030] Figure 5 is a schematic diagram of key point coordinates obtained by performing spot detection on a machine vision trajectory image provided by an embodiment of the present invention;

[0031] Figure 6 is an inertial schematic diagram provided according to an embodiment of the present invention;

[0032] Figure 7 is a convexity schematic diagram provided according to an embodiment of the present invention;

[0033] Figure 8is a schematic diagram of graphic filtering attribute information provided according to an embodiment of the present invention;

[0034] Fig. 9 is a spot detection result provided by an embodiment of the present invention;

[0035] Fig.10 is a schematic diagram of arc trajectory detection based on spot detection according to an embodiment of the present invention;

[0036] Fig.11 is a flow chart of another machine vision line patrol method based on spot detection provided according to an embodiment of the present invention;

[0037] Fig.12 is a structural schematic diagram of a machine vision line patrol device based on spot detection provided according to an embodiment of the present invention;

[0038] Fig.13 It is a structural schematic diagram of an electronic device for implementing the machine vision line patrol method based on spot detection according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0040] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0041] Figure 1This is a flow chart of a machine vision line patrol method based on spot detection provided according to an embodiment of the present invention. This embodiment can be applied to track line recognition in unmanned driving to control the movement of the machine along the track line. The method can be executed by a machine vision line patrol device based on spot detection. The machine vision line patrol device based on spot detection can be implemented in the form of hardware and / or software. The machine vision line patrol device based on spot detection can be configured in electronic equipment such as a computer or a controller. Figure 1 As shown, the method includes:

[0042] Step 110: Acquire a machine vision trajectory image, and set a plurality of preset one-dimensional coordinate values ​​in the machine vision trajectory image.

[0043] The machine vision trajectory image may be a picture captured by a camera device, which includes a predetermined route of the machine. Figure 2 is a machine vision trajectory image provided according to an embodiment of the present invention. Exemplarily, the machine vision trajectory image is a frame of 640×480. Figure 2 In the image shown, the middle black part is the track that needs to be identified.

[0044] The preset one-dimensional coordinate value can be a horizontal coordinate value or a vertical coordinate value. Figure 2 For the image shown, if the horizontal coordinates corresponding to different vertical coordinates of the center line of the track can be obtained, or if the vertical coordinates corresponding to different horizontal coordinates of the center line of the track can be obtained, machine vision line patrol can be successfully implemented.

[0045] There are many ways to preset the number of one-dimensional coordinate values. For example, the number of preset one-dimensional coordinate values ​​can be directly specified. Alternatively, more preset one-dimensional coordinate values ​​can be determined for a larger trajectory curvature; fewer preset one-dimensional coordinate values ​​can be determined for a smaller trajectory curvature. Exemplarily, only two preset one-dimensional coordinate values ​​are required for a straight line to accurately identify the position and slope of the straight line. In actual machine operation, the number of preset one-dimensional coordinate values ​​can be adjusted in real time according to the trajectory curvature. The distance between each preset one-dimensional coordinate value is not limited. For example, when the trajectory curvature is large, multiple preset one-dimensional coordinate values ​​with a smaller distance can be used; when the trajectory curvature is small, multiple preset one-dimensional coordinate values ​​with a larger distance can be used. Alternatively, the distance between the preset one-dimensional coordinate values ​​can be averaged.

[0046] In an optional implementation of an embodiment of the present invention, multiple preset one-dimensional coordinate values ​​are set in the machine vision trajectory image, including: determining the number of position samples according to the trajectory curvature in the machine vision trajectory image; setting a preset one-dimensional coordinate value equal to the number of position samples in the machine vision trajectory image; wherein the preset one-dimensional coordinate value is a preset horizontal coordinate or a preset vertical coordinate.

[0047] In the embodiment of the present invention, when determining the number of position samples, there is no need for an accurate method for determining the arc of the trajectory. Therefore, there are multiple methods for determining the arc of the trajectory that can be used in the embodiment of the present invention. For example, the trajectory can be compared with a preset graphic, and the number of position samples can be determined based on the arc of the matching graphic. For another example, a machine learning model can be used to identify the arc of the trajectory of the image. The embodiment of the present invention does not make specific limitations on this.

[0048] by Figure 2 Taking the image shown as an example, three preset vertical coordinates are set. For example, the preset vertical coordinates are set to Y_0=120; Y_1=280; and Y_2=440 respectively.

[0049] Step 120 : performing image extraction on the machine vision trajectory image according to each preset one-dimensional coordinate value to obtain a machine vision trajectory sub-image corresponding to each preset one-dimensional coordinate value.

[0050] For example, Figure 2 For example, images at positions corresponding to Y_0=120; Y_1=280; and Y_2=440 can be extracted as machine vision trajectory sub-images.

[0051] Specifically, in an optional implementation of an embodiment of the present invention, image extraction is performed on the machine vision trajectory image according to each preset one-dimensional coordinate value to obtain machine vision trajectory sub-images corresponding to each preset one-dimensional coordinate value, including: determining an image extraction range with each preset one-dimensional coordinate value as the center; performing rectangular window image extraction on the machine vision trajectory image according to each extraction range to obtain machine vision trajectory sub-images corresponding to each preset one-dimensional coordinate value.

[0052] Exemplarily, the rectangular window width (Y_H) is set to 60, and the image extraction range can be determined based on the preset one-dimensional coordinate value as the center: Image extraction range = preset one-dimensional coordinate value ± rectangular window width / 2. Figure 3 FIG. 1 is a schematic diagram of image extraction according to an embodiment of the present invention. Figure 3 As shown in FIG. 1 , the image extraction ranges corresponding to Y_0=120; Y_1=280; and Y_2=440 are ordinates 90 to 150; 250 to 310; and 410 to 470. Thus, according to the image extraction range, the image extraction of the rectangular window can be obtained as follows: Figure 3 Multiple machine vision track sub-images shown.

[0053] Optionally, when performing image extraction, the outer border of the image can be removed. Figure 2When extracting the image shown in FIG. 1 , the borders (X_L) on the left and right sides can be removed. For example, 30 on each side can be removed. Figure 3 As shown, a sub-image with a horizontal coordinate ranging from 30 to 580 can be obtained.

[0054] Step 130 : Perform spot detection on each machine vision trajectory sub-image to obtain key point coordinates of multiple spots.

[0055] In an embodiment of the present invention, spot detection can be performed on each machine vision trajectory sub-image according to the coordinate information. In order to improve the efficiency and accuracy of spot detection, each machine vision trajectory sub-image can be copied to a canvas. Specifically, in an optional implementation of an embodiment of the present invention, spot detection is performed on each machine vision trajectory sub-image to obtain key point coordinates of multiple spots, including: copying each machine vision trajectory sub-image according to the corresponding preset one-dimensional coordinate value to a preset canvas of the same size as the machine vision trajectory image; spot detection is performed on the preset canvas to obtain key point coordinates of multiple spots.

[0056] In order to reduce the complexity of spot detection and improve accuracy, the preset canvas can be white. Figure 3 shown.

[0057] In the embodiment of the present invention, Figure 2 The code for extracting the image shown and copying it to the preset canvas can be as follows:

[0058] int X_L = 30;

[0059] int X_W = 640-X_L*2; / / horizontal axis range

[0060] cv::Rect roi[3]; / / Number of position samples

[0061] roi[0] = cv::Rect(X_L, Y_0-Y_H / 2, X_W, Y_H); / / Machine vision trajectory sub-image

[0062] roi[1] = cv::Rect(X_L,Y_1-Y_H / 2,X_W,Y_H); / / Machine vision trajectory sub-image

[0063] roi[2] = cv::Rect(X_L,Y_2-Y_H / 2,X_W,Y_H); / / Machine vision trajectory sub-image

[0064] cv::Mat canvasWhite(src.rows,src.cols,CV_8UC3,cv::Scalar(255,255,255)); / / Create a white preset canvas of the specified size

[0065] for(int i=0;i<3;i++){

[0066] cv::Mat regionOfInterest=src(roi[i]).clone();

[0067] regionOfInterest.copyTo(canvasWhite(roi[i])); / / Copy each machine vision trajectory sub-image to the preset canvas

[0068] }

[0069] Blob detection can use the blob feature detection algorithm (SimpleBlobDetector) provided by the OpenCV vision library. OpenCV is an open source cross-platform computer vision library that provides a large number of application programming interfaces (APIs) to achieve image recognition and processing. The SimpleBlobDetector algorithm can filter the attributes of blobs and then determine the coordinates of the key points of the blobs.

[0070] In an optional implementation of an embodiment of the present invention, spot detection is performed on a preset canvas to obtain key point coordinates of multiple spots, including: setting filtering attribute information for spot detection according to trajectory features in a machine vision trajectory image and a color of the preset canvas; constructing a spot detector according to the filtering attribute information, and using the spot detector to perform spot detection on the preset canvas to obtain key point coordinates of multiple spots.

[0071] by Figure 3 Taking the preset canvas shown as an example, the filtering attribute information may include pixel value range, spot spacing, and grayscale. According to the filtering attribute information, the constructed spot detector may be a blob, and the specific spot detector construction code may be:

[0072] cv::SimpleBlobDetector::Params pBLOBDetector;

[0073] pBLOBDetector.thresholdStep=10; / / binarization threshold step

[0074] pBLOBDetector.minThreshold = 50; / / Binarization starting threshold

[0075] pBLOBDetector.maxThreshold = 100; / / binarization termination threshold

[0076] pBLOBDetector.minRepeatability = 2; / / A spot is detected at least twice under different conditions and is considered valid

[0077] pBLOBDetector.minDistBetweenBlobs = 5; / / Minimum distance between blobs

[0078] pBLOBDetector.filterByColor = true; / / Spot color limit variable, true means support color filtering

[0079] pBLOBDetector.blobColor = 0; / / The grayscale is 0, that is, the black spots are detected

[0080] pBLOBDetector.filterByArea = false; / / Spot area limit variable, false means not passing area filtering

[0081] pBLOBDetector.filterByCircularity = false; / / Blob circularity limit variable, false means not passing circularity filtering

[0082] pBLOBDetector.filterByInertia = false; / / Blob inertia limit variable, false means not passing inertia filtering

[0083] pBLOBDetector.filterByConvexity = false; / / Blob convexity limit variable, false means not passing convexity filtering

[0084] blob->detect(canvasWhite,key_points); / / Key points

[0085] cv::drawKeypoints(canvasWhite,key_points,outImg,cv::Scalar(0,0,255)); / / Mark the key points with red circles

[0086] Figure 4 FIG. 1 is a schematic diagram of spot detection according to an embodiment of the present invention. Figure 4 As shown in the figure, the red circle is Figure 3Perform spot detection to obtain multiple key points. In the embodiment of the present invention, the horizontal and vertical coordinates of the key points can be obtained by the function KeyPoint::convert() To convert. The code is as follows:

[0087] std::vector <cv::point2f>points2f; / / Point2f represents a point containing floating point coordinates

[0088] cv::KeyPoint::convert(key_points,points2f); / / Determine the coordinates of each key point

[0089] Figure 5 Schematic diagram of key point coordinates obtained by performing spot detection on a machine vision trajectory image according to an embodiment of the present invention. Figure 5 As shown in the figure, the ordinate of the center point of the spot (i.e. the key point) may have a slight deviation from the previously calibrated ordinate value. However, in the end, as long as the lateral displacement of different distances is obtained, the steering angle of the machine can be controlled.

[0090] Without loss of generality, when performing spot detection, optionally, filtering attribute information includes at least one of the following: pixel value range, spot spacing, area, grayscale, roundness, inertia, and convexity.

[0091] Among them, pixel value range, spot spacing, and area are common concepts. Grayscale (Thresholds) or binarization, which ranges from 0 to 255, represents the brightness distribution of pixel points. Among them, 0 represents the darkest state and 255 represents the brightest state. Circularity is a measure of the degree to which the spot is close to a circle, and the value range is 0 to 1. The circularity of a circle is 1, the circularity of a square is 0.785, and so on. The formula for calculating circularity is 4π×Area / (circumference×circumference), where Area is the area. Inertia is a measure of the elongation of a shape. Simply put, for a circle, the inertia value is 1; the inertia value of an ellipse is between 0 and 1; for a straight line, the inertia is 0. Figure 6 is an inertial schematic diagram provided according to an embodiment of the present invention. Figure 6 As shown, the ellipse on the left has high inertia, while the ellipse on the right has low inertia. Convexity is defined as the area of ​​the spot / the area of ​​the convex hull of the spot. The convex hull of a shape is the tightest convex shape that completely encloses the shape. To filter by convexity, you can set filterByConvexity = 1, and then you can set 0≤minConvexity≤1 and maxConvexity≤1. Figure 7 is a convexity schematic diagram provided according to an embodiment of the present invention. Figure 7 As shown, the convexity of the left graph is high, and the convexity of the right graph is low.

[0092] Figure 8 FIG. 1 is a schematic diagram of a graphic filtering attribute information provided according to an embodiment of the present invention. Figure 8 The example code for spot detection of the image shown is as follows:

[0093] #include<opencv2 / core.hpp> / / Import dependencies

[0094] cv::Mat src;

[0095] src = cv::imread("blob.png"); / / Read the image

[0096] cv::SimpleBlobDetector::Params pBLOBDetector; / / Create a parameter object pBLOBDetector

[0097] / / Set filter attribute information and range

[0098] pBLOBDetector.thresholdStep=10; / / Detection threshold step,

[0099] pBLOBDetector.minThreshold = 50; / / Set the minimum grayscale or color value for BLOB detection

[0100] pBLOBDetector.maxThreshold = 100; / / Set the maximum value of grayscale or color for BLOB detection

[0101] pBLOBDetector.minRepeatability = 2; / / The minimum number of times a BLOB is considered valid under different conditions

[0102] pBLOBDetector.minDistBetweenBlobs = 5; / / Minimum distance between two BLOBs

[0103] pBLOBDetector.filterByColor = true; / / Enable color filtering

[0104] pBLOBDetector.blobColor = 0; / / Detect black Blob color

[0105] pBLOBDetector.filterByArea = true; / / Enable area filtering

[0106] pBLOBDetector.minArea = 1500; / / minimum area filter value

[0107] pBLOBDetector.maxArea=6000; / / Maximum area filter value

[0108] pBLOBDetector.filterByCircularity = true; / / Enable circularity filtering value

[0109] pBLOBDetector.minCircularity = 0.8f; / / minimum circularity filter value

[0110] pBLOBDetector.maxCircularity=std::numeric_limits <float>::max(); / / Maximum roundness is unlimited

[0111] pBLOBDetector.filterByInertia = true; / / Enable inertia filtering

[0112] pBLOBDetector.minInertiaRatio = 0.1f; / / minimum value of inertia filtering

[0113] pBLOBDetector.maxInertiaRatio=std::numeric_limits <float>::max(); / / Maximum inertia is unlimited

[0114] pBLOBDetector.filterByConvexity = true; / / Enable convexity filtering

[0115] pBLOBDetector.minConvexity = 0.95f; / / minimum convexity filter value

[0116] pBLOBDetector.maxConvexity=std::numeric_limits <float>::max(); / / Unlimited maximum convexity

[0117] cv::Ptr <cv::simpleblobdetector>blob = cv::SimpleBlobDetector::create(pBLOBDetector); / / Call the function cv::SimpleBlob Detector::create() as an input parameter to create the detector object blob

[0118] std::vector <cv::keypoint>key_points;

[0119] blob->detect(canvasWhite,key_points); / / Call the function blob->detect() to detect, find the detected spots that meet the range set by the parameter pBLOBDetector, and save them to the key point set key_points

[0120] cv::Mat outImg;

[0121] cv::drawKeypoints(src,key_points,outImg,cv::Scalar(0,0,255)); / / Mark the key point positions of the spots that meet the conditions with red circles on the original image

[0122] imwrite("dst.png",outImg);

[0123] A key point can be a set of information about a valid spot area obtained through computer vision feature detection and description algorithms such as Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), or OrientedFAST and Rotated BRIEF (ORB). Its structure is defined by the class cv::KeyPoint, which contains information such as the location and size of the spot.

[0124] In specific application scenarios, you can refer to the above code to flexibly adjust the filter attribute information to obtain spot detection results that meet the requirements. Figure 8 The image shown is used as the image to be detected, and the above code is used for blob detection. When performing blob detection, the filter properties filterByColor, filterByArea, filterByCircularity, filterByInertia, and filterByConvexity are all set to true. Blob filtering is performed based on the range of these properties, and the KeyPoint of the blob that meets the conditions is marked with a red circle. Fig. 9 is a spot detection result provided according to an embodiment of the present invention.

[0125] In the embodiment of the present invention, image extraction may be performed on the machine vision trajectory image according to a plurality of preset one-dimensional coordinate values, and then spot detection may be performed on the machine vision trajectory sub-image to obtain key point coordinates of a plurality of spots. Fig.10 FIG. 1 is a schematic diagram of arc trajectory detection based on spot detection according to an embodiment of the present invention. Fig.10 As shown, the left side of the figure is the original image. After image extraction, the above code is used for spot detection, and the detection result shown on the right side can be obtained.

[0126] After spot detection and obtaining the coordinates of the key points of the spots, the machine operation can be controlled according to the coordinates of each key point.

[0127] Step 140: Determine the real-time steering angle of the trajectory in the machine vision trajectory image according to the coordinates of each key point.

[0128] After determining the coordinates of the key points, the machine can be controlled to run along the curve where the key points are located. The determination of the trajectory from the key point coordinates can be realized through a variety of mathematical algorithms. For example, the trajectory function is calculated based on the coordinate points, or the trajectory function is determined by curve fitting based on the coordinate points. According to the trajectory function, the steering angle of the machine can be adjusted in real time to make the machine run along the predetermined trajectory.

[0129] In an optional implementation of an embodiment of the present invention, the real-time steering angle of the trajectory in the machine vision trajectory image is determined according to the coordinates of each key point, including: performing curve fitting according to the coordinates of each key point to obtain the trajectory function of the trajectory in the machine vision trajectory image; determining the real-time tangent direction according to the trajectory function, and determining the real-time steering angle of the trajectory according to the tangent direction.

[0130] There are many ways of curve fitting, such as polynomial fitting, spline interpolation, Bezier curve or other curve fitting methods. The tangent direction of the current machine position can be calculated based on the trajectory function, and the real-time steering angle can be obtained based on the deviation between the tangent direction and the current machine running direction. The tangent direction can be obtained by calculating the derivative of the trajectory function at the location.

[0131] Step 150: Control the machine to move along the trajectory in the machine vision trajectory image according to the real-time steering angle.

[0132] The technical solution of this embodiment is to obtain a machine vision trajectory image and set multiple preset one-dimensional coordinate values ​​in the machine vision trajectory image; perform image extraction on the machine vision trajectory image according to each preset one-dimensional coordinate value to obtain machine vision trajectory sub-images corresponding to each preset one-dimensional coordinate value; perform spot detection on each machine vision trajectory sub-image to obtain key point coordinates of multiple spots; determine the real-time steering angle of the trajectory in the machine vision trajectory image according to each key point coordinate; control the machine to move according to the trajectory in the machine vision trajectory image according to the real-time steering angle, thereby solving the problem of track line recognition in machine vision line patrol, reducing the amount of trajectory recognition calculation, improving recognition real-time performance, being suitable for situations where CPU performance is low, and having strong applicability.

[0133] Fig.11 FIG. 1 is a flow chart of another machine vision line inspection method based on spot detection according to an embodiment of the present invention. Fig.11 As shown, an application process of the machine vision line patrol method based on spot detection provided by an embodiment of the present invention is: obtain a machine vision trajectory image, extract n ordinate value positions, and a machine vision trajectory sub-image of a rectangular window of a specific width according to specific needs; establish a white canvas of the same size as the machine vision trajectory image, and copy each machine vision trajectory sub-image to the white canvas. Construct a spot detector, and set the filtering attribute information of the spot detector according to the specific scene; use the spot detector to detect, obtain the spot position, and determine the key point coordinates; then obtain the real-time steering angle according to the key point coordinates to control the operation of the machine. By using the spot detection method for machine vision line patrol, the amount of trajectory recognition calculation can be reduced, the real-time recognition can be improved, it is suitable for situations with low CPU performance, and has strong applicability.

[0134] Fig.12 FIG. 1 is a schematic diagram of a machine vision line patrol device based on spot detection according to an embodiment of the present invention. Fig.12 As shown, the device includes: a preset one-dimensional coordinate value setting module 1210, an image extraction module 1220, a spot detection module 1230, a steering angle determination module 1240 and a motion control module 1250. Among them:

[0135] A preset one-dimensional coordinate value setting module 1210 is used to obtain a machine vision trajectory image and set a plurality of preset one-dimensional coordinate values ​​in the machine vision trajectory image;

[0136] An image extraction module 1220 is used to extract the machine vision trajectory image according to each preset one-dimensional coordinate value to obtain a machine vision trajectory sub-image corresponding to each preset one-dimensional coordinate value;

[0137] The spot detection module 1230 is used to perform spot detection on each machine vision track sub-image to obtain key point coordinates of multiple spots;

[0138] A steering angle determination module 1240 is used to determine the real-time steering angle of the track in the machine vision track image according to the coordinates of each key point;

[0139] The motion control module 1250 is used to control the machine to move along the trajectory in the machine vision trajectory image according to the real-time steering angle.

[0140] Optionally, the preset one-dimensional coordinate value setting module 1210 includes:

[0141] A position sampling number determination unit, used to determine the position sampling number according to the trajectory arc in the machine vision trajectory image;

[0142] A preset one-dimensional coordinate value setting unit, used to set a preset one-dimensional coordinate value equal to the number of position sampling in the machine vision trajectory image;

[0143] The preset one-dimensional coordinate value is a preset horizontal coordinate or a preset vertical coordinate.

[0144] Optionally, the spot detection module 1230 includes:

[0145] An image copying unit, used to copy each machine vision track sub-image to a preset canvas of the same size as the machine vision track image according to the corresponding preset one-dimensional coordinate value;

[0146] The spot detection unit is used to perform spot detection on a preset canvas to obtain key point coordinates of multiple spots.

[0147] Optional, spot detection unit, including:

[0148] The filter attribute information setting subunit is used to set the filter attribute information of spot detection according to the track features in the machine vision track image and the color of the preset canvas;

[0149] The spot detection subunit is used to construct a spot detector according to the filtering attribute information, and use the spot detector to perform spot detection on the preset canvas to obtain the key point coordinates of multiple spots.

[0150] Optionally, the filtering attribute information includes at least one of the following: pixel value range, spot spacing, area, grayscale, roundness, inertia, and convexity.

[0151] Optionally, the steering angle determination module 1240 includes:

[0152] A curve fitting unit, used for performing curve fitting according to the coordinates of each key point to obtain a trajectory function of the trajectory in the machine vision trajectory image;

[0153] The steering angle determination unit is used to determine the real-time tangent direction according to the trajectory function, and determine the real-time steering angle of the trajectory according to the tangent direction.

[0154] Optionally, the image extraction module 1220 includes:

[0155] An extraction range determination unit, used to determine the image extraction range based on each preset one-dimensional coordinate value as the center;

[0156] The image extraction unit is used to perform rectangular window image extraction on the machine vision trajectory image according to each extraction range to obtain machine vision trajectory sub-images corresponding to each preset one-dimensional coordinate value.

[0157] The machine vision line patrol device based on spot detection provided in the embodiment of the present invention can execute the machine vision line patrol method based on spot detection provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0158] Fig.13 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0159] like Fig.13 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0160] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0161] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a machine vision line patrol method based on spot detection.

[0162] In some embodiments, the machine vision line patrol method based on spot detection can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the machine vision line patrol method based on spot detection described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the machine vision line patrol method based on spot detection in any other appropriate manner (for example, by means of firmware).

[0163] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0164] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0165] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0166] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0167] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0168] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0169] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0170] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.< / cv::keypoint> < / cv::simpleblobdetector> < / float> < / float> < / float>

Claims

1. A machine vision line inspection method based on spot detection, characterized in that: include: Acquire a machine vision trajectory image, and set a plurality of preset one-dimensional coordinate values ​​in the machine vision trajectory image; According to each of the preset one-dimensional coordinate values, image extraction is performed on the machine vision trajectory image to obtain machine vision trajectory sub-images corresponding to each of the preset one-dimensional coordinate values; Performing spot detection on each of the machine vision trajectory sub-images to obtain key point coordinates of a plurality of spots; Determining the real-time steering angle of the track in the machine vision track image according to the coordinates of each of the key points; The machine is controlled to move according to the real-time steering angle along the trajectory in the machine vision trajectory image.

2. The method according to claim 1, characterized in that In the machine vision trajectory image, a plurality of preset one-dimensional coordinate values ​​are set, including: Determining the number of position samplings according to the trajectory arc in the machine vision trajectory image; In the machine vision trajectory image, setting a preset one-dimensional coordinate value equal to the number of position samplings; The preset one-dimensional coordinate value is a preset horizontal coordinate or a preset vertical coordinate.

3. The method according to claim 1, characterized in that Perform spot detection on each of the machine vision trajectory sub-images to obtain key point coordinates of multiple spots, including: Copying each of the machine vision trajectory sub-images to a preset canvas of the same size as the machine vision trajectory image according to the corresponding preset one-dimensional coordinate value; Spot detection is performed on the preset canvas to obtain key point coordinates of a plurality of spots.

4. The method according to claim 3, characterized in that Perform spot detection on the preset canvas to obtain key point coordinates of multiple spots, including: According to the trajectory features in the machine vision trajectory image and the color of the preset canvas, setting the filtering attribute information of the spot detection; A spot detector is constructed according to the filtering attribute information, and the spot detector is used to perform spot detection on the preset canvas to obtain key point coordinates of a plurality of spots.

5. The method according to claim 4, characterized in that The filtering attribute information includes at least one of the following: pixel value range, spot spacing, area, grayscale, roundness, inertia, and convexity.

6. The method according to claim 1, characterized in that Determining the real-time steering angle of the track in the machine vision track image according to the coordinates of each of the key points includes: Performing curve fitting according to the coordinates of each of the key points to obtain a trajectory function of the trajectory in the machine vision trajectory image; The real-time tangent direction is determined according to the trajectory function, and the real-time steering angle of the trajectory is determined according to the tangent direction.

7. The method according to claim 1, characterized in that According to each of the preset one-dimensional coordinate values, image extraction is performed on the machine vision trajectory image to obtain machine vision trajectory sub-images corresponding to each of the preset one-dimensional coordinate values, including: Determine the image extraction range with each of the preset one-dimensional coordinate values ​​as the center respectively; According to each of the extraction ranges, rectangular window image extraction is performed on the machine vision trajectory image to obtain machine vision trajectory sub-images corresponding to each of the preset one-dimensional coordinate values.

8. A machine vision line patrol device based on spot detection, characterized in that: include: A preset one-dimensional coordinate value setting module is used to obtain a machine vision trajectory image and set a plurality of preset one-dimensional coordinate values ​​in the machine vision trajectory image; An image extraction module, used to perform image extraction on the machine vision trajectory image according to each of the preset one-dimensional coordinate values, to obtain machine vision trajectory sub-images corresponding to each of the preset one-dimensional coordinate values; A spot detection module, used to perform spot detection on each of the machine vision trajectory sub-images to obtain key point coordinates of multiple spots; A steering angle determination module, used to determine the real-time steering angle of the track in the machine vision track image according to the coordinates of each key point; A motion control module is used to control the machine to move along the trajectory in the machine vision trajectory image according to the real-time steering angle.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the machine vision line patrol method based on spot detection according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the machine vision line patrol method based on spot detection according to any one of claims 1 to 7 when executed.

Images