Corner positioning method and device based on neural network and detection algorithm
By combining neural network and edge detection algorithm, the precise positioning of the edge corners of the steel plate is achieved, solving the problems of position deviation and instability in the edge detection during the injection coding process, and improving the accuracy and stability of the injection coding.
Patent Information
- Application Number
- CN202210059925.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-19
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-01-19
AI Technical Summary
In the prior art, the steel plates have deviations in the position angle deviation during the injection coding process, and the traditional edge and angle detection algorithms are not stable enough and are easily affected by the environment. Deep learning methods are also prone to edges. Point false detection.
Using edge and corner positioning method based on neural network and detection algorithm, by collecting sample images, building training sample sets, building neural network models, input real-time images to obtain edge and corner data, and corrected and fused with edge detection algorithms, outputting the corrected edges The angle data adjusts the coded position adaptively.
It improves the accuracy and stability of edge point detection, reduces environmental interference, ensures the accuracy of the injection code position, and avoids leakage and mis-spraying.
Smart Images

Figure CN114463751B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the field of control algorithms, and in particular to an algorithm for positioning the corners of a workpiece. Background Art
[0002] The main frames of engineering machinery such as vehicles and ships are welded from various workpieces, and these workpieces are mainly cut from steel plates. Usually, the workpieces cut from steel plates come in a variety of sizes and shapes, and the processes and uses of different workpieces are different, so it is necessary to spray code and mark various workpieces. Generally, mechanical spraying is used, which requires first locating the spraying position, and then controlling the spraying device to the designated spraying point to start the spraying process. The efficiency is faster than the manual method, but the steel plate may have a position angle deviation when it reaches the spraying point, which may cause the spraying position to deviate, or even cause missed spraying or wrong spraying.
[0003] Therefore, how to determine the real-time position of the steel plate and whether there is an angle deviation (corner data) to adaptively adjust the coding position is a key step in the mechanical coding process of the workpiece. In the prior art, due to the possibility of certain patterns and scratches on the steel plate, the traditional corner detection algorithm is not stable enough and is easily affected by the environment; deep learning methods are also prone to false detection of edge points. Therefore, how to provide a high-precision edge point detection algorithm to determine the real-time posture of the steel plate to adaptively adjust the coding position is an important technical problem that needs to be solved in the mechanical coding of workpieces. Summary of the invention
[0004] In view of the above technical problems, the present invention provides a corner positioning method based on a neural network and a detection algorithm, comprising:
[0005] S21: Collect sample images of the subject to be detected, and annotate corner data of the sample images to construct a training sample set;
[0006] S22: Build a neural network model for learning edge data;
[0007] S23: inputting the training sample set into the neural network model to obtain a trained neural network model;
[0008] S24: collecting a real-time image of the subject to be detected;
[0009] S25: inputting the real-time image into the trained neural network model and outputting the first corner data;
[0010] S26: According to the first corner data, intercepting a region image including the first corner data in the real-time image;
[0011] S27: Detect the area image to obtain second corner data;
[0012] S28: Correcting and fusing the first corner data with the second corner data to obtain corrected corner data as the corner positioning output result.
[0013] Further, the first corner data includes a first corner point; the second corner data includes a second corner point;
[0014] Step S27 includes:
[0015] S271: Input the region image into the edge detection algorithm model to obtain edge segments in the region image;
[0016] S272: Determine pre-selected corner points in the regional image according to the edge line segments;
[0017] S273: Determine whether there is a pre-selected corner point among the pre-selected corner points whose position difference with the first corner point is within a first set range;
[0018] S274: If it exists, returning the pre-selected corner point whose position difference is within the first set range as the second corner point;
[0019] S275: If it does not exist, the second corner point is not returned.
[0020] Further, step S28, specifically, if the second corner point is returned, the cluster center point of the first corner point and the second corner point is calculated as the corrected edge and corner data; if the second corner point is not returned, the first corner point is used as the corrected edge and corner data.
[0021] Furthermore, the first corner data further includes a first corner point; the second corner data further includes a second corner point; and step S27 further includes:
[0022] S276: Determine, according to the edge line segment, a point in the edge line segment that is closest to the first edge point as a pre-selected edge point;
[0023] S277: Determine whether the position difference between the pre-selected edge point and the first edge point is within a second set range;
[0024] S278: If yes, returning the pre-selected edge point whose position difference is within the second set range as the second edge point;
[0025] S279: If not, the second edge point is not returned.
[0026] Furthermore, step S28 specifically calculates the cluster center point of the first edge point and the second edge point as the corrected edge angle data.
[0027] Furthermore, step S271: further includes: calculating the edge area enclosed by the edge points in the regional image;
[0028] Step S272 includes:
[0029] S2721: Find the image center moment corresponding to the maximum edge area;
[0030] S2722: Calculate the distance between the image center moment and the edge line segment, merge the edges within a certain distance range, and eliminate discontinuous edges;
[0031] S2723: Determine pre-selected corner points within the region image based on edge line segments that eliminate discontinuous edges.
[0032] Furthermore, it also includes:
[0033] S29: Outputting the tilt angle according to the corrected edge angle data.
[0034] Further, step S25 includes:
[0035] S251: input the real-time image into the trained neural network model;
[0036] S252: Determine whether the first corner data is obtained;
[0037] S253: If yes, output the first corner data of the workpiece to be inspected;
[0038] S254: If not, enhance the real-time image, input the trained neural network model, and output the first corner data.
[0039] Furthermore, the method for enhancing the real-time image includes: brightness enhancement, HSV color space enhancement, up / down / horizontal inversion enhancement, and Gaussian Laplace enhancement.
[0040] On the other hand, the present invention further provides a corner positioning device based on a neural network and a detection algorithm, characterized in that the corner positioning device is used to implement any of the above corner positioning methods, including:
[0041] The sample collection module is used to collect sample images and annotate the edge and corner data of the sample images to construct a training sample set;
[0042] A model building module for building a neural network model for learning edge data;
[0043] A model training module is used to input the training sample set into the neural network model to obtain a trained neural network model;
[0044] An image acquisition module, used to acquire real-time images of the subject to be detected;
[0045] A first corner data output module, used to input the real-time image into the trained neural network model and output the first corner data of the workpiece to be detected;
[0046] An image capture module, configured to capture a region image including the first corner data in the real-time image according to the first corner data;
[0047] A second corner data output module is used to detect the regional image and obtain the second corner data of the workpiece to be detected;
[0048] The correction module is used to correct and fuse the second corner data with the first corner data to obtain the corrected corner data as the corner positioning output result.
[0049] The present invention provides a corner positioning method and device based on a neural network and a detection algorithm, which integrates the advantages of a neural network model and a traditional edge detection algorithm. First, the first corner data of a real-time image is obtained through the neural network model, and then a local image in the real-time image is intercepted according to the first corner data to obtain the second corner data of the real-time image through an edge detection algorithm. The two are corrected and fused to obtain corrected corner data, which can optionally include but is not limited to edge point data, corner point data, tilt angle, etc. Compared with a single neural network model or edge detection algorithm, the method and device have higher stability and accuracy and are less susceptible to environmental interference. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 A distribution diagram of parts on a steel plate according to an example of the present invention;
[0051] Figure 2 A flow chart of an embodiment of a method for correcting workpiece coding according to the present invention;
[0052] Figure 3 It is a schematic diagram of coding a workpiece to be coded on a non-offset steel plate according to an example of the present invention;
[0053] Figure 4 It is a schematic diagram of coding a workpiece to be coded on a deflected steel plate according to an example of the present invention;
[0054] Figure 5 The coding point map of the workpiece to be coded on the steel plate of the present invention;
[0055] Figure 6 A flowchart of an embodiment of step S1 of the workpiece coding correction method of the present invention;
[0056] Figure 7 It is a schematic diagram of the self-coding information of the workpiece to be coded according to the present invention;
[0057] Figure 8 This is a flowchart of step S11 of the workpiece coding correction method provided by the present invention, that is, an embodiment of the workpiece coding recognition method;
[0058] Fig. 9It is a flow chart of an embodiment of step S111 of the workpiece coding recognition method of the present invention;
[0059] Fig.10 It is a flow chart of an embodiment of step S114 of the workpiece coding recognition method of the present invention;
[0060] Fig.11 This is a flowchart of step S2 of the workpiece coding correction method of the present invention, that is, an embodiment of the corner positioning method based on a neural network and a detection algorithm;
[0061] Fig.12 A structural block diagram of an embodiment of a neural network model of the present invention;
[0062] Figure 13-14 A schematic diagram of a rectangular frame marking corner points in the corner positioning method of the present invention;
[0063] Figure 15-16 A schematic diagram of a rectangular frame marking edge points in the edge and corner positioning method of the present invention;
[0064] Fig.17 This is a flowchart of an embodiment of step S23 in the corner positioning method of the present invention;
[0065] Fig.18 This is a flow chart of an embodiment of step S25 in the corner positioning method of the present invention;
[0066] Fig.19 This is a flowchart of an embodiment of step S27 in the corner positioning method of the present invention;
[0067] Fig. 20 This is a flow chart of an embodiment of step S272 in the corner positioning method of the present invention;
[0068] Fig.21 A flowchart of another embodiment of the corner positioning method of the present invention;
[0069] Fig. 22 A flowchart of an embodiment of the coding path control method of the present invention;
[0070] Fig.23 A schematic diagram of a coding path under the coding path control method of the present invention;
[0071] Fig.24 A structural block diagram of an embodiment of a system for correcting workpiece coding according to the present invention;
[0072] Fig.25 It is a structural block diagram of an embodiment of a first acquisition device of a correction system for workpiece coding of the present invention;
[0073] Fig.26 A structural block diagram of an embodiment of a workpiece coding recognition device of the present invention;
[0074] Fig. 27 is a structural block diagram of an embodiment of the corner positioning device of the present invention;
[0075] Fig.28 It is a structural block diagram of an embodiment of the coding path planning device of the present invention;
[0076] Fig.29 It is a schematic structural diagram of an embodiment of the overall coding system of the present invention. DETAILED DESCRIPTION
[0077] To explain the technical idea of the present invention in detail, Figure 1 As shown, taking the coding main body as a steel plate and the workpiece to be coded as the parts scattered on the steel plate as an example, in the prior art, the steel plate is usually cut into each part before or after, and the coding mark is sprayed according to the type of the parts to distinguish different parts, and then assembled into engineering machinery such as ships and vehicles. It is worth noting that the above is only an example description and is not specifically limited. The coding main body can be selected but not limited to the main materials such as steel plates and alloy parts, and the workpiece to be coded can be selected but not limited to scattered single or multiple types. To illustrate other application scopes of the present invention, the coding main body can also be selected but not limited to a plastic plate, on which parts for assembling children's toys are scattered, and the names and numbers of different parts are distinguished by coding marks, so as to facilitate the subsequent assembly into a complete children's toy; the coding main body can also be selected but not limited to cloth, on which there are scattered parts for assembling a whole piece of clothing, and partial puzzles for assembling a whole picture. It can be understood by those skilled in the art that the examples of using coding to identify scattered parts to assemble into a whole piece are all within the scope of protection of the present invention.
[0078] like Figure 2-4 As shown, the present invention provides a method for correcting workpiece coding, comprising:
[0079] S1: Get the initial information of the workpiece to be coded on the coding body (such as Figure 3For example, the rectangular outer frame in the figure is the coding body, such as a steel plate, and its shape and size can be set arbitrarily; the pentagonal pattern in the figure is the workpiece to be coded, and its shape and size can be set arbitrarily. There are other workpieces to be coded scattered on the steel plate, which are not shown here; the dotted box in the figure is the coding area, and the solid circle is the coding center. This is the initial coding information that needs to be obtained in our example. It will be pointed out later with the example of the center of gravity position. It can be the center point / starting point / end point of the coding; the coding area can be a circle or rectangular box made by the center of gravity position. Specifically, the initial coding information can be selected but not limited to the coding coordinates of the workpiece to be coded on the coding body (the coding center / starting point / end point example can be used), the coding angle, etc., that is, assuming that the corner point of the steel plate is at the origin (0, 0) and the angle is 0 (under the ideal condition that the steel plate is placed without any deviation), the position where each workpiece to be coded should be coded (assuming that the steel plate has no deviation, the initial determined position and angle). For example, the initial coding information of the Nth workpiece can be selected but not limited to the coordinates (x n ,y n ), angle r n It means that (x n ,y n , r n ).
[0080] S2: Get the position information of the coding subject (such as Figure 4 Specifically, the position information may be expressed as corner point coordinates, offset angles, etc. for the coding subject, but is not limited to the corner point coordinates, offset angles, etc. for the coding subject. For example, when the coding subject is a steel plate, that is, when the steel plate is transported to the coding point by a mobile pallet, its corner point may not be located at the origin (0, 0), and the angle is not 0, but there is a displacement deviation and angle deviation, which may be expressed as corner point coordinates (x c ,y c ), angle r c It means that (x c ,y c , r c More preferably, as shown in the specific embodiment of S2 below, the corner point coordinates (x c ,y c ) and edge point coordinates (M c , N c ) Calculate its offset angle r c .
[0081] S3: According to the initial coding information of the workpiece to be coded on the coding body (the coding information determined under ideal conditions without deviation of the steel plate) and the position information of the coding body (real-time information of the steel plate, deviations in position and angle, etc.), determine the coding correction information (corrected coding position angle, etc.) of the workpiece to be coded on the coding body. That is, according to the position information of the coding body (optional but not limited to displacement deviation and angle deviation), correct the initially determined coding information to obtain the corrected coding information (coding correction information), that is, the actual position and angle of the current coding point that needs to be coded, expressed as (x n1 ,y n1 , r n1 ),(like Figure 4 shown).
[0082] In this embodiment, a specific embodiment of the calibration method for workpiece coding of the present invention is given, which fully considers that the position information of the coding subject (optionally but not limited to the steel plate) when arriving at the coding point may have deviations in position, posture, angle, etc. (such as Figure 4 As shown in the figure, it is not an ideal case where a square is placed under the coding point. Its corner point is not the origin and the angle is not zero. By detecting the position information of the coding body itself, the initially determined coding position is corrected to obtain the actual position where the workpiece to be coded should be coded, ensuring that the coding position and angle are accurate. It is worth noting that during the correction process, it is actually necessary to correct the initial coding information of N workpieces to be coded in turn according to the number of workpieces to be coded on the coding body, determine the coding correction information of each workpiece to be coded, and obtain the following: Figure 5 The inkjet dot diagram shown (compared to Figure 1 The added points are the coding correction information for each workpiece to be coded, that is, the actual coding point position).
[0083] Specifically, Figure 6 As shown, step S1 may optionally include but is not limited to:
[0084] S11: Determine the coding information of the workpiece to be coded (such as Figure 7 As shown), that is, regardless of the position of the workpiece to be coded in the coding body, the code should be marked on the workpiece itself according to its shape, size, etc., which can be selected but not limited to the coordinates (x na ,y na )express.
[0085] S12: Determine the placement information of the workpiece to be coded on the coding body (such as Figure 3As shown, it can be optionally but not limited to the coordinates of a key point such as a corner point or center point of the workpiece to be coded). Specifically, the placement position of the workpiece to be coded on the coding body can be optionally but not limited to the coordinates (x ) of the feature points of the workpiece to be coded (such as the upper, lower, left, right, center point of the workpiece, etc., the example in the figure is a corner point of the workpiece). nb ,y nb ) is indicated. More specifically, the placement position may be optionally, but not limited to, parsing and extracting material data (optionally, but not limited to, information data including the number and name of each workpiece) and line data (optionally, but not limited to, graphic data including the shape, size, and line segment composition of each workpiece) from the production nesting diagram (such as dxf, json, etc.) through a data parsing module; then the material data is stored as file data in the form of a file, and the line data is combined and converted into the frame data of the steel plate cutting and the workpiece data is stored as picture data in the form of a picture; finally, the file data and the picture data are matched one by one to determine the position, shape, size, etc. of each workpiece on the coding body (steel plate).
[0086] S13: According to the self-printing information of the workpiece to be printed (optional but not limited to using coordinates (x na ,y na ) indicates the location of the workpiece to be coded on the coding body (x nb ,y nb ), determine the initial information of the workpiece to be coded on the coding body (x n ,y n ). Angle r n An angle may be set by a person skilled in the art based on the appearance, etc. Generally, for easy viewing, it is generally set to be parallel to the workpiece, that is, the angle is 0.
[0087] In this embodiment, a specific embodiment of how to obtain the initial information of coding is given in step S1. The initial information of coding is determined by obtaining the coding information of the workpiece to be coded and the placement information of the workpiece to be coded on the coding body, that is, the coordinate conversion from the small coordinate system of the workpiece to be coded to the large coordinate system of the coding body is completed. The structure is simple and the calculation is accurate, which can ensure the accuracy of the coding position.
[0088] Specifically, in order to accurately identify and find the self-printing information of the workpiece to be printed (such as Figure 7 As shown, any information representing the coding position such as the coding area (dashed box) / coding starting point / end point / center point (solid circle), step S11, as another inventive concept of the present invention, also provides a method for identifying the coding of a workpiece. Specifically, step S11 may be optional but not limited to determining the coding information of the workpiece to be coded according to the center of gravity of the workpiece to be coded. More specifically, as Figure 8 As shown, S11 may include but is not limited to:
[0089] S111: Calculate the center of gravity position of the whole workpiece and the local workpiece of the workpiece to be coded. Specifically, the center of gravity position can be calculated based on, but not limited to, the binary image of the workpiece to be coded. Specifically, it can be extracted from, but not limited to, the file data and image data obtained from the production nesting map to form the contour data of each workpiece, and then the contour of the workpiece is filled to obtain the binary image of each workpiece.
[0090] S112: Determine whether the center of gravity position is a solid area of the workpiece to be coded; specifically, taking the partial workpiece including 4 quadrants as an example, it is necessary to determine whether 5 center of gravity positions (1 center of gravity position of the whole workpiece + 4 center of gravity positions of the partial workpiece) are solid areas. More specifically, it is optional but not limited to determine whether each center of gravity position (coordinate) calculated from the file data and image data obtained from the production nesting diagram is a solid area of the workpiece to be coded.
[0091] S113: If not, then delete the center of gravity position that is not a solid area; specifically, if the center of gravity position is not a solid area, it indicates that the center of gravity position is not on the workpiece to be coded (such as the center of gravity position of a ring is within the center circle), and the hollow area cannot be used as a pre-selected position for the coding area and needs to be deleted. Specifically, it is optional but not limited to removing the calculated center of gravity positions from the storage sequence.
[0092] S114: If yes, then the center of gravity position of the solid area is used as the key point to determine the self-coding information of the workpiece to be coded. Specifically, the self-coding information may be selected but not limited to any information that characterizes the position to be coded, such as the center of gravity position of the solid area as the starting point / end point / center point of the area to be coded. More specifically, since multiple center of gravity positions are calculated in step S111, there may be multiple center of gravity positions in the solid area in step S114. At this time, those skilled in the art can understand that one of them is selected as the coding information. The following is an improved preferred solution.
[0093] In this embodiment, a method for identifying workpiece coding of the present invention is given. By calculating the center of gravity positions of the overall workpiece and the local workpiece to be coded, the center of gravity position in the solid area is selected as the key point. Compared with the prior art, it can avoid the situation where the overall center of gravity position of certain workpieces (for example, hollow workpieces such as rings, other special-shaped parts, etc.) is not in the solid area of the workpiece, belongs to the hollow part, and cannot be used as a coding area, effectively avoiding the phenomenon of missed coding and wrong coding, and providing strong guarantee for subsequent workpiece identification, assembly and other processes.
[0094] Specifically, Fig. 9 As shown, step S111 may optionally include but is not limited to:
[0095] S1111: Calculate the center of gravity position of the entire workpiece to be coded;
[0096] S1112: Taking the center of gravity position of the whole workpiece as the dividing point, determining the dividing line through the dividing point, and dividing the whole workpiece to be coded into several partial workpieces. Specifically, it is optional but not limited to taking the center of gravity position as the dividing point, and making the horizontal axis parallel to the horizontal plane and the vertical axis perpendicular to the horizontal plane through the center of gravity position (center of gravity coordinate point), and dividing the workpiece to be coded into four quadrants of partial workpieces. It is worth noting that the specific form and position of the dividing line can be arbitrarily set according to the actual shape, size, coding position accuracy requirements, etc. of the workpiece to be coded. It is optional but not limited to selecting only one dividing line to divide the workpiece to be coded into two quadrants of partial workpieces; or selecting more dividing lines to divide the workpiece to be coded into more quadrants of partial workpieces. More specifically, when the workpiece to be coded is a symmetrical structure, it is optional but not limited to passing the center of gravity position, using the symmetry axis of the workpiece to be coded as the dividing line to divide the workpiece to be coded into several partial workpieces.
[0097] S1113: Calculate the centroid positions of the local workpieces of the workpiece to be coded respectively; specifically, taking the local workpiece including the local workpieces in 4 quadrants as an example, a total of 5 centroid positions are obtained (1 centroid position of the whole workpiece + 4 centroid positions of the local workpieces). More specifically, the centroid position can be optionally but not limited to being calculated by the binary image of the workpiece to be coded.
[0098] In this embodiment, step S111 is given, which is a specific embodiment of how to calculate the center of gravity position of the overall workpiece and the local workpiece to be coded. The center of gravity position of the overall workpiece is used as a dividing point, and a dividing line is selected to divide the workpiece to be coded into several local workpieces, and then the center of gravity position of the local workpiece is determined. The center of gravity position in the solid area can be selected from multiple center of gravity positions as the coding information, which can further avoid the situation where the overall center of gravity position of some workpieces is not in the solid area of the workpiece (belonging to the hollow part) and cannot be used as the coding area, effectively avoiding the phenomenon of missed spraying, wrong spraying, etc., and providing strong guarantee for subsequent workpiece identification, assembly and other processes.
[0099] More specifically, Fig.10 As shown, step S114 may optionally include but is not limited to:
[0100] S1141: Taking the center of gravity of the solid area as the key point, determine the area around the key point that can be coded (pre-coding area); specifically, S1141, it is optional but not limited to taking the center of gravity of the solid area as the center of coding, drawing a circle with r as the radius or a as the side length to draw a rectangular frame, and determining the solid area around the center of gravity as the area that can be coded. It is worth noting that the key point is selected as the center of coding and the surrounding solid area is the area that can be coded. It is only an illustrative example. Those skilled in the art can understand that the key point is selected as the starting point (landing point) of the coding area, and the solid area below the left (upper left, lower right, upper right) of the key point is used to calculate its coding area, etc., which can all be used as alternatives to the present invention.
[0101] More specifically, step S114 may also optionally include but is not limited to: S1142: Calculate the area of the area that can be coded. Specifically, the area of the area that can be coded around each center of gravity position in the solid area is calculated as the key point, that is, the area of the solid area around the center of gravity position (non-hollow area that can be coded) is calculated as the pre-coding area.
[0102] S1143: Compare the sizes of the coding areas, and select the key points corresponding to the largest coding area as the coding information itself. That is, the largest coding area (maximum solid area) is selected as the area to be coded, and its corresponding key points are the coding center point / starting point / end point, etc. It is worth noting that the coding information determined here is the coding information of the workpiece to be coded, and has nothing to do with its position on the coding body. More specifically, it is optional but not limited to using the center of gravity as the center of the coding area, selecting a circle with a radius of r, a rectangular frame with a side length of a, etc. as the area to be coded.
[0103] In this embodiment, step S114 is given, which is a specific embodiment of how to select the best self-inking information with the center of gravity position of the solid area as the key point. It can select the area with the largest area of the inkjet area and its corresponding key point as the inkjet information. On the one hand, it can avoid the situation of incomplete inkjet (part of the inkjet is not displayed due to insufficient inkjet position); on the other hand, it can expand the inkjet font and expand the inkjet content to the greatest extent, so that the staff can more clearly and completely identify the workpiece to be coded. It is worth noting that the largest area here is only an exemplary example of the best self-inking information. Those skilled in the art can also select but not be limited to selecting the most conspicuous position as the evaluation standard of the best self-inking information, and make the best choice for multiple center of gravity positions in the solid area.
[0104] Specifically, in order to obtain the position information of the coding body (optionally but not limited to the corner point coordinates, tilt angle, etc. of the coding body that characterize the position of the coding body), step S2, as another inventive concept of the present invention, also provides a corner positioning method based on a neural network and a detection algorithm, which is used to locate the position of the edge points and corner points of the steel plate and the tilt angle of the steel plate (such as Figure 4 As shown, here we take steel plate coding as an example for explanation. The corner coordinates of the steel plate are not the origin. The corner coordinates are (X c , Y c ), the edge point coordinates are (M c , N c )). It is worth noting that the corner positioning method based on the neural network and the detection algorithm is only a preferred example of step S2 and is not limited to this. More importantly, the corner positioning method based on the neural network and the detection algorithm is only applied to the steel plate inkjet position determination as an exemplary example. The method can also be applied to other steps that require corner positioning and position information, such as steel plate cutting and object grabbing. Its application objects and application scope are not limited to this example. Specifically, Fig.11 As shown, step S2 proposes a corner positioning method based on a neural network and a detection algorithm, which may include but is not limited to:
[0105] S21: Collect sample images and annotate the edge and corner data of the sample images (optionally but not limited to annotating with a rectangular frame, and then returning the rectangular frame to calculate and determine the coordinates of the edge points and corner points) to construct a training sample set; specifically, it is optional but not limited to using cameras, video cameras and other image acquisition devices to collect a large number of steel plate images at the construction site as sample images, especially including but not limited to photographed images of edge points / corner points. More specifically, in order to fully restore the on-site environment and actual conditions to improve the accuracy of subsequent model training, the sample images, optionally but not limited to covering the conditions of such lighting and dust as much as possible, obtain sample images under various complex conditions, so that the steel plates and workpieces have more realistic texture details, so that the neural network model obtained by subsequent training also has better generalization.
[0106] S22: Construct a neural network model for learning edge data; specifically, it is optional but not limited to using a high-precision improved version of yolov4 deep learning model (such as Fig.12As shown). In the Prediction area, YOLOv4 will output images with resolutions of 76×76 / 256, 38×38 / 512, and 19×19 / 1024, respectively, as the number of feature maps changes. In the present invention, it is optional but not limited to using a small input resolution to accelerate network training, and output images with scales of 52×52, 26×26, and 13×13. More specifically, the present invention is optional but not limited to adjusting the last layer activation function Mish and replacing it with the Relu activation function, so as to produce a fast model that is more suitable for field deployment while ensuring accuracy.
[0107] S23: inputting the training sample set into the neural network model, optimizing the parameters to obtain the trained neural network model; specifically, by inputting the training sample set, continuously iteratively reducing the loss function to optimize the parameters, and obtaining a neural network model with better performance.
[0108] S24: Collecting a real-time image of the subject to be detected; specifically, it is optional but not limited to collecting the current real-time image of the steel plate (taking inkjet coding as an example, that is, the subject of inkjet coding) through image acquisition devices such as cameras and webcams.
[0109] S25: Input the real-time image into the trained neural network model, and output the first corner data of the subject to be detected (the coding subject, such as a steel plate); specifically, the neural network model may optionally, but not limited to, return the same type of output data to represent the corner data of the real-time image in the manner of annotating the sample image in step S21. Specifically, it may optionally, but not limited to, return a rectangular box representing a corner point and a rectangular box representing an edge point, with the center point of each rectangular box as the first corner point coordinate and the first edge point coordinate.
[0110] S26: According to the first corner data, capture the area image including the first corner data in the real-time image; specifically, obtain the first corner point coordinates and the first side point coordinates according to the first corner data (optionally but not limited to the position and size of the rectangular frame represented by the center point and frame height or the diagonal coordinates of the rectangular frame), perform cropping processing centered on the recognition point, and obtain a cropped area map of a fixed size as the area image of the real-time image.
[0111] S27: Detect the regional image to obtain second corner data; specifically, the cropped regional image may be input into an edge detection algorithm model to calculate the second corner data. More specifically, the edge detection algorithm model may be, but is not limited to, a canny operator.
[0112] S28: Correct and fuse the second corner data with the first corner data to obtain the corrected corner data as the final corner positioning output result. Detection area imageThe second corner data obtained is corrected and fused by Neural Network Model The first corner data obtained are matched and fused with each other to obtain the corrected corner data.
[0113] In this embodiment, a preferred embodiment of step S2 of the present invention is given, which integrates the advantages of the neural network model and the traditional edge detection algorithm. First, the first corner data of the real-time image is obtained through the neural network model, and then the local image (corner image) in the real-time image is intercepted according to the first corner data to obtain the second corner data of the real-time image through the edge detection algorithm. The two are corrected and fused to obtain the corrected corner data, that is, the edge point, corner point, deflection angle, etc., and the position information of the coding body is fed back. The calculation results of the edge point and corner point are more accurate. Compared with a single neural network algorithm or edge detection algorithm, its stability and accuracy are higher and it is less susceptible to environmental interference. It is worth noting that this method is only a preferred example of step S2 and is not limited to this. Those skilled in the art can understand that any other method that can determine the position information of the coding body (steel plate) can be adopted, such as a single neural network model method, a single edge detection algorithm, etc.
[0114] More specifically, in step S21, when marking the corner data of the sample image, it is optional but not limited to the following: Figure 13-14 The rectangular frame in the example marks the corner points of the sample image (specifically, the solid line is the steel plate; the dotted line is the grid on which the steel plate is placed; the dotted line is the rectangular frame used as the mark); Figure 15-16 The rectangular box in the example is used to mark the edge points of the sample image. More specifically, when the edge data of the sample image is marked with a rectangular box, it is optional but not limited to being represented by parameters representing the position and size of the rectangular box. For example, the position and size parameters can be optionally but not limited to being represented by the center point position and frame height of the rectangular box; or by the coordinates of the two diagonal points of the rectangular box, etc.
[0115] More specifically, when annotating the edge point data of the sample image, special attention should be paid to the position and size of the rectangular box, because these two parameters jointly determine the proportion of the target object (corner point or edge point) in the rectangular box, which plays a vital role in training the neural network model and parameter optimization, and can improve the accuracy of the subsequent detection of the corner data of the real-time image using the trained neural network model.
[0116] For example, in the rectangular box marked with corner points, Fig.14 The selected rectangle is better than Fig.13 The selected rectangle. Fig.13As shown in the figure, the left image uses a 100x100 pixel rectangular frame for annotation. The pixel point is located in the center of the frame, and a large area around the corner point is marked, which means that nearly 1 / 3 of the area in the frame comes from the background grille. During the training process, if the network can find a frame with a large number of grilles and a small number of steel plates with uniform textures, it will be considered that a high degree of match has been achieved in terms of texture similarity, which is inconsistent with our actual needs. The right image marks the entire steel plate. The large-area frame selection method is affected by the placement method and the size of the area exposed by the steel plate in the entire image, which will lead to a certain probability of mismatching / missing matching. Fig.13 The marking method first finds the area around the target point, and then performs secondary refined matching in subsequent operations. This two-step calculation method brings a larger amount of calculation, and the result of the secondary matching depends on the accuracy of the first rough positioning. If the target point cannot be identified in the rough positioning stage, then the subsequent steps will be hindered. Fig.14 In the rectangular frame annotation in the left image, most of it is the main body of the steel plate, and the rest of the frame needs to ensure that it clearly contains the two right-angled sides that constitute the corner points, so that the deep learning network can distinguish the corner points of the steel plate from the corner points of the background grille without mismatching, and at the same time ensure that no matter what angle and exposed area the steel plate is in the photographed image, the rectangular frame can accurately find the corner points. The annotation method in the right image is used to deal with the situation where the texture in the image is clear, the light source is bright, but there are multiple "suspected corner points". The smaller rectangular frame will filter out many corner points in the background. The reason why the small frame has this advantage is that the rectangular frame has fewer pixels, which means that any mismatch of each pixel value will cause greater losses in the network calculation, thereby prompting the deep learning network to review the matching results with more stringent standards. Therefore, the present invention is preferred to adopt Fig.14 The rectangular box of the type is used to mark the corner point data to cope with the labeling of different sample images. Specifically, for low-quality sample images, Fig.14 The rectangular frame shown in the left figure includes corner points and two right-angled sides, and the main body of the inkjet printer occupies more than 2 / 3 of the area of the rectangular frame. When the neural network model is input and the first corner data is returned, the midpoint between the center point of the rectangular frame and the corner point is used as the actual corner point coordinate; for high-quality sample images, Fig.14 The rectangular box shown in the right figure includes corner points and two right-angled sides, but the pixel size (rectangular box size) is smaller. When the neural network model is input and the first corner data is returned, the center point of the rectangular box is used as the actual corner point coordinate.
[0117] In the rectangular box marked with edge points, the main technical difficulty is how to make the network reduce mismatching. Compared with the detection of corner points, there are more interference points in the edge points in the image, such as all grilles, background metal bars, scratches on the steel plate itself, water stains, etc., all of which may be misdetected as edge points. Fig.15There are four annotation methods, but the trained neural network models all have a certain degree of mismatch. Fig.16 The rectangular box shown is a new method for marking edge points. It is a long rectangular box to mark the edge of the steel plate. The reserved positions on the left and right sides of the marking box remain equal. At the same time, the marking box cannot be too long, and a certain amount of space must be reserved above and below to ensure that when the edge of the steel plate is too short in the figure, there will not be too much volume difference between the marking box and the marking boxes of other long edge samples.
[0118] More specifically, Fig.17 As shown, step S23 may optionally include but is not limited to: S231: data cleaning of the training sample set. That is, discovering and correcting identifiable errors in the data file, removing or repairing "dirty samples" that affect subsequent model training. More specifically, the data cleaning step may optionally include but is not limited to checking data consistency, processing invalid values and missing values, etc. S232: inputting the training sample set after data cleaning into the neural network model, iterating continuously to reduce the loss function of the model output, and then back-propagating to tune the model parameters to obtain a trained neural network model. The trained neural network model has better performance.
[0119] More specifically, Fig.18 As shown, step S25 may also optionally include but is not limited to:
[0120] S251: input the real-time image into the trained neural network model;
[0121] S252: Determine whether the first corner data is obtained;
[0122] S253: If yes, output the first edge and corner data of the workpiece to be inspected; this situation means that the quality of the current input real-time image is relatively good, and the neural network model can actively identify the first edge and corner data such as edge points and corner points.
[0123] S254: If not, then enhance the real-time image, input the trained neural network model, and output the first corner data. This situation is that the quality of the current input real-time image is poor, and the neural network model cannot automatically identify the first corner data such as edge points and corner points. It is necessary to perform enhancement processing using multiple synthesis algorithms with preset values according to the lighting characteristics of the current image to obtain a set of enhanced images, which are then input into the neural network model to obtain the first corner data.
[0124] More specifically, in step S254, the step of enhancing the real-time image may optionally include but is not limited to: brightness enhancement, HSV color space enhancement, up / down / horizontal inversion enhancement, Gaussian Laplace enhancement, and effective enhancement calculation parameters, which can efficiently and specifically enhance the original positioning and patrol data, thereby expanding the data set and improving the generalization and accuracy of the deep learning network.
[0125] Specifically, those skilled in the art will know that there are hundreds of enhancement algorithms for image processing to expand the collected data. According to the actual environment of the steel plate, the inventors have continuously studied and tested, and made creative efforts to select several preferred enhancement methods to combine and enhance the real-time image.
[0126] More specifically, the calculation formula for brightness enhancement is:
[0127] g(x)=αf(x)+β
[0128] The parameters α>0 and β are called gain and bias parameters, which control the brightness respectively. f(x) is the source image pixel and g(x) is the output image pixel, which means that the brightness of the input image is enhanced to obtain a brighter or darker image than the original image.
[0129] In HSV color space enhancement, H represents hue, with a value range of (0, 180); S represents saturation, with a value range of (0, 255); and V represents brightness, with a value range of (0, 255). The algorithm first converts the real-time image from RGB space to HSV space, then performs numerical amplification or reduction operations on the three components of H, S, and V based on experience, and finally converts it back to RGB space to achieve the effect of image enhancement.
[0130] The up-down / horizontal inversion enhancement method randomly crops some areas in the real-time image, and then stitches the cropped n images together to make one image contain as many concentrated objects as possible, thereby increasing the richness of samples and providing the network with more complex recognition scenarios. However, it is especially important to note that from an empirical point of view, scaling should not be performed during the cropping process. The image should be cropped according to the original size to avoid scaling the image, which causes the edge points and corners to shift out of the cropped area of the image, resulting in missing samples.
[0131] Laplace enhancement calculation formula:
[0132]
[0133]
[0134] Among them, G(x,y) represents the Gaussian function, and δ represents the standard deviation. That is, the image is first smoothed with the Gaussian kernel to reduce the high-frequency noise in the image and facilitate the subsequent Laplace operation. This means that a quadratic partial division operation is performed on it to obtain the LoG function of the Gaussian Laplace. Because the Gaussian Laplace is a derivative operator, it will highlight the sharp grayscale changes in the image and suppress the areas with slow grayscale changes, often resulting in gray edges and discontinuous images against a dark background. By superimposing the Laplace image with the original image, an image with a sharpening effect can be obtained.
[0135] More specifically, the steps to enhance the real-time image may include, but are not limited to, contrast enhancement, scaling enhancement, and Mosaic enhancement. However, due to the particularity of the scene, inappropriate enhancement processing algorithms may improve the visual effect of the human eye, but are not suitable for subsequent algorithm calculations, and will ultimately lead to a decrease in the performance of the neural network model. Especially in the positioning and patrolling scenarios of the edge points and corner points of steel plates, it has obvious advantages, mainly reflected in: the contrast difference brought by the contrast enhancement method will increase the distinction between the steel plate and the background, thereby reducing the generalization of the deep learning network, which is not conducive to identifying steel plates under complex conditions; scaling enhancement and Mosaic enhancement methods are usually applicable, but due to the uncertain position of the edge points and corner points of the steel plate, they may appear at the edge of the image, and common scaling and splicing can easily lose part of the annotation information. When there are too many blank unlabeled samples in the training data of the deep learning network, it is actually equivalent to reducing the number of training samples;
[0136] More specifically, the corner data may include, but is not limited to, corner point data, and the side point data may include, but is not limited to, corner point coordinates (X c , Y c ) and edge point coordinates (M c , N c ). More specifically, in the neural network model, the coordinates of the first corner point are determined by the coordinates of the center point of the rectangular box marking the corner points, and the coordinates of the first side point are determined by the coordinates of the center point of the rectangular box marking the side points. In the detection algorithm, the coordinates of the second corner point and the second side point are determined by the edge line segment. Specifically, Fig.19 As shown, step S27 may optionally include but is not limited to corner point data determination:
[0137] S271: Input the regional image into the edge detection algorithm model to obtain the edge segments in the regional image; it is worth noting that the edge segments may be the right-angled edges of the steel plates (the edge segments we need to detect), or they may be scratches on the steel plates, the edges of the grilles where the steel plates are placed, etc., which need to be subsequently eliminated.
[0138] S272: Determine the pre-selected corner points in the regional image according to the edge line segments; specifically, it is optional but not limited to finding all corner points (i.e., pre-selected corner points) existing in the regional image according to the intersection of right-angled sides; specifically, the corner points are corner points determined by the edge line segments. Since the edge line segments may be right-angled sides of the steel plate, the determined corner points are actual corner points of the steel plate and can be returned as the second corner data; it may also be scratches on the steel plate or the edges of the grilles on which the steel plate is placed. The determined corner points are not actual corner points of the steel plate and cannot be returned as the second corner data; therefore, in order to fully consider the situations such as scratches on the steel plate, it is necessary to exclude the corner points that are not actual corner points of the steel plate from the pre-selected corner points to improve the accuracy of corner positioning; specifically:
[0139] S273: Determine whether there is a pre-selected corner point among the pre-selected corner points, and the position difference with the first corner point is within the first set range; specifically, since the pre-selected corner point calculated in step S272 may be the corner point of the steel plate (the corner point that actually needs to be found), or it may be a scratch on the steel plate, a corner point of the grille, etc. (corner points that may be mistaken for corner points of the steel plate), it is necessary to compare the position difference between the pre-selected corner point and the first corner point. If the position is closer, it means that one of the pre-selected corner points determined by the detection algorithm is a real corner point of the steel plate, and the two need to be fused and corrected; if the position is farther, it means that the pre-selected corner point determined by the detection algorithm does not have the real corner point of the steel plate, but is a corner point of other line segments such as scratches, which needs to be excluded, and there is no need to fuse the two to correct.
[0140] S274: If it exists, the pre-selected corner point whose position difference is within the first set range is returned as the second corner point; when the characterization detection algorithm detects the regional image, the actual corner point of the steel plate is found.
[0141] S275: If it does not exist, the second corner point is not returned. When the characterization detection algorithm detects the regional image, no actual corner point of the steel plate is found.
[0142] More specifically, based on the above-mentioned corner point data determination, the corner point correction step S28 is as follows: if the second corner point is returned, the cluster center point of the first corner point and the second corner point is calculated as the corrected corner point data; if the second corner point is not returned, the first corner point is used as the corrected corner point data.
[0143] In this embodiment, the corner data is given as corner point data, and specific steps are given for calculating the second corner point and correcting and fusing the first corner point. This can eliminate unreasonable outliers (corner points of scratches, etc. may be mistaken for steel plate corner points), further optimize corner point coordinates, and improve the accuracy of corner positioning.
[0144] More specifically, S27 may optionally include, but is not limited to, determining edge point data. Specifically, the edge point data determination has similar characteristics to the above-mentioned corner point data determination, and may optionally include, but is not limited to:
[0145] S276: Determine, according to the edge line segment, a point in the edge line segment that is closest to the first edge point as a pre-selected edge point;
[0146] S277: Determine whether the position difference between the pre-selected edge point and the first edge point is within a second set range;
[0147] S278: If yes, returning the pre-selected edge point whose position difference is within the second set range as the second edge point;
[0148] S279: If not, the second edge point is not returned. The characterization detection algorithm detects the regional image and does not find the actual edge point of the steel plate.
[0149] More specifically, based on the determination of the above edge point data, the step of edge point correction in step S28 is specifically: calculating the cluster center point of the first edge point and the second edge point as the corrected edge point data.
[0150] In this embodiment, the corner data is provided as edge point data, and specific steps are provided for calculating the second edge point and correcting and fusing the first edge point, so as to further optimize the edge point coordinates and improve the accuracy of corner positioning.
[0151] More specifically, S28 may optionally include, but is not limited to, further optimizing the obtained coordinate position through various parameters such as distance, edge slope, and side length of the point after calculating the corner point and the edge point.
[0152] In this embodiment, the positioning method of the present invention is given, which is a specific embodiment of how to integrate the first edge point data determined by the neural network model and the second edge point data determined by the detection algorithm, so as to combine the advantages of both, improve the positioning accuracy of corner points and edge points, and further optimize the positioning accuracy of the coding points.
[0153] Specifically, Fig.21 As shown, in order to improve the accuracy of corner point determination, S272: may also optionally include but is not limited to: S2721: calculate the edge area enclosed by the edge points in the regional image; S2722, find the image center moment corresponding to the maximum edge area; S2723: calculate the distance between the image center moment and the edge line segment, merge the edges within a certain range of distances, and eliminate discontinuous edges; S2724: calculate the pre-selected corner points in the regional image based on the edge line segments that eliminate the discontinuous edges.
[0154] Specifically, in practical applications, the corner positioning method may optionally include, but is not limited to, converting the obtained corner data from the image coordinate system to the space coordinate system. Specifically, taking the acquisition of the real-time image of the workpiece to be detected as a steel plate as an example, it is optional but not limited to inputting the calculated corner data into the conversion algorithm, calling the external parameter matrix of the current camera, calculating the real coordinates of the corner points / edge points of the steel plate, and returning the results.
[0155] More specifically, in order to further obtain the current position of the coding subject, such as Fig.21 As shown, the corner positioning method of the present invention may also optionally but not limited to include: S29 calculating the inclination angle of the coding body (the body to be detected) according to the corrected corner data. Specifically, it may be optional but not limited to combining the corner point data (X c , Y c ), edge point data (M c , N c ) and the relevant parameters of the current camera position, and calculate the inclination angle r of the steel plate c .
[0156] In this embodiment, the inclination angle of the subject to be detected (optionally but not limited to a steel plate) can be further determined based on the corner point data and the edge point data, so as to provide effective guarantee for further accurate positioning of the coding point and further optimization of the coding effect.
[0157] Specifically, step S3 specifically uses formulas (1)-(3) to determine the coding correction information of the workpiece to be coded on the coding body.
[0158] x n1 =x n cos r c -y n sin r c +x c (1)
[0159] y n1 =y n sin r c +y n cos r c +y c (2)
[0160] r n1 =r n +r c (3)
[0161] Specifically, Fig. 22 As shown, the present invention also provides a coding path control method, comprising:
[0162] T1: Get the coding information of the coding body. Specifically, the coding information can be selected but limited to the coding information of each workpiece to be coded on the coding body (N workpieces to be coded with the same or different types and names, and the coding information of each workpiece can be selected but not limited to be distinguished by serial number). More specifically, in order to ensure the accuracy of the coding position and avoid missing or wrong coding, it is optional but not limited to fully consider the displacement and angle deviation when the coding body arrives at the coding point, and determine the coding information as the calibrated coding information of each workpiece to be coded, that is, the coding correction information, which can be selected but not limited to the coordinate angle (x n1 ,y n1 , r n1 ) indicates that the specific method for obtaining the coding information of each workpiece to be coded may be determined by, but is not limited to, the calibration method for workpiece coding described above, wherein steps S1-S3 may be determined by, but are not limited to, referring to the above and will not be described in detail here.
[0163] T2: Taking the corner point of the coding body as the starting point, select the workpiece to be coded that is closest to the corner point as the first coding workpiece, set i=1, and start the coding process; specifically, the distance calculation can be selected but not limited to using the Euclidean formula.
[0164] T3: When the coding process of the i-th workpiece to be coded is completed, the i-th workpiece to be coded is deleted in the sequence to be coded; specifically, the subsequent workpiece to be coded that can be understood by those skilled in the art is the workpiece to be coded that has not completed the coding process, that is, the workpiece to be coded that has completed the coding process needs to be deleted here, and no subsequent distance comparison is performed to avoid the coding device returning to the position where the coding has been coded again and coding again, which on the one hand reduces time efficiency and increases cost; on the other hand, it may cause the coding device to travel back and forth between two points, causing coding confusion and inability to proceed.
[0165] T4: Select the workpiece to be coded that is closest to the i-th workpiece to be coded as the next workpiece to be coded and start coding, let i=i+1;
[0166] T5: Determine whether i is equal to N; N is the total number of workpieces to be coded; that is, whether all workpieces to be coded have completed the coding process;
[0167] T6: If not, return to step T3;
[0168] T7: If yes, the coding is finished.
[0169] In this embodiment, a specific embodiment of the coding path planning of the present invention is given, which takes the shortest distance as the evaluation standard and arranges the entire coding process of the coding device according to the optimal planning path (such as Fig.23The method is shown as the line), which minimizes the distance the truss needs to move, shortens the coding time, improves the coding efficiency, and reduces the coding cost. Compared with the path control method in the prior art that codes in rows / columns in sequence, the method is significantly more efficient. Although the method seems simple, it breaks the conventional way of the industry and obtains obvious beneficial effects.
[0170] On the other hand, the present invention also provides a corresponding system based on the above method.
[0171] Specifically, Fig.24 As shown, corresponding to the above-mentioned workpiece coding correction method, the present invention also provides a workpiece coding correction system, which can optionally include but is not limited to:
[0172] The first acquisition device is used to obtain the initial information of the workpiece to be coded on the coding body (such as Figure 3 For example, the rectangular outer frame is the coding body, such as a steel plate, and its shape and size can be set arbitrarily; the pentagonal pattern inside it is the workpiece to be coded, and its shape and size can also be set arbitrarily. There are other workpieces to be coded scattered on the steel plate, which are not shown here; the dotted box is the coding area, and the solid circle is the coding center. This is the initial coding information that needs to be obtained in our example. It will be pointed out in the following example with the center of gravity position. It can be the center point / starting point / end point of the coding; the coding area can be a circle or rectangular box made by the center of gravity position. Specifically, the initial coding information can be selected but not limited to the coding coordinates and coding angles of the workpiece to be coded on the coding body. For example, assuming that the corner point of the steel plate is at the origin (0, 0) and the angle is 0 (in the ideal case where the steel plate is placed without any deviation), the position where each workpiece to be coded should be coded (assuming that the steel plate has no deviation, the position and angle initially determined). For example, the initial coding information of the Nth workpiece can be selected but not limited to the coordinates (x n ,y n ), angle r n It means that (x n ,y n , r n ). More specifically, the first acquisition module may be optionally but not limited to being connected to the storage unit to directly acquire the initial information of the inkjet coding preset in the nesting diagram.
[0173] The second acquisition device is used to obtain the position information of the coding body (such as Figure 4 ). Specifically, the position information may be, but is not limited to, the position coordinates and offset angle of the coding body. For example, when the coding body is a steel plate, when the steel plate is transported to the coding point by a mobile pallet, its corner point is not located at the origin (0, 0), and the angle is not 0, but there is a displacement deviation and angle deviation. For example, the position information of the coding body may be, but is not limited to, the corner point coordinates (xc ,y c ), angle r c It means that (x c ,y c , r c ). More specifically, the second acquisition module may optionally include, but is not limited to, an image acquisition device to acquire a real-time image of the coding body and determine the current position, posture, etc. of the coding body through image recognition.
[0174] The correction device is connected to the first acquisition device and the second acquisition device, and is used to determine the coding correction information (corrected information) of the workpiece to be coded on the coding body according to the initial coding information of the workpiece to be coded on the coding body (the steel plate has no deviation, and the coding information is determined under ideal conditions) and the position information of the coding body (the real-time information of the steel plate, and there are deviations in position, angle, etc.). That is, according to the position information of the coding body (optional but not limited to displacement deviation and angle deviation), the initially determined coding information is corrected to obtain the corrected coding information (coding correction information), that is, the actual position and angle of the current coding point that needs to be coded, expressed as (x n1 ,y n1 , r n1 ),(like Figure 4 shown).
[0175] In this embodiment, a specific embodiment of the workpiece coding correction system of the present invention is given, which fully considers the position information of the coding subject (optionally but not limited to the steel plate) when it reaches the coding point (there may be deviations in position, posture, angle, etc. (such as Figure 4 As shown in the figure, it is not an ideal case where a square is placed under the coding point. Its corner point is not the origin and the angle is not zero. By detecting the position information of the coding body itself, the initially determined coding position is corrected to obtain the actual position where the workpiece to be coded should be coded, ensuring that the coding position and angle are accurate. It is worth noting that during the correction process, it is actually necessary to correct the initial coding information of N workpieces to be coded in turn according to the number of workpieces to be coded on the coding body, determine the coding correction information of each workpiece to be coded, and obtain the following: Figure 5 The inkjet dot diagram shown (compared to Figure 1 , the added points are the coding correction information of each workpiece to be coded, i.e., the coding point position). It is more noteworthy that the above-mentioned devices and subsequent systems and modules are not divided in an absolute physical sense. They are only functional definitions and do not specifically limit their physical structures. The modules, devices, and systems can be connected, interacted, and combined in any form.
[0176] Specifically, Fig.25 As shown, the first acquisition device may optionally include but is not limited to:
[0177] The first determination module is used to determine the self-coding information of the workpiece to be coded (such as Figure 7 As shown), that is, regardless of the position of the workpiece to be coded in the coding body, the code should be marked on the workpiece itself according to its shape, size, etc., which can be selected but not limited to the coordinates (x na ,y na )express.
[0178] The second determination module is used to determine the placement position information of the workpiece to be coded on the coding body (such as Figure 3 As shown, it can be optionally but not limited to the coordinates of a key point such as a corner point or center point of the workpiece to be coded). Specifically, the placement position of the workpiece to be coded on the coding body can be optionally but not limited to the coordinates (x ) of the feature points of the workpiece to be coded (such as the upper, lower, left, right, center point of the workpiece, etc., the example in the figure is a corner point of the workpiece). nb ,y nb ) is indicated. More specifically, the placement position may be optionally, but not limited to, parsing and extracting material data (optionally, but not limited to, information data including the number and name of each workpiece) and line data (optionally, but not limited to, graphic data including the shape, size, and line segment composition of each workpiece) from the production nesting diagram (such as dxf, json, etc.) through a data parsing module; then the material data is stored as file data in the form of a file, and the line data is combined and converted into the frame data of the steel plate cutting and the workpiece data is stored as picture data in the form of a picture; finally, the file data and the picture data are matched one by one to determine the position, shape, size, etc. of each workpiece on the coding body (steel plate).
[0179] The third determining module is connected to the first determining module and the second determining module, and is used to determine the coding information of the workpiece to be coded (optionally but not limited to the coordinates (x na ,y na ) indicates the location of the workpiece to be coded on the coding body (x nb ,y nb ), determine the initial information of the workpiece to be coded on the coding body (x n ,y n ). Angle r n An angle may be set by a person skilled in the art based on the appearance, etc. Generally, for easy viewing, it is generally set to be parallel to the workpiece, that is, the angle is 0.
[0180] In this embodiment, a specific embodiment of the first acquisition device component module is given. It determines the initial information of the coding by acquiring the coding information of the workpiece to be coded and the placement information of the workpiece to be coded on the coding body, that is, completes the coordinate conversion from the small coordinate system of the workpiece to be coded to the large coordinate system of the coding body. It has a simple structure and accurate calculation, and can ensure that the coding position is accurate.
[0181] Specifically, in order to accurately identify and find the self-printing information of the workpiece to be printed (such as Figure 7 As shown, the coding area (dashed box) / coding starting point / end point / center point (solid circle) and other information representing the coding position) are provided. The present invention also provides a workpiece coding recognition device (used as the first determination module in the correction system). The specific workpiece coding recognition device can optionally but not limited to determine the coding information of the workpiece to be coded according to the center of gravity of the workpiece to be coded. More specifically, as Fig.26 As shown, the identification device may optionally include but is not limited to:
[0182] The center of gravity calculation module is used to calculate the center of gravity position of the whole workpiece and the local workpiece to be coded. Specifically, the center of gravity position can be optionally but not limited to be calculated based on the binary image of the workpiece to be coded. Specifically, it is optional but not limited to extracting the contour data of each workpiece from the file data and image data obtained from the production nesting diagram, and then filling the workpiece contour to obtain the binary image of each workpiece. More specifically, the center of gravity calculation module can be optionally but not limited to a control unit with image recognition, processing, data reception, processing, transmission and other functions to complete the above corresponding functions.
[0183] The solid area judgment module is connected to the center of gravity calculation module to judge whether the center of gravity position is the solid area of the workpiece to be coded; specifically, taking the local workpiece including 4 quadrants as an example, it is necessary to judge whether 5 center of gravity positions (1 center of gravity position of the whole workpiece + 4 center of gravity positions of the local workpiece) are solid areas. More specifically, it is optional but not limited to judging whether each center of gravity position (coordinate) calculated from the file data and image data obtained from the production nesting diagram is the solid area of the workpiece to be coded.
[0184] The deletion module is connected to the solid area judgment module, and is used to delete the center of gravity position that is not a solid area; specifically, if the center of gravity position is not a solid area, it indicates that the center of gravity position is not on the workpiece to be coded (such as the center of gravity position of a ring is within the center circle). The hollow area cannot be used as a pre-selected position for the coding area and needs to be deleted. Specifically, it is optional but not limited to excluding the center of gravity position from the storage sequence among the many calculated center of gravity positions. Specifically, it is optional but not limited to excluding the center of gravity position from the storage sequence among the many calculated center of gravity positions, and deleting the number, name, coding content, etc. corresponding to the workpiece to be coded in the storage unit.
[0185] The coding information determination module is connected to the solid area judgment module, and is used to determine the self-coding information of the workpiece to be coded, taking the center of gravity position of the solid area as the key point. Specifically, the self-coding information can be selected but not limited to any information that characterizes the position to be coded, such as the center of gravity position of the solid area as the starting point / end point / center point of the area to be coded. More specifically, since multiple center of gravity positions are calculated in step S111, there may be multiple center of gravity positions in the solid area in step S114. At this time, those skilled in the art can understand that one of them is selected as the coding information, and there is an improved preferred scheme below.
[0186] In this embodiment, an identification device for workpiece coding of the present invention is provided. By calculating the center of gravity positions of the overall workpiece and the local workpiece to be coded, the center of gravity position in the solid area is selected as the key point. Compared with the prior art, it can avoid the situation that the overall center of gravity position of certain workpieces (for example, hollow workpieces such as rings, other special-shaped parts, etc.) is not in the solid area of the workpiece, belongs to the hollow part, and cannot be used as a coding area, effectively avoiding the phenomenon of missed coding and wrong coding, and providing strong guarantee for subsequent workpiece identification, assembly and other processes.
[0187] Specifically, the center of gravity calculation module may include but is not limited to:
[0188] The first calculation unit is used to calculate the center of gravity position of the entire workpiece to be coded;
[0189] The segmentation unit uses the center of gravity position of the whole workpiece as the segmentation point, determines the segmentation line through the segmentation point, and segments the whole workpiece to be coded into several partial workpieces. Specifically, it is optional but not limited to using the center of gravity position as the segmentation point, and using the center of gravity position (center of gravity coordinate point) as the horizontal axis parallel to the horizontal plane and the vertical axis perpendicular to the horizontal plane to segment the workpiece to be coded into four quadrants of partial workpieces. It is worth noting that the specific form and position of the segmentation line can be arbitrarily set according to the actual shape, size, coding position accuracy requirements, etc. of the workpiece to be coded. It is optional but not limited to selecting only one segmentation line to segment the workpiece to be coded into two quadrants of partial workpieces; or selecting more segmentation lines to segment the workpiece to be coded into more quadrants of partial workpieces. More specifically, when the workpiece to be coded is a symmetrical structure, it is optional but not limited to using the symmetry axis of the workpiece to be coded as the segmentation line through the center of gravity position to segment the workpiece to be coded into several partial workpieces.
[0190] The second calculation unit calculates the centroid position of the local workpiece of the workpiece to be coded respectively; specifically, taking the local workpiece including the local workpiece of 4 quadrants as an example, a total of 5 centroid positions are obtained (1 centroid position of the whole workpiece + 4 centroid positions of the local workpiece). More specifically, the centroid position can be optionally but not limited to being calculated by the binary image of the workpiece to be coded.
[0191] In this embodiment, a small module for calculating the center of gravity is provided, which is a specific embodiment of how to calculate the center of gravity position of the overall workpiece and the local workpiece to be coded. The center of gravity position of the overall workpiece is used as a dividing point, and a dividing line is selected to divide the workpiece to be coded into several local workpieces, and then the center of gravity position of the local workpiece is determined. The center of gravity position in the solid area can be selected from multiple center of gravity positions as the coding information, which can further avoid the situation where the overall center of gravity position of some workpieces is not in the solid area of the workpiece (belonging to the hollow part) and cannot be used as the coding area, effectively avoiding the phenomenon of missed spraying, wrong spraying, etc., and providing strong guarantee for subsequent workpiece identification, assembly and other processes.
[0192] More specifically, the coding information determination module may include, but is not limited to:
[0193] The unit for determining the area that can be coded is used to determine the area that can be coded (pre-coding area) around the key point with the center of gravity of the solid area as the key point; specifically, the unit for determining the area that can be coded can be selected but not limited to taking the center of gravity of the solid area as the center of coding, drawing a circle with r as the radius or drawing a rectangular frame with a as the side length, and determining the solid area around the center of gravity as the area that can be coded. It is worth noting that the key point is selected as the center of coding and the surrounding solid area is the area that can be coded. It is only an illustrative example. Those skilled in the art can understand that the key point is selected as the starting point (landing point) of the coding area, and the solid area below the left (upper left, lower right, upper right) of the key point is used to calculate its coding area, etc., which can all be used as alternatives to the present invention.
[0194] More specifically, the coding information determination module may also optionally include, but is not limited to: an area calculation unit connected to the coding area determination unit, for calculating the area of the coding area. Specifically, the center of gravity position of each solid area is taken as the key point, and the area of the coding area around it is calculated, that is, the area of the solid area around the center of gravity position (non-hollow area that can be coded) is calculated as the pre-coding area.
[0195] The self-inking information determination unit is connected to the area calculation unit, and is used to compare the size of the inkjet-able area, and select the key point corresponding to the largest inkjet-able area as the self-inking information. That is, the largest inkjet-able area (maximum solid area) is selected as the area to be inkjet-coded, and its corresponding key point is the inkjet center point / starting point / end point, etc. It is worth noting that the inkjet information determined here is the self-inking information of the workpiece to be inkjet-coded, and has nothing to do with its position on the inkjet body. More specifically, it is optional but not limited to using the center of gravity as the center of the inkjet area, selecting a circle with a radius of r, a rectangular frame with a side length of a, etc. as the area to be inkjet-coded.
[0196] In this embodiment, a small module for determining the coding information is provided. It is a specific embodiment of how to select the best self-coding information with the center of gravity position of the solid area as the key point. It can select the area with the largest area of the coding area and its corresponding key point as the coding information. On the one hand, it can avoid the situation of incomplete coding to the greatest extent (part of the coding is not displayed due to insufficient coding position); on the other hand, it can expand the coding font and content to the greatest extent, so that the staff can more clearly and completely identify the workpiece to be coded. It is worth noting that the largest area here is only an exemplary example of the best self-coding information. Those skilled in the art can also select but not be limited to selecting the most conspicuous position as the evaluation standard of the best self-coding information, and make the best choice for multiple center of gravity positions in the solid area.
[0197] Specifically, Fig. 27As shown, for the above-mentioned corner positioning method, the present invention also provides a corner positioning device based on a neural network and a detection algorithm (serving as the above-mentioned second determination module), which may optionally include but is not limited to:
[0198] The sample collection module is used to collect sample images and annotate the corner data of the sample images to construct a training sample set;
[0199] A small model building module for building neural network models for learning edge data;
[0200] The model training module is used to input the training sample set into the neural network model to obtain the trained neural network model;
[0201] Image acquisition module, used to acquire real-time images of the workpiece to be inspected;
[0202] The first corner data output module is used to input the real-time image into the trained neural network model and output the first corner data of the workpiece to be detected;
[0203] An image capture module, used to capture a region image including the first corner data in the real-time image according to the first corner data;
[0204] The second corner data output module is used to detect the regional image and obtain the second corner data of the workpiece to be detected;
[0205] The correction module is used to correct and fuse the second corner data with the first corner data to obtain the corrected corner data as the corner positioning output result.
[0206] The specific details of the second determination module structure are optional but not limited to corresponding to the method, and will not be repeated here.
[0207] Specifically, the third determination module specifically uses formulas (1)-(3) to determine the coding correction information of the workpiece to be coded on the coding body.
[0208] x n1 =x n cos r c -y n sin r c +x c (1)
[0209] y n1 =y n sin r c +y n cos r c +y c (2)
[0210] r n 1 = rn +r c (3)
[0211] Specifically, Fig.28 As shown, the present invention also provides a coding path planning device, comprising:
[0212] The coding information acquisition module is used to obtain the coding information of the coding body. Specifically, the coding information can be selected but limited to the coding information of each workpiece to be coded on the coding body (N workpieces to be coded with the same or different types and names, and the coding information of each workpiece can be selected but not limited to be distinguished by serial number). More specifically, in order to ensure the accuracy of the coding position and avoid missed or wrong coding, it is optional but not limited to fully consider the displacement and angle deviation when the coding body arrives at the coding point, and determine the coding information as the corrected coding information for each workpiece to be coded, that is, the coding correction information, which can be selected but not limited to the coordinate angle (x n1 ,y n1 , r n1 ) indicates that the coding information is obtained by connecting the above-mentioned correction system, which will not be repeated here.
[0213] The first workpiece selection small module takes the corner point of the coding body as the starting point, selects the workpiece to be coded that is closest to the corner point as the first coding workpiece, and i=1, and starts the coding process; specifically, the distance calculation can be selected but not limited to using the Euclidean formula.
[0214] The workpiece deletion module deletes the i-th workpiece to be coded in the sequence to be coded when the coding process of the i-th workpiece to be coded is completed; specifically, the subsequent workpiece to be coded that can be understood by those skilled in the art is the workpiece to be coded that has not completed the coding process, that is, the workpiece to be coded that has completed the coding process needs to be deleted here, and no subsequent distance comparison is performed to avoid the coding device returning to the position where the coding has been coded and coding again, which on the one hand reduces timeliness and increases costs; on the other hand, it may cause the coding device to travel back and forth between two points, causing coding confusion and inability to proceed.
[0215] The next workpiece selection module is used to select the workpiece to be coded that is closest to the i-th workpiece to be coded as the next workpiece to be coded to start coding, let i=i+1;
[0216] The coding completion judgment module determines whether i is equal to N; N is the total number of workpieces to be coded; that is, whether all the workpieces to be coded have completed the coding process; if not, continue coding; if so, the coding is finished.
[0217] In this embodiment, a specific embodiment of the coding path planning device of the present invention is given. It takes the shortest distance as the evaluation standard and arranges the entire coding process of the coding device according to the optimal planned path, so that the distance that the truss needs to move is shortened, the coding time is shortened, the coding efficiency is improved, and the coding cost consumption is reduced. Compared with the path control method in the prior art that codes in sequence according to rows / columns, its efficiency is significantly faster. Although this method seems simple, it breaks the conventional way of the industry and obtains obvious beneficial effects.
[0218] More specifically, Fig.29 As shown, the present invention also provides a total inkjet coding system, including a truss, an inkjet coding system, a control system and a correction system; the inventive point is that the control system is connected to the above-mentioned correction system, and is used to control the truss to drive the inkjet coding system to move to the inkjet coding point specified by the inkjet coding correction information according to the inkjet coding correction information, and start the inkjet coding process.
[0219] In this embodiment, the inventive concept of the overall coding system of the present invention is created based on the above-mentioned correction system, and the purpose is to obtain coding correction information so as to adaptively adjust the coding system to the correction position to start coding according to the real-time position and posture of the steel plate to avoid wrong coding and missed coding. The key point is to capture the real-time image of the coding body (steel plate) through an image acquisition small module (optional but not limited to a camera), identify its actual position and posture, and adaptively adjust the coding position to ensure that the coding position is more accurate. It is a total coding system that is based on visual guidance and adaptively adjusts the coding point position according to the position and posture of the coding body. The specific form, drive mode, etc. of the coding system and truss are not the focus of the invention of the present invention and will not be described here. Any method of realizing coding and truss movement that can be understood by those skilled in the art is acceptable.
[0220] More specifically, the overall coding system may also optionally include, but is not limited to, a coding path planning device, which is connected to the control system and is used to provide path planning for the control system; the control system controls the truss according to the path planning, and controls the truss to complete the coding process in sequence. Specifically, the truss drives the coding device to move to the designated coding positions in sequence according to the movement trajectory of the path planning. When the truss moves to the coding position, it triggers the coding machine to start coding, and the process is repeated to the next coding position until all the workpieces to be coded on the coding body are completely coded. More specifically, the coding system may also optionally include, but is not limited to, a sensing device, which can automatically sense the height of the steel plate during the coding process and transmit it to the control system to control the height of the truss, thereby maintaining a suitable coding height and ensuring that the coding effect is clear and visible. More specifically, the specific structure, drive method, etc. of the truss and the coding system are as follows. Fig.29 For example.
[0221] The above-mentioned systems and devices are created based on the above-mentioned methods, and their technical functions and beneficial effects are not repeated here. It is worth noting that the various systems, devices, modules, units, etc. are only functional divisions, and there is no limitation on their physical structure. Those skilled in the art can understand that the various technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the various technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0222] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several deformations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims. It should also be noted that the correction method and system of the workpiece coding and the total coding system are the overall inventive concept of the present invention. The key lies in obtaining the position information of the coding body and correcting the initial coding information. It is a visually guided, adaptive coding device. The identification method and device of the workpiece coding, as step S1 of the correction method, is another inventive concept of the present invention. The key lies in how to correctly determine the coding information of the workpiece to be coded to avoid the situation where the coding position is not on the workpiece. The corner positioning method and device, as step S2 of the correction method, is another aspect of the present invention. The key lies in how to integrate the deep learning network model algorithm and the traditional detection algorithm to determine the real-time posture of the steel plate. The three invention points are both complementary and independent of each other. To avoid the problem of unity in this case, the above invention points are protected as a series of applications, which is specially explained here.
Claims
1. A corner positioning method based on a neural network and a detection algorithm, characterized in that: include: S21: Collect sample images of the subject to be detected, and annotate corner data of the sample images to construct a training sample set; S22: Build a neural network model for learning edge data; S23: inputting the training sample set into the neural network model to obtain a trained neural network model; S24: Acquire a real-time image of the subject to be detected; S25: inputting the real-time image into the trained neural network model and outputting the first corner data; S26: According to the first corner data, intercepting a region image including the first corner data in the real-time image; S27: Detect the area image to obtain second corner data; S28: Correcting and fusing the first corner data with the second corner data to obtain corrected corner data as the corner positioning output result; The first corner data includes a first corner point; the second corner data includes a second corner point; Step S27 includes: S271: Input the region image into the edge detection algorithm model to obtain edge segments in the region image; S272: Determine pre-selected corner points in the regional image according to the edge line segments; S273: Determine whether there is a pre-selected corner point among the pre-selected corner points whose position difference with the first corner point is within a first set range; S274: If it exists, returning the pre-selected corner point whose position difference is within the first set range as the second corner point; S275: If it does not exist, the second corner point is not returned.
2. The corner positioning method according to claim 1, characterized in that: Step S28, specifically, if the second corner point is returned, the cluster center point of the first corner point and the second corner point is calculated as the corrected edge and corner data; if the second corner point is not returned, the first corner point is used as the corrected edge and corner data.
3. The corner positioning method according to claim 2, characterized in that: The first corner data also includes a first corner point; the second corner data also includes a second corner point; step S27 also includes: S276: Determine, according to the edge line segment, a point in the edge line segment that is closest to the first edge point as a pre-selected edge point; S277: Determine whether the position difference between the pre-selected edge point and the first edge point is within a second set range; S278: If yes, returning the pre-selected edge point whose position difference is within the second set range as the second edge point; S279: If not, the second edge point is not returned.
4. The corner positioning method according to claim 3, characterized in that: Step S28 specifically calculates the cluster center point of the first edge point and the second edge point as the corrected edge angle data.
5. The corner positioning method according to claim 1, characterized in that: Step S271: further comprising: calculating the edge area enclosed by the edge points in the region image; Step S272 includes: S2721: Find the image center moment corresponding to the maximum edge area; S2722: Calculate the distance between the image center moment and the edge line segment, merge the edges within a certain distance range, and eliminate discontinuous edges; S2723: Determine pre-selected corner points within the region image based on edge line segments that eliminate discontinuous edges.
6. The corner positioning method according to any one of claims 1 to 5, characterized in that: Also includes: S29: Outputting the tilt angle of the subject to be detected according to the corrected edge angle data.
7. The corner positioning method according to claim 6, characterized in that: Step S25 includes: S251: input the real-time image into the trained neural network model; S252: Determine whether the first corner data is obtained; S253: If yes, output the first corner data of the workpiece to be inspected; S254: If not, enhance the real-time image, input the trained neural network model, and output the first corner data.
8. The corner positioning method according to claim 7, characterized in that: Methods for enhancing real-time images include: brightness enhancement, HSV color space enhancement, up / down / horizontal inversion enhancement, and Gaussian Laplace enhancement.
9. A corner positioning device based on a neural network and a detection algorithm, characterized in that: The corner positioning device is used to implement the corner positioning method according to any one of claims 1 to 8, comprising: The sample collection module is used to collect sample images and annotate the corner data of the sample images to construct a training sample set; A small model building module for building neural network models for learning edge data; The model training module is used to input the training sample set into the neural network model to obtain the trained neural network model; An image acquisition module is used to acquire real-time images of the subject to be detected; The first corner data output module is used to input the real-time image into the trained neural network model and output the first corner data of the workpiece to be detected; An image capture module, used to capture a region image including the first corner data in the real-time image according to the first corner data; The second corner data output module is used to detect the regional image and obtain the second corner data of the workpiece to be detected; The correction module is used to correct and fuse the second corner data with the first corner data to obtain the corrected corner data as the corner positioning output result.
Citation Information
Patent Citations
Welding spot image detection method based on in-depth learning
CN107610111A
Picture book corner positioning method and device, computer device and readable storage medium
CN111401266A
Steel plate corner point and edge point positioning method, workpiece grabbing method and production line
CN111768369A
Target Detection Method, Training Method, Electronic Device, and Computer-Readable Medium
US20210256680A1
