Mobile visual inspection device and related computer program
Through the combination of mobile vision detection devices and deep convolution generation adversarial network, the efficient and accurate problems of intelligent vision detection in 3C product manufacturing are solved, rapid adaptability and high-precision detection are achieved, and application needs of higher standards are met. .
Patent Information
- Application Number
- CN202411064714.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-14
- Publication Date
- 2024-11-19
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The prior art is difficult to achieve efficient and accurate intelligent visual inspection in the manufacturing process of 3C products, especially when there is insufficient defect data and difficult to match the production line, and there is a lack of effective multi-scene image acquisition and high-precision appearance detection solutions.
Using mobile vision detection devices, including AVG mobile devices, robotic arms, image data acquisition equipment, 5G industrial CPE and intelligent control equipment, to achieve rapid transfer and adaptation through small sample learning, establish a visual detection reference database, and generate using deep convolution High-precision training for adversarial networks, combined with 5G networks for real-time data transmission and self-learning updates.
It realizes efficient and accurate product appearance defect detection, can quickly adapt to higher standard application scenarios, improves the reliability and applicability of inspection, and reduces the difficulty of matching the production line.
Smart Images

Figure CN118980685A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of electronic component detection, and in particular to a mobile visual detection device and a related computer program for implementing intelligent detection on a 3C product manufacturing production line. Background Art
[0002] In the process of producing 3C products, there are clear standards for their appearance, size, and defects. For example, the appearance scratches, size and height, and the presence of defects such as solder feet, collapse, and missing are all specific quantitative requirements. Optimizing the process is the only way to make 3C products meet the standards of appearance, size, and defects, and visual inspection of 3C products is the only way to filter out the outflow of products that do not meet the standards. Existing technology can implement appearance screening inspections through two methods: manual inspection and intelligent inspection.
[0003] Compared with manual inspection, appearance screening by intelligent visual inspection system is more convenient and labor-saving, and it has the advantages of convenience, speed and reliable results. Therefore, in the manufacturing field of large and complex structural parts involving aircraft and automobile mainframes, engines, components, etc., appearance screening is often carried out through intelligent visual inspection systems.
[0004] The existing production lines are mostly modular in design, and can quickly switch to produce different models or even different types of 3C products, which will lead to higher requirements for the update of the database and execution standards of the intelligent visual inspection system. The industry has not yet proposed an effective inspection solution to achieve multi-scene image acquisition and high-precision appearance inspection in response to industry problems such as insufficient industrial product defect data, difficulty in matching production lines, and poor replicability of solutions. Summary of the invention
[0005] The technical problem to be solved by the present invention is to provide a mobile visual inspection method, system and storage medium in response to the above-mentioned defects of the prior art, which can not only perform intelligent visual inspection efficiently and accurately to check product appearance defects, but also achieve rapid transfer and adaptation based on small sample learning, effectively meeting higher standard application scenarios.
[0006] The technical solution adopted by the present invention to solve the technical problem is as follows:
[0007] A mobile visual inspection method, the method is based on a mobile visual inspection device, the device at least includes an AVG mobile device, a mechanical arm, an image data acquisition device, a 5G industrial CPE and an intelligent control device, the AVG mobile device is used to control the mobile visual inspection device to move within a user-defined area, the mechanical arm is used to adjust the position of the image data acquisition device so that the image data acquisition device is aligned with the inspection area, the image data acquisition device is used to collect image data, the 5G industrial CPE is used to perform data transmission, and the intelligent control device is used to realize the overall control function, the method includes the following steps:
[0008] S1. Acquire a three-dimensional coordinate system of a user-defined area and position data of a user-defined area, wherein the position data of the user-defined area includes a coordinate set of a moving area and a coordinate set of a detection area;
[0009] S2. Obtain the three-dimensional coordinate system of the robot arm, perform coordinate system mapping according to the three-dimensional coordinate system of the user-defined area and the three-dimensional coordinate system of the robot arm, and obtain the coordinate conversion relationship between the three-dimensional coordinate system of the user-defined area and the three-dimensional coordinate system of the robot arm;
[0010] S3. Control the movement of the AVG mobile device according to the moving area coordinate set, control the mechanical arm to adjust the position according to the detection area coordinate set, control the image data acquisition device to collect actual image sample data, and annotate the actual image sample data to obtain the actual appearance data, actual size data and actual defect data of the corresponding product;
[0011] S4. Obtaining preset ideal image sample data, annotating the ideal image sample data to obtain the ideal appearance data, ideal size data and ideal defect data of the corresponding product;
[0012] S5. Perform small sample high-precision training based on actual appearance data, actual size data, actual defect data and ideal appearance data, ideal size data, and ideal defect data to establish a visual inspection reference database, wherein the visual inspection reference database includes appearance reference data, size reference data, and defect reference data;
[0013] S6. Control the movement of the AVG mobile device according to the moving area coordinate set, control the mechanical arm to adjust the position according to the detection area coordinate set, control the image data acquisition device to perform real-time product quality inspection, and collect real-time visual inspection data, wherein the real-time visual inspection data includes real-time appearance data, real-time size data, and real-time defect data;
[0014] S7. Compare the appearance reference data with the appearance real-time data to obtain the appearance inspection data, compare the size reference data with the size real-time data to obtain the size inspection data, compare the defect reference data with the defect real-time data to obtain the defect inspection data, and analyze the visual inspection result according to the appearance inspection data, the size inspection data and the defect inspection data, wherein the visual inspection result is one of the following: inspection passed, inspection failed or unable to be determined;
[0015] S8. When the visual inspection result is unable to judge, output the manual auxiliary prompt, and receive the manual inspection result input from the outside, the manual inspection result is one of the three: pass, fail or unable to judge;
[0016] S9. Perform self-learning update on the visual inspection reference database according to the manual inspection results;
[0017] S10. Control the 5G industrial CPE to transmit the detection results to the central server.
[0018] Compared with the existing technology, the beneficial effects of this technical solution are: a visual inspection reference database is obtained by training with actual image sample data and ideal image sample data, and then the real-time visual inspection data is analyzed according to the visual inspection reference database to perform visual inspection on the product, check for product appearance defects, and perform self-learning and updating after the inspection is completed, so as to achieve rapid transfer and rapid adaptation, and effectively meet higher standard application scenarios.
[0019] Further, in step S2, coordinate system mapping is performed according to the three-dimensional coordinate system of the user-defined area and the three-dimensional coordinate system of the robotic arm, and the coordinate conversion relationship between the three-dimensional coordinate system of the user-defined area and the three-dimensional coordinate system of the robotic arm is obtained, which specifically includes:
[0020] S201. Determine the origin and coordinate axis direction of the three-dimensional coordinate system of the user-defined area, obtain the three-dimensional point cloud data of the user-defined area, and convert the three-dimensional point cloud data of the user-defined area into coordinate values in the three-dimensional coordinate system of the user-defined area;
[0021] S202. Determine the origin and coordinate axis direction of the three-dimensional coordinate system of the robot arm, obtain the coordinate value of the end of the robot arm, and convert the coordinate value of the end of the robot arm into the coordinate value of the three-dimensional coordinate system of the robot arm;
[0022] S203. Select a calibration point with identification features, obtain the coordinate value (X1, Y1, Z1) of the calibration point in the three-dimensional coordinate system of the user-defined area and the coordinate value (X2, Y2, Z2) of the calibration point in the three-dimensional coordinate system of the robot arm, and pair the coordinate value (X1, Y1, Z1) in the three-dimensional coordinate system of the user-defined area and the coordinate value (X2, Y2, Z2) in the three-dimensional coordinate system of the robot arm;
[0023] S204. According to the coordinate values (X1, Y1, Z1) of the same calibration point in the three-dimensional coordinate system of the user-defined area and the coordinate values (X2, Y2, Z2) of the three-dimensional coordinate system of the robotic arm after pairing, calculate the translation vector T between the three-dimensional coordinate system of the user-defined area and the three-dimensional coordinate system of the robotic arm, where:
[0024]
[0025] S205. According to the coordinate values (X1, Y1, Z1) of the same calibration point in the three-dimensional coordinate system of the user-defined area and the coordinate values (X2, Y2, Z2) of the three-dimensional coordinate system of the robotic arm after pairing, the rotation matrix R between the three-dimensional coordinate system of the user-defined area and the three-dimensional coordinate system of the robotic arm is calculated, wherein the unit rotation matrix I is used as the rotation matrix R, then:
[0026]
[0027] S206. Calculate the coordinate transformation matrix M according to the translation vector T and the rotation matrix R, where:
[0028]
[0029] The beneficial effect of adopting the above scheme is: after selecting calibration points with identification features, the three-dimensional coordinate system of the user-defined area and the three-dimensional coordinate system of the robotic arm are paired based on the coordinate values of the calibration points in the three-dimensional coordinate system of the user-defined area and the three-dimensional coordinate system of the robotic arm, and the translation vector T, rotation matrix R and coordinate transformation matrix M are analyzed to facilitate data exchange between the two three-dimensional coordinate systems. By knowing the coordinate value of a point in a certain coordinate system, the coordinate value of the point in another coordinate system can be obtained.
[0030] Further, in step S5, small sample high-precision training is performed to establish a visual inspection reference database, specifically including:
[0031] S501. Construct a GAN model based on a deep convolutional generative adversarial network, and configure a generator and a discriminator in the GAN model; wherein the generator is used to generate fake samples similar to real samples, and the discriminator is used to distinguish whether the input sample is real or generated;
[0032] S502. Performing data fusion based on actual appearance data, actual size data, actual defect data and ideal appearance data, ideal size data, and ideal defect data to obtain an expanded data set;
[0033] S503. Performing data enhancement on the expanded data set by image rotation, image flipping, image scaling, and image translation, and performing image preprocessing on the expanded data set after data enhancement, wherein the image preprocessing specifically includes image cleaning operation, feature labeling operation, and normalization operation;
[0034] S504. Perform GAN model training, control the generator to generate fake samples based on the expanded data set after image preprocessing, control the discriminator to identify the fake samples generated by the generator and output the judgment result, and further perform feature distribution analysis on the actual appearance data, actual size data, actual defect data and the ideal appearance data, ideal size data, and ideal defect data based on the judgment result of the discriminator to obtain appearance reference data, size reference data, and defect reference data.
[0035] The beneficial effect of adopting the above scheme is: by building a GAN model configuration generator and discriminator based on the adversarial network, the distribution characteristics can be better analyzed from the actual appearance data, actual size data, actual defect data and ideal appearance data, ideal size data, ideal defect data, etc. of the small sample, thereby improving the data reliability of the appearance reference data, size reference data and defect reference data, and improving the small sample training results.
[0036] Further, in step S6, controlling the movement of the AVG mobile device according to the moving area coordinate set specifically includes:
[0037] S601. Construct an AVG moving map according to the moving area coordinate set, and perform AVG path planning based on the AVG moving map to obtain an AVG planned path;
[0038] S602. Dynamically obtain the AVG real-time coordinates of the AVG mobile device (X V1 , Y V1 );
[0039] S603. Match the real-time coordinates of the AVG mobile device to the AVG planned path, and obtain the coordinates of the planned path points (X V2 , Y V2 );
[0040] S604. Based on AVG real-time coordinates (X V1 , Y V1 ) and the coordinates of the planned path points (X V2 , Y V2 ), calculate the distance d between the two, where:
[0041]
[0042] S605. Traverse to find the nearest neighbor d min , calculate the coordinates of the nearest neighbor point (X Vi , YVi ),in:
[0043]
[0044] Specifically, n is the number of points on the AVG planning path;
[0045] S606. According to the AVG real-time coordinates (X V1 , Y V1 ) and the coordinates of the nearest neighbor (X Vi , Y Vi ) generating an AVG control instruction, and controlling the movement of the AVG mobile device according to the AVG control instruction;
[0046] In step S6, controlling the robot arm to adjust the position according to the detection area coordinate set specifically includes:
[0047] S611. Analyze the end path of the robot arm according to the detection area coordinate set;
[0048] S612. Determine the target coordinates (X) of the end of the robot arm based on the end path of the robot arm Mi , Y Mi , Z Mi ) and the target posture of the end of the robot arm;
[0049] S613. Get the real-time coordinates of the end of the robot arm (X M1 , Y M2 , Z M2 ) and the real-time posture of the end of the robotic arm;
[0050] S614. Based on the target coordinates of the end of the robot arm, the target posture of the end of the robot arm, the real-time coordinates of the end of the robot arm and the real-time posture of the end of the robot arm, inverse kinematics calculation is performed to obtain the robot arm joint control data Θ t =IK(P t ,R t ), where Θ t is the joint angle vector control data, IK is the inverse kinematics function, P t is the target coordinate of the end of the robot arm, R t is the joint rotation matrix, specifically: P t =(X t ,Y t ,Z t ), X t =l1cos(θ1)+l2cos(θ1+θ2), Y t =l1sin(θ1)+l2sin(θ1+θ2), R t =R z (θ1)·R z (θ2), Z tis the target value of the Z axis, l1 is the length of the first connecting rod in the robot arm, l2 is the length of the second connecting rod in the robot arm, θ1 is the deflection angle of the first connecting rod in the robot arm relative to the Z axis, θ2 is the deflection angle of the second connecting rod in the robot arm relative to the Z axis, R z (θ1) is the rotation matrix of the angle θ1 around the Z axis, R z (θ2) is the rotation matrix of the angle θ2 around the Z axis;
[0051] S615. Control data Θ according to joint angle vector t Control the robotic arm to adjust its position.
[0052] The beneficial effect of adopting the above scheme is: ensuring that in the process of controlling the movement of the AVG mobile device and controlling the position adjustment of the mechanical arm, both the AVG mobile device and the mechanical arm can be in the optimal movement path, ensuring that the detection process is efficient and safe.
[0053] Further, in step S9, the self-learning update of the visual inspection reference database according to the manual inspection result specifically includes:
[0054] S901. Obtaining actual image sample data corresponding to the manual detection result;
[0055] S902. Perform denoising preprocessing on the actual image sample data;
[0056] S903. Perform feature extraction on the actual image sample data after denoising preprocessing to obtain appearance feature data, size feature data and actual feature data;
[0057] S904. Perform algorithm optimization on the visual inspection reference database according to the appearance feature data, the size feature data and the actual feature data to obtain optimized appearance reference data, size reference data and defect reference data.
[0058] The beneficial effect of adopting the above scheme is: when intelligent visual detection is unable to make a judgment, manual intervention inspection is carried out, and the visual detection reference database is self-learned and updated after obtaining the manual detection results, which can improve the reliability and applicability of this technical solution.
[0059] Correspondingly, a mobile visual inspection system is provided, the system is based on a mobile visual inspection device, the device at least includes an AVG mobile device, a mechanical arm, an image data acquisition device, a 5G industrial CPE and an intelligent control device, the AVG mobile device is used to control the mobile visual inspection device to move within a user-defined area, the mechanical arm is used to adjust the position of the image data acquisition device so that the image data acquisition device is aligned with the inspection area, the image data acquisition device is used to collect image data, the 5G industrial CPE is used to perform data transmission, the intelligent control device is used to realize the overall control function, and the system includes:
[0060] A position data acquisition module, used to acquire a three-dimensional coordinate system of a user-defined area and position data of the user-defined area, wherein the position data of the user-defined area includes a moving area coordinate set and a detection area coordinate set;
[0061] A coordinate system mapping module is used to obtain the three-dimensional coordinate system of the robot arm, perform coordinate system mapping according to the three-dimensional coordinate system of the user-defined area and the three-dimensional coordinate system of the robot arm, and obtain the coordinate conversion relationship between the three-dimensional coordinate system of the user-defined area and the three-dimensional coordinate system of the robot arm;
[0062] The first sample acquisition module is used to control the movement of the AVG mobile device according to the moving area coordinate set, control the mechanical arm to adjust the position according to the detection area coordinate set, control the image data acquisition device to collect actual image sample data, and annotate the actual image sample data to obtain the actual appearance data, actual size data and actual defect data of the corresponding product;
[0063] The second sample acquisition module is used to obtain preset ideal image sample data, and annotate the ideal image sample data to obtain ideal appearance data, ideal size data and ideal defect data of the corresponding product;
[0064] A database establishment module is used to perform small sample high-precision training based on actual appearance data, actual size data, actual defect data and ideal appearance data, ideal size data, and ideal defect data to establish a visual inspection reference database, wherein the visual inspection reference database includes appearance reference data, size reference data, and defect reference data;
[0065] An image acquisition module is used to control the movement of the AVG mobile device according to the moving area coordinate set, control the mechanical arm to adjust the position according to the detection area coordinate set, control the image data acquisition device to perform real-time product quality detection, and collect visual detection real-time data, wherein the visual detection real-time data includes appearance real-time data, size real-time data, and defect real-time data;
[0066] A visual inspection module is used to compare the appearance reference data with the appearance real-time data to obtain the appearance inspection data, compare the size reference data with the size real-time data to obtain the size inspection data, compare the defect reference data with the defect real-time data to obtain the defect detection data, and analyze the visual inspection result according to the appearance inspection data, the size inspection data and the defect detection data, wherein the visual inspection result is one of the following three: inspection passed, inspection failed or unable to be determined;
[0067] The auxiliary judgment module is used to output a manual auxiliary prompt when the visual inspection result is unable to judge, and receive the manual inspection result input from the outside, wherein the manual inspection result is one of the following three: the inspection is passed, the inspection is failed, or the inspection is unable to judge;
[0068] A self-learning module, used for self-learning and updating the visual inspection reference database according to manual inspection results;
[0069] The data transmission module is used to control the 5G industrial CPE to transmit the detection results to the central server.
[0070] Furthermore, the coordinate system mapping module specifically includes:
[0071] A first coordinate value conversion unit is used to determine the origin and coordinate axis directions of the three-dimensional coordinate system of the user-defined area, obtain the three-dimensional point cloud data of the user-defined area, and convert the three-dimensional point cloud data of the user-defined area into coordinate values in the three-dimensional coordinate system of the user-defined area;
[0072] A second coordinate value conversion unit is used to determine the origin and coordinate axis direction of the three-dimensional coordinate system of the robot arm, obtain the coordinate value of the execution end of the robot arm, and convert the coordinate value of the execution end of the robot arm into the coordinate value of the three-dimensional coordinate system of the robot arm;
[0073] A calibration point pairing unit is used to select a calibration point with an identification feature, obtain the coordinate value (X1, Y1, Z1) of the calibration point in the three-dimensional coordinate system of the user-defined area and the coordinate value (X2, Y2, Z2) of the calibration point in the three-dimensional coordinate system of the robot arm, and pair the coordinate value (X1, Y1, Z1) in the three-dimensional coordinate system of the user-defined area and the coordinate value (X2, Y2, Z2) in the three-dimensional coordinate system of the robot arm;
[0074] The translation vector analysis unit is used to calculate the translation vector T between the three-dimensional coordinate system of the user-defined area and the three-dimensional coordinate system of the robotic arm according to the coordinate values (X1, Y1, Z1) of the same calibration point in the three-dimensional coordinate system of the user-defined area and the coordinate values (X2, Y2, Z2) of the three-dimensional coordinate system of the robotic arm after pairing; the translation vector T describes the position of the origin of the three-dimensional coordinate system of the robotic arm in the three-dimensional coordinate system of the user-defined area, where:
[0075]
[0076] The rotation matrix calculation unit is used to calculate the rotation matrix R between the user-defined area three-dimensional coordinate system and the robotic arm three-dimensional coordinate system according to the coordinate values (X1, Y1, Z1) of the same calibration point in the user-defined area three-dimensional coordinate system and the coordinate values (X2, Y2, Z2) in the robotic arm three-dimensional coordinate system after pairing, wherein the unit rotation matrix I is used as the rotation matrix R, then:
[0077]
[0078] The transformation matrix analysis unit is used to calculate the coordinate transformation matrix M according to the translation vector T and the rotation matrix R, where:
[0079]
[0080] Furthermore, the database establishment module specifically includes:
[0081] A model building unit, used to build a GAN model based on a deep convolutional generative adversarial network, and configure the generator and discriminator in the GAN model; wherein the generator is used to generate fake samples similar to real samples, and the discriminator is used to distinguish whether the input sample is real or generated;
[0082] A data fusion unit, used for fusing the actual appearance data, the actual size data, the actual defect data and the ideal appearance data, the ideal size data, the ideal defect data to obtain an expanded data set;
[0083] An image preprocessing unit, used to perform data enhancement on the expanded data set by image rotation, image flipping, image scaling, and image translation, and perform image preprocessing on the expanded data set after data enhancement, wherein the image preprocessing specifically includes image cleaning operation, feature labeling operation, and normalization operation;
[0084] A model training unit is used to perform GAN model training, control the generator to generate fake samples based on the image preprocessed and expanded data set, control the discriminator to identify the fake samples generated by the generator and output a judgment result, and further perform feature distribution analysis on the actual appearance data, actual size data, actual defect data and the ideal appearance data, ideal size data, and ideal defect data according to the judgment result of the discriminator to obtain appearance reference data, size reference data, and defect reference data;
[0085] The self-learning module specifically includes:
[0086] A manual detection result acquisition unit, used to acquire actual image sample data corresponding to the manual detection result;
[0087] A preprocessing unit, used for performing denoising preprocessing on actual image sample data;
[0088] A feature extraction unit is used to extract features from the actual image sample data after denoising preprocessing to obtain appearance feature data, size feature data and actual feature data;
[0089] The algorithm optimization unit is used to perform algorithm optimization on the visual inspection reference database according to the appearance feature data, the size feature data and the actual feature data to obtain optimized appearance reference data, size reference data and defect reference data.
[0090] Furthermore, the image acquisition module includes an AVG control module and a robotic arm control module:
[0091] The AVG control module specifically includes:
[0092] An AVG map construction unit is used to construct an AVG moving map according to a moving area coordinate set, and perform AVG path planning based on the AVG moving map to obtain an AVG planned path;
[0093] AVG positioning unit, used to dynamically obtain the AVG real-time coordinates of the AVG mobile device (X V1 , Y V1 );
[0094] AVG path planning unit, used to match the real-time coordinates of the AVG mobile device to the AVG planned path, and obtain the coordinates of the planned path points (X V2 , Y V2 );
[0095] Distance calculation unit, used to calculate the distance based on the AVG real-time coordinates (X V1 , Y V1 ) and the coordinates of the planned path points (X V2 , Y V2 ), calculate the distance d between the two, where:
[0096]
[0097] The nearest neighbor analysis unit is used to traverse and find the nearest neighbor d min , calculate the coordinates of the nearest neighbor point (X Vi , Y Vi ),in:
[0098]
[0099] Specifically, n is the number of points on the AVG planning path;
[0100] AVG driving unit, used to calculate the real-time coordinates of AVG (X V1 , YV1 ) and the coordinates of the nearest neighbor (X Vi , Y Vi ) generating an AVG control instruction, and controlling the movement of the AVG mobile device according to the AVG control instruction;
[0101] The robot arm control module specifically includes:
[0102] A robot arm path analysis unit, used for analyzing the end path of the robot arm according to the detection area coordinate set;
[0103] The end target pose analysis unit is used to determine the end target coordinates (X Mi , Y Mi , Z Mi ) and the target posture of the end of the robot arm;
[0104] The inverse kinematics analysis unit is used to perform inverse kinematics calculation based on the target coordinates of the end of the robot arm, the target posture of the end of the robot arm, the real-time coordinates of the end of the robot arm, and the real-time posture of the end of the robot arm to obtain the robot arm joint control data θ t =IK(P t ,R t ), where Θ t is the joint angle vector control data, IK is the inverse kinematics function, P t is the target coordinate of the end of the robot arm, R t is the joint rotation matrix, specifically: P t =(X t ,Y t ,Z t ), X t =l1 cos(θ1)+l2cos(θ1+θ2), Y t =l1sin(θ1)+l2sin(θ1+θ2), R t =R z (θ1)·R z (θ2), Z t is the target value of the Z axis, l1 is the length of the first connecting rod in the robot arm, l2 is the length of the second connecting rod in the robot arm, θ1 is the deflection angle of the first connecting rod in the robot arm relative to the Z axis, θ2 is the deflection angle of the second connecting rod in the robot arm relative to the Z axis, R z (θ1) is the rotation matrix of the angle θ1 around the Z axis, R z (θ2) is the rotation matrix of the angle θ2 around the Z axis;
[0105] The robot drive unit is used to control the data Θ according to the joint angle vector t Control the robotic arm to adjust its position.
[0106] Correspondingly, a storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the mobile visual inspection method as described above. BRIEF DESCRIPTION OF THE DRAWINGS
[0107] Figure 1 It is a schematic flow chart of the mobile visual inspection method of the present invention.
[0108] Figure 2 It is a structural schematic diagram of the mobile visual inspection system of the present invention.
[0109] In the figure, the components represented by the reference numerals are shown as follows:
[0110] Position data acquisition module 1, coordinate system mapping module 2, first sample acquisition module 3, second sample acquisition module 4, database establishment module 5, image acquisition module 6, visual detection module 7, auxiliary judgment module 8, self-learning module 9, data transmission module 10. DETAILED DESCRIPTION
[0111] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0112] In the description of the present invention, it should be understood that the orientations or positional relationships indicated by the terms "center", "upper", "lower", "front", "back", "left", "right", etc. are based on the orientations or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or component referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.
[0113] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two components. When a component is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other component or there can also be a central component. When a component is considered to be "connected" to another element, it can be directly connected to the other element or there can be a central element at the same time. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0114] In the process of producing 3C products, there are clear standards for their appearance, size, and defects. For example, the appearance scratches, size and height, and the presence of defects such as solder feet, collapse, and missing are all specific quantitative requirements. Optimizing the process is the only way to make 3C products meet the standards of appearance, size, and defects, and visual inspection of 3C products is the only way to filter out the outflow of products that do not meet the standards. Existing technology can implement appearance screening inspections through two methods: manual inspection and intelligent inspection.
[0115] Compared with manual inspection, appearance screening by intelligent visual inspection system is more convenient and labor-saving, and it has the advantages of convenience, speed and reliable results. Therefore, in the manufacturing field of large and complex structural parts involving aircraft and automobile mainframes, engines, components, etc., appearance screening is often carried out through intelligent visual inspection systems.
[0116] The existing production lines are mostly modular in design, and can quickly switch to produce different models or even different types of 3C products, which will lead to higher requirements for the update of the database and execution standards of the intelligent visual inspection system. The industry has not yet proposed an effective inspection solution to achieve multi-scene image acquisition and high-precision appearance inspection in response to industry problems such as insufficient industrial product defect data, difficulty in matching production lines, and poor replicability of solutions.
[0117] In order to solve the above technical problems, the technical solution of the present application is based on a mobile visual inspection device and proposes a mobile visual inspection method, system and storage medium.
[0118] The mobile visual inspection device at least includes an AVG mobile device, a robotic arm, an image data acquisition device, a 5G industrial CPE and an intelligent control device. Specifically, the AVG mobile device is used to control the movement of the mobile visual inspection device within a user-defined area; the robotic arm is used to adjust the position of the image data acquisition device so that the image data acquisition device is aligned with the inspection area; the image data acquisition device is used to collect image data, such as collecting actual image sample data, to mark the actual appearance data, actual size data and actual defect data of the corresponding product; the 5G industrial CPE is used for data transmission, and the intelligent control device is used to realize the overall control function. In this technical solution, the moving area can be understood as the area where the production line is located, and the inspection area can be understood as the area where the product is placed for appearance inspection.
[0119] The working process of the mobile visual inspection device is as follows: first, a command is issued to control the AVG mobile device to move the mobile visual inspection device as a whole to the user-defined area, and then the mechanical arm is controlled to adjust the angle so that the image data acquisition device can collect the image data of the product to be inspected. When performing appearance inspection, the image data acquisition device collects the image data and then processes and analyzes the data through the intelligent control device, and obtains the inspection results to determine whether the product is qualified. In the above process, the data collected, processed and generated during the working process are uploaded to the quality information management system through the 5G industrial CPE to facilitate subsequent product quality optimization.
[0120] like Figure 1 As shown, a mobile visual inspection method comprises the following steps:
[0121] S1. Obtain a user-defined area three-dimensional coordinate system and user-defined area position data, wherein the user-defined area position data includes a moving area coordinate set and a detection area coordinate set. In step S1, the user-defined area three-dimensional coordinate system refers to the three-dimensional coordinate system used by the production line in the industrial control process, and the user-defined area position data represents the area position data built and defined for adaptive design of the production line in order to produce specific products under the user-defined area three-dimensional coordinate system, specifically including a moving area coordinate set and a detection area coordinate set. Among them, the moving area coordinate set can be understood as the range within which the mobile visual inspection device can move on the production line, and the detection area coordinate set can be understood as the area where the product is placed when the appearance inspection is to be performed. After obtaining the moving area coordinate set and the detection area coordinate set, the position of the production line can be grasped for movement and detection.
[0122] S2. Obtain the three-dimensional coordinate system of the robotic arm, perform coordinate system mapping based on the three-dimensional coordinate system of the user-defined area and the three-dimensional coordinate system of the robotic arm, and obtain the coordinate conversion relationship between the three-dimensional coordinate system of the user-defined area and the three-dimensional coordinate system of the robotic arm. The robotic arm is an independent system with an independent control system and coordinate system. In step S2, first obtain the preset three-dimensional coordinate system of the robotic arm, and then further perform coordinate system mapping based on the three-dimensional coordinate system of the user-defined area and the three-dimensional coordinate system of the robotic arm, so as to obtain the coordinate conversion relationship between the two. After completing step S2, the corresponding data in the three-dimensional coordinate system of the robotic arm can be obtained through the specific coordinate points on the three-dimensional coordinate system of the user-defined area, which is essential for operating the robotic arm.
[0123] S3. Control the movement of the AVG mobile device according to the moving area coordinate set, control the mechanical arm to adjust the position according to the detection area coordinate set, control the image data acquisition device to collect actual image sample data, and annotate the actual image sample data to obtain the actual appearance data, actual size data and actual defect data of the corresponding product. The technical purpose of step S3 is to control the AVG mobile device to move to a specific area according to the actual layout of the production line, so that the image data acquisition device can be aimed at the product to be detected, so as to facilitate the actual image sample data collection, that is, to take pictures or videos of the product to be detected to obtain the actual image sample data. Based on this, the actual appearance data, actual size data and actual defect data of the corresponding product can be obtained through image processing technology and product design specifications. For example, whether there are scratches and grooves on the appearance of the product to be detected, how much is the specific data of its length, width and height, whether there are defects such as empty weld collapse angle, etc. The product targeted by this step has been clearly identified as having specific problems in appearance detection, so the actual presentation status of specific problems on the production line can be reflected through actual appearance data, actual size data and actual defect data. It should be noted that step S3 is not a normal detection step, but a data preparation before the detection to adapt to the new production line before implementing the present technical solution.
[0124] S4. Obtain the preset ideal image sample data, and annotate the ideal image sample data to obtain the ideal appearance data, ideal size data, and ideal defect data of the corresponding product. In step S4, the ideal image sample data can be understood as the most representative image selected during the design process or the production process, and then annotated based on this specific graphic to obtain the ideal appearance data, ideal size data, and ideal defect data of the corresponding product. The difference between step S4 and step S3 lies in the different data sources. Step S4 is aimed at the most representative products selected, whether in appearance, size or defects, they are all ideal and very easy to identify; step S3 is aimed at the products selected before the small sample training, some of which may be in a marginal state and difficult to judge, and some may be easier to judge. Collecting training samples through steps S3 and S4 can fully reflect the situations encountered in the actual production process. The larger the number of samples, the more comprehensive the coverage.
[0125] S5. Perform small sample high-precision training based on actual appearance data, actual size data, actual defect data and ideal appearance data, ideal size data, and ideal defect data, and establish a visual inspection reference database, which includes appearance reference data, size reference data, and defect reference data. After the data preparation of steps S3 and S4, perform small sample high-precision training based on the data at two levels of actual state and ideal state and establish a visual inspection reference database. According to this visual inspection reference database, appearance reference data, size reference data, and defect reference data can be retrieved, wherein the appearance reference data gives a quantitative judgment interval for whether the appearance of the product is qualified, such as 3 or less scratches on a certain side are acceptable; the size reference data gives a quantitative judgment interval for whether the size of the product is qualified, such as the product thickness is acceptable at 3.20mm-3.24mm; the defect reference data gives a quantitative judgment interval for the defect of the product, such as the length of the crack is acceptable below 300um. In general, this is a data quantification standard for judging the images collected during the inspection process. Before step S5, it is the preliminary preparation work for visual inspection.
[0126] S6. Control the movement of the AVG mobile device according to the coordinate set of the moving area, control the mechanical arm to adjust its position according to the coordinate set of the detection area, control the image data acquisition device to perform real-time product quality detection, and collect visual detection real-time data, which includes appearance real-time data, size real-time data, and defect real-time data. Starting from step S6, enter the visual inspection process. Compared with step S3, step S6 is performed in the actual production inspection process, and is also aimed at the assembled products to be inspected. Therefore, the data collected by step S6 are appearance real-time data, size real-time data, and defect real-time data.
[0127] S7. Compare the appearance reference data and the appearance real-time data to obtain the appearance detection data, compare the size reference data and the size real-time data to obtain the size detection data, compare the defect reference data and the defect real-time data to obtain the defect detection data, and analyze the visual detection result according to the appearance detection data, the size detection data and the defect detection data. The visual detection result is one of the three: detection passed, detection failed, or unable to judge. In step S7, the appearance real-time data, size real-time data and defect real-time data collected in step S6 can be used to determine whether a certain product to be inspected is qualified based on the appearance reference data, size reference data and defect reference data. If it meets the requirements, it is determined to pass the inspection, and if it does not meet the requirements, it is determined to fail the inspection. When an extreme "pressing line" situation occurs, the feedback is that it cannot be judged, so as to remind the operator to perform manual verification.
[0128] S8. When the visual inspection result is unable to judge, output manual auxiliary prompts, and receive the manual inspection results input from the outside, the manual inspection result is one of the three: passed, failed, or unable to judge. In this step, in addition to passed and failed, the visual inspection results also include "unable to judge", which is one of the innovative points of this technical solution. By feeding back "unable to judge", manual intervention and verification based on this feedback can, on the one hand, screen out defective products to the greatest extent, and more importantly, on the other hand, provide valuable data for subsequent self-learning. The next time a similar situation occurs, a judgment can be made directly based on the manual verification results. Based on this, the more samples tested, the more perfect the process, and the more reliable the results.
[0129] S9. The visual inspection reference database is updated through self-learning according to the manual inspection results. In step S9, the purpose of the self-learning update is to bind the extreme "line pressing" situation with the manual verification results so that when similar situations are encountered in the subsequent production inspection process, relevant judgments can be made quickly and accurately.
[0130] S10. Control the 5G industrial CPE to transmit the test results to the central server. The purpose of step S10 is to share data and upload the data collected, processed and generated during the work process to the quality information management system, which is particularly important for large production lines to facilitate subsequent product quality optimization.
[0131] Based on the above technical solution, the AVG mobile device and the robotic arm are controlled according to the moving area coordinate set and the detection area coordinate set to move the visual inspection device as a whole to the corresponding position on the production line, and adjust the posture of the robotic arm so that the image data acquisition device points to a specific position and collects image data of the product to be inspected, thereby realizing mobile inspection. The visual inspection reference database is then obtained through training with actual image sample data and ideal image sample data. The real-time visual inspection data is analyzed according to the visual inspection reference database to perform visual inspection on the product, check for product appearance defects, and perform self-learning and updating after the inspection is completed, so as to achieve rapid transfer and rapid adaptation, and effectively meet higher standard application scenarios.
[0132] Preferably, in step S2, coordinate system mapping is performed according to the three-dimensional coordinate system of the user-defined area and the three-dimensional coordinate system of the robotic arm, and obtaining the coordinate conversion relationship between the three-dimensional coordinate system of the user-defined area and the three-dimensional coordinate system of the robotic arm specifically includes:
[0133] S201. Determine the origin and coordinate axis direction of the three-dimensional coordinate system of the user-defined area, obtain the three-dimensional point cloud data of the user-defined area, and convert the three-dimensional point cloud data of the user-defined area into coordinate values in the three-dimensional coordinate system of the user-defined area. In step S201, the three-dimensional point cloud data is converted into corresponding coordinate values according to the origin and coordinate axis direction of the three-dimensional coordinate system of the user-defined area. This step is the process of data conversion, which prepares for the subsequent establishment of a mapping relationship between the two coordinate systems.
[0134] S202. Determine the origin and coordinate axis direction of the three-dimensional coordinate system of the robot arm, obtain the coordinate value of the end of the robot arm, and convert the coordinate value of the end of the robot arm into the coordinate value of the three-dimensional coordinate system of the robot arm. In step S202, the coordinate value of the end of the robot arm is obtained according to the origin and coordinate axis direction of the three-dimensional coordinate system of the robot arm. Step S202 is the same as step S201, which is a data conversion process to prepare for the subsequent establishment of a mapping relationship between the two coordinate systems.
[0135] S203. Select calibration points with identification features, obtain the coordinate values (X1, Y1, Z1) of the calibration points in the three-dimensional coordinate system of the user-defined area and the coordinate values (X2, Y2, Z2) in the three-dimensional coordinate system of the robotic arm, and pair the coordinate values (X1, Y1, Z1) in the three-dimensional coordinate system of the user-defined area and the coordinate values (X2, Y2, Z2) in the three-dimensional coordinate system of the robotic arm. In step S203, the identification features refer to points with prominent features in both the three-dimensional coordinate system of the user-defined area and the three-dimensional coordinate system of the robotic arm, such as a fixed point on the production line. After analyzing and obtaining the coordinate values of the calibration points in the three-dimensional coordinate system of the user-defined area and the three-dimensional coordinate system of the robotic arm, pairing is performed between the two, and a mapping relationship analysis can be performed based on the paired coordinate values. It should be noted that in the process of implementing this technical solution, it is recommended to select two or more calibration points. By pairing the coordinate values of the same calibration point in different coordinate systems one by one, the mapping relationship between the two coordinate systems can be further analyzed.
[0136] S204. Calculate the translation vector T between the three-dimensional coordinate system of the user-defined area and the three-dimensional coordinate system of the robotic arm according to the coordinate values (X1, Y1, Z1) of the same calibration point in the three-dimensional coordinate system of the user-defined area and the coordinate values (X2, Y2, Z2) of the robotic arm after pairing; the translation vector T describes the position of the origin of the three-dimensional coordinate system of the robotic arm in the three-dimensional coordinate system of the user-defined area, where:
[0137]
[0138] S205. According to the coordinate values (X1, Y1, Z1) of the same calibration point in the three-dimensional coordinate system of the user-defined area and the coordinate values (X2, Y2, Z2) of the three-dimensional coordinate system of the robotic arm after pairing, the rotation matrix R between the three-dimensional coordinate system of the user-defined area and the three-dimensional coordinate system of the robotic arm is calculated, wherein the unit rotation matrix I is used as the rotation matrix R, then:
[0139]
[0140] S206. Calculate the coordinate transformation matrix M according to the translation vector T and the rotation matrix R, where:
[0141]
[0142] In steps S201-S206, after pairing the coordinate values of the same calibration point in different coordinate systems, the translation vector T and the rotation matrix R are analyzed in turn, and then the coordinate transformation matrix M is obtained based on the translation vector T and the rotation matrix R. Based on the coordinate transformation matrix M, when the coordinate value of a point in the three-dimensional coordinate system of the user-defined area is known, the corresponding coordinate value of the point in the three-dimensional coordinate system of the robot arm can be obtained, and vice versa.
[0143] Preferably, in step S5, performing small sample high-precision training and establishing a visual inspection reference database specifically includes:
[0144] S501. Construct a GAN model based on a deep convolutional generative adversarial network, and configure the generator and discriminator in the GAN model; wherein the generator is used to generate fake samples similar to real samples, and the discriminator is used to distinguish whether the input sample is real or generated. In step S501, a fake sample similar to a real sample is generated by the generator, and the discriminator is used to distinguish whether the input sample is real or generated. Therefore, a GAN model is constructed based on a deep convolutional generative adversarial network. Through the generator and the discriminator, small sample high-precision training can be performed when the number of samples is insufficient, thereby improving the applicability of this technical solution on different production lines.
[0145] S502. Perform data fusion based on actual appearance data, actual size data, actual defect data and ideal appearance data, ideal size data, and ideal defect data to obtain an expanded data set. In step S502, the actual appearance data, actual size data, actual defect data and ideal appearance data, ideal size data, and ideal defect data are integrated to obtain an expanded data set, and different samples are classified and efficiently utilized.
[0146] S503. The expanded data set is enhanced by image rotation, image flipping, image scaling, and image translation, and image preprocessing is performed on the expanded data set after data enhancement, wherein the image preprocessing specifically includes image cleaning operation, feature labeling operation, and normalization operation. The function of step S503 is to strengthen the features in the sample through data enhancement, thereby making full use of the expanded data set with a small number of samples.
[0147] S504. Perform GAN model training, control the generator to generate fake samples based on the expanded data set after image preprocessing, control the discriminator to identify the fake samples generated by the generator and output the judgment result, and further perform feature distribution analysis on the actual appearance data, actual size data, actual defect data and the ideal appearance data, ideal size data, and ideal defect data based on the judgment result of the discriminator to obtain appearance reference data, size reference data, and defect reference data.
[0148] Based on the above technical solution, it is possible to effectively train with less data, better learn the distribution and characteristics of data from small sample data, and thus improve the effect of generating models. At the same time, the data set can be expanded on small sample data, and the diversity of training can be increased, thereby improving the generalization ability of the model and improving the results of small sample training.
[0149] Preferably, in step S6, controlling the movement of the AVG mobile device according to the moving area coordinate set specifically includes:
[0150] S601. Construct an AVG moving map according to the moving area coordinate set, and perform AVG path planning based on the AVG moving map to obtain an AVG planned path. In step S601, an AVG moving map is constructed by using the moving area coordinate set, the production line entity is digitized, and AVG path planning is performed based on the digitized AVG moving map, so that the AVG planned path can be obtained according to the visual inspection requirements.
[0151] S602. Dynamically obtain the AVG real-time coordinates of the AVG mobile device (X V1 , Y V1 ). It is understandable that navigation of the AVG mobile device is inseparable from the real-time position of the AVG mobile device, and the function of step S602 is to obtain the real-time position of the AVG mobile device.
[0152] S603. Match the real-time coordinates of the AVG mobile device to the AVG planned path, and obtain the coordinates of the planned path points (X V2 , Y V2 In step S603, based on the AVG real-time coordinates (X V1 , Y V1 ), combined with the AVG planning path, we can get the coordinates of the planned path points (XV2 , Y V2 ), which is the coordinate point where the AVG mobile device will go next.
[0153] S604. Based on AVG real-time coordinates (X V1 , Y V1 ) and the coordinates of the planned path points (X V2 , Y V2 ), calculate the distance d between the two, where:
[0154]
[0155] S605. Traverse to find the nearest neighbor d min , calculate the coordinates of the nearest neighbor point (X Vi , Y Vi ),in:
[0156]
[0157] Specifically, n is the number of points on the AVG planning path.
[0158] S606. According to the AVG real-time coordinates (X V1 , Y V1 ) and the coordinates of the nearest neighbor (X Vi , Y Vi ) generates an AVG control instruction, and controls the movement of the AVG mobile device according to the AVG control instruction.
[0159] Based on the above technical solution, the path of the AVG mobile device can be divided into multiple small segments during the visual detection process, using the AVG real-time coordinates (X V1 , Y V1 ) is the current point, and the coordinates of the planned path point (X V2 , Y V2 ) is the next step point, controlling the movement of the AVG mobile device so that the AVG mobile device can accurately reach the predetermined position.
[0160] Preferably, in step S6, controlling the mechanical arm to adjust the position according to the detection area coordinate set specifically includes:
[0161] S611. Analyze the end path of the robot arm according to the detection area coordinate set. It can be understood that the detection area is a spatial range, and the role of the robot arm is to enable the image data acquisition device to align with the detection area. The role of step S611 is to reversely obtain the end path of the robot arm through the detection area coordinate set, that is, to calculate the ideal position to be reached by the robot arm given the detection area coordinate set.
[0162] S612. Determine the target coordinates (X) of the end of the robot arm based on the end path of the robot arm Mi, Y Mi , Z Mi ) and the target posture of the end of the robot arm. The robot arm includes multiple joints and ends. In the present technical solution, the image data acquisition device is fixedly arranged on the end. Therefore, the coordinates and posture of the end of the robot arm are extremely important for the position of the image data acquisition device. The function of step S612 is to analyze and determine the target coordinates (X) of the end of the robot arm through the ideal position to be reached by the robot arm. Mi , Y Mi , Z Mi ) and the target posture of the end of the robot arm, so that image data acquisition can be in the ideal position.
[0163] S613. Get the real-time coordinates of the end of the robot arm (X M1 , Y M2 , Z M2 ) and the real-time posture of the end of the robot arm. As in the general navigation process, navigation needs to be performed based on the real-time position and the target position. Step S612 has obtained the target coordinates (X Mi , Y Mi , Z Mi ) and the target posture of the end of the robot arm. The purpose of step 613 is to obtain the real-time coordinates (X M1 , Y M2 , Z M2 ) and the real-time posture of the end of the robotic arm to provide a data basis for the navigation of the robotic arm.
[0164] S614. Based on the target coordinates of the end of the robot arm, the target posture of the end of the robot arm, the real-time coordinates of the end of the robot arm and the real-time posture of the end of the robot arm, inverse kinematics calculation is performed to obtain the robot arm joint control data Θ t =IK(P t ,R t ), where Θ t is the joint angle vector control data, IK is the inverse kinematics function, P t is the target coordinate of the end of the robot arm, R t is the joint rotation matrix, specifically: P t =(X t ,Y t ,Z t ), X t =l1cos(θ1)+l2cos(θ1+θ2), Y t =l1sin(θ1)+l2sin(θ1+θ2), R t =R z (θ1)·R z (θ2), Z tis the target value of the Z axis, l1 is the length of the first connecting rod in the robot arm, l2 is the length of the second connecting rod in the robot arm, θ1 is the deflection angle of the first connecting rod in the robot arm relative to the Z axis, θ2 is the deflection angle of the second connecting rod in the robot arm relative to the Z axis, R z (θ1) is the rotation matrix of the angle θ1 around the Z axis, R z (θ2) is the rotation matrix of the angle θ2 around the Z axis. The robot arm is a complex mechanical structure involving multiple connecting rods, and adjacent connecting rods are connected by joints. To adjust the robot arm to an ideal position and posture, it is necessary to control the joints. The purpose of step S614 is to obtain the robot arm joint control data through inverse kinematics calculation based on the target coordinates of the end of the robot arm, the target posture of the end of the robot arm, the real-time coordinates of the end of the robot arm, and the real-time posture analysis of the end of the robot arm.
[0165] S615. Control data Θ according to joint angle vector t Control the robotic arm to adjust its position.
[0166] Based on the above technical solution, the control data of the robotic arm can be obtained according to the target coordinates of the robotic arm end, the target posture of the robotic arm end, the real-time coordinates of the robotic arm end and the real-time posture analysis of the robotic arm end, and the different connecting rods and joint movements on the robotic arm can be further controlled accordingly, so that the robotic arm reaches the ideal state.
[0167] Preferably, in step S9, performing self-learning updating on the visual inspection reference database according to the manual inspection result specifically includes:
[0168] S901. Obtain the actual image sample data corresponding to the manual inspection result. Only when the visual inspection result is unable to be judged, the system will output a manual assistance prompt to remind the operator to intervene, and at the same time provide the actual image sample data that the system cannot judge, so as to facilitate manual observation and inspection.
[0169] S902. Perform denoising preprocessing on the actual image sample data. The denoising preprocessing in step S902 is a common technique in the field of image processing, and is intended to eliminate noise points in the actual image sample data that may have a negative impact on subsequent analysis results.
[0170] S903. Perform feature extraction on the actual image sample data that has undergone denoising preprocessing to obtain appearance feature data, size feature data, and actual feature data.
[0171] S904. Perform algorithm optimization on the visual inspection reference database according to the appearance feature data, the size feature data and the actual feature data to obtain optimized appearance reference data, size reference data and defect reference data.
[0172] Based on the above technical solution, when intelligent visual inspection is unable to make a judgment, manual intervention inspection is performed, and the visual inspection reference database is self-learned and updated after obtaining the manual inspection results, which can improve the reliability and applicability of this technical solution.
[0173] like Figure 2 As shown, the technical solution also includes a mobile visual inspection system, which includes a position data acquisition module 1, a coordinate system mapping module 2, a first sample acquisition module 3, a second sample acquisition module 4, a database establishment module 5, an image acquisition module 6, a visual inspection module 7, an auxiliary judgment module 8, a self-learning module 9 and a data transmission module 10, wherein:
[0174] Position data acquisition module 1, used to acquire a user-defined area three-dimensional coordinate system and user-defined area position data, wherein the user-defined area position data includes a moving area coordinate set and a detection area coordinate set;
[0175] Coordinate system mapping module 2 is used to obtain the three-dimensional coordinate system of the robot arm, perform coordinate system mapping according to the three-dimensional coordinate system of the user-defined area and the three-dimensional coordinate system of the robot arm, and obtain the coordinate conversion relationship between the three-dimensional coordinate system of the user-defined area and the three-dimensional coordinate system of the robot arm;
[0176] The first sample acquisition module 3 is used to control the movement of the AVG mobile device according to the moving area coordinate set, control the mechanical arm to adjust the position according to the detection area coordinate set, control the image data acquisition device to collect actual image sample data, and annotate the actual image sample data to obtain the actual appearance data, actual size data and actual defect data of the corresponding product;
[0177] The second sample acquisition module 4 is used to obtain preset ideal image sample data, and annotate the ideal image sample data to obtain ideal appearance data, ideal size data and ideal defect data of the corresponding product;
[0178] A database establishment module 5 is used to perform small sample high-precision training based on actual appearance data, actual size data, actual defect data and ideal appearance data, ideal size data, and ideal defect data to establish a visual inspection reference database, wherein the visual inspection reference database includes appearance reference data, size reference data, and defect reference data;
[0179] Image acquisition module 6, used to control the movement of AVG mobile device according to the moving area coordinate set, control the mechanical arm to adjust the position according to the detection area coordinate set, control the image data acquisition device to perform real-time product quality detection, and collect visual detection real-time data, wherein the visual detection real-time data includes appearance real-time data, size real-time data and defect real-time data;
[0180] A visual inspection module 7 is used to compare the appearance reference data with the appearance real-time data to obtain the appearance inspection data, compare the size reference data with the size real-time data to obtain the size inspection data, compare the defect reference data with the defect real-time data to obtain the defect inspection data, and analyze the visual inspection result according to the appearance inspection data, the size inspection data and the defect inspection data, wherein the visual inspection result is one of the following three: inspection passed, inspection failed or unable to be determined;
[0181] The auxiliary judgment module 8 is used to output a manual auxiliary prompt when the visual inspection result is unable to judge, and receive an external manual inspection result, wherein the manual inspection result is one of the following three: passing the inspection, failing the inspection, or unable to judge;
[0182] A self-learning module 9, used for self-learning and updating the visual inspection reference database according to the manual inspection results;
[0183] The data transmission module 10 is used to control the 5G industrial CPE to transmit the detection results to the central server.
[0184] Preferably, the coordinate system mapping module 2 specifically includes:
[0185] A first coordinate value conversion unit is used to determine the origin and coordinate axis directions of the three-dimensional coordinate system of the user-defined area, obtain the three-dimensional point cloud data of the user-defined area, and convert the three-dimensional point cloud data of the user-defined area into coordinate values in the three-dimensional coordinate system of the user-defined area;
[0186] A second coordinate value conversion unit is used to determine the origin and coordinate axis direction of the three-dimensional coordinate system of the robot arm, obtain the coordinate value of the execution end of the robot arm, and convert the coordinate value of the execution end of the robot arm into the coordinate value of the three-dimensional coordinate system of the robot arm;
[0187] A calibration point pairing unit is used to select a calibration point with an identification feature, obtain the coordinate value (X1, Y1, Z1) of the calibration point in the three-dimensional coordinate system of the user-defined area and the coordinate value (X2, Y2, Z2) of the calibration point in the three-dimensional coordinate system of the robot arm, and pair the coordinate value (X1, Y1, Z1) in the three-dimensional coordinate system of the user-defined area and the coordinate value (X2, Y2, Z2) in the three-dimensional coordinate system of the robot arm;
[0188] The translation vector analysis unit is used to calculate the translation vector T between the three-dimensional coordinate system of the user-defined area and the three-dimensional coordinate system of the robotic arm according to the coordinate values (X1, Y1, Z1) of the same calibration point in the three-dimensional coordinate system of the user-defined area and the coordinate values (X2, Y2, Z2) of the three-dimensional coordinate system of the robotic arm after pairing; the translation vector T describes the position of the origin of the three-dimensional coordinate system of the robotic arm in the three-dimensional coordinate system of the user-defined area, where:
[0189]
[0190] The rotation matrix calculation unit is used to calculate the rotation matrix R between the user-defined area three-dimensional coordinate system and the robotic arm three-dimensional coordinate system according to the coordinate values (X1, Y1, Z1) of the same calibration point in the user-defined area three-dimensional coordinate system and the coordinate values (X2, Y2, Z2) in the robotic arm three-dimensional coordinate system after pairing, wherein the unit rotation matrix I is used as the rotation matrix R, then:
[0191]
[0192] The transformation matrix analysis unit is used to calculate the coordinate transformation matrix M according to the translation vector T and the rotation matrix R, where:
[0193]
[0194] Preferably, the database establishment module 5 specifically includes:
[0195] A model building unit, used to build a GAN model based on a deep convolutional generative adversarial network, and configure the generator and discriminator in the GAN model; wherein the generator is used to generate fake samples similar to real samples, and the discriminator is used to distinguish whether the input sample is real or generated;
[0196] A data fusion unit, used for fusing the actual appearance data, the actual size data, the actual defect data and the ideal appearance data, the ideal size data, the ideal defect data to obtain an expanded data set;
[0197] An image preprocessing unit, used to perform data enhancement on the expanded data set by image rotation, image flipping, image scaling, and image translation, and perform image preprocessing on the expanded data set after data enhancement, wherein the image preprocessing specifically includes image cleaning operation, feature labeling operation, and normalization operation;
[0198] A model training unit is used to perform GAN model training, control the generator to generate fake samples based on the image preprocessed and expanded data set, control the discriminator to identify the fake samples generated by the generator and output a judgment result, and further perform feature distribution analysis on the actual appearance data, actual size data, actual defect data and the ideal appearance data, ideal size data, and ideal defect data according to the judgment result of the discriminator to obtain appearance reference data, size reference data, and defect reference data;
[0199] The self-learning module 9 specifically includes:
[0200] A manual detection result acquisition unit, used to acquire actual image sample data corresponding to the manual detection result;
[0201] A preprocessing unit, used for performing denoising preprocessing on actual image sample data;
[0202] A feature extraction unit is used to extract features from the actual image sample data after denoising preprocessing to obtain appearance feature data, size feature data and actual feature data;
[0203] The algorithm optimization unit is used to perform algorithm optimization on the visual inspection reference database according to the appearance feature data, the size feature data and the actual feature data to obtain optimized appearance reference data, size reference data and defect reference data.
[0204] Preferably, the image acquisition module 6 includes an AVG control module and a robotic arm control module:
[0205] The AVG control module specifically includes:
[0206] An AVG map construction unit is used to construct an AVG moving map according to a moving area coordinate set, and perform AVG path planning based on the AVG moving map to obtain an AVG planned path;
[0207] AVG positioning unit, used to dynamically obtain the AVG real-time coordinates of the AVG mobile device (X V1 , Y V1 );
[0208] AVG path planning unit, used to match the real-time coordinates of the AVG mobile device to the AVG planned path, and obtain the coordinates of the planned path points (X V2 , Y V2 );
[0209] Distance calculation unit, used to calculate the distance based on the AVG real-time coordinates (X V1 , Y V1 ) and the coordinates of the planned path points (X V2 , Y V2 ), calculate the distance d between the two, where:
[0210]
[0211] The nearest neighbor analysis unit is used to traverse and find the nearest neighbor d min , calculate the coordinates of the nearest neighbor point (X Vi , Y Vi ),in:
[0212]
[0213] Specifically, n is the number of points on the AVG planning path;
[0214] AVG driving unit, used to calculate the real-time coordinates of AVG (X V1 , YV1 ) and the coordinates of the nearest neighbor (X Vi , Y Vi ) generating an AVG control instruction, and controlling the movement of the AVG mobile device according to the AVG control instruction;
[0215] The robot arm control module specifically includes:
[0216] A robot arm path analysis unit, used for analyzing the end path of the robot arm according to the detection area coordinate set;
[0217] The end target pose analysis unit is used to determine the end target coordinates (X Mi , Y Mi , Z Mi ) and the target posture of the end of the robot arm;
[0218] The inverse kinematics analysis unit is used to perform inverse kinematics calculation based on the target coordinates of the end of the robot arm, the target posture of the end of the robot arm, the real-time coordinates of the end of the robot arm, and the real-time posture of the end of the robot arm to obtain the robot arm joint control data θ t =IK(P t ,R t ), where Θ t is the joint angle vector control data, IK is the inverse kinematics function, P t is the target coordinate of the end of the robot arm, R t is the joint rotation matrix, specifically: P t =(X t ,Y t ,Z t ), X t =l1 cos(θ1)+l2cos(θ1+θ2), Y t =l1sin(θ1)+l2sin(θ1+θ2), R t =R z (θ1)·R z (θ2), Z t is the target value of the Z axis, l1 is the length of the first connecting rod in the robot arm, l2 is the length of the second connecting rod in the robot arm, θ1 is the deflection angle of the first connecting rod in the robot arm relative to the Z axis, θ2 is the deflection angle of the second connecting rod in the robot arm relative to the Z axis, R z (θ1) is the rotation matrix of the angle θ1 around the Z axis, R z (θ2) is the rotation matrix of the angle θ2 around the Z axis;
[0219] The robot drive unit is used to control the data Θ according to the joint angle vector t Control the robotic arm to adjust its position.
[0220] Correspondingly, the technical solution also includes a storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the mobile visual inspection method as described above.
[0221] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.
Claims
1. A mobile visual inspection device, characterized in that: The device at least includes an AVG mobile device, a mechanical arm, an image data acquisition device, an industrial CPE and an intelligent control device. The AVG mobile device is used to control the mobile visual inspection device to move within a user-defined area. The mechanical arm is used to adjust the position of the image data acquisition device so that the image data acquisition device is aligned with the inspection area. The image data acquisition device is used to collect image data. The industrial CPE is used to transmit data. The intelligent control device is used to realize the overall control function.
2. The device according to claim 1, characterized in that The method for executing the device comprises the following steps: S1. Acquire a three-dimensional coordinate system of a user-defined area and position data of a user-defined area, wherein the position data of the user-defined area includes a coordinate set of a moving area and a coordinate set of a detection area; S2. Obtain the three-dimensional coordinate system of the robot arm, perform coordinate system mapping according to the three-dimensional coordinate system of the user-defined area and the three-dimensional coordinate system of the robot arm, and obtain the coordinate conversion relationship between the three-dimensional coordinate system of the user-defined area and the three-dimensional coordinate system of the robot arm; S3. Control the movement of the AVG mobile device according to the moving area coordinate set, control the mechanical arm to adjust the position according to the detection area coordinate set, control the image data acquisition device to collect actual image sample data, and annotate the actual image sample data to obtain the actual appearance data, actual size data and actual defect data of the corresponding product; S4. Obtaining preset ideal image sample data, annotating the ideal image sample data to obtain the ideal appearance data, ideal size data and ideal defect data of the corresponding product; S5. Perform small sample high-precision training based on actual appearance data, actual size data, actual defect data and ideal appearance data, ideal size data, and ideal defect data to establish a visual inspection reference database, wherein the visual inspection reference database includes appearance reference data, size reference data, and defect reference data; S6. Control the movement of the AVG mobile device according to the moving area coordinate set, control the mechanical arm to adjust the position according to the detection area coordinate set, control the image data acquisition device to perform real-time product quality inspection, and collect real-time visual inspection data, wherein the real-time visual inspection data includes real-time appearance data, real-time size data, and real-time defect data; S7. Compare the appearance reference data with the appearance real-time data to obtain the appearance inspection data, compare the size reference data with the size real-time data to obtain the size inspection data, compare the defect reference data with the defect real-time data to obtain the defect inspection data, and analyze the visual inspection result according to the appearance inspection data, the size inspection data and the defect inspection data, wherein the visual inspection result is one of the following: inspection passed, inspection failed or unable to be determined; S8. When the visual inspection result is unable to judge, output the manual auxiliary prompt, and receive the manual inspection result input from the outside, the manual inspection result is one of the three: pass, fail or unable to judge; S9. Perform self-learning update on the visual inspection reference database according to the manual inspection results; S10. Control the 5G industrial CPE to transmit the detection results to the central server.
3. The device according to claim 2, characterized in that In step S2, coordinate system mapping is performed according to the three-dimensional coordinate system of the user-defined area and the three-dimensional coordinate system of the robotic arm, and the coordinate conversion relationship between the three-dimensional coordinate system of the user-defined area and the three-dimensional coordinate system of the robotic arm is obtained, which specifically includes: S201. Determine the origin and coordinate axis direction of the three-dimensional coordinate system of the user-defined area, obtain the three-dimensional point cloud data of the user-defined area, and convert the three-dimensional point cloud data of the user-defined area into coordinate values in the three-dimensional coordinate system of the user-defined area; S202. Determine the origin and coordinate axis direction of the three-dimensional coordinate system of the robot arm, obtain the coordinate value of the end of the robot arm, and convert the coordinate value of the end of the robot arm into the coordinate value of the three-dimensional coordinate system of the robot arm; S203. Select a calibration point with identification features, obtain the coordinate value (X1, Y1, Z1) of the calibration point in the three-dimensional coordinate system of the user-defined area and the coordinate value (X2, Y2, Z2) of the calibration point in the three-dimensional coordinate system of the robot arm, and pair the coordinate value (X1, Y1, Z1) in the three-dimensional coordinate system of the user-defined area and the coordinate value (X2, Y2, Z2) in the three-dimensional coordinate system of the robot arm; S204. According to the coordinate values (X1, Y1, Z1) of the same calibration point in the three-dimensional coordinate system of the user-defined area and the coordinate values (X2, Y2, Z2) of the three-dimensional coordinate system of the robotic arm after pairing, calculate the translation vector T between the three-dimensional coordinate system of the user-defined area and the three-dimensional coordinate system of the robotic arm, where: S205. According to the coordinate values (X1, Y1, Z1) of the same calibration point in the three-dimensional coordinate system of the user-defined area and the coordinate values (X2, Y2, Z2) of the three-dimensional coordinate system of the robotic arm after pairing, the rotation matrix R between the three-dimensional coordinate system of the user-defined area and the three-dimensional coordinate system of the robotic arm is calculated, wherein the unit rotation matrix I is used as the rotation matrix R, then: S206. Calculate the coordinate transformation matrix M according to the translation vector T and the rotation matrix R, where:
4. The device according to claim 2 or 3, characterized in that In step S5, small sample high-precision training is performed to establish a visual inspection reference database, specifically including: S501. Construct a GAN model based on a deep convolutional generative adversarial network, and configure a generator and a discriminator in the GAN model; wherein the generator is used to generate fake samples similar to real samples, and the discriminator is used to distinguish whether the input sample is real or generated; S502. Performing data fusion based on actual appearance data, actual size data, actual defect data and ideal appearance data, ideal size data, and ideal defect data to obtain an expanded data set; S503. Performing data enhancement on the expanded data set by image rotation, image flipping, image scaling, and image translation, and performing image preprocessing on the expanded data set after data enhancement, wherein the image preprocessing specifically includes image cleaning operation, feature labeling operation, and normalization operation; S504. Perform GAN model training, control the generator to generate fake samples based on the expanded data set after image preprocessing, control the discriminator to identify the fake samples generated by the generator and output the judgment result, and further perform feature distribution analysis on the actual appearance data, actual size data, actual defect data and the ideal appearance data, ideal size data, and ideal defect data based on the judgment result of the discriminator to obtain appearance reference data, size reference data, and defect reference data.
5. The device according to any one of claims 2 to 4, characterized in that: In step S6, controlling the movement of the AVG mobile device according to the moving area coordinate set specifically includes: S601. Construct an AVG moving map according to the moving area coordinate set, and perform AVG path planning based on the AVG moving map to obtain an AVG planned path; S602. Dynamically obtain the AVG real-time coordinates of the AVG mobile device (X V1 , Y V1 ); S603. Match the real-time coordinates of the AVG mobile device to the AVG planned path, and obtain the coordinates of the planned path points (X V2 , Y V2 ); S604. Based on AVG real-time coordinates (X V1 , Y V1 ) and the coordinates of the planned path points (X V2 , Y V2 ), calculate the distance d between the two, where: S605. Traverse to find the nearest neighbor d min , calculate the coordinates of the nearest neighbor point (X Vi , Y Vi ),in: Specifically, n is the number of points on the AVG planning path; S606. According to the AVG real-time coordinates (X V1 , Y V1 ) and the coordinates of the nearest neighbor (X Vi , Y Vi ) generating an AVG control instruction, and controlling the movement of the AVG mobile device according to the AVG control instruction; In step S6, controlling the robot arm to adjust the position according to the detection area coordinate set specifically includes: S611. Analyze the end path of the robot arm according to the detection area coordinate set; S612. Determine the target coordinates (X) of the end of the robot arm based on the end path of the robot arm Mi , Y Mi , Z Mi ) and the target posture of the end of the robot arm; S613. Get the real-time coordinates of the end of the robot arm (X M1 , Y M2 , Z M2 ) and the real-time posture of the end of the robotic arm; S614. Based on the target coordinates of the end of the robot arm, the target posture of the end of the robot arm, the real-time coordinates of the end of the robot arm and the real-time posture of the end of the robot arm, inverse kinematics calculation is performed to obtain the robot arm joint control data Θ t =IK(P t ,R t ), where Θ t is the joint angle vector control data, IK is the inverse kinematics function, P t is the target coordinate of the end of the robot arm, R t is the joint rotation matrix, specifically: P t =(X t ,Y t ,Z t ), X t =l1cos(θ1)+l2cos(θ1+θ2), Y t =l1sin(θ1)+l2sin(θ1+θ2), R t =R z (θ1)·R z (θ2), Z t is the target value of the Z axis, l1 is the length of the first connecting rod in the robot arm, l2 is the length of the second connecting rod in the robot arm, θ1 is the deflection angle of the first connecting rod in the robot arm relative to the Z axis, θ2 is the deflection angle of the second connecting rod in the robot arm relative to the Z axis, R z (θ1) is the rotation matrix of the angle θ1 around the Z axis, R z (θ2) is the rotation matrix of the angle θ2 around the Z axis; S615. Control data Θ according to joint angle vector t Control the robotic arm to adjust its position.
6. The device according to any one of claims 2 to 5, characterized in that In step S9, the self-learning update of the visual inspection reference database according to the manual inspection result specifically includes: S901. Obtaining actual image sample data corresponding to the manual detection result; S902. Perform denoising preprocessing on the actual image sample data; S903. Perform feature extraction on the actual image sample data after denoising preprocessing to obtain appearance feature data, size feature data and actual feature data; S904. Perform algorithm optimization on the visual inspection reference database according to the appearance feature data, the size feature data and the actual feature data to obtain optimized appearance reference data, size reference data and defect reference data.
7. A computer program, characterized in that The computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the method performed by the mobile visual inspection device according to any one of claims 1 to 7.
Citation Information
Patent Citations
Method for uniquely solving inverse kinematics numerical value of joint type mechanical arm
CN109895101A
Five-degree-of-freedom mechanical arm inverse kinematics solving method
CN110434851A
Remote control system of hot-line work robot and control method thereof
CN110900600A
Visual point cloud path automatic planning system and method based on intelligent subway purging
CN115283360A
Flexible production line AGV scheduling control method, device and system
CN115328046A
