Nut classification method, device and equipment based on machine vision and storage medium

Abstract

The application discloses a nut classification method and device based on machine vision, equipment and a storage medium. The method comprises the following steps: acquiring a nut image of a nut to be classified; performing feature extraction on the nut image to obtain at least one nut feature image, wherein the nut feature image comprises a plurality of reference pixel points; determining gradient information of each reference pixel point, and determining a target nut radius according to the gradient information; and classifying the nut to be classified according to the target nut radius. The target nut radius is measured according to the nut image, so that the subjective influence of artificial measurement can be avoided. Then, the nut to be classified is classified according to the target nut radius, so that automatic detection and classification of the nut can be realized. Compared with the technical solution in the prior art, in which the nut is measured by contact through a vernier caliper, and then classified according to the measurement result, the accuracy of nut classification can be effectively improved.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of parts classification technology, and in particular to a method, apparatus, device, and storage medium for nut classification based on machine vision. Background Technology

[0002] With the development of modern industrial technology, mechanical equipment is increasingly widely used in various fields, and nuts are indispensable fasteners in mechanical equipment. However, in mechanical assembly work, the frequent disassembly and replacement of nuts can easily lead to the mixing of different types of nuts. Furthermore, nuts are also subjected to lateral loads during disassembly and replacement, causing plastic deformation and bore wear. Using incompatible nuts for mechanical assembly will jeopardize the fixing effect of the nuts and the efficiency of the mechanical equipment.

[0003] In related technologies, nuts are usually measured manually using vernier calipers, and then classified according to the measurement results. However, the measurement accuracy of vernier calipers is limited, and manual measurement is subject to a large degree of subjectivity, making it difficult to guarantee the accuracy of the measurement results, resulting in low accuracy in nut classification. Summary of the Invention

[0004] This application provides a machine vision-based nut classification method, apparatus, device, and storage medium, which can effectively improve the accuracy of nut classification.

[0005] In a first aspect, embodiments of this application provide a machine vision-based nut classification method, applied to a nut classification system, comprising:

[0006] Obtain the image of the nut to be classified;

[0007] Feature extraction is performed on the nut image to obtain at least one nut feature image, wherein the nut feature image includes multiple reference pixels;

[0008] Determine the gradient information of each of the reference pixels, and determine the radius of the target nut based on the gradient information;

[0009] The nuts to be classified are classified according to the target nut radius.

[0010] According to some embodiments of the first aspect of this application, the step of extracting features from the nut image to obtain at least one nut feature image includes:

[0011] The nut image is subjected to perspective transformation to obtain a first nut image;

[0012] The first nut image is filtered using a bilateral filter, and the filtered first nut image is then binarized to obtain the second nut image.

[0013] The second nut image is subjected to convolution and image segmentation to obtain at least one nut feature image.

[0014] According to some embodiments of the first aspect of this application, the step of performing convolution operation and image segmentation processing on the second nut image to obtain at least one nut feature image includes:

[0015] Based on the Sobel operator, a convolution operation is performed on the second nut image to obtain a first first-order gradient component in the vertical direction, a second first-order gradient component in the horizontal direction, a third first-order gradient component in the first direction, and a fourth first-order gradient component in the second direction, wherein the angle between the first direction and the vertical direction is 45 degrees, and the second direction is perpendicular to the first direction.

[0016] Based on the gradient magnitude formula, the first first-order gradient component, the second first-order gradient component, the third first-order gradient component, and the fourth first-order gradient component are calculated to determine the gradient magnitude of the second nut image.

[0017] Based on the gradient magnitude and the preset magnitude threshold, the second nut image is subjected to non-maximum suppression processing to obtain the reference nut contour;

[0018] Based on the reference nut contour, the second nut image is segmented using a preset Otsu algorithm and the grayscale information of the second nut image to obtain at least one nut feature image.

[0019] According to some embodiments of the first aspect of this application, the gradient information includes the gradient direction, and determining the gradient information of each of the reference pixels and determining the target nut radius based on the gradient information includes:

[0020] Determine the pixel value of each of the reference pixels, and determine non-zero pixels among the reference pixels based on the pixel values;

[0021] Based on the Scharr operator, the nut feature image is convolved to obtain the fifth first-order gradient component in the vertical direction and the sixth first-order gradient component in the horizontal direction.

[0022] Based on the gradient direction formula, the fifth first-order gradient component and the sixth first-order gradient component are calculated to determine the gradient direction of each non-zero pixel, and the gradient direction line of the non-zero pixel is determined according to the gradient direction.

[0023] Traverse the gradient direction lines and take the intersection of any one gradient direction line with the rest of the gradient direction lines as a candidate point, wherein the candidate points are all located on the nut feature image;

[0024] The target circle center is determined based on the candidate points, and the target nut radius is determined based on the target circle center.

[0025] According to some embodiments of the first aspect of this application, determining the target circle center based on the candidate points and determining the target nut radius based on the target circle center includes:

[0026] Determine whether any candidate point overlaps with the other candidate points. Based on the overlap determination result, take the candidate point with the most overlaps as the center of the target circle.

[0027] Calculate the candidate nut radius between each of the non-zero pixels and the center of the target circle;

[0028] Determine whether any candidate nut radius is equal to the remaining candidate nut radii. Based on the equality determination result, the candidate nut radius with the largest number of equal radii is taken as the target nut radius.

[0029] According to some embodiments of the first aspect of this application, classifying the nuts to be classified based on the target nut radius includes:

[0030] Based on the target nut radius and a preset type mapping table, the target nut type and standard nut radius of the nut to be classified are determined, wherein the type mapping table is used to indicate the mapping relationship between nut type and nut radius;

[0031] Calculate the target difference between the target nut radius and the standard nut radius;

[0032] The quality information of the nuts to be classified is determined based on the target difference.

[0033] The nuts to be classified are classified according to the target nut type and the quality information.

[0034] According to some embodiments of the first aspect of this application, the nut sorting system is connected to a camera device, the camera device being used to acquire images of the nuts, and the method further includes:

[0035] Obtain the first position information of the camera device and the second position information of the nut to be classified;

[0036] Based on the first position information and the second position information, establish at least one reference coordinate system;

[0037] Based on at least one of the reference coordinate systems, a first parameter is received, and a second parameter is determined from a preset parameter library; the nut classification system is calibrated according to the first parameter or the second parameter.

[0038] Secondly, embodiments of this application provide a nut sorting device based on machine vision, comprising:

[0039] The nut image acquisition module is used to acquire images of the nuts to be classified.

[0040] A nut feature extraction module is used to extract features from the nut image to obtain at least one nut feature image, wherein the nut feature image includes multiple reference pixels.

[0041] The target nut radius determination module is used to determine the gradient information of each of the reference pixels and determine the target nut radius based on the gradient information.

[0042] The classification module is used to classify the nuts to be classified according to the radius of the target nut.

[0043] Thirdly, embodiments of this application provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the machine vision-based nut classification method as described in the first aspect.

[0044] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the machine vision-based nut sorting method as described in the first aspect.

[0045] This application proposes a machine vision-based nut classification method, apparatus, device, and storage medium. Embodiments of this application include: acquiring a nut image of a nut to be classified; extracting features from the nut image to obtain at least one nut feature image, wherein the nut feature image includes multiple reference pixels; determining the gradient information of each reference pixel and determining the target nut radius based on the gradient information; and classifying the nut to be classified according to the target nut radius. According to the scheme provided in this application, feature extraction is performed on the nut image to extract the small diameter contour of the nut to be classified, obtaining at least one nut feature image, which facilitates the simultaneous detection and classification of multiple nuts to be classified in the nut image. Subsequently, the gradient information of each reference pixel in the nut feature image is determined, and the nut radius is filtered and the target nut radius is determined based on the gradient information to ensure the accuracy of nut size detection. Then, the model and quality information of the nut to be classified are judged based on the target nut radius, and the nut to be classified is classified according to the judgment result, which can effectively improve the accuracy of nut classification. As can be seen, the machine vision-based nut classification method provided in this application measure the radius of the target nut from the nut image, which avoids the subjective influence of human measurement. Then, the nuts to be classified are classified according to the radius of the target nut, so as to realize the automatic detection and classification of nuts. Compared with the related technology, which uses a vernier caliper to measure the nuts by contact and then classifies them according to the measurement results, this method can effectively improve the accuracy of nut classification. Attached Figure Description

[0046] The accompanying drawings are provided to further understand the technical solutions of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the technical solutions of the present invention, and do not constitute a limitation on the technical solutions of the present invention.

[0047] Figure 1 This is an optional flowchart of the nut classification method based on machine vision provided in the embodiments of this application;

[0048] Figure 2 This is a flowchart of an optional step in the nut feature image extraction method provided in this application embodiment;

[0049] Figure 3 This is another optional flowchart of the nut feature image extraction method provided in the embodiments of this application;

[0050] Figure 4 This is a flowchart of the steps in the method for determining the radius of a target nut provided in an embodiment of this application;

[0051] Figure 5 This is a flowchart illustrating the steps of a method for determining the radius of a target nut based on the center of a target circle, as provided in an embodiment of this application.

[0052] Figure 6 This is a flowchart of the steps of the nut classification method provided in the embodiments of this application;

[0053] Figure 7 This is an optional flowchart of the nut classification system calibration method provided in the embodiments of this application;

[0054] Figure 8 This is a schematic diagram of the camera device provided in the embodiments of this application;

[0055] Figure 9 This is a reference schematic diagram of the coordinate system establishment provided in the embodiments of this application;

[0056] Figure 10 This is a schematic diagram of the nut classification results provided in the embodiments of this application;

[0057] Figure 11 This is a schematic diagram of a machine vision-based nut sorting device provided in an embodiment of this application;

[0058] Figure 12 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0060] In the description of this application, "several" means one or more, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0061] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, or the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0062] With the development of modern industrial technology, mechanical equipment is increasingly widely used in various fields, and nuts are indispensable fasteners in mechanical equipment. However, in mechanical assembly work, the frequent disassembly and replacement of nuts can easily lead to the mixing of different types of nuts. Furthermore, nuts are also subjected to lateral loads during disassembly and replacement, causing plastic deformation and bore wear. Using incompatible nuts for mechanical assembly will jeopardize the fixing effect of the nuts and the efficiency of the mechanical equipment.

[0063] In related technologies, nuts are usually measured manually using vernier calipers, and then classified according to the measurement results. However, the measurement accuracy of vernier calipers is limited, and manual measurement is subject to a large degree of subjectivity, making it difficult to guarantee the accuracy of the measurement results, resulting in low accuracy in nut classification.

[0064] To address the issue of low accuracy in nut classification, this application provides a machine vision-based nut classification method, apparatus, device, and storage medium. The method includes: acquiring a nut image of the nuts to be classified; extracting features from the nut image to obtain at least one nut feature image, wherein the nut feature image includes multiple reference pixels; determining the gradient information of each reference pixel and determining the target nut radius based on the gradient information; and classifying the nuts to be classified based on the target nut radius. According to the scheme provided in this application, feature extraction is performed on the nut image to extract the small diameter contour of the nuts to be classified, obtaining at least one nut feature image, which facilitates the simultaneous detection and classification of multiple nuts to be classified in the nut image. Subsequently, the gradient information of each reference pixel in the nut feature image is determined, and the nut radius is filtered and the target nut radius is determined based on the gradient information to ensure the accuracy of nut size detection. Then, the model and quality information of the nuts to be classified are judged based on the target nut radius, and the nuts to be classified are classified based on the judgment result, which can effectively improve the accuracy of nut classification. As can be seen, the machine vision-based nut classification method provided in this application measure the radius of the target nut from the nut image, which avoids the subjective influence of human measurement. Then, the nuts to be classified are classified according to the radius of the target nut, so as to realize the automatic detection and classification of nuts. Compared with the related technology, which uses a vernier caliper to measure the nuts by contact and then classifies them according to the measurement results, this method can effectively improve the accuracy of nut classification.

[0065] The nut classification method, apparatus, device, and storage medium based on machine vision provided in this application are specifically described through the following embodiments. First, the nut classification method based on machine vision in this application is described.

[0066] The machine vision-based nut classification method provided in this application relates to the field of parts classification technology. This machine vision-based nut classification method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the machine vision-based nut classification method, but is not limited to the above forms.

[0067] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0068] It should be noted that in various specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data for the normal operation of the embodiments of this application obtained. The embodiments of this application will be further described below with reference to the accompanying drawings.

[0069] Reference Figure 1 , Figure 1This is an optional flowchart of a machine vision-based nut classification method provided in this application embodiment. This machine vision-based nut classification method is applied to a nut classification system and includes, but is not limited to, the following steps:

[0070] Step S110: Obtain the image of the nut to be classified;

[0071] Step S120: Extract features from the nut image to obtain at least one nut feature image, wherein the nut feature image includes multiple reference pixels;

[0072] Step S130: Determine the gradient information of each reference pixel and determine the radius of the target nut based on the gradient information;

[0073] Step S140: Classify the nuts to be classified according to the target nut radius.

[0074] It should be noted that the embodiments of this application do not limit the specific number of nuts to be classified in the nut image. There can be one nut to be classified, or two or more nuts to be classified. It is understood that detecting and classifying multiple nuts to be classified at the same time based on the nut image can effectively improve the efficiency of nut detection and classification.

[0075] It is understandable that acquiring an image of the nut to be classified and extracting features from the image to extract the small diameter contour of the nut to be classified, resulting in at least one nut feature image, reduces the subsequent detection area. This nut feature image includes multiple reference pixels to facilitate simultaneous detection and classification of multiple nuts in the image, effectively improving the efficiency of nut detection and classification. Subsequently, the gradient information of each reference pixel in the nut feature image is determined. Based on the gradient information, the nut radius is filtered and the target nut radius is determined to ensure the accuracy of nut size detection. Then, based on the target nut radius, the model and quality information of the nuts to be classified are judged, and the nuts are classified according to the judgment result, effectively improving the accuracy of nut classification. Therefore, the machine vision-based nut classification method provided in this application, which measures the target nut radius from the nut image, avoids the subjective influence of human measurement. Classifying the nuts according to the target nut radius achieves automatic nut detection and classification. Compared to related technologies that rely on manual contact measurement with calipers and then classification based on the measurement results, this method effectively improves the accuracy of nut classification.

[0076] Additionally, refer to Figure 2 In one embodiment, Figure 1 Step S120 in the illustrated embodiment also includes, but is not limited to, the following steps:

[0077] Step S210: Perform perspective transformation on the nut image to obtain the first nut image;

[0078] Step S220: Filter the first nut image using a bilateral filter, and binarize the filtered first nut image to obtain the second nut image.

[0079] Step S230: Perform convolution operation and image segmentation on the second nut image to obtain at least one nut feature image.

[0080] Understandably, bilateral filters are nonlinear filtering methods that can simultaneously consider the spatial proximity and pixel value similarity of an image, thus preserving edge information while reducing noise. Therefore, using a bilateral filter to filter the first nut image can ensure the integrity of the edge information in the subsequently obtained nut feature image.

[0081] Understandably, when there are multiple nuts to be classified, or when the position of the corresponding nut image is not exactly in the center, the various nuts in the nut image may have some degree of shape variation. Therefore, perspective transformation is performed on the nut image to perform geometric transformation and achieve image correction, resulting in a first nut image that ensures its reliability. Subsequently, a bilateral filter is used to filter the first nut image, eliminating noise and making it smoother, sharper, and with enhanced boundaries. The filtered first nut image is then binarized to remove the influence of uneven lighting, resulting in a second nut image. This second nut image is then used for convolution and image segmentation to extract the minor diameter contour of the nuts to be classified, obtaining at least one nut feature image.

[0082] It is understandable that the perspective transformation formula can be used to perform perspective transformation on the nut image. This perspective transformation formula is h. i =Th j , where h i =[ui,vi,1],h i h represents the coordinates of the corresponding pixel after perspective transformation, that is, the pixel coordinates of the first nut image. j =[uj,vj,1],h j This represents the coordinates of a single pixel in the original image, specifically the coordinates of a single pixel in the nut image. T represents the perspective transformation matrix. Assume the four coordinates of the original image are (x1, y1), (x2, y2), (x3, y3), and (x4, y4), and the four coordinates after the perspective transformation are (x1, y1), (x2, y2), (x3, y3), and (x4, y4). Let a... 34 =0, and the perspective transformation matrix T can be calculated using the following formula:

[0083]

[0084] Additionally, refer to Figure 3 In one embodiment, Figure 2 Step S230 in the illustrated embodiment also includes, but is not limited to, the following steps:

[0085] Step S310: Based on the Sobel operator, perform convolution operation on the second nut image to obtain the first first-order gradient component in the vertical direction, the second first-order gradient component in the horizontal direction, the third first-order gradient component in the first direction, and the fourth first-order gradient component in the second direction. The angle between the first direction and the vertical direction is 45 degrees, and the second direction is perpendicular to the first direction.

[0086] Step S320: Based on the gradient magnitude formula, calculate the first first-order gradient component, the second first-order gradient component, the third first-order gradient component, and the fourth first-order gradient component to determine the gradient magnitude of the second nut image.

[0087] Step S330: Based on the gradient magnitude and a preset magnitude threshold, perform non-maximum suppression processing on the second nut image to obtain the reference nut profile;

[0088] Step S340: Based on the reference nut profile, the second nut image is segmented using a preset Otsu algorithm and the grayscale information of the second nut image to obtain at least one nut feature image.

[0089] It should be noted that the embodiments of this application do not limit the specific value of the preset amplitude threshold, which can be adjusted according to the gradient amplitude or measurement requirements.

[0090] Understandably, based on the Sobel operator, convolution is performed on the second nut image to obtain the first first-order gradient component in the vertical direction, the second first-order gradient component in the horizontal direction, the third first-order gradient component in the first direction, and the fourth first-order gradient component in the second direction. The first direction makes a 45-degree angle with the vertical direction, and the second direction is perpendicular to the first direction. For example, the first first-order gradient component is G. y (x, y), the second first-order gradient component is G x (x, y), the third first-order gradient component is G 45(x, y), the fourth first-order gradient component is G 135 (x, y). Subsequently, based on the gradient magnitude formula, the first, second, third, and fourth gradient components are calculated to determine the gradient magnitude of the second nut image, where the gradient magnitude formula is M(x, y) = sqrt(G). x 2 +G y 2 +G 45 2 +G 135 2 Then, based on the gradient magnitude and a preset magnitude threshold, non-maximum suppression processing is performed on the second nut image to filter out pixels with small rates of change, thereby removing redundant data and obtaining a reference nut contour. Based on the reference nut contour, the second nut image is then segmented using a preset Otsu algorithm and the grayscale information of the second nut image to obtain at least one nut feature image.

[0091] Additionally, refer to Figure 4 In one embodiment, the gradient information includes the gradient direction. Figure 1 Step S130 in the illustrated embodiment also includes, but is not limited to, the following steps:

[0092] Step S410: Determine the pixel value of each reference pixel, and determine non-zero pixels among each reference pixel based on the pixel value;

[0093] Step S420: Based on the Scharr operator, perform convolution operation on the nut feature image to obtain the fifth first-order gradient component in the vertical direction and the sixth first-order gradient component in the horizontal direction.

[0094] Step S430: Based on the gradient direction formula, calculate the fifth and sixth gradient components to determine the gradient direction of each non-zero pixel and determine the gradient direction line of the non-zero pixel based on the gradient direction.

[0095] Step S440: Traverse the gradient direction lines and take the intersection of any gradient direction line with the other gradient direction lines as candidate points. All candidate points are located on the nut feature image.

[0096] Step S450: Determine the target circle center based on the candidate points, and determine the target nut radius based on the target circle center.

[0097] Understandably, the Scharr operator uses a smaller convolution kernel, can perform convolution operations in different directions, has better noise resistance, and can better preserve the details in the image. Furthermore, the Scharr operator is fast, effectively improving the detection efficiency of nuts to be classified. Moreover, the Scharr operator provides more accurate edge detection results for slanted edges, better capturing edge details. Therefore, using the Scharr operator to perform convolution operations on nut feature images can reduce noise interference, improve computation speed, and enhance edge detection accuracy, thereby improving the efficiency and accuracy of determining the radius of the target nut.

[0098] Understandably, the pixel values ​​of each reference pixel are determined, and non-zero pixels are identified within each reference pixel based on these values. These non-zero pixels represent the contour boundary of the nut to be classified. Using the Scharr operator, a convolution operation is performed on the nut feature image to obtain the fifth-order gradient component in the vertical direction and the sixth-order gradient component in the horizontal direction. For example, the fifth-order gradient component is G. y The sixth-order gradient component is G′(x, y). x Based on the gradient direction formula, the fifth and sixth order gradient components are calculated to determine the gradient direction of each non-zero pixel. The gradient direction formula is as follows: Since the gradient direction of non-zero pixels points towards the center of the circle, gradient direction lines for non-zero pixels are determined based on the gradient direction. These gradient direction lines are then traversed, and the intersection of any gradient direction line with the remaining lines is taken as a candidate point. All candidate points are located on the nut feature image. Based on the obtained candidate points, they are filtered to determine the target circle center, ensuring its accuracy. The radius of the target nut is then determined based on the target circle center.

[0099] Additionally, refer to Figure 5 In one embodiment, Figure 4 Step S450 in the illustrated embodiment also includes, but is not limited to, the following steps:

[0100] Step S510: Determine whether any candidate point overlaps with the other candidate points. Based on the overlap determination result, take the candidate point with the most overlaps as the center of the target circle.

[0101] Step S520: Calculate the candidate nut radius between each non-zero pixel and the center of the target circle;

[0102] Step S530: Determine whether the radius of any candidate nut is equal to the radius of the other candidate nuts. Based on the result of the equality determination, take the radius of the candidate nut with the largest number of equal nuts as the target nut radius.

[0103] Understandably, the process involves determining whether any candidate point overlaps with other candidate points. If a candidate point overlaps with others, the overlap count is accumulated, and the candidate points are sorted based on this count to obtain the overlap determination result. Based on this result, the candidate point with the highest overlap count is selected as the target circle center, effectively ensuring the accuracy of the target circle center. The radius of the candidate nut between each non-zero pixel and the target circle center is calculated. It is then determined whether any candidate nut radius is equal to the remaining candidate nut radii. If so, the number of equalities is accumulated, and the candidate nut radii are sorted based on this count to obtain the equality determination result. According to the equality determination result, the higher the number of equalities, the more likely the candidate nut radius is to be the radius of the nut to be classified. Using the candidate nut radius with the highest number of equalities as the target nut radius ensures the accuracy of the target nut radius, thereby guaranteeing the accuracy of subsequent classification of the nuts to be classified based on the target nut radius.

[0104] Additionally, refer to Figure 6 In one embodiment, Figure 1 Step S140 in the illustrated embodiment also includes, but is not limited to, the following steps:

[0105] Step S610: Determine the target nut type and standard nut radius of the nut to be classified based on the target nut radius and the preset type mapping table, wherein the type mapping table is used to indicate the mapping relationship between nut type and nut radius;

[0106] Step S620: Calculate the target difference between the target nut radius and the standard nut radius;

[0107] Step S630: Determine the quality information of the nuts to be classified based on the target difference;

[0108] Step S640: Classify the nuts to be classified according to the target nut type and quality information.

[0109] Understandably, different nut types correspond to different nut radii. Based on the target nut radius and a preset type mapping table, the target nut type and standard nut radius of the nuts to be classified are determined. This facilitates subsequent classification of nuts according to the target nut type, avoiding the mixing of different types of nuts. The type mapping table indicates the mapping relationship between nut type and nut radius. Nuts are also subjected to lateral loads during disassembly and replacement, leading to plastic deformation and bore wear. The target difference between the target nut radius and the standard nut radius is calculated to assess the damage level of the nuts to be classified, determining their quality information. Then, the nuts to be classified are categorized according to the target nut type and quality information. This allows for automatic classification by nut type and automatic classification of nuts of the same type according to quality information, distinguishing nuts of different ages and improving classification efficiency and accuracy. This results in clear nut classification, facilitating subsequent assembly work.

[0110] It is understood that, in one embodiment, the machine vision-based nut classification method further includes, but is not limited to, the following steps: when the target difference belongs to a preset first threshold interval, the quality information of the nut to be classified is determined to be slightly damaged; when the target difference belongs to a preset second threshold interval, the quality information of the nut to be classified is determined to be moderately damaged; when the target difference belongs to a preset third threshold interval, the quality information of the nut to be classified is determined to be severely damaged.

[0111] Additionally, refer to Figure 7 In one embodiment, the nut classification system is connected to a camera device for acquiring images of nuts. The machine vision-based nut classification method further includes, but is not limited to, the following steps:

[0112] Step S710: Obtain the first position information of the camera device and the second position information of the nuts to be classified;

[0113] Step S720: Establish at least one reference coordinate system based on the first position information and the second position information;

[0114] Step S730: Based on at least one reference coordinate system, receive a first parameter and determine a second parameter from a preset parameter library; calibrate the nut classification system according to the first parameter or the second parameter.

[0115] It is understandable that during the acquisition of nut images by the nut classification system, the lens of the camera device may experience distortion, resulting in distorted nut images. Therefore, based on the embodiments provided in this application, a first position information of the camera device and a second position information of the nut to be classified are acquired. At least one reference coordinate system is established based on the first and second position information to accurately describe the state of the nut classification system in three-dimensional space. The state of the nut classification system may include position information and orientation information. Subsequently, based on at least one reference coordinate system, a first parameter is received, and a second parameter is determined from a preset parameter library. The nut classification system is calibrated according to the first or second parameter to avoid lens distortion problems and improve the accuracy of feature extraction, target nut radius confirmation, and classification of the nut images. Furthermore, the calibration of the nut classification system facilitates accurate gripping and classification of the nuts by a robotic arm.

[0116] In one embodiment, the machine vision-based nut classification method may further include, but is not limited to, the following steps:

[0117] The nut classification system is calibrated.

[0118] Obtain images of the nuts to be classified and preprocess the images;

[0119] Feature extraction is performed on the preprocessed nut image to obtain at least one nut feature image;

[0120] Determine the radius of the target nut based on the nut feature image;

[0121] Nuts to be classified are classified according to the target nut radius.

[0122] Understandably, the structure of the camera device in the nut sorting system can be as follows: Figure 8 As shown, the imaging device includes an industrial camera 810, an aperture 820, and a conveyor belt 930. The relative positions of the industrial camera 810 and the aperture 820 remain constant to ensure that the captured image of the nut is clear, complete, and free from obvious distortion. Based on the imaging device, a coordinate system is first established, such as... Figure 9As shown, a pixel coordinate system (PCS), an image coordinate system (IPCS), a camera coordinate system (CCS), a robot arm coordinate system (RCS), and a world coordinate system (SCS) can be established. Based on the established coordinate systems, the nut classification system is calibrated, and the intrinsic and extrinsic parameters of the camera device are adjusted. Adjusting the intrinsic parameters of the camera device avoids lens distortion affecting the accuracy of subsequent feature extraction from the nut images, while adjusting the extrinsic parameters facilitates the classification of the nuts to be classified. Subsequently, nut images of the nuts to be classified are acquired and preprocessed to solve the perspective distortion problem, ensuring the accuracy of the nut images and facilitating feature extraction from the preprocessed images. Edge detection and feature extraction are performed on the preprocessed nut images using the Canny edge detection algorithm and the Sobel operator, resulting in at least one nut feature image. Subsequently, the gradient direction of all non-zero pixels in the nut feature image is calculated using the HoughCircle function and the Scharr operator, and a straight line is drawn along the gradient direction to obtain the gradient direction line. The gradient direction lines are traversed, and the intersection point of any gradient direction line with the others is taken as a candidate point. All candidate points lie on the nut feature image. The target circle center is then determined based on the candidate points, and the target nut radius is determined based on the target circle center. Finally, the nuts to be classified are classified according to their target nut radius, for example, as... Figure 10 As shown, nut No. 1 has a radius of 10.010mm, its corresponding nut type is M12, and its quality information is OLD; nut No. 2 has a radius of 8.190mm, its corresponding nut type is M10, and its quality information is NEW; nut No. 3 has a radius of 6.552mm, its corresponding nut type is M8, and its quality information is NEW.

[0123] Reference Figure 11 , Figure 11 This is a schematic diagram of a machine vision-based nut sorting device according to an embodiment of this application. The machine vision-based nut sorting device 1100 includes:

[0124] Nut image acquisition module 1110 is used to acquire nut images of nuts to be classified;

[0125] The nut feature extraction module 1120 is used to extract features from a nut image to obtain at least one nut feature image, wherein the nut feature image includes multiple reference pixels.

[0126] The target nut radius determination module 1130 is used to determine the gradient information of each reference pixel and determine the target nut radius based on the gradient information.

[0127] The classification module 1140 is used to classify nuts to be classified according to the target nut radius.

[0128] The specific implementation of the machine vision-based nut sorting device 1100 in this application is basically the same as the specific implementation of the machine vision-based nut sorting method described above, and will not be repeated here.

[0129] Additionally, refer to Figure 12 , Figure 12 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0130] The processor 1201 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0131] The memory 1202 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1202 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1202 and is called and executed by the processor 1201 to execute the machine vision-based nut classification method applied to the nut classification system of this application, for example, executing the above-described... Figure 1 Method steps S110 to S140 in the middle Figure 2 Method steps S210 to S230 in the middle Figure 3 Method steps S310 to S340 in the middle Figure 4 Method steps S410 to S450 in the middle Figure 5 Method steps S510 to S530 in the middle Figure 6 Method steps S610 to S640 and Figure 7 Method steps S710 to S730;

[0132] The input / output interface 1203 is used to implement information input and output;

[0133] The communication interface 1204 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0134] Bus 1205 transmits information between various components of the device (e.g., processor 1201, memory 1202, input / output interface 1203, and communication interface 1204);

[0135] The processor 1201, memory 1202, input / output interface 1203 and communication interface 1204 are connected to each other within the device via bus 1205.

[0136] This application embodiment also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, which can be executed by one or more processors to implement the machine vision-based nut classification method applied to the nut classification system described above. For example, it executes the above-described... Figure 1 Method steps S110 to S140 in the middle Figure 2 Method steps S210 to S230 in the middle Figure 3 Method steps S310 to S340 in the middle Figure 4 Method steps S410 to S450 in the middle Figure 5 Method steps S510 to S530 in the middle Figure 6 Method steps S610 to S640 and Figure 7 Method steps S710 to S730.

[0137] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0138] The nut classification method, apparatus, device, and storage medium based on machine vision provided in this application embodiment acquire nut images of nuts to be classified; perform feature extraction on the nut images to obtain at least one nut feature image, wherein the nut feature image includes multiple reference pixels; determine the gradient information of each reference pixel, and determine the target nut radius based on the gradient information; classify the nuts to be classified according to the target nut radius. According to the scheme provided in this application embodiment, feature extraction is performed on the nut image to extract the small diameter contour of the nuts to be classified, obtaining at least one nut feature image, so as to facilitate the simultaneous detection and classification of multiple nuts to be classified in the nut image. Subsequently, the gradient information of each reference pixel in the nut feature image is determined, and the nut radius is filtered and the target nut radius is determined based on the gradient information to ensure the accuracy of nut size detection. Then, the model and quality information of the nuts to be classified are judged based on the target nut radius, and the nuts to be classified are classified according to the judgment result, which can effectively improve the accuracy of nut classification. As can be seen, the machine vision-based nut classification method provided in this application measure the radius of the target nut from the nut image, which avoids the subjective influence of human measurement. Then, the nuts to be classified are classified according to the radius of the target nut, so as to realize the automatic detection and classification of nuts. Compared with the related technology, which uses a vernier caliper to measure the nuts by contact and then classifies them according to the measurement results, this method can effectively improve the accuracy of nut classification.

[0139] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0140] It will be understood by those skilled in the art that Figures 1 to 7 The technical solutions shown do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0141] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0142] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0143] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0144] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0145] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0146] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0147] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0148] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0149] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A machine vision-based nut classification method, applied to a nut classification system, characterized in that, include: Obtain the image of the nut to be classified; Feature extraction is performed on the nut image to obtain at least one nut feature image, wherein the nut feature image includes multiple reference pixels; Determine the gradient information of each of the reference pixels, and determine the radius of the target nut based on the gradient information; The nuts to be classified are classified according to the target nut radius; The step of extracting features from the nut image to obtain at least one nut feature image includes: The nut image is subjected to perspective transformation to obtain a first nut image; The first nut image is filtered using a bilateral filter, and the filtered first nut image is then binarized to obtain the second nut image. Perform convolution and image segmentation on the second nut image to obtain at least one nut feature image; The step of performing convolution and image segmentation on the second nut image to obtain at least one nut feature image includes: Based on the Sobel operator, a convolution operation is performed on the second nut image to obtain a first first-order gradient component in the vertical direction, a second first-order gradient component in the horizontal direction, a third first-order gradient component in the first direction, and a fourth first-order gradient component in the second direction, wherein the angle between the first direction and the vertical direction is 45 degrees, and the second direction is perpendicular to the first direction. Based on the gradient magnitude formula, the first first-order gradient component, the second first-order gradient component, the third first-order gradient component, and the fourth first-order gradient component are calculated to determine the gradient magnitude of the second nut image. Based on the gradient magnitude and the preset magnitude threshold, the second nut image is subjected to non-maximum suppression processing to obtain the reference nut contour; Based on the reference nut contour, the second nut image is segmented using a preset Otsu algorithm and the grayscale information of the second nut image to obtain at least one nut feature image.

2. The nut classification method based on machine vision according to claim 1, characterized in that, The gradient information includes the gradient direction. Determining the gradient information of each reference pixel and determining the target nut radius based on the gradient information includes: Determine the pixel value of each of the reference pixels, and determine non-zero pixels among the reference pixels based on the pixel values; Based on the Scharr operator, the nut feature image is convolved to obtain the fifth first-order gradient component in the vertical direction and the sixth first-order gradient component in the horizontal direction. Based on the gradient direction formula, the fifth first-order gradient component and the sixth first-order gradient component are calculated to determine the gradient direction of each non-zero pixel, and the gradient direction line of the non-zero pixel is determined according to the gradient direction. Traverse the gradient direction lines and take the intersection of any one gradient direction line with the rest of the gradient direction lines as a candidate point, wherein the candidate points are all located on the nut feature image; The target circle center is determined based on the candidate points, and the target nut radius is determined based on the target circle center.

3. The machine vision-based nut classification method according to claim 2, characterized in that, The step of determining the target circle center based on the candidate points and determining the target nut radius based on the target circle center includes: Determine whether any candidate point overlaps with the other candidate points. Based on the overlap determination result, take the candidate point with the most overlaps as the center of the target circle. Calculate the candidate nut radius between each of the non-zero pixels and the center of the target circle; Determine whether any candidate nut radius is equal to the remaining candidate nut radii. Based on the equality determination result, the candidate nut radius with the largest number of equal radii is taken as the target nut radius.

4. The machine vision-based nut classification method according to claim 1, characterized in that, The step of classifying the nuts to be classified according to the target nut radius includes: Based on the target nut radius and a preset type mapping table, the target nut type and standard nut radius of the nut to be classified are determined, wherein the type mapping table is used to indicate the mapping relationship between nut type and nut radius; Calculate the target difference between the target nut radius and the standard nut radius; The quality information of the nuts to be classified is determined based on the target difference. The nuts to be classified are classified according to the target nut type and the quality information.

5. The machine vision-based nut classification method according to claim 1, characterized in that, The nut sorting system is connected to a camera device, which is used to acquire images of the nuts. The method further includes: Obtain the first position information of the camera device and the second position information of the nut to be classified; Based on the first position information and the second position information, establish at least one reference coordinate system; Based on at least one of the reference coordinate systems, a first parameter is received, and a second parameter is determined from a preset parameter library; the nut classification system is calibrated according to the first parameter or the second parameter.

6. A nut sorting device based on machine vision, characterized in that, include: The nut image acquisition module is used to acquire images of the nuts to be classified. A nut feature extraction module is used to extract features from the nut image to obtain at least one nut feature image, wherein the nut feature image includes multiple reference pixels. The target nut radius determination module is used to determine the gradient information of each of the reference pixels and determine the target nut radius based on the gradient information. A classification module is used to classify the nuts to be classified according to the radius of the target nut; The step of extracting features from the nut image to obtain at least one nut feature image includes: The nut image is subjected to perspective transformation to obtain a first nut image; The first nut image is filtered using a bilateral filter, and the filtered first nut image is then binarized to obtain the second nut image. Perform convolution and image segmentation on the second nut image to obtain at least one nut feature image; The step of performing convolution and image segmentation on the second nut image to obtain at least one nut feature image includes: Based on the Sobel operator, a convolution operation is performed on the second nut image to obtain a first first-order gradient component in the vertical direction, a second first-order gradient component in the horizontal direction, a third first-order gradient component in the first direction, and a fourth first-order gradient component in the second direction, wherein the angle between the first direction and the vertical direction is 45 degrees, and the second direction is perpendicular to the first direction. Based on the gradient magnitude formula, the first first-order gradient component, the second first-order gradient component, the third first-order gradient component, and the fourth first-order gradient component are calculated to determine the gradient magnitude of the second nut image. Based on the gradient magnitude and the preset magnitude threshold, the second nut image is subjected to non-maximum suppression processing to obtain the reference nut contour; Based on the reference nut contour, the second nut image is segmented using a preset Otsu algorithm and the grayscale information of the second nut image to obtain at least one nut feature image.

7. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the machine vision-based nut classification method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the machine vision-based nut sorting method as described in any one of claims 1 to 5.

Images