Unsupervised Visual Segmentation Method and System for Metal Bolts

Abstract

The present invention is based on an unsupervised metal bolt visual segmentation method and system, which performs convolutional neural network feature extraction on the collected metal bolt images to obtain high-abstraction feature maps and low-abstraction feature maps; performs dot multiplication operations on the foreground feature matrix and the background feature matrix, and performs logistic regression operations to obtain a probability distribution matrix containing the metal bolt body, and performs weighted summation calculations at each two-dimensional spatial position according to its corresponding category probability distribution to obtain a joint feature map; uses the low-abstraction feature map and the joint feature map to iteratively calculate the mutual information, and the mutual information tends to be maximized, thereby optimizing the representation of the feature map and obtaining a refined metal bolt body segmentation result; after the above-mentioned unsupervised learning, the present invention can distinguish between reflective areas and shadow areas in the pictures for bolt body extraction, thereby effectively solving the interference of shadows and reflections without the need for additional manual labeling.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to an unsupervised metal bolt visual segmentation method and system. Background Art

[0002] In the intelligent manufacturing industry, algorithms for visual inspection of good metal parts, visual intelligent sorting, and visual defect detection play an important role in industrial quality inspection. Among existing algorithmic approaches, extracting accurate segmentation information for the main body of a metal part from an image is often a crucial step in the algorithm. A high-quality segmentation image of the main body of a part also profoundly impacts the subsequent steps of the algorithm. Due to the material of the metal parts themselves, shadows and reflections often occur during visual acquisition; therefore, effectively distinguishing shadows, reflections, and the main body of the part in the image becomes a significant challenge. Furthermore, compared to supervised deep learning semantic segmentation algorithms, extensive manual labeling is not a worry-free and labor-saving intelligent automation solution. Therefore, it is necessary to propose a method that can effectively address the reflection and shadow issues in visual imaging for scenarios such as visual inspection of good metal parts, visual intelligent sorting, and defect detection. Summary of the Invention

[0003] The purpose of the present invention is to propose an unsupervised metal bolt visual segmentation method and system that can obtain accurate metal bolt segmentation images without labeling, which can effectively solve the interference of shadows and reflections without the need for additional manual labeling.

[0004] The present invention is based on an unsupervised metal bolt visual segmentation method, comprising the following steps:

[0005] Step 100, training step

[0006] Performing convolutional neural network feature extraction on the collected metal bolt image to obtain a high-abstraction feature map and a low-abstraction feature map; the convolutional neural network includes two sub-branches, one for extracting high-abstraction features to obtain a high-abstraction feature map; the other for extracting low-abstraction features to obtain a low-abstraction feature map; the high-abstraction feature map includes a foreground feature matrix and a background feature matrix;

[0007] Perform a dot product operation on the foreground feature matrix and the background feature matrix, and perform a logistic regression operation to obtain a probability distribution matrix containing the metal bolt body. A weighted summation calculation is performed at each two-dimensional spatial position according to its corresponding category probability distribution to obtain a joint feature map. The probability distribution matrix is a probability distribution matrix for regional clustering. It is pre-divided into several categories based on the semantic similarity of different regions in the image, including a probability distribution matrix representing the bolt body region, a background probability distribution matrix representing the metal reflective region, and a background probability distribution matrix representing the shadow region. Each pixel position in the probability distribution matrix represents a category and its corresponding probability.

[0008] The mutual information is calculated using a measure of the interdependence between two random variables in the low-abstraction feature map and the joint feature map, and the mutual information is iteratively calculated. During the iterative calculation process, the low-abstraction feature map obtained is updated through backpropagation of the convolutional neural network, thereby updating the model weights of the convolutional neural network. The joint feature map is used to maximize the use of global information to assist the unsupervised iterative training process. When the mutual information is maximized, training is stopped, and the trained model is obtained.

[0009] Step 200, reasoning step

[0010] The model trained in step 100 is used to perform convolutional neural network feature extraction on the collected metal bolt image to obtain a high-abstraction feature map and a low-abstraction feature map; the obtained high-abstraction feature map and the low-abstraction feature map are respectively subjected to a dot multiplication operation to obtain a feature map product result; the feature map product result is subjected to a logistic regression operation to obtain a probability distribution matrix for regional clustering; the probability distribution matrix for regional clustering is subjected to a probability transformation to obtain a foreground image and a background image, thereby extracting the bolt body from the input metal bolt image, obtaining a refined bolt body segmentation map and outputting it.

[0011] The present invention is based on an unsupervised metal bolt visual segmentation system, comprising:

[0012] Image acquisition module, used to collect metal bolt images and output them to the image segmentation module;

[0013] An image segmentation module includes a convolutional neural network and an image segmentation model; the convolutional neural network includes two sub-branches, one for extracting high-abstraction features to obtain a high-abstraction feature map; the other for extracting low-abstraction features to obtain a low-abstraction feature map; the high-abstraction feature map includes a foreground feature matrix H1 and a background feature matrix H0; the convolutional neural network extracts features from an input metal bolt image to obtain a high-abstraction feature map and a low-abstraction feature map;

[0014] The image segmentation model is obtained according to the following training process: a dot multiplication operation is performed on the foreground feature matrix and the background feature matrix obtained by feature extraction through a convolutional neural network, and a logistic regression operation is performed to obtain a probability distribution matrix containing the metal bolt body, and a weighted summation calculation is performed at each two-dimensional spatial position according to its corresponding category probability distribution to obtain a joint feature map; the probability distribution matrix is a probability distribution matrix for regional clustering, which is pre-divided into several categories according to the semantic similarity of different regions in the image, including a probability distribution matrix representing the bolt body region, a background probability distribution matrix representing the metal reflective region, and a background probability distribution matrix representing the shadow region, and each pixel position in the probability distribution matrix represents a category and its corresponding probability; the mutual information amount is calculated by measuring the mutual dependence between the two random variables of the low-abstraction feature map and the joint feature map, and the mutual information is iteratively calculated; in the iterative calculation process, the low-abstraction feature map obtained is updated by back propagation of the convolutional neural network, thereby updating the model weights of the convolutional neural network; the global information is maximized through the joint feature map to assist the unsupervised iterative training process, and the training is stopped when the mutual information amount is maximized to obtain a trained model;

[0015] During use, the convolutional neural network is used to extract features from the collected metal bolt image to obtain a high-abstraction feature map and a low-abstraction feature map. The image segmentation model then performs dot multiplication on the obtained high-abstraction feature map and the low-abstraction feature map to obtain the feature map product result. The feature map product result is subjected to a logistic regression operation to obtain a probability distribution matrix for regional clustering. The probability distribution matrix for regional clustering is subjected to a probability transformation to obtain a foreground image and a background image, thereby extracting the bolt body from the input metal bolt image and obtaining a refined bolt body segmentation map for output.

[0016] A computer device includes a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the computer program is used to implement the above-mentioned unsupervised metal bolt visual segmentation method.

[0017] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned unsupervised metal bolt visual segmentation method.

[0018] The convolutional neural network of the present invention includes two sub-branches, which respectively extract high-abstraction features and low-abstraction features, and perform convolutional neural network feature extraction on the collected metal bolt image to obtain a high-abstraction feature map and a low-abstraction feature map; perform dot multiplication operation on the foreground feature matrix and the background feature matrix of the high-abstraction feature map, and perform logistic regression operation to obtain a probability distribution matrix containing the metal bolt body, and perform weighted summation calculation according to the corresponding category probability distribution at each two-dimensional spatial position to obtain a joint feature map; use the low-abstraction feature map and the joint feature map to perform iterative calculation of mutual information, and finally achieve the maximization of the mutual information between the low-abstraction feature map and the joint feature map, thereby optimizing the representation of the feature map and obtaining a refined metal bolt body segmentation result; after the above-mentioned unsupervised learning, the present invention can distinguish the reflective area and the shadow area of the picture for bolt body extraction, thereby obtaining a refined bolt body segmentation map, which can effectively solve the interference of shadows and reflections without the need for additional manual labeling.

[0019] Since the joint feature map is obtained by logistic regression of the high-abstract feature map, the category probability distribution is obtained from the corresponding pixel probability of the category in each two-dimensional space. Therefore, the joint feature map preserves the aggregate information of each automatic clustering category result and can maximize the use of global information to assist the unsupervised iterative training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is the logic diagram of the image segmentation algorithm in the present invention;

[0021] Figure 2 It is the overall flow chart of the present invention. DETAILED DESCRIPTION

[0022] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features therein can be combined with each other without conflict.

[0023] The following description sets forth numerous specific details to facilitate a thorough understanding of the present invention. The embodiments described are merely some, not all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0025] The terms "first," "second," and "third," etc., in the specification and claims of the present invention and the accompanying drawings are used to distinguish between different items, not to describe a specific order. Furthermore, the terms "comprise," "comprising," and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0026] Example 1

[0027] See also Figures 1 to 2 In the first embodiment, the present invention is based on an unsupervised metal bolt visual segmentation method, comprising the following steps:

[0028] Step 100, training step

[0029] Performing convolutional neural network feature extraction on the collected metal bolt image to obtain a high-abstraction feature map and a low-abstraction feature map; the convolutional neural network includes two sub-branches, one for extracting high-abstraction features to obtain a high-abstraction feature map; the other for extracting low-abstraction features to obtain a low-abstraction feature map; the high-abstraction feature map includes a foreground feature matrix H1 and a background feature matrix H0;

[0030] A dot product operation is performed on the foreground feature matrix H1 and the background feature matrix H0 of the high-abstraction feature map, and a logistic regression operation is performed to obtain a probability distribution matrix containing the metal bolt body. A weighted summation calculation is performed at each two-dimensional spatial position according to its corresponding category probability distribution to obtain a joint feature map. The probability distribution matrix is a probability distribution matrix for regional clustering and is pre-divided into K categories based on the semantic similarity of different regions in the image. In the embodiment of the present invention, K=2, including a probability distribution matrix representing the bolt body region, a background probability distribution matrix representing the metal reflective region, and a background probability distribution matrix representing the shadow region. Each pixel position in the probability distribution matrix represents a category and its corresponding probability.

[0031] The mutual information is calculated using a measure of the interdependence between two random variables in the low-abstraction feature map and the joint feature map, and the mutual information is iteratively calculated. During the iterative calculation process, the low-abstraction feature map obtained is updated through backpropagation of the convolutional neural network, thereby updating the model weights of the convolutional neural network. The joint feature map is used to maximize the use of global information to assist the unsupervised iterative training process. When the mutual information is maximized, training is stopped, and the trained model is obtained.

[0032] The above model eliminates the uncertainty of random variables in the process of continuously iterating mutual information. When the mutual information is maximized, the gap between the foreground feature matrix H1 representing the bolt body area and the background feature matrix H0 containing the metal reflective area is the largest, thus achieving the complete segmentation of the bolt body and ultimately obtaining a refined metal bolt body segmentation result.

[0033] Step 200, reasoning step

[0034] The model trained in step 100 is used to perform convolutional neural network feature extraction on the collected metal bolt image to obtain a high-abstraction feature map and a low-abstraction feature map; the obtained high-abstraction feature map and the low-abstraction feature map are respectively subjected to a dot multiplication operation to obtain a feature map product result; the feature map product result is subjected to a logistic regression operation to obtain a probability distribution matrix for regional clustering; the probability distribution matrix for regional clustering is subjected to a probability transformation to obtain a foreground image and a background image, thereby extracting the bolt body from the input image, obtaining a refined bolt body segmentation map and outputting it.

[0035] Example 2

[0036] In the second embodiment, the present invention is based on an unsupervised metal bolt visual segmentation system, including:

[0037] Image acquisition module, used to collect metal bolt images and output them to the image segmentation module;

[0038] Image segmentation module, including convolutional neural network and image segmentation model;

[0039] The convolutional neural network includes two sub-branches, one of which is used to extract high-abstraction features to obtain a high-abstraction feature map; the other is used to extract low-abstraction features to obtain a low-abstraction feature map; the high-abstraction feature map includes a foreground feature matrix H1 and a background feature matrix H0; the convolutional neural network performs convolutional neural network feature extraction on the input metal bolt image to obtain a high-abstraction feature map and a low-abstraction feature map;

[0040] The image segmentation model is obtained according to the following training process: a dot multiplication operation is performed on the foreground feature matrix H1 and the background feature matrix H0 obtained by feature extraction through a convolutional neural network, and a logistic regression operation is performed to obtain a probability distribution matrix containing the metal bolt body, and a weighted summation calculation is performed at each two-dimensional spatial position according to its corresponding category probability distribution to obtain a joint feature map; the probability distribution matrix is a probability distribution matrix for regional clustering, which is pre-divided into several categories according to the semantic similarity of different regions in the image, including a probability distribution matrix representing the bolt body region, a background probability distribution matrix representing the metal reflective region, and a background probability distribution matrix representing the shadow region, and each pixel position in the probability distribution matrix represents a category and its corresponding probability; the mutual information amount is calculated by measuring the mutual dependence between the two random variables of the low-abstraction feature map and the joint feature map, and the mutual information is iteratively calculated; in the iterative calculation process, the low-abstraction feature map obtained is updated by back propagation of the convolutional neural network, thereby updating the model weights of the convolutional neural network; the global information is maximized through the joint feature map to assist the unsupervised iterative training process, and the training is stopped when the mutual information amount is maximized to obtain a trained model;

[0041] During use, the convolutional neural network first extracts features from the collected metal bolt image to obtain a high-abstraction feature map and a low-abstraction feature map; then the image segmentation model performs dot multiplication on the obtained high-abstraction feature map and the low-abstraction feature map respectively to obtain the feature map product result; the feature map product result is subjected to a logistic regression operation to obtain a probability distribution matrix for regional clustering, and a probability transformation is performed on the probability distribution matrix for regional clustering to obtain a foreground image and a background image, thereby extracting the bolt body from the input metal bolt image, obtaining a refined bolt body segmentation map and outputting it.

[0042] Example 3

[0043] In a third embodiment, the present invention provides a computer device comprising a processor, memory, a network interface, a display screen, and an input device connected via a system bus. The processor of the computer device is configured to provide computing and control capabilities, and the memory comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. When executed by the processor, the computer program implements the unsupervised metal bolt visual segmentation method of the first embodiment. Embodiments 1 and 3 have the same functions and benefits.

[0044] Example 4

[0045] In a fourth embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the unsupervised metal bolt visual segmentation method of the first embodiment is implemented. The first and fourth embodiments have the same functions and beneficial effects.

[0046] The present invention can be applied to any terminal device, which can be a mobile phone, a tablet, a server, etc. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above method embodiments. The non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0047] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units or modules as needed, that is, the internal structure of the storage device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the implementation method can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other and are not used to limit the scope of protection of this application.

[0048] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0049] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. Based on unsupervised metal bolt visual segmentation method, it is characterized by The steps include: Step 100, training step Performing convolutional neural network feature extraction on the collected metal bolt image to obtain a high-abstraction feature map and a low-abstraction feature map; the convolutional neural network includes two sub-branches, one of which is used to extract high-abstraction features to obtain the high-abstraction feature map; The other is used to extract low-abstraction features and obtain a low-abstraction feature map; The high-abstraction feature map includes a foreground feature matrix and a background feature matrix; Perform a dot product operation on the foreground feature matrix and the background feature matrix, and perform a logistic regression operation to obtain a probability distribution matrix containing the metal bolt body. Perform a weighted sum calculation based on the probability distribution of its corresponding category at each two-dimensional spatial position to obtain a joint feature map. The probability distribution matrix is a probability distribution matrix for regional clustering. It is pre-divided into several categories based on the semantic similarity of different regions in the image, including a probability distribution matrix representing the bolt body area, a background probability distribution matrix representing the metal reflective area, and a background probability distribution matrix representing the shadow area. Each pixel position in the probability distribution matrix represents a category and its corresponding probability. The mutual information is calculated by using the measure of mutual dependence between two random variables of the low-abstraction feature map and the joint feature map, and the mutual information is iteratively calculated; During the iterative calculation process, the low-abstraction feature map obtained is updated through the back propagation of the convolutional neural network, thereby updating the model weights of the convolutional neural network. By combining the feature maps, global information is maximized to assist the unsupervised iterative training process. When the mutual information is maximized, stop training and get the trained model; Step 200, reasoning step The model trained in step 100 is used to perform convolutional neural network feature extraction on the collected metal bolt image to obtain a high-abstraction feature map and a low-abstraction feature map; the obtained high-abstraction feature map and the low-abstraction feature map are respectively subjected to a dot multiplication operation to obtain a feature map product result; the feature map product result is subjected to a logistic regression operation to obtain a probability distribution matrix for regional clustering; the probability distribution matrix for regional clustering is subjected to a probability transformation to obtain a foreground image and a background image, thereby extracting the bolt body from the input metal bolt image, obtaining a refined bolt body segmentation map and outputting it.

2. Based on unsupervised metal bolt visual segmentation system, it is characterized by include: Image acquisition module, used to collect metal bolt images and output them to the image segmentation module; An image segmentation module includes a convolutional neural network and an image segmentation model; the convolutional neural network includes two sub-branches, one for extracting high-abstraction features to obtain a high-abstraction feature map; the other for extracting low-abstraction features to obtain a low-abstraction feature map; the high-abstraction feature map includes a foreground feature matrix H1 and a background feature matrix H0; the convolutional neural network extracts features from an input metal bolt image to obtain a high-abstraction feature map and a low-abstraction feature map; The image segmentation model is obtained according to the following training process: a dot product operation is performed on the foreground feature matrix and the background feature matrix obtained by feature extraction through a convolutional neural network, and a logistic regression operation is performed to obtain a probability distribution matrix containing the metal bolt body. A weighted summation calculation is performed at each two-dimensional spatial position according to the probability distribution of its corresponding category to obtain a joint feature map; The probability distribution matrix is a probability distribution matrix for regional clustering. It is pre-divided into several categories based on the semantic similarity of different regions in the image, including a probability distribution matrix representing the bolt body area, a background probability distribution matrix representing the metal reflective area, and a background probability distribution matrix representing the shadow area. Each pixel position in the probability distribution matrix represents a category and its corresponding probability. The mutual information is calculated using a measure of the interdependence between two random variables in the low-abstraction feature map and the joint feature map, and the mutual information is iteratively calculated. During the iterative calculation process, the low-abstraction feature map is updated through backpropagation of the convolutional neural network, thereby updating the model weights of the convolutional neural network. The joint feature map is used to maximize the use of global information to assist the unsupervised iterative training process. When the mutual information is maximized, training is stopped, resulting in a trained model. During use, the convolutional neural network is used to extract features from the collected metal bolt image to obtain a high-abstraction feature map and a low-abstraction feature map. The image segmentation model then performs a dot multiplication operation on the obtained high-abstraction feature map and the low-abstraction feature map to obtain the feature map product result. The product of the feature maps is subjected to a logistic regression operation to obtain the probability distribution matrix of regional clustering. The probability distribution matrix of regional clustering is subjected to a probability transformation to obtain the foreground image and background image, thereby extracting the bolt body from the input metal bolt image and obtaining a refined bolt body segmentation map and outputting it.

3. A computer device comprising a processor and a memory, wherein the memory stores a computer program, wherein: When the computer program is executed by a processor, the unsupervised metal bolt visual segmentation method according to claim 1 is implemented.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the unsupervised metal bolt visual segmentation method according to claim 1 is implemented.

Images