A visual segmentation counting method and system suitable for jumbled stacked parts

Abstract

The present application relates to the field of image processing, and proposes a visual segmentation and counting method and system suitable for disordered stacked parts, which avoids the influence of complex construction environment by performing dynamic Gamma correction and overlapping area enhancement processing on multiple view part images, simultaneously highlights and enhances the features of the overlapping area, makes it easier to identify the specific details of the occluded part and small target, improves the accuracy and robustness of identification, then performs basic semantic segmentation through a lightweight CNN model, greatly reduces the calculation amount while ensuring the identification accuracy, improves the identification efficiency, and further produces seed points and region growth through dynamic threshold segmentation, combines the semantic and geometric features of the image, more accurately segments the occluded and small-sized parts, effectively reduces the missed detection rate, and exhibits stronger adaptability and detection accuracy in complex scenes, and the present application improves the accuracy and efficiency of the visual segmentation and counting method of disordered stacked parts.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a visual segmentation counting method and system suitable for jumbled stacked parts. BACKGROUND

[0002] With the rapid development of industrial production, the production speed of parts is gradually increasing, thus a large number of jumbled stacked parts are produced, which brings great challenges to the production quantity calculation and inspection of parts.

[0003] In the prior art, the traditional manual counting method has been difficult to meet the efficient rhythm of production, not only the counting time is long and the counting accuracy is poor, therefore, the image recognition processing algorithm based on computer vision gradually becomes the mainstream, but the existing image processing algorithm, such as the watershed algorithm based on edge detection, when facing a large number of parts in dense occlusion state (overlapping rate greater than 50%), the error is still high and the time consumption is long, especially when the volume of parts is small (pixel area is less than 100px²), small targets cause more missed detection situations, and the robustness is poor, which is easily affected by the complex production counting environment, resulting in further decline of counting accuracy.

[0004] Therefore, if a visual segmentation counting method suitable for jumbled stacked parts is designed to avoid the influence of dense occlusion and small targets and improve the accuracy of counting. SUMMARY

[0005] Based on this, the present application provides a visual segmentation counting method and system suitable for jumbled stacked parts, which avoids the influence of complex construction environment by dynamic Gamma correction and overlapping area enhancement processing of multiple view part images, and highlights and enhances the features of the overlapping area, so that the specific detail information of the occluded part and small target is more easily identified, the accuracy and robustness of identification are improved, a lightweight CNN model is used for basic semantic segmentation, the calculation amount is greatly reduced while ensuring the identification accuracy, the identification efficiency is improved, and dynamic threshold segmentation, seed point production and region growing are used in combination with the semantic and geometric features of the image, so that the occluded and small size parts can be more accurately segmented, the missed detection rate is effectively reduced, and stronger adaptability and detection accuracy are exhibited in complex scenes, the present application improves the accuracy and efficiency of the visual segmentation counting method of jumbled stacked parts.

[0006] The visual segmentation counting method suitable for jumbled stacked parts provided by the present application comprises:

[0007] The multi-view part image based on the top view image, the front view image, the left view image and the right view image of the pipeline detection area is acquired and preprocessed, and the preprocessing includes dynamic Gamma correction and overlapping area enhancement processing;

[0008] The multi-view part image is subjected to semantic segmentation processing according to a lightweight CNN model to acquire a part semantic probability graph, and the lightweight CNN model is based on a depth separable convolution and a cross-layer connection;

[0009] The part semantic probability graph is subjected to dynamic threshold segmentation to acquire a target seed point set, the dynamic threshold segmentation is based on the mean and the standard deviation of the part semantic probability graph, and the target seed point set is based on a connected domain centroid;

[0010] Iteration is performed according to the target seed point set to acquire a part instance set, and the part instance set is based on a part instance mask;

[0011] The part instance set is subjected to overlapping separation processing to acquire an effective part instance collection, so as to calculate the number of parts according to a connected domain analysis algorithm, and the overlapping separation processing is based on a Canny algorithm and a watershed algorithm.

[0012] In summary, according to the above-mentioned visual segmentation counting method for randomly stacked parts, by performing dynamic Gamma correction and overlapping area enhancement processing on multiple view part images, the influence of complex construction environment is avoided, and the features of the overlapping area are highlighted and enhanced, so that the specific detail information of the occluded part and the small target is easier to be identified, the accuracy and robustness of identification are improved, and then a lightweight CNN model is used for basic semantic segmentation, which greatly reduces the calculation amount while ensuring the identification accuracy, improves the identification efficiency, and through dynamic threshold segmentation, seed point production and region growing, combined with the semantic and geometric features of the image, the occluded and small size parts can be more accurately segmented, the missed detection rate is effectively reduced, and stronger adaptability and detection accuracy are exhibited in complex scenes, and the accuracy and efficiency of the visual segmentation counting method for randomly stacked parts are improved. Specifically, multiple view part images are obtained and preprocessed, the multiple view part images are based on top view images, front view images, left view images and right view images of pipeline detection regions, the preprocessing includes dynamic Gamma correction and overlapping area enhancement processing, which avoids the influence of complex construction environment, highlights and enhances the features of the overlapping area, so that the specific detail information of the occluded part and the small target is easier to be identified, the accuracy and robustness of identification are improved, and a lightweight CNN model is used for semantic segmentation processing on the multiple view part images to obtain a part semantic probability graph, the lightweight CNN model is based on depth separable convolution and cross-layer connection, which greatly reduces the calculation amount while ensuring the identification accuracy, improves the identification efficiency, and the part semantic probability graph is subjected to dynamic threshold segmentation to obtain a target seed point set, the dynamic threshold segmentation is based on the mean and standard deviation of the part semantic probability graph, the target seed point set is based on connected domain centroid, and iteration is performed according to the target seed point set to obtain a part instance set, the part instance set is based on part instance mask, the part instance set is subjected to overlapping separation processing to obtain an effective part instance set, and the number of parts is calculated according to the connected domain analysis algorithm, the overlapping separation processing is based on Canny algorithm and watershed algorithm, combined with the semantic and geometric features of the image, the occluded and small size parts can be more accurately segmented, the missed detection rate is effectively reduced, stronger adaptability and detection accuracy are exhibited in complex scenes, and the accuracy and efficiency of the visual segmentation counting method for randomly stacked parts are improved.

[0013] Further, the step of obtaining multiple view part images and preprocessing, specifically includes:

[0014] obtaining multiple view part images;

[0015] performing dynamic Gamma correction on the multiple view part images, and the specific algorithm of the dynamic Gamma correction is as follows:

[0016] ,

[0017] wherein, represents a corrected pixel value, represents an original pixel value, represents a dynamic correction parameter, represents an image brightness mean value;

[0018] According to The spatial kernel and the 15-value range kernel are used to filter the multi-view part image after dynamic Gamma correction, so as to filter and reserve the edge detail information of the multi-view part image.

[0019] The filtered multi-view part image is further subjected to overlapping region enhancement processing, and the specific algorithm of the overlapping region enhancement processing is as follows:

[0020] ,

[0021] wherein, represents a weighted fused multi-view part image, , , , represents a top view image, a front view image, a left view image and a right view image of a pipeline detection region, , , , represents a weight of a different view image, and the weight of the different view image is based on a part type.

[0022] Further, the step of performing semantic segmentation processing on the multi-view part image according to the lightweight CNN model to obtain a part semantic probability map specifically comprises:

[0023] The multi-view part image is subjected to scaling processing, the scaling processing adjusts the multi-view part image to 512x512 pixels, and the multi-view part image is subjected to normalization processing, and the specific algorithm of the normalization processing is as follows:

[0024] ,

[0025] wherein, represents a weighted fused multi-view part image, represents a normalized multi-view part image;

[0026] The normalized multi-view part image is input into a lightweight CNN model, the lightweight CNN model comprising 8 layers of depth separable convolution, the lightweight CNN model comprising a cross-layer connection structure for retaining and enhancing edge features of a bottom layer to output a part semantic probability map, the part semantic probability map being 128x128 pixels, the part semantic probability map having a channel number of , wherein N represents the number of part types, the part types including bolts and gears.

[0027] Further, the step of performing dynamic threshold segmentation on the part semantic probability map to obtain a target seed point set specifically comprises:

[0028] The part semantic probability map is divided into 16x16 regions, each of the regions comprising a 32x32 pixel sub-block, and the mean value and standard deviation of each sub-block in the part semantic probability map are calculated, the specific algorithm of the mean value and standard deviation being as follows:

[0029] ,

[0030] ,

[0031] wherein, represents the mean value, represents the standard deviation, represents the part semantic probability map, represents the sub-block serial number;

[0032] The dynamic threshold of each sub-block is calculated, and the specific algorithm of the dynamic threshold is as follows:

[0033] ,

[0034] wherein, represents the dynamic threshold;

[0035] A binary mask is obtained according to the dynamic threshold and the part semantic probability map for comparison and judgment, if the part semantic probability map of any region is greater than or equal to the dynamic threshold, the value of the binary mask in the region is 1, and if the part semantic probability map of any region is less than the dynamic threshold, the value of the binary mask in the region is 0.

[0036] According to morphological image processing, the binary mask is eroded, and the specific algorithm of the erosion is as follows:

[0037] ,

[0038] wherein, represents the eroded mask, represents a binary mask, represents nuclei;

[0039] After the mask is corroded, 8-neighbor connected domain marking is performed to calculate the connected domain centroid of each connected domain, and the set of connected domain centroids is taken as the target seed point set. The specific algorithm of the connected domain centroid is as follows:

[0040]

[0041]

[0042] wherein, represents the horizontal coordinate of the seed point corresponding to the connected domain, represents the horizontal coordinate of the seed point corresponding to the connected domain, represents the number of pixel points in the connected domain, represents the index ordinal of the pixel point in the connected domain, and represents the horizontal and vertical coordinates of the pixel point in the connected domain.

[0043] Further, the step of iterating according to the target seed point set to obtain the part instance set specifically comprises:

[0044] Extract the RGB color value of each seed point in the target seed point set, which is taken as the iteration growth reference color. The seed point is iteratively grown, and the iteration growth is expanded according to the upper neighborhood, lower field, left neighborhood and right neighborhood of the seed point. The iteration growth stops when the preset maximum iteration round or the preset maximum non-update round is reached, to obtain the part instance set. Each part instance in the part instance set has a unique corresponding part instance mask.

[0045] Further, the step of performing overlap separation processing on the part instance set to obtain the effective part instance set specifically comprises:

[0046] According to the Canny algorithm, the contour of each part instance mask in the part instance set is extracted to obtain a closed contour set. The minimum pixel distance between any two closed contours in the closed contour set is calculated. If the minimum pixel distance is less than 20 pixel values, the overlap separation processing is performed on the part instance mask corresponding to the closed contour based on the sobel gradient graph.

[0047] ​​The area of the part instance mask after the overlap separation processing is screened, if the area of the part instance mask is less than a preset mask area threshold, the part instance mask is deleted, the solid degree of the remaining part instance mask after the deletion is calculated, and the specific algorithm of the solid degree calculation is as follows:

[0048] ,

[0049] Wherein, Solid degree, Mask area, Convex hull area;

[0050] According to the solid degree, the solid degree screening is performed, if the solid degree of the part instance mask is less than or equal to a preset solid degree threshold, the part instance mask is deleted;

[0051] The deleted part instance mask forms an effective part instance set.

[0052] Further, the step of forming the effective part instance set after the deletion of the part instance mask includes:

[0053] According to the connected domain analysis algorithm, all connected regions in the effective part instance set are identified, the number of parts is calculated according to the number of connected regions, each connected region has a unique corresponding part, and the number is divided according to the type of the part.

[0054] The application provides a visual segmentation and counting system suitable for disordered stacked parts, which comprises:

[0055] A preprocessing module is configured to acquire and preprocess multi-view part images, wherein the multi-view part images are based on top-view images, front-view images, left-view images and right-view images of a pipeline detection area, and the preprocessing includes dynamic Gamma correction and overlapping area enhancement processing.

[0056] A semantic segmentation module is configured to perform semantic segmentation processing on the multi-view part images based on a lightweight CNN model to acquire part semantic probability maps, wherein the lightweight CNN model is based on depth separable convolution and cross-layer connection.

[0057] A dynamic threshold segmentation module is configured to perform dynamic threshold segmentation on the part semantic probability maps to acquire a target seed point set, wherein the dynamic threshold segmentation is based on the mean and standard deviation of the part semantic probability maps, and the target seed point set is based on connected domain centroids.

[0058] a seed iteration module configured to iterate according to the set of target seed points to obtain a set of part instances, the set of part instances being based on a part instance mask;

[0059] a separation counting module configured to perform an overlap separation process on the set of part instances to obtain a valid part instance collection, and to calculate a part quantity according to a connected component analysis algorithm, the overlap separation process being based on a Canny algorithm and a watershed algorithm.

[0060] The application further provides a storage medium storing one or more programs, which, when executed by a processor, implement the visual segmentation counting method for parts in a cluttered stack as described above.

[0061] The application further provides a computer device comprising a memory and a processor, wherein:

[0062] The memory is configured to store a computer program;

[0063] The processor is configured to execute the computer program stored in the memory to implement the visual segmentation counting method for parts in a cluttered stack as described above. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 a flowchart of the visual segmentation counting method for parts in a cluttered stack according to the first embodiment of the application;

[0065] Figure 2 a structural schematic diagram of the visual segmentation counting system for parts in a cluttered stack according to the second embodiment of the application.

[0066] The following detailed description will further illustrate the application in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0067] In order to facilitate the understanding of the application, the application will be described more fully below with reference to the accompanying drawings. The application is illustrated in several embodiments in the drawings. However, the application can be realized in many different forms and is not limited to the embodiments described herein. On the contrary, these embodiments are provided to make the disclosure of the application more thorough and comprehensive.

[0068] It should be noted that when an element is referred to as being "fixedly attached" to another element, it can be directly on the other element or there can be an intervening element. When an element is referred to as being "connected" to another element, it can be directly connected to the other element or intervening elements can be present. The terms "vertical", "horizontal", "left", "right", and the like as used herein are for purposes of illustration and description only.

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application. The use herein of the terms "and / or" includes a set of one or more associated listed items.

[0070] Referring to FIG. 1, which shows a flow chart of a visual segmentation counting method for randomly stacked parts according to a first embodiment of the present application, the visual segmentation counting method for randomly stacked parts includes steps S01 to S05, wherein: Figure 1

[0071] Step S01: Obtain multi-view part images and perform preprocessing;

[0072] It should be noted that in this embodiment, the multi-view part images are based on top view, front view, left view and right view images of the pipeline detection area, and the preprocessing includes dynamic Gamma correction and overlapping area enhancement processing.

[0073] In this embodiment, four industrial area array cameras are installed horizontally above and around the industrial pipeline detection station, respectively in top view, front view, left view and right view, wherein the camera in top view is 50 cm high from the stacked table, and its optical axis is perpendicular to the table, while the optical axes of the remaining cameras are parallel to the table, and the three cameras are spaced 90 degrees apart. The four cameras can be Basler-acA1920-155um, with a resolution of 1920x1280 and a frame rate of 15fps. A ring-shaped shadowless light source is arranged below the top camera, and a strip-shaped light source is arranged near the left and right cameras. The ring-shaped shadowless light source is white light, and the strip-shaped light source is blue light with a wavelength of 450nm. When the photoelectric sensor at the end of the conveyor belt detects that the part stacking tray is in place, the four cameras are triggered to start synchronously by a TTL signal, and the ring-shaped shadowless light source and the two side strip-shaped light sources are turned on simultaneously. The exposure is completed within 10ms, and finally four timestamped RGB images are obtained, named Top_Image.jpg, Front_Image.jpg, Left_Image.jpg and Right_Image.jpg, with a resolution of 1920x1280, and saved to the edge computing unit.

[0074] Obtain multi-view part images;

[0075] Perform dynamic Gamma correction on the multi-view part images, and the specific algorithm of the dynamic Gamma correction is as follows:

[0076] ​ ,

[0077] wherein, represents a corrected pixel value, represents an original pixel value, represents a dynamic correction parameter, represents an image brightness mean value;

[0078] According to The spatial kernel and the 15-value range kernel are used to filter the multi-view part image after dynamic Gamma correction, so as to filter and reserve the edge detail information of the multi-view part image.

[0079] The multi-view part image after filtering is further subjected to overlapping area enhancement processing, and the specific algorithm of the overlapping area enhancement processing is as follows:

[0080] ,

[0081] wherein, represents a weighted fused multi-view part image, , , , represents a top view image, a front view image, a left view image and a right view image of a pipeline detection area, , , , represents a weight of different view images, and the weight of the different view images is based on the part type.

[0082] Step S02: performing semantic segmentation processing on the multi-view part image according to the lightweight CNN model to obtain a part semantic probability map;

[0083] It should be noted that in the embodiment, the lightweight CNN model is based on deep separable convolution and cross-layer connection, and the multi-view part image is subjected to scaling processing, which adjusts the multi-view part image to 512x512 pixels, and the multi-view part image is subjected to normalization processing, and the specific algorithm of the normalization processing is as follows:

[0084] ,

[0085] wherein, represents a weighted fused multi-view part image, represents a normalized multi-view part image;

[0086] The normalized multi-view part image is input into a lightweight CNN model, the lightweight CNN model comprising 8 layers of depth separable convolution, the lightweight CNN model comprising a cross-layer connection structure for retaining and enhancing edge features of a bottom layer to output a part semantic probability map, the part semantic probability map being 128x128 pixels, the part semantic probability map having a channel number of , wherein N represents a number of part types, the part types including bolts and gears.

[0087] Step S03: performing dynamic threshold segmentation on the part semantic probability map to obtain a target seed point set;

[0088] It should be noted that in the embodiment, the dynamic threshold segmentation is based on the mean and standard deviation of the part semantic probability map, the target seed point set is based on the connected domain centroid, the part semantic probability map is divided into 16x16 regions, each of the regions comprising a 32x32 pixel sub-block, the mean and standard deviation of each sub-block in the part semantic probability map are calculated, and the specific algorithm of the mean and standard deviation is as follows:

[0089] ,

[0090] ,

[0091] wherein represents the mean, represents the standard deviation, represents the part semantic probability map, represents a sub-block serial number;

[0092] The dynamic threshold of each sub-block is calculated, and the specific algorithm of the dynamic threshold is as follows:

[0093] ,

[0094] wherein represents the dynamic threshold;

[0095] A binary mask is obtained according to the dynamic threshold and the part semantic probability map for comparison and judgment, if the part semantic probability map of any region is greater than or equal to the dynamic threshold, the value of the binary mask in the region is 1, and if the part semantic probability map of any region is less than the dynamic threshold, the value of the binary mask in the region is 0;

[0096] According to morphological image processing, the binary mask is eroded, and the specific algorithm of the erosion is as follows:

[0097] ,

[0098] wherein, represents the mask after erosion, represents the binary mask, represents nuclei;

[0099] The mask after erosion is subjected to 8-neighbor connected component labeling to calculate the connected component centroid of each connected component, and the set of connected component centroids is taken as the target seed point set. The specific algorithm of the connected component centroid is as follows:

[0100] ,

[0101] ,

[0102] wherein, represents the horizontal coordinate of the seed point corresponding to the connected component, represents the horizontal coordinate of the seed point corresponding to the connected component, represents the number of pixel points in the connected component, represents the index ordinal of the pixel point in the connected component, and represents the horizontal coordinate and the vertical coordinate of the pixel point in the connected component.

[0103] Step S04: iteration is performed according to the target seed point set to obtain a part instance set;

[0104] It should be noted that in the embodiment, the part instance set is based on the part instance mask, and the RGB color value of each seed point in the target seed point set is extracted, which is taken as the iteration growth reference color. The seed point is subjected to iteration growth, which is expanded according to the upper neighborhood, lower field, left neighborhood and right neighborhood of the seed point. The iteration growth stops when the preset maximum iteration round or the preset maximum non-updating round is reached, so as to obtain the part instance set. Each part instance in the part instance set has a unique corresponding part instance mask.

[0105] In the embodiment, the preset maximum iteration round is 50 generations, and the preset maximum non-updating round is 5 generations. The non-updating round refers to the round of continuous non-addition of new pixels.

[0106] Step S05: overlapping separation processing is performed on the part instance set to obtain an effective part instance collection, so as to calculate the number of parts according to the connected component analysis algorithm.

[0107] It should be noted that in the embodiment, the overlap separation processing is based on the Canny algorithm and the watershed algorithm. According to the Canny algorithm, the contour of each part instance mask in the part instance set is extracted to obtain a closed contour set. The minimum pixel distance between any two closed contours in the closed contour set is calculated. If the minimum pixel distance is less than 20 pixel values, the overlap separation processing is performed on the part instance mask corresponding to the closed contour based on the sobel gradient map.

[0108] The area of the part instance mask is screened. If the area of the part instance mask is less than a preset mask area threshold, the part instance mask is deleted. The solidity of the remaining part instance mask after deletion is calculated. The specific algorithm of the solidity calculation is as follows:

[0109] ,

[0110] wherein, solidity, mask area, convex hull area;

[0111] The solidity is screened according to the solidity. If the solidity of the part instance mask is less than or equal to a preset solidity threshold, the part instance mask is deleted.

[0112] In the embodiment, the preset mask area threshold is 100, and the preset solidity threshold is 0.4.

[0113] The deleted part instance mask forms an effective part instance set;

[0114] According to the connected domain analysis algorithm, all connected regions in the effective part instance set are identified. The number of parts is calculated according to the number of connected regions. Each connected region has a unique corresponding part. The number is divided according to the type of the part.

[0115] In summary, according to the above-mentioned visual segmentation counting method suitable for disordered stacked parts, by performing dynamic Gamma correction and overlapping area enhancement processing on multiple view part images, the influence of complex construction environment is avoided, and the features of the overlapping area are highlighted and enhanced, so that the specific detail information of the occluded part and the small target is easier to be identified, the accuracy and robustness of identification are improved, and then a lightweight CNN model is used for basic semantic segmentation, while ensuring the identification accuracy, the calculation amount is greatly reduced, and the identification efficiency is improved, and then through dynamic threshold segmentation and seed point production and region growing, combined with the semantic and geometric features of the image, the occluded and small size parts can be more accurately segmented, the missed detection rate is effectively reduced, and stronger adaptability and detection accuracy are shown in complex scenes, and the accuracy and efficiency of the visual segmentation counting method of the disordered stacked parts are improved. Specifically, multiple view part images are obtained and preprocessed, the multiple view part images are based on top view images, front view images, left view images and right view images of pipeline detection regions, and the preprocessing includes dynamic Gamma correction and overlapping area enhancement processing, which avoids the influence of complex construction environment, highlights and enhances the features of the overlapping area, so that the specific detail information of the occluded part and the small target is easier to be identified, the accuracy and robustness of identification are improved, and a lightweight CNN model is used for semantic segmentation processing on the multiple view part images to obtain a part semantic probability graph, the lightweight CNN model is based on depth separable convolution and cross-layer connection, while ensuring the identification accuracy, the calculation amount is greatly reduced, and the identification efficiency is improved, dynamic threshold segmentation is performed on the part semantic probability graph to obtain a target seed point set, the dynamic threshold segmentation is based on the mean and standard deviation of the part semantic probability graph, the target seed point set is based on the connected domain centroid, and iteration is performed according to the target seed point set to obtain a part instance set, the part instance set is based on a part instance mask, overlapping separation processing is performed on the part instance set to obtain an effective part instance set, and the number of parts is calculated according to a connected domain analysis algorithm, the overlapping separation processing is based on Canny algorithm and watershed algorithm, combined with the semantic and geometric features of the image, the occluded and small size parts can be more accurately segmented, the missed detection rate is effectively reduced, stronger adaptability and detection accuracy are shown in complex scenes, and the accuracy and efficiency of the visual segmentation counting method of the disordered stacked parts are improved.

[0116] Please refer to Figure 2 , which is a structure schematic diagram of the visual segmentation counting system suitable for disordered stacked parts according to the second embodiment of the present application, and the system comprises:

[0117] The preprocessing module 10 is used for acquiring and preprocessing multi-view part images, the multi-view part images are based on top-view images, front-view images, left-view images and right-view images of a pipeline detection area, and the preprocessing includes dynamic Gamma correction and overlapping area enhancement processing;

[0118] The semantic segmentation module 20 is used for performing semantic segmentation processing on the multi-view part images according to a lightweight CNN model to acquire part semantic probability maps, and the lightweight CNN model is based on deep separable convolution and cross-layer connection.

[0119] The dynamic threshold segmentation module 30 is used for performing dynamic threshold segmentation on the part semantic probability maps to acquire a target seed point set, and the dynamic threshold segmentation is based on a mean value and a standard deviation of the part semantic probability maps, and the target seed point set is based on connected domain centroids.

[0120] The seed iteration module 40 is used for performing iteration according to the target seed point set to acquire a part instance set, and the part instance set is based on part instance masks.

[0121] The separation counting module 50 is used for performing overlapping separation processing on the part instance set to acquire an effective part instance set, and the number of parts is calculated according to a connected domain analysis algorithm, and the overlapping separation processing is based on Canny algorithm and watershed algorithm.

[0122] The application further provides a computer storage medium, which stores one or more programs, and the programs are executed by a processor to implement the visual segmentation counting method for parts in a cluttered stack.

[0123] The application further provides a computer device, which comprises a memory and a processor, wherein the memory is used for storing a computer program, and the processor is used for executing the computer program stored in the memory to implement the visual segmentation counting method for parts in a cluttered stack.

[0124] Those skilled in the art can understand that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequence list of executable instructions for implementing the logic function, which can be specifically implemented in any computer readable medium for use by or in conjunction with an instruction execution system, device or equipment, such as a computer-based system, a system including a processor or other system that can fetch and execute instructions from the instruction execution system, device or equipment. For the purpose of the present description, the "computer readable medium" can be any device that can contain a storage, communication, propagation or transmission program for use by or in conjunction with the instruction execution system, device or equipment or in conjunction with these instruction execution systems, devices or equipment.

[0125] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can also be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via an optical scanner, then compiled, interpreted, or otherwise processed, and stored in a computer memory in a form that is then employable by a computer. In some embodiments, the example computer-readable storage medium can be non-transitory. For example, in some embodiments, the example computer-readable storage medium can be a non-transitory medium.

[0126] It should be understood that aspects of the application can be implemented in hardware, software, firmware, or combinations thereof. In the embodiments described above, various steps or methods can be implemented, for example, by software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following techniques, which are well known in the art, can be used to implement the application: a hybrid of the above techniques, discrete logic circuit(s) having logic gates for implementing logic functions upon request pins of the discrete logic circuit(s), programmable logic array(s) (PLAs), field programmable gate array(s) (FPGAs), etc.

[0127] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific feature, structure, material or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any suitable manner in one or more embodiments or examples.

[0128] The above-described embodiments are merely some embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as limiting the scope of the patent of the present application. It should be noted that, for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the scope of protection of the patent of the present application should be subject to the appended claims.

Claims

1. A method for visual part counting suitable for a jumbled stack of parts, characterized by, The method comprises the following steps: acquiring multi-view part images and performing preprocessing, the multi-view part images being based on top-view images, front-view images, left-view images and right-view images of a pipeline detection area, and the preprocessing comprising dynamic Gamma correction and overlapping area enhancement processing; performing semantic segmentation processing on the multi-view part images according to a lightweight CNN model to acquire part semantic probability maps, the lightweight CNN model being based on deep separable convolution and cross-layer connection; performing dynamic threshold segmentation on the part semantic probability maps to acquire a target seed point set, the dynamic threshold segmentation being based on the mean value and standard deviation of the part semantic probability maps, and the target seed point set being based on connected domain centroids; the step of performing dynamic threshold segmentation on the part semantic probability maps to acquire a target seed point set specifically comprises: dividing the part semantic probability maps into 16*16 regions, each of the regions comprising a 32*32-pixel sub-block, calculating the mean value and standard deviation of each sub-block in the part semantic probability maps, and the specific algorithm of the mean value and standard deviation being as follows: ， ， wherein, denotes the mean, denotes the standard deviation, denotes the part semantic probability map, denotes the sub-block ordinal number; calculating the dynamic threshold value of each sub-block, and the specific algorithm of the dynamic threshold value being as follows: ， wherein represents a dynamic threshold value; acquiring a binary mask according to the dynamic threshold value and the part semantic probability maps to perform comparison and judgment, if the part semantic probability maps of any region are greater than or equal to the dynamic threshold value, the value of the binary mask in the region is 1, and if the part semantic probability maps of any region are less than the dynamic threshold value, the value of the binary mask in the region is 0; performing corrosion on the binary mask according to morphological image processing, and the specific algorithm of the corrosion being as follows: ， wherein, represents the mask after etching, represents the binary mask, represents nucleus; performing 8-neighbor connected domain labeling on the corroded mask to calculate the connected domain centroids of each connected domain, and taking the set of the connected domain centroids as the target seed point set, and the specific algorithm of the connected domain centroids being as follows: ， ， wherein, represents the horizontal coordinate of the seed point corresponding to the connected domain, represents the horizontal coordinate of the seed point corresponding to the connected domain, represents the number of pixel points within the connected domain, represents the index ordinal of the pixel point within the connected domain, and represents the horizontal and vertical coordinates of the pixel point within the connected domain; iterating according to the target seed point set to acquire a part instance set, the part instance set being based on part instance masks; performing overlapping separation processing on the part instance set to acquire an effective part instance set, and calculating the number of parts according to a connected domain analysis algorithm, the overlapping separation processing being based on Canny algorithm and watershed algorithm.

2. The method for visual part count suitable for jumbled stacked parts of claim 1, wherein, The step of acquiring multi-view part images and performing preprocessing specifically comprises: acquiring multi-view part images; performing dynamic Gamma correction on the multi-view part images, and the specific algorithm of the dynamic Gamma correction being as follows: ， wherein, denotes the corrected pixel value, denotes the original pixel value, denotes the dynamic correction parameter, denotes the image brightness mean value; According to The spatial kernel and the 15-value range kernel filter the multi-view part image after dynamic gamma correction to filter and reserve edge detail information of the multi-view part image. performing overlapping area enhancement processing on the multi-view part images after the filtering processing, and the specific algorithm of the overlapping area enhancement processing being as follows: ， wherein, denotes the weighted fused multi-view part image, , , , denotes the top view image, the front view image, the left view image and the right view image of the pipeline detection area, , , , denotes the weight of different view images, the weight of different view images is based on part type.

3. The method for visual part count suitable for jumbled stacked parts of claim 1, wherein, The step of performing semantic segmentation processing on the multi-view part images according to a lightweight CNN model to acquire part semantic probability maps specifically comprises: performing scaling processing on the multi-view part images, the scaling processing adjusting the multi-view part images to 512*512 pixels, and performing normalization processing on the multi-view part images, and the specific algorithm of the normalization processing being as follows: ， wherein, denotes the weighted fused multi-view part image, denotes the normalized multi-view part image; The normalized multi-view part image is input into a lightweight CNN model, the lightweight CNN model includes 8 layers of depth separable convolution, the lightweight CNN model includes a cross-layer connection structure, the cross-layer connection structure is used for retaining and enhancing edge features of a bottom layer, and a part semantic probability map is output, the part semantic probability map is 128*128 pixels, and the number of channels of the part semantic probability map is , the N indicates the number of part types, and the part types include bolts and gears.

4. The method for visual part count suitable for jumbled stacked parts of claim 1, wherein, The step of performing iteration according to the target seed point set to obtain a part instance set specifically comprises: extracting an RGB color value of each seed point in the target seed point set, the RGB color value serving as an iteration growth reference color, performing iteration growth on the seed point, the iteration growth being expanded according to an upper neighborhood, a lower field, a left neighborhood and a right neighborhood of the seed point, the iteration growth being stopped at a preset maximum iteration round or a preset maximum non-updating round to obtain a part instance set, each part instance in the part instance set having a unique corresponding part instance mask.

5. The method for visual part count suitable for jumbled stacked parts of claim 1, wherein, The step of performing overlap separation processing on the part instance set to obtain an effective part instance set specifically comprises: performing contour extraction on each part instance mask in the part instance set according to a Canny algorithm to obtain a closed contour set, calculating a minimum pixel distance between any two closed contours in the closed contour set, and performing screening judgment on the minimum pixel distance, if the minimum pixel distance is less than 20 pixel values, performing overlap separation processing on the part instance mask corresponding to the closed contour based on a sobel gradient graph; performing area screening on the part instance mask after the overlap separation processing, if the area of the part instance mask is less than a preset mask area threshold, deleting the part instance mask, and performing solidity calculation on the part instance mask remaining after the deletion, a specific algorithm of the solidity calculation being as follows: ， wherein, represents the solid degree, represents the mask area, represents the convex hull area; performing solidity screening according to the solidity, if the solidity of the part instance mask is less than or equal to a preset solidity threshold, deleting the part instance mask; forming an effective part instance set from the deleted part instance masks.

6. The method for visual part count suitable for jumbled stacked parts of claim 5, wherein, The step of forming an effective part instance set from the deleted part instance masks further comprises: identifying all connected regions in the effective part instance set according to a connected domain analysis algorithm, calculating the number of parts according to the number of the connected regions, each connected region having a unique corresponding part, and dividing the number of parts according to the types of the parts.

7. A vision part counting system suitable for use with a jumbled stack of parts, the system comprising: a conveyor for moving the stack of parts; a camera for capturing images of the stack of parts; a processor for processing the images; and a display for displaying the processed images. It comprises: a preprocessing module configured to obtain and preprocess multi-view part images, the multi-view part images being based on top-view images, front-view images, left-view images and right-view images of a pipeline detection region, the preprocessing comprising dynamic Gamma correction and overlap region enhancement processing; a semantic segmentation module configured to perform semantic segmentation processing on the multi-view part images according to a lightweight CNN model to obtain part semantic probability maps, the lightweight CNN model being based on depth separable convolution and cross-layer connection; a dynamic threshold segmentation module configured to perform dynamic threshold segmentation on the part semantic probability maps to obtain a target seed point set, the dynamic threshold segmentation being based on a mean value and a standard deviation of the part semantic probability maps, and the target seed point set being based on connected domain centroids. The step of performing dynamic threshold segmentation on the part semantic probability maps to obtain a target seed point set specifically comprises: The part semantic probability map is divided into 16*16 regions, each of which includes a 32*32 pixel sub-block, the mean and standard deviation of each sub-block in the part semantic probability map are calculated, and the specific algorithm of the mean and standard deviation is as follows: ， ， wherein, denotes the mean, denotes the standard deviation, denotes the part semantic probability graph, denotes the sub-block ordinal number; The dynamic threshold of each sub-block is calculated, and the specific algorithm of the dynamic threshold is as follows: ， wherein represents a dynamic threshold value; A binary mask is obtained according to the dynamic threshold and the part semantic probability map for comparison and judgment, if the part semantic probability map of any region is greater than or equal to the dynamic threshold, the value of the binary mask in the region is 1, if the part semantic probability map of any region is less than the dynamic threshold, the value of the binary mask in the region is 0; According to morphological image processing, the binary mask is eroded, and the specific algorithm of the erosion is as follows: ， wherein, represents the mask after etching, represents the binary mask, represents nucleus; The mask after erosion is marked in 8-neighbor connected domain to calculate the connected domain centroid of each connected domain, and the set of connected domain centroids is taken as the target seed point set, and the specific algorithm of the connected domain centroid is as follows: ， ， wherein, represents the horizontal coordinate of the seed point corresponding to the connected domain, represents the horizontal coordinate of the seed point corresponding to the connected domain, represents the number of pixel points within the connected domain, represents the index ordinal of the pixel point within the connected domain, and represents the horizontal and vertical coordinates of the pixel point within the connected domain; A seed iteration module is used to iterate according to the target seed point set to obtain a part instance set, and the part instance set is based on a part instance mask. A separation counting module is used to perform overlapping separation processing on the part instance set to obtain an effective part instance set, and the number of parts is calculated according to a connected domain analysis algorithm, and the overlapping separation processing is based on Canny algorithm and watershed algorithm.

8. A storage medium, characterized by The storage medium stores one or more programs, which are executed by the processor to implement the visual segmentation counting method for parts in a cluttered stack according to any one of claims 1-6.

9. A computer device, comprising: The computer device includes a memory and a processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer programs stored in the memory to implement the visual segmentation counting method for parts in a cluttered stack according to any one of claims 1-6. The computer device includes a memory and a processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer programs stored in the memory to implement the visual segmentation counting method for parts in a cluttered stack according to any one of claims 1-6.

Images