Connecting piece surface defect detection device, method and equipment based on multi-channel model

Abstract

The invention discloses a connecting piece surface defect detection device, method and equipment based on a multi-channel model, and belongs to the technical field of electric power facilities. The device comprises an image acquisition module used for acquiring a color image and a height image of the connecting piece; the feature extraction module is used for extracting detail features and semantic features of the color image through a multi-channel model and extracting detail features and semantic features of the height image; the feature fusion module is used for fusing the detail features to obtain a detail feature fusion result and fusing the semantic features to obtain a semantic feature fusion result; and the defect detection module is used for determining whether a defect exists on the surface of the connecting piece based on an aggregation result of the detail feature fusion result and the semantic feature fusion result. According to the technical scheme, the influence of different features in the color image and the height image on defect detection can be comprehensively considered, whether defects exist on the surface of the connecting piece or not is recognized according to the fused features, and the accuracy of surface defect detection of the connecting piece is improved.

Description

Technical Field

[0001] The present application belongs to the technical field of electric power facilities, and specifically relates to a device, method and equipment for detecting surface defects of connectors based on a multi-channel model. Background Art

[0002] Cable connectors are components that connect cables in cable lines. They can provide stable electrical connections, reduce contact resistance, and prevent faults such as cable heating caused by poor connections.

[0003] Cable connectors need to undergo yield testing before leaving the factory to ensure that the produced cable connectors meet the actual use requirements. However, the current yield testing mainly relies on manual testing to identify defective connectors. This method has low detection efficiency and poor accuracy, which affects the factory quality of cable connectors.

[0004] Therefore, how to accurately and efficiently detect whether there are defects on the surface of the connecting part is a technical problem that needs to be solved urgently by technical personnel in this field. Summary of the invention

[0005] The embodiments of the present application provide a device, method and equipment for detecting surface defects of connectors based on a multi-channel model, the purpose of which is to comprehensively consider the impact of different features in color images and height images on defect detection, identify whether there are defects on the surface of connectors based on the fused features, and improve the accuracy of surface defect detection of connectors.

[0006] In a first aspect, an embodiment of the present application provides a device for detecting surface defects of a connector based on a multi-channel model, the device comprising:

[0007] An image acquisition module, used for acquiring a color image and a height image of the connection piece;

[0008] A feature extraction module, used to extract detail features of the color image through the first channel of the multi-channel model; extract semantic features of the color image through the second channel of the multi-channel model; extract detail features of the height image through the third channel of the multi-channel model; extract semantic features of the height image through the fourth channel of the multi-channel model;

[0009] A feature fusion module is used to fuse the detail features of the color image and the height image to obtain a detail feature fusion result, and to fuse the semantic features of the color image and the height image to obtain a semantic feature fusion result;

[0010] The defect detection module is used to aggregate the detail feature fusion result and the semantic feature fusion result, and determine whether there is a defect on the surface of the connecting part based on the aggregation result.

[0011] Furthermore, the feature fusion module is specifically used for:

[0012] Extracting height information of each pixel in the detail feature of the height image, and fusing the height information with feature information of the same pixel in the detail feature of the color image to obtain a detail feature fusion result;

[0013] as well as,

[0014] The first weight of the semantic feature of the color image and the second weight of the semantic feature of the height image are obtained, and the semantic information of each image region is determined according to the first weight and the second weight to obtain a semantic feature fusion result.

[0015] Furthermore, the feature fusion module is specifically used for:

[0016] Extract edge height information of edge pixels in the detail feature fusion result, identify whether there is a mutation value in the edge height information, and if so, take the edge pixel corresponding to the mutation value as an abnormal pixel;

[0017] Acquire height information of pixels on the same edge as the abnormal pixel, and acquire a gradual change rule of the height information of the pixels on the same edge;

[0018] The edge height information replacement value of the abnormal pixel point is determined according to the gradual change rule.

[0019] Furthermore, the defect detection module is specifically used to:

[0020] According to the detail feature fusion result, the edge of the connector image area corresponding to each semantic information in the semantic feature fusion result is optimized, and the height information of each pixel point in the connector image area is labeled to obtain the aggregation result of the detail feature fusion result and the semantic feature fusion result.

[0021] Furthermore, the defect detection module is also used to:

[0022] identifying the edge shape of the connection hole on the surface of the connector according to the polymerization result;

[0023] The edge shape is fitted with a standard arc shape, and the neatness of the shape of the connection hole is determined according to the fitting result. If the neatness does not meet the production requirements of the connection piece, it is determined that there are defects on the surface of the connection piece.

[0024] Furthermore, the defect detection module is also used to:

[0025] Obtaining an average height of the surface of the connector according to the polymerization result;

[0026] Calculating the difference between the height information in each position area on the surface of the connecting member and the average height;

[0027] If the difference is smaller than the first preset error, it is determined that the current position area is a surface concave defect; if the difference exceeds the second preset error, it is determined that the current position area is a surface convex defect.

[0028] Furthermore, the defect detection module is specifically used to:

[0029] Pre-acquire the aggregation results of multiple connection parts surfaces as training samples, mark the defect conditions of the aggregation results in the training samples according to whether there are defects on the connection parts surfaces, obtain the defect labels in the training samples, use the training samples with the defect labels as input variables, perform supervised training, and construct a defect condition classifier;

[0030] The aggregation result of the current connection part is input into the defect condition classifier, and whether there is a defect on the surface of the connection part is determined according to the output result of the defect condition classifier.

[0031] In a second aspect, an embodiment of the present application provides a method for detecting surface defects of a connector based on a multi-channel model, the method comprising:

[0032] Acquire color images and height images of the connection parts;

[0033] Extracting detail features of the color image through the first channel of the multi-channel model; extracting semantic features of the color image through the second channel of the multi-channel model; extracting detail features of the height image through the third channel of the multi-channel model; extracting semantic features of the height image through the fourth channel of the multi-channel model;

[0034] The detail features of the color image and the height image are fused to obtain a detail feature fusion result, and the semantic features of the color image and the height image are fused to obtain a semantic feature fusion result;

[0035] The detail feature fusion result and the semantic feature fusion result are aggregated, and whether there is a defect on the surface of the connecting part is determined based on the aggregation result.

[0036] Further, the detail features of the color image and the height image are fused to obtain a detail feature fusion result, and the semantic features of the color image and the height image are fused to obtain a semantic feature fusion result, including:

[0037] Extracting height information of each pixel in the detail feature of the height image, and fusing the height information with feature information of the same pixel in the detail feature of the color image to obtain a detail feature fusion result;

[0038] as well as,

[0039] The first weight of the semantic feature of the color image and the second weight of the semantic feature of the height image are obtained, and the semantic information of each image region is determined according to the first weight and the second weight to obtain a semantic feature fusion result.

[0040] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0041] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0042] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0043] In an embodiment of the present application, an image acquisition module is used to acquire a color image and a height image of a connector; a feature extraction module is used to extract detail features of the color image through a first channel of a multi-channel model; extract semantic features of the color image through a second channel of the multi-channel model; extract detail features of the height image through a third channel of the multi-channel model; and extract semantic features of the height image through a fourth channel of the multi-channel model; a feature fusion module is used to fuse the detail features of the color image with the detail features of the height image to obtain a detail feature fusion result, and to fuse the semantic features of the color image with the height image to obtain a semantic feature fusion result; and a defect detection module is used to aggregate the detail feature fusion result and the semantic feature fusion result, and determine whether there is a defect on the surface of the connector based on the aggregation result.

[0044] Through the above-mentioned connector surface defect detection device based on the multi-channel model, the influence of different features in the color image and the height image on defect detection can be comprehensively considered, and whether there are defects on the connector surface can be identified according to the fused features, thereby improving the accuracy of connector surface defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1It is a structural schematic diagram of a connection surface defect detection device based on a multi-channel model provided in Example 1 of the present application;

[0046] Figure 2 It is a structural schematic diagram of a connection surface defect detection device based on a multi-channel model provided in Example 2 of the present application;

[0047] Figure 3 is a schematic diagram of the structure of a connection surface defect detection device based on a multi-channel model provided in Example 3 of the present application;

[0048] Figure 4 It is a flow chart of a method for detecting surface defects of connectors based on a multi-channel model provided in the fourth embodiment of the present application;

[0049] Figure 5 It is a structural diagram of an electronic device provided in Example 5 of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0051] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0052] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0053] In conjunction with the accompanying drawings, the following detailed description is given of the device, method and equipment for detecting surface defects of connectors based on a multi-channel model provided in the embodiments of the present application through specific embodiments and their application scenarios.

[0054] Embodiment 1

[0055] Figure 1 Schematic diagram of the structure of the device for detecting surface defects of connectors based on a multi-channel model provided in the first embodiment of the present application. Figure 1 As shown, the specific steps include:

[0056] An image acquisition module 101 is used to acquire a color image and a height image of the connection member;

[0057] The feature extraction module 102 is used to extract the detail features of the color image through the first channel of the multi-channel model; extract the semantic features of the color image through the second channel of the multi-channel model; extract the detail features of the height image through the third channel of the multi-channel model; and extract the semantic features of the height image through the fourth channel of the multi-channel model;

[0058] The feature fusion module 103 is used to fuse the detail features of the color image and the height image to obtain a detail feature fusion result, and to fuse the semantic features of the color image and the height image to obtain a semantic feature fusion result;

[0059] The defect detection module 104 is used to aggregate the detail feature fusion result and the semantic feature fusion result, and determine whether there is a defect on the surface of the connecting part based on the aggregation result.

[0060] The application scenario of this solution may be a scenario where defects of a cable connector are detected after the connector is manufactured.

[0061] Based on the above scenario, it can be understood that the executor of this scenario can be a smart terminal that identifies whether the connector has defects, such as a desktop computer, a laptop computer, a tablet computer, and an interactive multimedia device, etc., and no excessive limitations are made here.

[0062] The image acquisition module 101 may be a program design for an intelligent terminal to acquire a color image and a height image of a cable connector. Specifically, a color image may be a special case of a multispectral image, corresponding to the three primary colors of human vision, namely, red, green, and blue bands, and is an approximation of the spectral quantitative properties of the human eye. A color image is an image in which each pixel is composed of R, G, and B components, wherein R, G, and B are described by different gray levels.

[0063] A height image can be a grayscale image that uses pixel grayscale levels to represent the height data of the surface of an object in the image. A grayscale image has only one channel and is a monochrome image with a 256-level grayscale color range or level from black to white. The pixel value in the image is one of the 256 grayscales between black and white, and different grayscale levels can reflect different height information.

[0064] The method of obtaining the color image of the connector may be to collect the color image of the connector using an image sensor. Specifically, when light is irradiated onto the photosensitive unit of the image sensor, the semiconductor material in the photosensitive unit absorbs photons and generates electron-hole pairs, forming a photoinduced electromotive force, thereby realizing photoelectric conversion. The converted electrical signal is processed by amplification, filtering, analog-to-digital conversion, etc., and finally outputs a digital image signal. In order to realize the acquisition of color images, image sensors usually use color filter arrays, such as Bayer filter arrays, to separate light of different colors. Each photosensitive unit only receives light of one color, and by measuring and calculating the intensity of light of different colors, a color image can be restored.

[0065] The method of obtaining the height image of the connector can be to use a binocular vision camera to shoot the depth image of the connector. Specifically, the two cameras in the binocular vision camera simultaneously obtain two images of the cable connector from different angles. The distance from the object to the camera can be calculated by the known baseline length (the horizontal distance between the two cameras), the focal length of the two cameras, and the parallax of the connector in the two images. Combined with the installation height of the camera and the conversion relationship between the pixel coordinates and the actual physical size, the height of the connector can be determined. The height of the connector is represented by the corresponding grayscale value, and then the height image of the connector is generated.

[0066] The feature extraction module 102 may be a program design for extracting image features of color images and height images by the intelligent terminal. Specifically, the multi-channel model may be a learning model that can process multiple tasks in parallel using multiple channels, and can provide interactive channels between the channels during the processing process to effectively integrate and coordinate the information of multiple channels.

[0067] Detail features may be features such as color, texture, and height data contained in an image that can be distinguished from other images or regions.

[0068] Semantic features can represent objects, scenes, or attributes in an image and convert them into information that can be understood by humans.

[0069] The detailed features of a color image can be extracted by inputting the color image into the first channel of a multi-channel model, and using the first channel to count the frequency of occurrence of different colors in the color image to obtain a color histogram that can reflect the overall color distribution of the image. The grayscale relationship between two pixels at a certain distance and direction in the image is calculated to obtain a grayscale co-occurrence matrix, from which texture features such as contrast, correlation, energy, and homogeneity are extracted. Based on the differences in color and texture of the obtained color image in different areas, the edge of the color image can be identified to obtain edge information in the color image.

[0070] The method of extracting the semantic features of the color image may be to input the color image into the second channel of the multi-channel model, obtain edge information in the color image from the second channel to the first channel, segment the color image according to the edge information, obtain the bag-of-words model through big data, and obtain the semantic information matching each segmented region in the bag-of-words model as the semantic features of the color image. The bag-of-words model may be a commonly used text representation method in natural language processing and information retrieval.

[0071] A method for extracting detailed features of a height image may be to input the height image into the third channel of a multi-channel model, obtain the height information of each pixel in the height image through the third channel, perform statistics on the height information of each pixel, obtain the distribution pattern of the height information of each pixel in the height image, and use the distribution pattern of the height information as the detailed features of the height image.

[0072] The method for extracting the semantic features of the height image can be to input the height image into the fourth channel of the multi-channel model, obtain the detail features of the height image of the third channel through the fourth channel, perform regional segmentation on the height information with the same change law in the height image according to the height distribution law in the detail features, obtain the word bag model through big data, and obtain the semantic information matching each segmented region in the word bag model as the semantic features of the height image.

[0073] The feature fusion module 103 may be a program design for the intelligent terminal to fuse image features of the same attribute. The method of obtaining the detail feature fusion result may be to obtain the feature description of each pixel in the detail features of the color image and the height image respectively, take the union of the feature descriptions of the detail features of the color image and the height image, that is, delete the repeated parts in the detail features of the color image and the height image, and use the retained features as the detail feature fusion result.

[0074] The method for obtaining the semantic feature fusion result can be to compare the semantic information in the same divided area in the semantic features of the color image and the height image, retain the semantic information that is consistent with the comparison, and when there is inconsistent semantic information, obtain the upper-level speech information that can cover the semantic information of the color image and the height image as the new semantic information of the area, thereby obtaining the semantic feature fusion result.

[0075] The defect detection module 104 may be a program design for the intelligent terminal to identify whether there are defects on the surface of the connector. Specifically, the method of aggregating the detail feature fusion results and the semantic feature fusion results may be to mark the edge information, color information and height information in the detail feature fusion results in the pixel points corresponding to the regions where each semantic feature is located in the semantic feature fusion results to obtain the aggregation result.

[0076] A method for determining whether there are defects on the surface of the connector may be to read the aggregation result and identify whether there is semantic information such as cracks, protrusions, and depressions therein. If so, it is determined that there are defects on the surface of the connector.

[0077] Based on the above solution, optionally, the defect detection module is specifically used to:

[0078] Pre-acquire the aggregation results of multiple connection parts surfaces as training samples, mark the defect conditions of the aggregation results in the training samples according to whether there are defects on the connection parts surfaces, obtain the defect labels in the training samples, use the training samples with the defect labels as input variables, perform supervised training, and construct a defect condition classifier;

[0079] The aggregation result of the current connection part is input into the defect condition classifier, and whether there is a defect on the surface of the connection part is determined according to the output result of the defect condition classifier.

[0080] The defect condition classifier may be a learning model that can predict whether there is a defect condition on the surface of the connector according to the aggregation result of the connector.

[0081] The training samples may be a collection of aggregated data that provides data support for the defect condition classifier to learn the classification criteria.

[0082] The defect label may be a label used to indicate whether the connector corresponding to the aggregated data has a defect.

[0083] The method of constructing a defect condition classifier can be to obtain the aggregated results of the surfaces of multiple cable connectors in advance as training samples, and have the staff detect the surface of each connector to identify whether there are defects on the connector surface, and mark the aggregated results corresponding to the connectors with defects in the training samples as defective data, and mark the aggregated results corresponding to the connectors without defects as normal data. The training samples with all the aggregated results marked are used as input variables and input into the basic framework of the defect condition classifier, so that the defect condition classifier can learn the distinguishing features between defect data and normal data, optimize its own classification algorithm, and then build a fully functional defect condition classifier.

[0084] The method of determining whether there is a defect on the surface of the connector can be to input the aggregated features of the current connector surface into a pre-built defect condition classifier, and the defect condition classifier identifies whether the current aggregated features belong to defect data or normal data according to the learned classification algorithm. If the output result of the defect condition classifier is normal data, it is determined that there is no defect on the surface of the connector; if the output result of the defect condition classifier is abnormal data, it is determined that there is a defect on the surface of the connector.

[0085] The advantage of this arrangement of the scheme is that the defect condition classifier can be used to determine whether there are defects on the surface of the connector, which is conducive to reducing the reliance of the defect detection program on manual labor and improving the efficiency and accuracy of surface defect identification in the future.

[0086] This solution provides a program code for extracting detail features and semantic features from color images and height images for reference:

[0087] import numpy as np

[0088] import tensorflow as tf

[0089] #Assume that the color image is color_image and the height image is height_image

[0090] #Define the feature extraction function of each channel of the multi-channel model

[0091] def extract_detail_feature_color(image):

[0092] #Here you can use convolutional neural networks and other methods to extract the detailed features of color images

[0093] conv1=tf.layers.conv2d(image, filters=32, kernel_size=(3,3), activation=tf.nn.relu)

[0094] conv2=tf.layers.conv2d(conv1, filters=64, kernel_size=(3,3), activation=tf.nn.relu)

[0095] return conv2

[0096] def extract_semantic_feature_color(image):

[0097] resnet=tf.keras.applications.ResNet50(include_top=False,weights='imagenet',input_tensor=image)

[0098] return resnet.output

[0099] def extract_detail_feature_height(image):

[0100] #Extract the detailed features of the height image, similar to the method of color image

[0101] conv1=tf.layers.conv2d(image, filters=32, kernel_size=(3,3), activation=tf.nn.relu)

[0102] conv2=tf.layers.conv2d(conv1, filters=64, kernel_size=(3,3), activation=tf.nn.relu)

[0103] return conv2

[0104] def extract_semantic_feature_height(image):

[0105] #Extract the semantic features of the height image

[0106] resnet=tf.keras.applications.ResNet50(include_top=False,weights='imagenet',input_tensor=image)

[0107] return resnet.output

[0108] #Extract features of color images and height images

[0109] detail_feature_color=extract_detail_feature_color(color_image)

[0110] semantic_feature_color=extract_semantic_feature_color(color_image)

[0111] detail_feature_height=extract_detail_feature_height(height_image)

[0112] semantic_feature_height=extract_semantic_feature_height(height_image)

[0113] In an embodiment of the present application, an image acquisition module is used to acquire a color image and a height image of a connector; a feature extraction module is used to extract detail features of the color image through a first channel of a multi-channel model; extract semantic features of the color image through a second channel of the multi-channel model; extract detail features of the height image through a third channel of the multi-channel model; and extract semantic features of the height image through a fourth channel of the multi-channel model; a feature fusion module is used to fuse the detail features of the color image with the detail features of the height image to obtain a detail feature fusion result, and to fuse the semantic features of the color image with the height image to obtain a semantic feature fusion result; and a defect detection module is used to aggregate the detail feature fusion result and the semantic feature fusion result, and determine whether there is a defect on the surface of the connector based on the aggregation result.

[0114] Through the above-mentioned connector surface defect detection device based on the multi-channel model, the influence of different features in the color image and the height image on defect detection can be comprehensively considered, and whether there are defects on the connector surface can be identified according to the fused features, thereby improving the accuracy of connector surface defect detection.

[0115] Embodiment 2

[0116] Figure 2 It is a structural diagram of a connector surface defect detection device based on a multi-channel model provided in Example 2 of the present application. This solution makes a better improvement on the above-mentioned embodiment, and the specific improvement is as follows: the feature fusion module is specifically used to: extract the height information of each pixel in the detail feature of the height image, fuse the height information with the feature information of the same pixel in the detail feature of the color image, and obtain the detail feature fusion result; and obtain the first weight of the semantic feature of the color image and the second weight of the semantic feature of the height image, determine the semantic information of each image area according to the first weight and the second weight, and obtain the semantic feature fusion result.

[0117] like Figure 2 As shown, specifically including the following:

[0118] An image acquisition module 201 is used to acquire a color image and a height image of the connection member;

[0119] The feature extraction module 202 is used to extract the detail features of the color image through the first channel of the multi-channel model; extract the semantic features of the color image through the second channel of the multi-channel model; extract the detail features of the height image through the third channel of the multi-channel model; and extract the semantic features of the height image through the fourth channel of the multi-channel model;

[0120] The feature fusion module 203 is used to fuse the detail features of the color image and the height image to obtain a detail feature fusion result, and to fuse the semantic features of the color image and the height image to obtain a semantic feature fusion result;

[0121] The defect detection module 204 is used to aggregate the detail feature fusion result and the semantic feature fusion result, and determine whether there is a defect on the surface of the connecting part based on the aggregation result.

[0122] The feature fusion module 203 is specifically used to extract the height information of each pixel in the detail feature of the height image, and fuse the height information with the feature information of the same pixel in the detail feature of the color image to obtain a detail feature fusion result;

[0123] as well as,

[0124] The first weight of the semantic feature of the color image and the second weight of the semantic feature of the height image are obtained, and the semantic information of each image region is determined according to the first weight and the second weight to obtain a semantic feature fusion result.

[0125] The height information may be a grayscale value used in a height image to represent the height of each position of an object in the image.

[0126] The first weight may be the influence of the height image on determining the semantic feature fusion result when performing semantic feature fusion.

[0127] The second weight may be the influence of the color image on determining the result of the semantic feature fusion when the semantic feature fusion is performed. Since the color image contains richer semantic information, the second weight is generally greater than the first weight when the semantic feature fusion is performed.

[0128] The method for obtaining the detail feature fusion result can be to obtain the position of each pixel point in the detail feature of the height image and the corresponding height information, identify the pixel points corresponding to the position of each pixel point in the detail feature of the height image in the detail feature of the color image, store the height information in the attribute information of the corresponding pixel point, and obtain the detail feature fusion result.

[0129] The method for obtaining the semantic feature fusion result is to extract the semantic information of the same position area in the height image and the color image, and perform similar semantic extraction on the semantic information of the height image and the color image according to the first weight and the second weight respectively, and use the extraction results of each position area as the semantic feature fusion result.

[0130] The advantage of this arrangement of the present invention is that the detail features and semantic features of the two images can be extracted separately first, thereby obtaining more accurate and comprehensive image features, thereby providing effective data for the subsequent identification of surface defects of connectors.

[0131] Based on the above solution, optionally, the feature fusion module is specifically used to:

[0132] Extract edge height information of edge pixels in the detail feature fusion result, identify whether there is a mutation value in the edge height information, and if so, take the edge pixel corresponding to the mutation value as an abnormal pixel;

[0133] Acquire height information of pixels on the same edge as the abnormal pixel, and acquire a gradual change rule of the height information of the pixels on the same edge;

[0134] The edge height information replacement value of the abnormal pixel point is determined according to the gradual change rule.

[0135] The edge pixel point may be a pixel point located at the edge of the surface of the connector in the image.

[0136] The mutation value can be edge height information that has a significant numerical difference from other data.

[0137] The gradual change rule may be a numerical change rule of the edge height information of each edge pixel point as the position of the edge pixel point changes. For example, the gradual change rule may be that the height information of each edge pixel point is the same, or that the height information decreases along the extension direction of the edge pixel point position.

[0138] The edge height information replacement value may be edge height information corresponding to a new edge pixel point that is used to replace an abnormal pixel point.

[0139] The method for determining edge pixels can be to extract edge pixels from the detail feature fusion result, obtain edge height information corresponding to each edge pixel, calculate the difference in edge height information of adjacent edge pixels according to the arrangement position of the edge pixels, and if there is edge height information that is significantly different from the previous difference, use the edge height information as a mutation value, and use the edge pixel corresponding to the edge height information as an abnormal pixel.

[0140] A method for obtaining the changing law of the height information of pixel points on the same edge can be to identify the edge position where the abnormal pixel point is located, extract the edge height information corresponding to the edge pixel point in the edge position, calculate the difference in edge height information of adjacent edge pixel points according to the arrangement position of the edge pixel points on the same edge, analyze all the differences, and obtain the changing law of the difference with the arrangement position as the changing law of the height information of the pixel points on the same edge.

[0141] The edge height information replacement value of the abnormal pixel point is determined by predicting the edge height information of the abnormal pixel point as the edge height information replacement value based on the change rule and the height information corresponding to the edge pixels near the abnormal pixel point. After determining the edge height information replacement value, a pixel point consistent with the edge height information replacement value can be obtained around the abnormal pixel point, and the pixel point is used to replace the abnormal pixel point as a new edge pixel point.

[0142] The advantage of this setting of the present invention is that the edge pixels in the detail feature fusion result can be proofread through the edge height information, so that the detail feature fusion result can more clearly reflect the actual situation of the connector, which is conducive to improving the accuracy of defect detection.

[0143] Embodiment 3

[0144] Figure 3It is a structural diagram of a connector surface defect detection device based on a multi-channel model provided in Example 3 of the present application. This solution makes a better improvement on the above-mentioned embodiment, and the specific improvement is as follows: the defect detection module is specifically used to optimize the edge of the connector image area corresponding to each semantic information in the semantic feature fusion result according to the detail feature fusion result, and mark the height information of each pixel point in the connector image area to obtain the aggregation result of the detail feature fusion result and the semantic feature fusion result.

[0145] like Figure 3 As shown, specifically including the following:

[0146] An image acquisition module 301 is used to acquire a color image and a height image of the connection member;

[0147] The feature extraction module 302 is used to extract the detail features of the color image through the first channel of the multi-channel model; extract the semantic features of the color image through the second channel of the multi-channel model; extract the detail features of the height image through the third channel of the multi-channel model; and extract the semantic features of the height image through the fourth channel of the multi-channel model;

[0148] The feature fusion module 303 is used to fuse the detail features of the color image and the height image to obtain a detail feature fusion result, and to fuse the semantic features of the color image and the height image to obtain a semantic feature fusion result;

[0149] The defect detection module 304 is used to aggregate the detail feature fusion result and the semantic feature fusion result, and determine whether there is a defect on the surface of the connecting part based on the aggregation result.

[0150] Among them, the defect detection module 304 is specifically used to optimize the edges of the connector image area corresponding to each semantic information in the semantic feature fusion result according to the detail feature fusion result, and to mark the height information of each pixel point in the connector image area to obtain the aggregation result of the detail feature fusion result and the semantic feature fusion result.

[0151] Semantic information can be meaningful information obtained by identifying the meaning of objects in an image and can eliminate the uncertainty of things.

[0152] The method for optimizing the edge of the connector image area can be to identify the area segmented by each semantic information in the semantic feature fusion result, obtain the edge pixel position of the area where the connector is located, and extract the edge information of the connector in the detail feature fusion result, and compare the edge pixel position of the area where the connector is located obtained in the semantic feature fusion result with the edge information of the connector extracted in the detail feature fusion result. If there is an inconsistency in the edge position, the edge pixel position in the semantic feature fusion result is adjusted according to the edge information in the detail feature fusion result to achieve optimization of the edge of the connector image area.

[0153] A method for obtaining an aggregated result of the detail feature fusion result and the semantic feature fusion result can be to extract the height information of each pixel in the detail feature fusion result after optimizing the edge of the connector image area corresponding to each semantic information in the semantic feature fusion result, and mark the height information in the attribute information of the corresponding pixel in the connector image area in the semantic feature fusion result to obtain the aggregated result of the detail feature fusion result and the semantic feature fusion result.

[0154] The advantage of this setting of the present scheme is that the detail feature fusion result and the semantic feature fusion result are aggregated to ensure that the feature information used to judge whether there are defects on the surface of the connector is more comprehensive, which is conducive to improving the accuracy of the defect detection results.

[0155] Based on the above solution, optionally, the defect detection module is further used to identify the edge shape of the connection hole on the surface of the connector according to the aggregation result;

[0156] The edge shape is fitted with a standard arc shape, and the neatness of the shape of the connection hole is determined according to the fitting result. If the neatness does not meet the production requirements of the connection piece, it is determined that there are defects on the surface of the connection piece.

[0157] The connection hole may be a connection hole in a connector used to fix and connect cables. If the connection hole has defects such as irregular shape or inconsistent size, it will have a serious impact on the laying of the cable.

[0158] The standard arc shape may be an arc of a standard shape having the same size as the connector production requirements.

[0159] The method for identifying the edge shape of the connection hole may be to extract the area with semantic information of the connection hole in the aggregation result, obtain the edge information of the area, identify the edge contour of the connection hole according to the edge information, and obtain the edge shape of the connection hole.

[0160] A method for determining whether there are defects on the surface of a connector may be to fit the edge shape of the current connector obtained with a standard arc shape, determine the uniformity of the shape of the connecting hole according to the degree of overlap between the edge shape and the standard arc shape, obtain the required range for the uniformity of the shape of the connecting hole of a qualified connector in the connector production requirements, compare the uniformity of the current connecting hole shape with the required range, and if the uniformity is within the required range, determine that there are defects on the surface of the connector.

[0161] The advantage of such a configuration of the present invention is that the shape of the connecting hole of the connecting piece can be inspected to ensure the factory quality of the connecting piece and avoid the problem of the cable being unable to be installed when the cable is laid.

[0162] Based on the above solution, optionally, the defect detection module is further used to:

[0163] Obtaining an average height of the surface of the connector according to the polymerization result;

[0164] Calculating the difference between the height information in each position area on the surface of the connecting member and the average height;

[0165] If the difference is smaller than the first preset error, it is determined that the current position area is a surface concave defect; if the difference exceeds the second preset error, it is determined that the current position area is a surface convex defect.

[0166] The average height of the connector surface can be obtained by extracting the height data of each pixel point on the connector surface in the aggregation result, and calculating the ratio of the sum of all height data to the number of pixels as the average height of the connector surface.

[0167] The first preset error may be the difference between the height of the concave position on the surface of the connector and the average height of the entire surface of the connector, for example, 1 cm or more less than the average height, that is, there is a concave at a position where the difference between the height information in each position area on the surface of the connector and the average height is less than -1.

[0168] The second preset error may be the difference between the height of the protrusion on the surface of the connector and the average height of the entire surface of the connector, for example, 1 cm or more greater than the average height, that is, there is a protrusion at a position where the difference between the height information of each position area on the surface of the connector and the average height is greater than 1.

[0169] The method for calculating the difference between the height information in each position area on the surface of the connector and the average height can be to divide each position on the surface of the connector into several small areas of equal area, calculate the regional average height in each position area, subtract the regional average height from the average height of the connector surface, and take the calculation result as the difference between the height information in each position area on the surface of the connector and the average height.

[0170] The method for determining whether the current position area is a surface depression defect is to compare the difference between the height information and the average height in each position area on the surface of the connecting part with a first preset error. If the difference is less than the first preset error, it is determined that the current position area is a surface depression defect.

[0171] The method for determining whether the current position area is a surface protrusion defect is to compare the difference between the height information and the average height in each position area on the surface of the connecting part with a second preset error. If the difference is greater than the second preset error, it is determined that the current position area is a surface protrusion defect.

[0172] The advantage of this arrangement of the present invention is that the concave and convex defects on the surface of the connector can be identified based on the average height and the average height of the area, which is conducive to accurately identifying connectors with uneven surfaces and ensuring the quality of the connectors before leaving the factory.

[0173] Embodiment 4

[0174] Figure 4 It is a flow chart of the method for detecting surface defects of connectors based on a multi-channel model provided in the fourth embodiment of the present application.

[0175] like Figure 4 As shown, the specific steps include:

[0176] S401, obtaining a color image and a height image of the connection part;

[0177] S402, extracting detail features of the color image through the first channel of the multi-channel model; extracting semantic features of the color image through the second channel of the multi-channel model; extracting detail features of the height image through the third channel of the multi-channel model; extracting semantic features of the height image through the fourth channel of the multi-channel model;

[0178] S403, fusing the detail features of the color image and the height image to obtain a detail feature fusion result, and fusing the semantic features of the color image and the height image to obtain a semantic feature fusion result;

[0179] S404: Aggregate the detail feature fusion result and the semantic feature fusion result, and determine whether there is a defect on the surface of the connecting part based on the aggregation result.

[0180] Optionally, the detail features of the color image and the height image are fused to obtain a detail feature fusion result, and the semantic features of the color image and the height image are fused to obtain a semantic feature fusion result, including:

[0181] Extracting height information of each pixel in the detail feature of the height image, and fusing the height information with feature information of the same pixel in the detail feature of the color image to obtain a detail feature fusion result;

[0182] as well as,

[0183] The first weight of the semantic feature of the color image and the second weight of the semantic feature of the height image are obtained, and the semantic information of each image region is determined according to the first weight and the second weight to obtain a semantic feature fusion result.

[0184] In an embodiment of the present application, a color image and a height image of a connector are acquired; detail features of the color image are extracted through a first channel of a multi-channel model; semantic features of the color image are extracted through a second channel of the multi-channel model; detail features of the height image are extracted through a third channel of the multi-channel model; semantic features of the height image are extracted through a fourth channel of the multi-channel model; the detail features of the color image and the height image are fused to obtain a detail feature fusion result, and the color image and the semantic features of the height image are fused to obtain a semantic feature fusion result; the detail feature fusion result and the semantic feature fusion result are aggregated, and based on the aggregated result, it is determined whether there are defects on the surface of the connector.

[0185] Through the above-mentioned method for detecting surface defects of connectors based on a multi-channel model, the influence of different features in the color image and the height image on defect detection can be comprehensively considered, and whether there are defects on the surface of the connector can be identified according to the fused features, thereby improving the accuracy of detecting surface defects of connectors.

[0186] The connection surface defect detection method based on a multi-channel model provided in the embodiment of the present application corresponds to the connection surface defect detection device based on a multi-channel model provided in the above-mentioned embodiment, and has the same functional modules and beneficial effects. To avoid repetition, they will not be repeated here.

[0187] Embodiment 5

[0188] like Figure 5 As shown, an embodiment of the present application also provides an electronic device 500, including a processor 501, a memory 502, and a program or instruction stored in the memory 502 and executable on the processor 501. When the program or instruction is executed by the processor 501, each process of the embodiment of the above-mentioned multi-channel model-based connector surface defect detection device is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0189] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0190] Embodiment 6

[0191] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned embodiment of the connection surface defect detection device based on a multi-channel model is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0192] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0193] Embodiment 7

[0194] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned embodiment of the connector surface defect detection device based on a multi-channel model, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0195] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0196] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be noted that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0197] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0198] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0199] The above are only preferred embodiments of the present application and the technical principles used. The present application is not limited to the specific embodiments described herein, and various obvious changes, readjustments and substitutions that can be made by those skilled in the art will not deviate from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the claims.

Claims

1. A device for detecting surface defects of connectors based on a multi-channel model, characterized in that: The device comprises: An image acquisition module, used for acquiring a color image and a height image of the connection piece; A feature extraction module, used to extract detail features of the color image through the first channel of the multi-channel model; extract semantic features of the color image through the second channel of the multi-channel model; extract detail features of the height image through the third channel of the multi-channel model; extract semantic features of the height image through the fourth channel of the multi-channel model; A feature fusion module is used to fuse the detail features of the color image and the height image to obtain a detail feature fusion result, and to fuse the semantic features of the color image and the height image to obtain a semantic feature fusion result; The defect detection module is used to aggregate the detail feature fusion result and the semantic feature fusion result, and determine whether there is a defect on the surface of the connecting part based on the aggregation result.

2. The device for detecting surface defects of connectors based on a multi-channel model according to claim 1, characterized in that: The feature fusion module is specifically used for: Extracting height information of each pixel in the detail feature of the height image, and fusing the height information with feature information of the same pixel in the detail feature of the color image to obtain a detail feature fusion result; as well as, The first weight of the semantic feature of the color image and the second weight of the semantic feature of the height image are obtained, and the semantic information of each image region is determined according to the first weight and the second weight to obtain a semantic feature fusion result.

3. The device for detecting surface defects of connectors based on a multi-channel model according to claim 2, characterized in that: The feature fusion module is specifically used for: Extract edge height information of edge pixels in the detail feature fusion result, identify whether there is a mutation value in the edge height information, and if so, take the edge pixel corresponding to the mutation value as an abnormal pixel; Acquire height information of pixels on the same edge as the abnormal pixel, and acquire a gradual change rule of the height information of the pixels on the same edge; The edge height information replacement value of the abnormal pixel point is determined according to the gradual change rule.

4. The device for detecting surface defects of connectors based on a multi-channel model according to claim 1, characterized in that: The defect detection module is specifically used for: According to the detail feature fusion result, the edge of the connector image area corresponding to each semantic information in the semantic feature fusion result is optimized, and the height information of each pixel point in the connector image area is labeled to obtain the aggregation result of the detail feature fusion result and the semantic feature fusion result.

5. The device for detecting surface defects of connectors based on a multi-channel model according to claim 4, characterized in that: The defect detection module is further used for: identifying the edge shape of the connection hole on the surface of the connector according to the polymerization result; The edge shape is fitted with a standard arc shape, and the neatness of the shape of the connection hole is determined according to the fitting result. If the neatness does not meet the production requirements of the connection piece, it is determined that there are defects on the surface of the connection piece.

6. The device for detecting surface defects of connectors based on a multi-channel model according to claim 4, characterized in that: The defect detection module is further used for: Obtaining an average height of the surface of the connector according to the polymerization result; Calculating the difference between the height information in each position area on the surface of the connecting member and the average height; If it is determined that the difference is less than a first preset error, it is determined that the current position area is a surface concave defect; If the difference exceeds the second preset error, it is determined that the current position area is a surface protrusion defect.

7. The device for detecting surface defects of connectors based on a multi-channel model according to claim 1, characterized in that: The defect detection module is specifically used for: Pre-acquire the aggregation results of multiple connection parts surfaces as training samples, mark the defect conditions of the aggregation results in the training samples according to whether there are defects on the connection parts surfaces, obtain the defect labels in the training samples, use the training samples with the defect labels as input variables, perform supervised training, and construct a defect condition classifier; The aggregation result of the current connection part is input into the defect condition classifier, and whether there is a defect on the surface of the connection part is determined according to the output result of the defect condition classifier.

8. A method for detecting surface defects of connectors based on a multi-channel model, characterized in that: The method comprises: Acquire color images and height images of the connection parts; Extracting detail features of the color image through the first channel of the multi-channel model; extracting semantic features of the color image through the second channel of the multi-channel model; extracting detail features of the height image through the third channel of the multi-channel model; extracting semantic features of the height image through the fourth channel of the multi-channel model; The detail features of the color image and the height image are fused to obtain a detail feature fusion result, and the semantic features of the color image and the height image are fused to obtain a semantic feature fusion result; The detail feature fusion result and the semantic feature fusion result are aggregated, and whether there is a defect on the surface of the connecting part is determined based on the aggregation result.

9. The method for detecting surface defects of connectors based on a multi-channel model according to claim 8, characterized in that: The detail features of the color image and the height image are fused to obtain a detail feature fusion result, and the semantic features of the color image and the height image are fused to obtain a semantic feature fusion result, including: Extracting height information of each pixel in the detail feature of the height image, and fusing the height information with feature information of the same pixel in the detail feature of the color image to obtain a detail feature fusion result; as well as, The first weight of the semantic feature of the color image and the second weight of the semantic feature of the height image are obtained, and the semantic information of each image region is determined according to the first weight and the second weight to obtain a semantic feature fusion result.

10. An electronic device, characterized in that: It includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method for detecting surface defects of connectors based on a multi-channel model as described in any one of claims 8 to 9.

Images