Defect detection method and device in image pair and electronic device

Abstract

The invention provides an image centering defect detection method and device and an electronic device.The method comprises the steps that a hub image to be detected is acquired; a pre-constructed centering frame template is adopted to carry out centering operation on the hub image, a centering result is obtained, and the centering result is used for representing whether centering of the hub image succeeds or not; according to a centering result, inputting the hub image into a pre-trained hub defect detection network to obtain a pre-selected candidate box, the pre-selected candidate box comprising a hub defect position, a defect category and a confidence coefficient in the hub image; the defect information in the hub image is generated according to the pre-selected candidate box and a pre-constructed target attention matrix, the attention matrix is used for representing probability information of defects appearing at the corresponding pixel positions, through the hub defect detection method and device, the technical problem that in the related technology, the precision of hub defect detection through machine vision is low is solved, and the hub defect detection accuracy is improved. The accuracy and speed of hub defect detection are improved, and the robustness and stability of the detection system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial control, and in particular, to a method and device for defect detection in image alignment, and an electronic device. Background Art

[0002] During the production process of automotive parts, the method of manually inspecting each wheel hub with the naked eye or the method of traditional machine vision template matching is used to detect external defects of the wheel hub.

[0003] In the related art, the defect detection rate is low and the false detection rate is very high by manual means. Since the scale grasped by each person is different and there is no quantitative standard, the detection standard varies from person to person and is not unified. The manual method is also affected by energy and working methods, resulting in low detection efficiency. By using the traditional machine vision template matching method, it is greatly affected by factors such as illumination, angle, position, picture size, texture, and complex background during photographing. Frequent template replacement is required during use, which greatly limits the implementation and deployment of this method. Especially when the product model changes, the entire algorithm needs to be re-adapted, resulting in poor robustness of the algorithm.

[0004] For the above problems existing in the related art, no efficient and accurate solution has been found yet. Summary of the Invention

[0005] The present invention provides a method and device for defect detection in image alignment, and an electronic device to solve the above technical problems existing in the related art.

[0006] According to an embodiment of the present invention, a method for defect detection in image alignment is provided, including: obtaining a wheel hub image to be detected; performing an alignment operation on the wheel hub image using a pre-constructed alignment frame template to obtain an alignment result, where the alignment result is used to characterize whether the alignment of the wheel hub image is successful; inputting the wheel hub image into a pre-trained wheel hub defect detection network according to the alignment result to obtain a preselected candidate box, where the preselected candidate box includes the position, defect category, and confidence level of the wheel hub defect in the wheel hub image; generating defect information in the wheel hub image according to the preselected candidate box and a pre-constructed target attention matrix, where the attention matrix is used to characterize the probability information of defects occurring at corresponding pixel positions.

[0007] Optionally, performing an alignment operation on the wheel hub image using a pre-constructed alignment frame template to obtain an alignment result includes: rotating the alignment frame template at a preset step length to generate a template set, where the template set includes alignment frame templates corresponding to multiple rotation angles; finding a target alignment frame template that best matches the wheel hub image in the template set; generating an alignment result of the wheel hub image based on the target alignment frame template.

[0008] Optionally, finding the target pair of middle frame templates that best match the hub image in the template set includes: finding the best matching region in the hub image; calculating the similarity with the best matching region for each template in the template set; and determining the template with the highest similarity as the target pair of middle frame templates that best match the hub image.

[0009] Optionally, generating the alignment result of the hub image based on the target pair of middle frame templates includes: calculating the horizontal offset and vertical offset between the target pair of middle frame templates and the best matching region in the hub image; determining whether both the horizontal offset and the vertical offset are less than a preset threshold; if both the horizontal offset and the vertical offset are less than the preset threshold, generating a first alignment result; if the horizontal offset or the vertical offset is greater than or equal to the preset threshold, generating a second alignment result, where the first alignment result is used to indicate that the alignment of the hub image is successful, and the second alignment result is used to indicate that the alignment of the hub image fails.

[0010] Optionally, inputting the hub image into a pre-trained hub defect detection network according to the alignment result includes: if the alignment of the hub image is successful, obtaining the alignment rotation angle and alignment translation amount of the hub image; rotating the hub image by the alignment rotation angle to obtain a first intermediate image; translating the first intermediate image by the alignment translation amount to obtain a second intermediate image; cutting the background pixels other than the hub in the second intermediate image to obtain a third intermediate image; inputting the third intermediate image into the pre-trained hub defect detection network; if the alignment of the hub image fails, inputting the hub image into the pre-trained hub defect detection network.

[0011] Optionally, generating defect information in the hub image according to the preselected candidate box and the pre-constructed target attention matrix includes: calculating the center of the box of the preselected candidate box; calculating the distances between the center of the box and the cluster centers of each attention matrix in the pre-constructed attention matrix set, and selecting the target attention matrix with the closest distance; multiplying the preselected candidate box by the corresponding coordinates in the target attention matrix and then accumulating to obtain an initial candidate box set, where each candidate box corresponds to a confidence level; filtering the initial candidate boxes with a confidence level less than a first threshold in the initial candidate box set to obtain an intermediate candidate box set; calculating the intersection area and union area in the intermediate candidate box set, and calculating the intersection over union based on the intersection area and the union area; filtering the intermediate candidate boxes with an intersection over union less than a second threshold in the intermediate candidate box set to obtain a defect box set, where each defect box includes the hub defect position, defect category, and confidence level.

[0012] Optionally, before performing the centering operation on the hub image using the pre-built centering frame template, the method further includes: selecting multiple grayscale sample pictures; sequentially performing the following steps on each grayscale sample picture among the multiple grayscale sample pictures: starting from the initial position of the current grayscale sample picture, selecting a sliding window of a fixed size, and selecting the window picture within the sliding window as the sliding template; using the sliding template to match the grayscale sample picture to find the lowest similarity degree of the match; in the grayscale sample picture, selecting the window area where the sliding window with the smallest similarity degree is located; after the multiple grayscale sample pictures are processed, reading the multiple window areas corresponding to the multiple grayscale sample pictures; calculating the average position value of the multiple window areas to obtain the optimal centering frame position; based on the optimal centering frame position, respectively extracting a best window picture of a centering frame from the multiple grayscale sample pictures to obtain multiple best window pictures; calculating the pixel grayscale average value of the multiple best window pictures to obtain the centering frame template.

[0013] Optionally, before generating the defect information in the hub image according to the preselected candidate box and the pre-built target attention matrix, the method further includes: obtaining multiple defect sample pictures; counting the calibration coordinates and defect sizes of the defect areas in each defect sample picture among the multiple defect sample pictures in a preset calibration direction; performing clustering of an initial number of categories on the center points of the defect sizes; iteratively performing the following steps until an intersection with an area larger than a preset number of pixels appears: calculating the intersection of multiple defect masks in each cluster; determining whether the number of pixels within the intersection is greater than a preset number; if the number of pixels within the intersection is less than or equal to the preset number, increasing the number of categories of the cluster; obtaining multiple clusters after the iteration ends and configuring an initial attention matrix for each cluster, where the elements of the initial attention matrix are all 0; for each cluster, resetting the element values in the initial attention matrix according to the number of overlapping defect masks at each element position to obtain a set of attention matrices, where the element values are positively correlated with the number of overlapping defect masks.

[0014] According to another embodiment of the present invention, there is provided a defect detection device in an image pair, including: a first acquisition module for acquiring a hub image to be detected; an alignment module for performing an alignment operation on the hub image by using a pre-constructed alignment frame template to obtain an alignment result, where the alignment result is used to characterize whether the alignment of the hub image is successful; a processing module for inputting the hub image into a pre-trained hub defect detection network according to the alignment result to obtain a preselected candidate box, where the preselected candidate box includes the hub defect position, defect category and its confidence level in the hub image; a generation module for generating defect information in the hub image according to the preselected candidate box and a pre-constructed target attention matrix, where the attention matrix is used to characterize the probability information of defects occurring at corresponding pixel positions.

[0015] Optionally, the alignment module includes: a rotation unit for rotating the alignment frame template according to a preset step size to generate a template set, where the template set includes alignment frame templates corresponding to multiple rotation angles; a search unit for searching for a target alignment frame template that best matches the hub image in the template set; a generation unit for generating an alignment result of the hub image based on the target alignment frame template.

[0016] Optionally, the search unit includes: a search subunit for searching for the best matching region in the hub image; a calculation subunit for calculating the similarity with the best matching region for each template in the template set respectively; a determination subunit for determining the template with the highest similarity as the target alignment frame template that best matches the hub image.

[0017] Optionally, the generation unit includes: a calculation subunit for calculating the horizontal offset and vertical offset between the target alignment frame template and the best matching region in the hub image; a judgment subunit for judging whether both the horizontal offset and the vertical offset are less than a preset threshold; a generation subunit for generating a first alignment result if both the horizontal offset and the vertical offset are less than the preset threshold; and generating a second alignment result if the horizontal offset or the vertical offset is greater than or equal to the preset threshold, where the first alignment result is used to characterize that the alignment of the hub image is successful, and the second alignment result is used to characterize that the alignment of the hub image fails.

[0018] Optionally, the processing module includes: a first processing unit, used to obtain the centering rotation angle and centering translation of the wheel hub image if the wheel hub image is successfully centered; rotate the wheel hub image by the centering rotation angle to obtain a first intermediate image; translate the first intermediate image by the centering translation to obtain a second intermediate image; cut background pixels except the wheel hub in the second intermediate image to obtain a third intermediate image; input the third intermediate image into a pre-trained wheel hub defect detection network; a second processing unit, used to input the wheel hub image into a pre-trained wheel hub defect detection network if the wheel hub image fails to be centered.

[0019] Optionally, the generation module includes: a first calculation unit, used to calculate the box center of the pre-selected candidate box; a second calculation unit, used to calculate the distance between the box center and the cluster center of each attention matrix in the pre-constructed attention matrix set, and select the target attention matrix with the closest distance; a first operation unit, used to multiply the pre-selected candidate box with the corresponding coordinates in the target attention matrix and then accumulate them to obtain an initial candidate box set, wherein each candidate box corresponds to a confidence level; a first filtering unit, used to filter the initial candidate boxes whose confidence levels are less than a first threshold in the initial candidate box set to obtain an intermediate candidate box set; a second operation unit, used to calculate the intersection area and the union area in the intermediate candidate box set, and calculate the intersection-union ratio based on the intersection area and the union area; a second filtering unit, used to filter the intermediate candidate boxes whose intersection-union ratios are less than a second threshold in the intermediate candidate box set to obtain a defect box set, wherein each defect box includes a hub defect position, a defect category, and a confidence level.

[0020] Optionally, the device also includes: a second acquisition module, which is used to select multiple grayscale sample images before the centering module uses a pre-built centering frame template to perform a centering operation on the wheel hub image; a first iteration module, which is used to perform the following steps on each of the multiple grayscale sample images in sequence: selecting a sliding window of a fixed size from an initial position for the current grayscale sample image, and selecting a window image in the sliding window as a sliding template; using the sliding template to match the grayscale sample image to find the similarity with the lowest matching degree; selecting a window area where the sliding window with the smallest similarity is located in the grayscale sample image; a reading module, which is used to read multiple window areas corresponding to the multiple grayscale sample images after the execution of the multiple grayscale sample images is completed; a first calculation module, which is used to calculate the average position value of the multiple window areas to obtain the best centering frame position; an extraction module, which is used to take out the best window image of a centering frame from the multiple grayscale sample images based on the best centering frame position to obtain multiple best window images; a second calculation module, which is used to calculate the pixel grayscale mean of the multiple best window images to obtain the centering frame template.

[0021] Optionally, the device further includes: a third acquisition module, configured to acquire multiple defect sample pictures before the generation module generates defect information in the hub image according to the preselected candidate box and the pre-constructed target attention matrix; a statistics module, configured to count the calibration coordinates and defect sizes of the defect areas in each defect sample picture in a preset calibration direction among the multiple defect sample pictures; a clustering module, configured to perform clustering of an initial number of categories on the center points of the defect sizes; a second iteration module, configured to iteratively execute the following steps until an intersection with an area greater than a preset number of pixels appears: calculate the intersection of multiple defect masks in each cluster; determine whether the number of pixels in the intersection is greater than a preset number; if the number of pixels in the intersection is less than or equal to the preset number, increase the number of categories of the cluster; a configuration module, configured to acquire multiple clusters after the iteration ends and configure an initial attention matrix for each cluster, where elements of the initial attention matrix are all 0; a reset module, configured to, for each cluster, reset the element values in the initial attention matrix according to the number of defect masks overlapping at each element position to obtain a set of attention matrices, where the element values are positively correlated with the number of overlapping defect masks.

[0022] According to another embodiment of the present invention, there is also provided a storage medium storing a computer program, where the computer program is configured to execute the steps in any one of the above device embodiments when running.

[0023] According to another embodiment of the present invention, there is also provided an electronic device including a memory and a processor, where the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above device embodiments.

[0024] Through the embodiments of the present invention, a hub image to be detected is acquired; a centering operation is performed on the hub image by using a pre-constructed centering frame template to obtain a centering result, where the centering result is used to represent whether the centering of the hub image is successful; the hub image is input into a pre-trained hub defect detection network according to the centering result to obtain a preselected candidate box, where the preselected candidate box includes the position, defect category, and confidence of the hub defect in the hub image; defect information in the hub image is generated according to the preselected candidate box and the pre-constructed target attention matrix, where the attention matrix is used to represent the probability information of a defect occurring at a corresponding pixel position. By centering the hub image by using a centering frame template and optimizing the preselected candidate box output by the network by using a target attention matrix, the technical problem of low accuracy in detecting hub defects in the related art is solved, the accuracy and speed of hub defect detection are improved, and the robustness and stability of the detection system are improved. Description of the Drawings

[0025] The accompanying drawings described herein are used to provide a further understanding of the present invention and form a part of this application. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 is a hardware structure block diagram of a computer according to an embodiment of the present invention; Figure 2 is a flowchart of a method for defect detection in an image pair according to an embodiment of the present invention; Figure 3 is a schematic diagram of a wheel hub defect detection network in an embodiment of the present invention; Figure 4 is an overall flowchart of an embodiment of the present invention; Figure 5 is a structure block diagram of a device for defect detection in an image pair according to an embodiment of the present invention. Detailed Embodiments

[0026] In order to enable those skilled in the art of the present technology to better understand the solutions of this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of this application. It should be noted that, without conflict, the embodiments in this application and the features in the embodiments can be combined with each other.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned accompanying drawings are used to distinguish similar objects and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of this application described here can be implemented in an order different from those illustrated or described here. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, product, or device that includes a series of steps or units does not necessarily need to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0028] Embodiment 1 The method embodiment provided in the first embodiment of this application can be executed on an operation device such as a server, a computer, or a camera. Taking running on a computer as an example, Figure 1 is a hardware structure block diagram of a computer according to an embodiment of the present invention. As Figure 1As shown, the computer may include one or more ( Figure 1 only one is shown in the figure) processors 102 (the processor 102 may include, but is not limited to, processing devices such as a microprocessor MCU or a field-programmable gate array FPGA) and a memory 104 for storing data. Optionally, the above computer may further include a transmission device 106 for communication functions and an input / output device 108. Those of ordinary skill in the art can understand that Figure 1 the structure shown is only schematic and does not limit the structure of the above computer. For example, the computer may further include more or fewer components than Figure 1 shown in the figure, or have a different configuration from Figure 1 shown in the figure.

[0029] The memory 104 can be used to store computer programs. For example, software programs and modules of application software, such as the computer program corresponding to the method for defect detection in an image pair in an embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implements the above method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory 104 may further include a memory remotely disposed relative to the processor 102, and these remote memories can be connected to the computer through a network. Examples of the above network include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0030] The transmission device 106 is used to receive or send data via a network. Specific examples of the above network may include a wireless network provided by a communication provider of the computer. In one instance, the transmission device 106 includes a network adapter (Network Interface Controller, abbreviated as NIC), which can be connected to other network devices through a base station and thus can communicate with the Internet. In one instance, the transmission device 106 may be a radio frequency (Radio Frequency, abbreviated as RF) module, which is used to communicate with the Internet wirelessly.

[0031] In this embodiment, a method for defect detection in an image pair is provided. Figure 2 is a flowchart of the method for defect detection in an image pair according to an embodiment of the present invention. As Figure 2 shown, the process includes the following steps: Step S202, obtain a hub image to be detected; Optionally, the hub image may be a hub of a workpiece such as an automotive part, such as an automotive hub.

[0032] Step S204: Perform centering operation on the hub image using a pre-built centering frame template to obtain a centering result, where the centering result is used to indicate whether the centering of the hub image is successful; Step S206: Input the hub image into a pre-trained hub defect detection network according to the centering result to obtain preselected candidate boxes, where the preselected candidate boxes include the positions, defect categories, and their confidence levels of the hub defects in the hub image; The defect data corresponding to the preselected candidate boxes is the initial defect data.

[0033] Step S208: Generate defect information in the hub image according to the preselected candidate boxes and a pre-built target attention matrix, where the attention matrix is used to represent the probability information of defects occurring at corresponding pixel positions.

[0034] Through the above steps, a hub image to be detected is obtained; perform centering operation on the hub image using a pre-built centering frame template to obtain a centering result, where the centering result is used to indicate whether the centering of the hub image is successful; input the hub image into a pre-trained hub defect detection network according to the centering result to obtain preselected candidate boxes, where the preselected candidate boxes include the positions, defect categories, and their confidence levels of the hub defects in the hub image; generate defect information in the hub image according to the preselected candidate boxes and a pre-built target attention matrix, where the attention matrix is used to represent the probability information of defects occurring at corresponding pixel positions. By centering the hub image using a centering frame template and optimizing the preselected candidate boxes output by the network using a target attention matrix, the technical problem of low accuracy in detecting hub defects in the related art is solved, the accuracy and speed of hub defect detection are improved, and the robustness and stability of the detection system are enhanced.

[0035] In an implementation manner of this embodiment, performing centering operation on the hub image using a pre-built centering frame template to obtain a centering result includes: rotating the centering frame template at a preset step length to generate a template set, where the template set includes centering frame templates corresponding to multiple rotation angles; searching for a target centering frame template in the template set that best matches the hub image; generating a centering result of the hub image based on the target centering frame template.

[0036] In this embodiment, before performing the centering operation on the hub image using the pre-constructed centering frame template, the following steps are further included: selecting multiple grayscale sample pictures; sequentially performing the following steps on each of the multiple grayscale sample pictures: starting from the initial position of the current grayscale sample picture, selecting a sliding window of a fixed size, and taking the window image within the sliding window as the sliding template; using the sliding template to match the grayscale sample picture to find the lowest similarity of the matching degree; selecting the window area where the sliding window with the smallest similarity is located in the grayscale sample picture; after the multiple grayscale sample pictures are processed, reading the multiple window areas corresponding to the multiple grayscale sample pictures; calculating the average position value of the multiple window areas to obtain the optimal centering frame position; based on the optimal centering frame position, respectively extracting an optimal window image of a centering frame from the multiple grayscale sample pictures to obtain multiple optimal window images; calculating the pixel grayscale average value of the multiple optimal window images to obtain the centering frame template.

[0037] Optionally, rotate the centering frame template in steps of 1 degree, rotating from 10 degrees counterclockwise to 10 degrees clockwise to generate 20 templates.

[0038] In one embodiment, the process of constructing the centering frame template includes: a1: Randomly select 10 pictures from the input single-channel grayscale picture of 2048 * 2048 as the grayscale sample pictures; a2: For each of the 10 grayscale sample pictures of the hub, select a 200*200 sliding window starting from the upper left corner, and take the small picture within the sliding window as the template; a3: Use the template in the previous step to match the entire picture and find the similarity value of the least matching point; a4: Slide the sliding window in step a2 in steps of 50 pixels from left to right and from top to bottom, and perform the matching in step a3; a5: Compare the similarity values of the least matching points of each sliding window, and find the coordinates of the sliding window with the smallest similarity value in the entire picture; a6: Repeat steps a2 to a5 for each of the 10 pictures, calculate the average values of the coordinates and width and height of the sliding windows with the smallest similarity values of the 10 pictures, and obtain the window position where the optimal centering frame is located; a7: According to the position where the optimal centering frame is located, extract the small pictures of the centering frame from the 10 pictures, and take the average of the pixel grayscale values to obtain the picture of the centering frame template.

[0039] In one example, finding the target centering box template that best matches the hub image in the template set includes: finding the best matching region in the hub image; calculating the similarity with the best matching region for each template in the template set; and determining the template with the highest similarity as the target centering box template that best matches the hub image.

[0040] Find the best matching region (from the hub image) with the centering box template in the currently input hub image, and calculate the similarity between the best matching region and the centering box template.

[0041] In one example, generating the centering result of the hub image based on the target centering box template includes: calculating the horizontal offset and vertical offset between the target centering box template and the best matching region in the hub image; determining whether both the horizontal offset and the vertical offset are less than a preset threshold; if both the horizontal offset and the vertical offset are less than the preset threshold, generating a first centering result; if the horizontal offset or the vertical offset is greater than or equal to the preset threshold, generating a second centering result, where the first centering result is used to represent that the centering of the hub image is successful, and the second centering result is used to represent that the centering of the hub image fails.

[0042] Obtain the rotation angle angle of the template with the highest similarity match and the offset (coordinates) of the best matching region. If the offset in the x direction and the offset in the y direction are both less than or equal to 100, the centering is successful; otherwise, the centering fails.

[0043] In an implementation scenario of this embodiment, inputting the hub image into a pre-trained hub defect detection network according to the centering result includes: if the centering of the hub image is successful, obtaining the centering rotation angle and centering translation amount of the hub image; rotating the hub image by the centering rotation angle to obtain a first intermediate image; translating the first intermediate image by the centering translation amount to obtain a second intermediate image; cutting the background pixels other than the hub in the second intermediate image to obtain a third intermediate image; inputting the third intermediate image into the pre-trained hub defect detection network; if the centering of the hub image fails, inputting the hub image into the pre-trained hub defect detection network.

[0044] In a scenario, if the centering is successful, according to the calculated horizontal offset and vertical offset between the target centering box template and the best matching region, obtain the translation amount offset of the current hub image. If the centering fails, set angle to 0 and offset to (0, 0), and directly input the original hub image into the pre-trained hub defect detection network. If the centering is successful, for the currently to-be-inspected hub image, rotate it by the angle angle and then translate it by offset to obtain the centered image, and cut off the pixels outside the hub in the image and supplement them according to the edge pixels.

[0045] Figure 3 It is the schematic diagram of the wheel hub defect detection network in the embodiment of the present invention. The whole detection algorithm process is a closed-loop feedback cycle. After the algorithm network is deployed on the wheel hub appearance defect detection device, it performs real-time detection on the input batch of wheel hub images, outputs the detected defect categories and the pixel positions in the images, and at the same time collects the images with poor detection effects for retraining and redeployment, and conducts a cycle to gradually improve the detection effect. The wheel hub defect detection network includes multiple convolutional layers and multiple fully connected layers. The first 5 convolutional layers and the last 3 fully connected layers need to be initialized, and their weights are initialized to a random normal distribution of 0-10^-2, bias = 0. Finally, the features of the wheel hub are extracted into a 1*1000 feature vector. The network structure uses 1*1 convolution, and 1*1 convolution first appeared in NIN (network in network). 1*1 convolution greatly reduces the number of parameters and dimensions, and can speed up the model training speed.

[0046] In an implementation manner of this embodiment, generating the defect information in the wheel hub image according to the preselected candidate box and the pre-constructed target attention matrix includes: calculating the center of the preselected candidate box; calculating the distances between the center of the box and the clustering centers of each attention matrix in the pre-constructed attention matrix set, and selecting the target attention matrix with the closest distance; multiplying the corresponding coordinates of the preselected candidate box and the target attention matrix and then accumulating to obtain an initial candidate box set, where each candidate box corresponds to a confidence level; filtering the initial candidate boxes with a confidence level less than the first threshold in the initial candidate box set to obtain an intermediate candidate box set; calculating the intersection area and the union area in the intermediate candidate box set, and calculating the intersection over union based on the intersection area and the union area; filtering the intermediate candidate boxes with an intersection over union less than the second threshold in the intermediate candidate box set to obtain a defect box set, where each defect box includes the wheel hub defect position, defect category, and confidence level.

[0047] In this embodiment, before generating the defect information in the hub image based on the preselected candidate boxes and the pre-constructed target attention matrix, the following steps are further included: obtaining a plurality of defect sample pictures; counting the calibration coordinates and defect sizes of the defect areas in each defect sample picture in the plurality of defect sample pictures in a preset calibration direction; performing clustering of an initial number of categories on the center points of the defect sizes; iteratively executing the following steps until an intersection with an area larger than a preset number of pixels appears: calculating the intersection of multiple defect masks in each cluster; determining whether the number of pixels in the intersection is greater than a preset number; if the number of pixels in the intersection is less than or equal to the preset number, increasing the number of categories of the cluster; obtaining a plurality of clusters after the iteration ends, and configuring an initial attention matrix for each cluster, where the elements of the initial attention matrix are all 0; for each cluster, resetting the element values in the initial attention matrix according to the number of overlapping defect masks at each element position, to obtain a set of attention matrices, where the element values are positively correlated with the number of overlapping defect masks.

[0048] After rotation, translation, and cutting, the centered picture, through the inference of the hub defect detection network, generates preselected candidate boxes, including information such as position, category, and corresponding confidence. Calculate the distance between the attention matrix and the clustering center of the preselected candidate boxes, select the attention matrix with the closest distance, multiply and add the elements (confidence and probability) corresponding to the coordinates in the preselected candidate box and the attention matrix, and after filtering by the candidate box confidence and the IOU (Intersection over Union) filtering of NMS (Non-Maximum Suppression), obtain the position, category, and corresponding confidence of the defects in the hub product.

[0049] Optionally, the value range of the element values of the reset attention matrix is [0, 1]. If all defect rectangles overlap at the position of pixel A, the value of the element of the attention matrix at pixel A is 1. 1 represents a 100% probability of a defect occurring at the corresponding pixel position, 0 represents a 0% probability of a defect occurring at the corresponding pixel position, and the higher the value, the higher the probability of a defect occurring, and vice versa.

[0050] In one embodiment, the process of constructing the attention matrix of the defect includes: b1: Count the upper left coordinates and the length and width of the defects in all hub pictures in the image data of the labeled defect sample pictures to obtain defect rectangles, that is, defect masks mask; b2: Perform clustering of 5 categories on the upper left coordinates and the length and width of the defects; Optionally, performing clustering of an initial number of categories on the center points of the defect sizes includes: minimizing the clustering using the following objective function J: ; is the defect center point of the i-th defect sample picture, is the probability that point i belongs to cluster j, is the center of cluster j, = 1, is the center of cluster l, n is the number of defect sample pictures, k is the number of initial categories, that is, the number of clusters, here it is taken as 5, and m is the fuzzy index, m = 2. and are updated according to the following formula: ; ; b3: Calculate the intersection of the defect masks in each cluster in the previous step. If there is no intersection with an area greater than 25 pixels, increase the number of categories of the cluster until there is an intersection with an area greater than 25 pixels; b4: Initialize an attention matrix with the same size as the hub image; the element values are set to all 0, and each cluster corresponds to the attention matrix; b5: Set the matrix elements in the intersection of each completely overlapping defect mask to 1, set the elements outside the rectangular box to 0, and generate other elements according to the number of overlapping rectangular boxes by linear interpolation; b6: All elements in the generated attention matrix are between [0, 1].

[0051] In the process of producing wheels, due to the temperature and humidity of the production environment, process flow, raw material factors, equipment or human factors, various types of wheel defects are generated, such as scratches, black lines, stripes, bubbles, inclusions, extrusion rings, mold scratches, scrapes, no film, rust spots, oil stains, color difference, aluminum chips, film explosion, burrs, mixed materials, etc. In this embodiment, according to the image features collected from the wheel and the location, size, length, density and other features of the defects in the image, an algorithm network based on the image centering template and the defect attention matrix is ​​designed to solve the problem of inconsistent location of the wheel on each photo when extracting image features. The defect attention matrix is ​​constructed based on the statistical regularity of the location of the defects on the wheel. This embodiment customizes the design of the wheel hub image centering and attention matrix algorithm network according to the characteristics of the wheel hub image, which can solve the speed and accuracy problems of inputting multiple wheel hub images into the defect detection network for parallel training and prediction. The use of this network training model does not require high computing power, CUDA (Compute Unified Device Architecture) cores and memory and other computing resources. After training and verification on the training data set, the model is exported and applied to the wheel hub defect detection device to achieve real-time wheel hub defect category, position detection and confidence output.

[0052] The present invention proposes a new method for constructing a centering frame template and a defect attention matrix. Based on the wheel hub big data collected by the wheel hub appearance defect detection equipment, after the wheel hub image is centered, a deep learning defect feature extraction network model based on image centering is trained, and then the attention matrix is ​​superimposed. A wheel hub defect detection system can be constructed using the algorithm model. Figure 4 It is an overall flow chart of an embodiment of the present invention.

[0053] By adopting the solution of this embodiment, defects on a wheel hub image with a resolution of 2048*2048 can be predicted in real time in an average of 11 ms. There are as many as 18 types of defects. The accuracy and speed of the system work meet the needs of actual applications. Compared with existing systems based on traditional machine vision or manual methods, the accuracy and detection speed are improved.

[0054] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases, the former is a better implementation. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions for causing a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of the present invention.

[0055] Embodiment 2 In this embodiment, a defect detection device for an image pair is further provided. This device is used to implement the above embodiments and preferred implementation manners, and those that have been described will not be repeated. As used hereinafter, the term "module" can be a combination of software and hardware that can implement a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, can also be conceived.

[0056] Figure 5 is a structural block diagram of a defect detection device for an image pair according to an embodiment of the present invention. As Figure 5 shown, the device includes: A first acquisition module 50, configured to acquire a hub image to be detected; An alignment module 52, configured to perform an alignment operation on the hub image by using a pre-constructed alignment frame template to obtain an alignment result, where the alignment result is used to characterize whether the alignment of the hub image is successful; A processing module 54, configured to input the hub image into a pre-trained hub defect detection network according to the alignment result to obtain a preselected candidate box, where the preselected candidate box includes the position, defect category, and confidence of the hub defect in the hub image; A generation module 56, configured to generate defect information in the hub image according to the preselected candidate box and a pre-constructed target attention matrix, where the attention matrix is used to characterize the probability information of a defect occurring at a corresponding pixel position.

[0057] Optionally, the alignment module includes: a rotation unit, configured to rotate the alignment frame template according to a preset step size to generate a template set, where the template set includes alignment frame templates corresponding to multiple rotation angles; a search unit, configured to search for a target alignment frame template that best matches the hub image in the template set; a generation unit, configured to generate an alignment result of the hub image based on the target alignment frame template.

[0058] Optionally, the lookup unit includes: a lookup subunit for looking up the best matching region in the hub image; a calculation subunit for calculating the similarity with the best matching region for each template in the template set; and a determination subunit for determining the template with the highest similarity as the target pair middle frame template that best matches the hub image.

[0059] Optionally, the generation unit includes: a calculation subunit for calculating the horizontal offset and vertical offset between the target pair middle frame template and the best matching region in the hub image; a judgment subunit for judging whether both the horizontal offset and the vertical offset are less than a preset threshold; and a generation subunit for generating a first alignment result if both the horizontal offset and the vertical offset are less than the preset threshold, and generating a second alignment result if the horizontal offset or the vertical offset is greater than or equal to the preset threshold, where the first alignment result is used to indicate that the hub image alignment is successful, and the second alignment result is used to indicate that the hub image alignment fails.

[0060] Optionally, the processing module includes: a first processing unit for, if the hub image alignment is successful, obtaining the alignment rotation angle and alignment translation amount of the hub image; rotating the hub image by the alignment rotation angle to obtain a first intermediate image; translating the first intermediate image by the alignment translation amount to obtain a second intermediate image; cutting the background pixels other than the hub in the second intermediate image to obtain a third intermediate image; and inputting the third intermediate image into a pre-trained hub defect detection network; and a second processing unit for, if the hub image alignment fails, inputting the hub image into the pre-trained hub defect detection network.

[0061] Optionally, the generation module includes: a first calculation unit for calculating the center of the box of the preselected candidate box; a second calculation unit for calculating the distances between the center of the box and the cluster centers of each attention matrix in a pre-constructed attention matrix set, and selecting the target attention matrix with the closest distance; a first operation unit for multiplying the preselected candidate box by the corresponding coordinates in the target attention matrix and then accumulating to obtain an initial candidate box set, where each candidate box corresponds to a confidence level; a first filtering unit for filtering the initial candidate boxes with a confidence level less than a first threshold in the initial candidate box set to obtain an intermediate candidate box set; a second operation unit for calculating the intersection area and union area in the intermediate candidate box set, and calculating the intersection over union based on the intersection area and the union area; and a second filtering unit for filtering the intermediate candidate boxes with an intersection over union less than a second threshold in the intermediate candidate box set to obtain a defect box set, where each defect box includes the hub defect position, defect category, and confidence level.

[0062] Optionally, the device further includes: a second acquisition module, configured to select multiple grayscale sample pictures before the alignment module performs alignment operation on the hub image by using a pre-constructed alignment frame template; a first iteration module, configured to sequentially perform the following steps on each grayscale sample picture in the multiple grayscale sample pictures: select a sliding window with a fixed size starting from an initial position for the current grayscale sample picture, and select the window picture within the sliding window as a sliding template; use the sliding template to match the grayscale sample picture to find the lowest similarity degree of matching; select the window area where the sliding window with the smallest similarity degree is located in the grayscale sample picture; a reading module, configured to read the multiple window areas corresponding to the multiple grayscale sample pictures after the multiple grayscale sample pictures are processed; a first calculation module, configured to calculate an average position value of the multiple window areas to obtain an optimal alignment frame position; an extraction module, configured to respectively extract an optimal window picture of an alignment frame from the multiple grayscale sample pictures based on the optimal alignment frame position to obtain multiple optimal window pictures; a second calculation module, configured to calculate a pixel grayscale average value of the multiple optimal window pictures to obtain an alignment frame template.

[0063] Optionally, the device further includes: a third acquisition module, configured to acquire multiple defect sample pictures before the generation module generates defect information in the hub image according to the preselected candidate frame and the pre-constructed target attention matrix; a statistics module, configured to count calibration coordinates and defect sizes of defect areas in each defect sample picture in a preset calibration direction in the multiple defect sample pictures; a clustering module, configured to perform clustering of an initial number of categories on central points of the defect sizes; a second iteration module, configured to iteratively perform the following steps until an intersection with an area larger than a preset number of pixels appears: calculate intersections of multiple defect masks in each cluster; determine whether the number of pixels within the intersection is greater than a preset number; if the number of pixels within the intersection is less than or equal to the preset number, increase the number of categories of the cluster; a configuration module, configured to acquire multiple clusters after the iteration ends and configure an initial attention matrix for each cluster, where elements of the initial attention matrix are all 0; a reset module, configured to, for each cluster, reset element values in the initial attention matrix according to the number of defect masks overlapping at each element position to obtain a set of attention matrices, where the element values are positively correlated with the number of overlapping defect masks.

[0064] It should be noted that the above-mentioned respective modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited thereto: all the above-mentioned modules are located in the same processor; or, the above-mentioned respective modules are respectively located in different processors in any combination form.

[0065] Embodiment 3 An embodiment of the present invention also provides a storage medium, in which a computer program is stored, and the computer program is configured to execute the steps in any one of the above method embodiments when running.

[0066] Optionally, in this embodiment, the above storage medium may be configured to store a computer program for execution: S1. Obtain a hub image to be detected; S2. Perform a centering operation on the hub image by using a pre-constructed centering frame template to obtain a centering result, where the centering result is used to indicate whether the centering of the hub image is successful; S3. Input the hub image into a pre-trained hub defect detection network according to the centering result to obtain a preselected candidate box, where the preselected candidate box includes the hub defect position, defect category, and its confidence level in the hub image; S4. Generate defect information in the hub image according to the preselected candidate box and a pre-constructed target attention matrix, where the attention matrix is used to represent the probability information of a defect occurring at a corresponding pixel position.

[0067] Optionally, in this embodiment, the above storage medium may include, but is not limited to: various media such as a USB flash drive, a read-only memory (ROM for short), a random access memory (RAM for short), a mobile hard disk, a magnetic disk, or an optical disc that can store a computer program.

[0068] An embodiment of the present invention also provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0069] Optionally, the above electronic device may further include a transmission device and an input / output device, where the transmission device is connected to the above processor, and the input / output device is connected to the above processor.

[0070] Optionally, in this embodiment, the above processor may be configured to execute the following steps through a computer program: S1. Obtain a hub image to be detected; S2. Perform a centering operation on the hub image by using a pre-constructed centering frame template to obtain a centering result, where the centering result is used to indicate whether the centering of the hub image is successful; S3. Input the hub image into a pre-trained hub defect detection network according to the centering result to obtain a preselected candidate box, where the preselected candidate box includes the hub defect position, defect category, and its confidence level in the hub image; S4. Generate the defect information in the hub image according to the preselected candidate boxes and the pre-constructed target attention matrix, where the attention matrix is used to represent the probability information of defects occurring at corresponding pixel positions.

[0071] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation manners, and will not be elaborated herein.

[0072] The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0073] In the above embodiments of the present application, the descriptions of the respective embodiments have their own emphases. For the parts not elaborated in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0074] In the several embodiments provided by the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces. The indirect coupling or communication connection of units or modules can be in an electrical or other form.

[0075] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0076] In addition, the functional units in the respective embodiments of the present application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0077] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in various embodiments of this application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), mobile hard disks, magnetic disks, or optical discs.

[0078] The above are only the preferred embodiments of this application. It should be noted that for those of ordinary skill in the art of this technology, without departing from the principle of this application, several improvements and refinements can still be made, and these improvements and refinements should also be regarded as the protection scope of this application.

Claims

1. A defect detection method for an image pair, characterized in that, Including: Obtain a hub image to be detected; Perform a centering operation on the hub image using a pre-built centering frame template to obtain a centering result, where the centering result is used to characterize whether the centering of the hub image is successful; Input the hub image into a pre-trained hub defect detection network according to the centering result to obtain preselected candidate boxes, where the preselected candidate boxes include the positions of hub defects, defect categories, and their confidence levels in the hub image; Generate defect information in the hub image according to the preselected candidate boxes and a pre-built target attention matrix, where the attention matrix is used to characterize the probability information of defects occurring at corresponding pixel positions.

2. The method according to claim 1, wherein Performing a centering operation on the hub image using a pre-built centering frame template to obtain a centering result, including: Rotate the centering frame template according to a preset step size to generate a template set, where the template set includes centering frame templates corresponding to multiple rotation angles; Find the target centering frame template that best matches the hub image in the template set; Generate a centering result for the hub image based on the target centering frame template.

3. The method according to claim 2, wherein Finding the target centering frame template that best matches the hub image in the template set includes: Find the best matching area in the hub image; Calculate the similarity with the best matching area for each template in the template set respectively; Determine the template with the highest similarity as the target centering frame template that best matches the hub image.

4. The method according to claim 2, wherein Generating a centering result for the hub image based on the target centering frame template includes: Calculate the horizontal offset and vertical offset between the target centering frame template and the best matching area in the hub image; Judge whether both the horizontal offset and the vertical offset are less than a preset threshold; If both the horizontal offset and the vertical offset are less than the preset threshold, generate a first centering result; if the horizontal offset or the vertical offset is greater than or equal to the preset threshold, generate a second centering result, where the first centering result is used to characterize that the centering of the hub image is successful, and the second centering result is used to characterize that the centering of the hub image fails.

5. The method according to claim 1, characterized in that, Inputting the hub image into a pre-trained hub defect detection network according to the centering result includes: If the centering of the hub image is successful, obtain the centering rotation angle and centering translation amount of the hub image; rotate the hub image by the centering rotation angle to obtain a first intermediate image; translate the first intermediate image by the centering translation amount to obtain a second intermediate image; cut the background pixels other than the hub in the second intermediate image to obtain a third intermediate image; input the third intermediate image into a pre-trained hub defect detection network; If the centering of the hub image fails, input the hub image into a pre-trained hub defect detection network.

6. The method according to claim 1, wherein Generating defect information in the hub image according to the preselected candidate boxes and a pre-built target attention matrix includes: Calculate the center of the preselected candidate box; Calculate the distance between the center of the box and the clustering center of each attention matrix in the pre-built attention matrix set, and select the target attention matrix with the closest distance; Multiply the preselected candidate boxes by the corresponding coordinates in the target attention matrix and then accumulate them to obtain an initial set of candidate boxes, where each candidate box corresponds to a confidence level; Filter the initial candidate boxes with confidence levels less than the first threshold in the initial set of candidate boxes to obtain an intermediate set of candidate boxes; Calculate the intersection area and union area in the intermediate set of candidate boxes, and calculate the intersection over union based on the intersection area and the union area; Filter the intermediate candidate boxes with an intersection over union less than the second threshold in the intermediate set of candidate boxes to obtain a set of defect boxes, where each defect box includes the hub defect position, defect category, and confidence level.

7. The method according to claim 1, characterized in that, Before performing the centering operation on the hub image using a pre-constructed centering box template, the method further includes: Select multiple grayscale sample images; For each of the multiple grayscale sample images, perform the following steps in sequence: select a sliding window of a fixed size starting from the initial position for the current grayscale sample image, and select the window image within the sliding window as the sliding template; use the sliding template to match the grayscale sample image and find the lowest similarity; select the window area where the sliding window with the smallest similarity is located in the grayscale sample image; After the multiple grayscale sample images are processed, read the multiple window areas corresponding to the multiple grayscale sample images; Calculate the average position value of the multiple window areas to obtain the optimal centering box position; Based on the optimal centering box position, extract a best window image of a centering box from each of the multiple grayscale sample images to obtain multiple best window images; Calculate the pixel grayscale mean value of the multiple best window images to obtain the centering box template.

8. The method according to claim 1, wherein Before generating the defect information in the hub image according to the preselected candidate boxes and the pre-constructed target attention matrix, the method further includes: Obtain multiple defect sample images; Statistically analyze the calibration coordinates and defect sizes of the defect areas in each defect sample image in the multiple defect sample images in a preset calibration direction; Cluster the centers of the defect sizes into an initial number of categories; Iteratively execute the following steps until an intersection with an area larger than a preset number of pixels appears: calculate the intersection of multiple defect masks in each cluster; determine whether the number of pixels in the intersection is greater than the preset number; if the number of pixels in the intersection is less than or equal to the preset number, increase the number of categories of the cluster; Obtain multiple clusters after the iteration ends and configure an initial attention matrix for each cluster, where the elements of the initial attention matrix are all 0; For each cluster, reset the element values in the initial attention matrix according to the number of overlapping defect masks at each element position to obtain a set of attention matrices, where the element values are positively correlated with the number of overlapping defect masks.

9. A defect detection device in an image pair, characterized in that, Including: A first acquisition module for acquiring a hub image to be detected; A centering module for performing a centering operation on the hub image using a pre-constructed centering box template to obtain a centering result, where the centering result is used to indicate whether the centering of the hub image is successful; A processing module, configured to input the hub image into a pre-trained hub defect detection network according to the centering result, so as to obtain preselected candidate boxes, where the preselected candidate boxes include the positions, defect categories and their confidence levels of the hub defects in the hub image; A generation module, configured to generate defect information in the hub image according to the preselected candidate boxes and a pre-constructed target attention matrix, where the attention matrix is used to represent the probability information of defects occurring at corresponding pixel positions.

10. An electronic device, comprising a memory and a processor, characterized in that, A computer program is stored in the memory, and the processor is configured to run the computer program to execute the method according to any one of claims 1 to 8.

Images