Image-based bolt performance detection method and device

Abstract

The invention provides an image-based bolt performance detection method and device, and belongs to the field of image classification and recognized.The method comprises the steps that a first image set is acquired, and the first image set is preprocessed to obtain a second image set; wherein the first image set comprises a plurality of images of different angles of the target bolt; the second image set is input into a trained MobileNetV3 classification network, a third image set is obtained, and the third image set is the second image set which is obtained through classification and is provided with category labels; and inputting the third image set into a defect identification network to obtain a detection result of the target bolt. According to the image-based bolt performance detection method and device provided by the invention, background interference can be eliminated through mask preprocessing, targeted defect detection is realized in combination with bolt category information, the recognition precision is remarkably improved, the false detection rate is reduced, meanwhile, the bolt performance is accurately evaluated by adopting quantitative calculation, and the efficient and accurate requirements of industrial quality inspection are met.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image classification and recognition, and more particularly relates to a bolt performance detection method and device based on images. BACKGROUND

[0002] As a core basic component for connecting structures in the industrial field, the performance reliability of a bolt directly affects the safe operation of key fields such as mechanical equipment, construction engineering and rail transportation.

[0003] Traditional detection methods are low in efficiency and strong in subjectivity, and automatic detection technology based on images becomes the key to solving this problem. However, although a general deep learning model can automatically extract features, it has computational redundancy and ignores the differences between bolt types, resulting in poor detection effect for small types.

[0004] Therefore, an intelligent detection scheme that fuses prior knowledge of bolt types and supports multi-angle information fusion is urgently needed. SUMMARY

[0005] The application aims to provide a bolt performance detection method and device based on images to improve recognition accuracy and reduce false detection rate.

[0006] The first aspect of the embodiment of the application provides a bolt performance detection method based on images, comprising: obtaining a first image set, and pre-processing the first image set to obtain a second image set; wherein the first image set comprises multiple images of a target bolt at different angles; inputting the second image set into a trained MobileNetV3 classification network to obtain a third image set, wherein the third image set is the second image set with a category label obtained through classification; inputting the third image set into a defect recognition network to obtain a detection result of the target bolt.

[0007] The second aspect of the embodiment of the application provides a bolt performance detection device based on images, comprising: a first data processing module configured to obtain a first image set, and pre-process the first image set to obtain a second image set; wherein the first image set comprises multiple images of a target bolt at different angles; a second data processing module configured to input the second image set into a trained MobileNetV3 classification network to obtain a third image set, wherein the third image set is the second image set with a category label obtained through classification; a third data processing module configured to input the third image set into a defect recognition network to obtain a detection result of the target bolt.

[0008] In a third aspect, the embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor implements the steps of the image-based bolt performance detection method when running the computer program.

[0009] In a fourth aspect, the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the image-based bolt performance detection method.

[0010] The image-based bolt performance detection method and device provided by the embodiment of the present application have the following advantages: The pre-processing of the bolt image reduces background interference and reduces the amount of calculation, the MobileNetV3 classification network is used to accurately identify the bolt type, the defect identification network is used to realize targeted defect detection combined with the category label, and the multi-angle image information is fused to improve the detection integrity, so that the efficient and accurate detection and quantitative evaluation of the bolt performance are realized, the problems of calculation redundancy, poor type adaptability and non-quantitative defect evaluation in the general model are effectively solved, and the actual needs of the industrial scene for bolt performance quality inspection are met. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0012] Figure 1 The flowchart of the image-based bolt performance detection method provided by an embodiment of the present application is shown in the figure. Figure 2 The flowchart of the image-based bolt performance detection method provided by an embodiment of the present application is shown in the figure. Figure 3 The structural block diagram of the image-based bolt performance detection device provided by an embodiment of the present application is shown in the figure. Figure 4 The schematic block diagram of the electronic device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0013] In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, technologies, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

[0014] In order to make the purposes, technical solutions and advantages of the present application clearer, the following will be described by specific embodiments in conjunction with the accompanying drawings.

[0015] Reference will be made to Figure 1 , Figure 1 The flowchart of the image-based bolt performance detection method provided by an embodiment of the present application can be executed by an electronic device, specifically a computer, a server, etc. The method can include the following steps. S101: acquiring a first image set and pre-processing the first image set to obtain a second image set; wherein the first image set includes multiple images of the target bolt from different angles.

[0016] In the present embodiment, the first image set is for the same target bolt, and images are collected from at least three different angles, covering axial (e.g. front view of bolt head), radial (e.g. side view of bolt), oblique (e.g. thread detail view at 45° to the axis), etc., to ensure complete capture of the appearance features of key areas such as bolt head, thread section, tail, etc. The image background of the first image set can contain common elements in industrial scenes (such as workbench, conveyor belt), but the target bolt should account for no less than 20% in the image to avoid too small target leading to feature loss.

[0017] In the present embodiment, the core of the pre-processing is to separate the bolt area and the background, which is realized by semantic segmentation and mask operation. The second image set after pre-processing only contains the effective area of the bolt, and the background is completely invalidated, and the bolt features (such as head shape, thread texture) are more prominent.

[0018] S102: inputting the second image set into a trained MobileNetV3 classification network to obtain a third image set, wherein the third image set is the second image set with class labels obtained through classification.

[0019] In the present embodiment, the output layer of the MobileNetV3 classification network is a multi-task classification head, including but not limited to being designed according to the features such as bolt head shape, thread design or functional type. The MobileNetV3 classification network gives different weights to the second image set images from different angles, and determines the final class through weighted voting.

[0020] Further, the bolt images of different perspectives are clustered according to perspective features and classified into three core perspectives: axial front view, radial side view and oblique view. Images of the same perspective share the basic weight of the perspective, and the total weight is 1, wherein the axial front view basic weight is 0.4; the radial side view basic weight is 0.3; and the oblique view basic weight is 0.3. For multiple images in the same perspective group, sub-weights can be further assigned according to image quality (such as definition, uniformity of illumination). The image quality score is calculated by definition evaluation (such as Laplacian variance) and illumination evaluation (such as brightness standard deviation) (range 0-1), and the total weight of the group is the basic weight of the group.

[0021] Illustratively, there are three bolt images of different angles, and the bolt images share the lightweight feature extraction backbone of MobileNetV3. During feature extraction, the network automatically captures key features such as bolt head profile, thread texture and functional structure. And for different angle images of the same bolt, the network gives dynamic weight according to the amount of perspective information. For the axial front view (head or tail directly facing the lens) which can fully present the head shape and functional features, the weight coefficient is set to 0.4; for the radial side view (parallel to the axis) which focuses on the thread segment and the overall length features, the weight coefficient is set to 0.3; for the oblique view which supplements the transition area features of the head and thread, the weight coefficient is set to 0.3. The classification results of the images of different angles are weighted and summed, and the class with the highest score is taken as the final determination result.

[0022] In this embodiment, the third image set adopts a multi-level classification label system. On the basis of retaining the original second image data, classification labels are added, including but not limited to head shape categories (hexagonal head, internal hexagonal, flange head), thread design categories (full thread, partial thread, self-tapping screw), and functional type categories (double-headed bolt, anti-theft bolt, eye bolt).

[0023] Illustratively, the multi-angle second image data set is input into the MobileNetV3 classification network. The head features of the front view image are clear (weight 0.4), the thread features of the side view image are complete (weight 0.3), and the eye structure features of the oblique view image are clear (weight 0.3). MobileNetV3 synchronously processes the classification tasks of three dimensions of head shape, thread design and functional type. An angle weighting strategy is adopted to give higher weight to the front view image. Through a weighted voting mechanism, the recognition results of multiple angles are integrated to finally accurately determine that the bolt is of the "flange head-full thread-eye" type.

[0024] S103: inputting the third image set into the defect recognition network to obtain a detection result of the target bolt.

[0025] In this embodiment, the defect recognition network adopts an improved U-Net defect recognition network, and a multi-branch detection module is adopted to select a corresponding branch according to the category label of the input image, dynamically adjust the attention area, and recognize the defect type.

[0026] Exemplarily, the third image set (containing the effective area of the bolt + category label) is input into the defect recognition network, the category label is converted into a low-dimensional vector through an embedding layer, and the decoder module of the network is injected, so as to provide category prior information for feature extraction. The detection result of the target bolt includes a bolt category label (such as an internal hexagon + a partially threaded bolt) and a defect type label (such as head deformation + thread damage).

[0027] From the above, it can be seen that in the preprocessing stage, the bolt area is separated from the background and the background is invalidated, which reduces irrelevant information interference and network calculation redundancy, and highlights the key features of the bolt, laying a high-quality data foundation for subsequent detection; through multi-angle image acquisition and weighted fusion, combined with the multi-level classification label system of MobileNetV3, accurate and multi-dimensional identification of the bolt type is realized; with the help of the improved U-Net defect recognition network, the corresponding detection branch is dynamically activated according to the category label and the vulnerable area is focused, so that the pertinence and accuracy of defect detection are improved; the overall scheme takes into account the comprehensiveness, efficiency and industrial scene adaptability of detection, can accurately output the bolt type and defect information, and meets the actual needs of industrial quality inspection for bolt quality grading and fault tracing.

[0028] In an embodiment of the present application, a first image set is obtained, and a second image set is obtained by preprocessing the first image set, including: The first image set is input into a semantic segmentation model to generate a binary mask of the target bolt area; The first image set is multiplied by the mask to obtain the second image set.

[0029] In this embodiment, the semantic segmentation model includes but is not limited to lightweight models such as BiSeNetV2 and DeepLabv3. In the binary mask generated by the semantic segmentation model, the pixel value of the target bolt area is 1, and the pixel value of the background area is 0. This feature makes the mask a "spatial filter" - when multiplied pixel by pixel with the first image set (original image), the original pixel information (color, texture, detail) of the bolt area will be completely retained (1 x original pixel = original pixel), and the pixels of the background area will be cleared (0 x original pixel = 0), naturally realizing "background invalidation". The second image set does not crop the image, although the invalid background area has a pixel value of 0, it still occupies the spatial position of the original image, maintaining the original layout of the bolt in the image (i.e. not deleting the background area, only making it visually invalid), avoiding the loss of position information caused by cropping.

[0030] Further, the model classifies the input image pixel by pixel, and the pixels with output values greater than a threshold value (usually 0.5) are marked as bolt regions (value 1), and the rest are background (value 0). To improve edge accuracy, an edge detection loss (such as BoundaryLoss) can be introduced to control the error between the segmentation boundary and the true edge of the bolt within 2 pixels.

[0031] For example, the first image set is a set of original images collected from an industrial scene, containing multiple images of the target bolt at different angles (such as axial front view, radial side view, oblique view, etc.). A lightweight semantic segmentation model (such as BiSeNetV2) is used to process each image in the first image set. The preprocessed second image set presents: the bolt region retains the original texture and details, and the background region becomes pure black (RGB value is 0), forming a high-contrast image of "bolt foreground + black background", which is convenient for subsequent network focus analysis.

[0032] As can be seen from the above, by generating a binary mask through a lightweight semantic segmentation model and multiplying it with the original image pixel by pixel, the original texture, details, and other key information of the bolt region can be completely preserved, the background region pixel value can be cleared to realize invalidation, a high-contrast image can be formed to reduce background interference; at the same time, the image is not cropped, the spatial position of the invalid background area is preserved, the original layout of the bolt is maintained to avoid loss of position information. This method balances processing efficiency and effect, and can adapt to the processing needs of bolt multi-angle images in industrial scenes, providing high-quality and reliable input basis for subsequent network analysis.

[0033] In an embodiment of the present application, before the second image set is input into the MobileNetV3 classification network to obtain the third image set, it further includes: The MobileNetV3 classification network is trained based on a classification training image set to obtain a trained MobileNetV3 classification network, wherein the classification training image set is a plurality of images of standard bolts at different angles after preprocessing.

[0034] In this embodiment, multi-angle images of the same bolt are input into the network at the same time, sharing the feature extraction layer, and different weights are assigned to different angle image sets to learn the weight distribution of different view images, for example, higher weight is assigned to the front view and lower weight is assigned to the side view. The final class is determined by weighted voting. The MobileNetV3 classification network adds multiple parallel classification heads in the network output layer, including but not limited to head shape, thread design, and function type classification tasks.

[0035] Further, the MobileNetV3 classification network uses a weighted multi-task loss function to balance the training of different classification tasks, and its loss function formula is:

[0036] wherein, 、 、 correspond to the cross-entropy loss of multiple parallel classification heads respectively (such as head shape, thread design and function type); is the loss of angle weight learning (calculated by comparing the consistency of different angle predictions); is the weight coefficient, which is adjusted according to the importance of the task.

[0037] For example, for each image in the classification training image set, three types of labels (hexagonal head, internal hexagonal, flange head) are annotated, and the image is input into the model. Set the multi-task loss function, train for 100 epochs, and make the verification set accuracy reach 98%. Output three-level classification labels for each image, such as "internal hexagonal - partial thread - anti-theft bolt".

[0038] From the above, by constructing a classification training set containing multiple angle preprocessed images, combining multi-angle shared feature extraction and weighted voting mechanism, different angle information can be effectively fused, avoiding misjudgment caused by single angle deviation, and improving the classification robustness; Provide detailed category prior for subsequent defect detection, significantly improve the pertinence and accuracy of the overall detection process, and the multi-angle weight learning mechanism adapts to the diversity of bolt placement posture in industrial scenes, enhancing the industrial practicability of the model.

[0039] In an embodiment of the present application, before the third image set is input into the defect recognition network to obtain the detection result of the target bolt, the following steps are further included: training the first defect recognition network based on the recognition training image set to obtain the trained defect recognition network, wherein the recognition training image set is the classification training image set with category labels obtained through classification.

[0040] In this embodiment, the recognition training image set carries classification labels (such as head shape = hexagonal head) and defect labels (such as defect type = crack) at the same time, forming multi-modal annotation data. The defect area is pixel-level labeled (such as crack position is labeled by polygon frame), and the defect type (such as crack, deformation, and missing) is labeled.

[0041] In this embodiment, the first defect recognition network can learn and extract feature information in the image by training a large number of bolt image data with labels, and then obtain a trained defect recognition network, thereby realizing performance detection on new bolts to be recognized.

[0042] Further, during training, the training data is grouped by bolt category (such as hexagonal head, flange), and each batch only contains samples of the same category, so that the network focuses on learning the defect features of that type of bolt. Independent detection branches are designed for different types of bolts, each branch optimizes parameters for common defects of a specific type of bolt (such as hexagonal head bolts prone to head cracks, threaded bolts prone to thread wear), and the sample weight is adjusted according to the failure rate of the bolt in industry.

[0043] From the above, it can be concluded that the combination of category grouping training and independent detection branch design realizes the fine and targeted recognition of bolt defects. Grouping training by category allows the network to focus on the defect features of a specific type of bolt, and adjusting the sample weight based on the industrial failure rate ensures the recognition sensitivity of high-risk defect types and enhances the practicality of the model. This training strategy significantly improves the accuracy of defect recognition.

[0044] In an embodiment of the present application, the loss function of the first defect recognition network is a multi-task perception loss function.

[0045] In this embodiment, a multi-task perception loss function is used to simultaneously optimize the defect detection and classification tasks, and its formula is:

[0046] wherein, is the defect segmentation loss, which evaluates the prediction accuracy of the defect area; is the defect type classification loss, which identifies types such as cracks and wear; is the metric learning loss (such as Triplet Loss), which enhances the clustering of defects of the same type; is the weight coefficient, .

[0047] From the above, by integrating the loss functions of the three tasks of defect segmentation, type classification, and metric learning, comprehensive and accurate recognition and feature extraction of bolt defects are realized.

[0048] In an embodiment of the present application, the third image set is input to the defect recognition network to obtain the detection result, which includes: The defect network is an improved U-Net defect recognition network, which includes a mask perception convolution module and a multi-branch detection module; The mask perception convolution module is used for shielding processing of invalid areas, and the invalid area is an area with a pixel value of 0 in the target image, and the target image is an image in the second image set; The multi-branch detection module provides independent detection branches for different types of bolts.

[0049] In this embodiment, a mask-aware convolution module and a multi-branch detection module are introduced on the basis of a conventional U-Net.

[0050] The mask-aware convolution module automatically ignores the background invalid area (pixel value is 0) in the feature extraction process, and only updates the weight of the pixel in the valid area when calculating the gradient, thereby avoiding the interference of background noise. That is, a mask gating unit is added before each convolution layer of the U-Net, and the background area with a pixel value of 0 is automatically skipped during calculation.

[0051] The multi-branch detection module encodes the category label (such as hexagonal head / inner hexagonal) output by MobileNetV3 into a vector, injects the vector into the decoder of the U-Net, and integrates the category information into the feature extraction process of the U-Net. According to the bolt category label (such as hexagonal head, full thread), the corresponding detection branch is dynamically activated to achieve targeted defect recognition.

[0052] Further, through the targeted detection guided by the category label, the limitation of the general network to different types of bolt defects is solved, and the accuracy of defect recognition is improved (such as avoiding misjudgment of the normal protrusion of the flange head as deformation).

[0053] For example, after inputting the third image set containing the bolt valid area, the background invalid area, and the category label into the improved U-Net defect recognition network, the background area is shielded by the mask gating unit before each layer of convolution, and only the bolt valid area is used for feature extraction. The mask gating mechanism is also applied to ensure that only the details of the bolt area are restored during the feature upsampling process, thereby avoiding the misamplification of background noise. The decoder fuses the category label vector and the multi-scale feature to guide the convolution kernel of the corresponding branch to focus on the target defect area. For example, if the target bolt is a “flange head bolt”, the branch network will enhance the feature suppression of the annular protrusion on the flange face (to avoid misjudgment of the normal protrusion as deformation), and at the same time, the detection sensitivity to the edge crack of the protrusion will be enhanced. If it is a “self-tapping screw”, the branch network distinguishes the normal spiral structure of the thread from the texture disorder caused by wear through a texture matching algorithm.

[0054] As can be seen from the above, the mask-aware convolution module shields the background invalid area and only updates the gradient of the valid area, thereby reducing the interference of background noise, improving the feature extraction efficiency and purity, and ensuring that the network focuses on the key features of the bolt. The multi-branch detection module integrates the category label into the feature extraction process, dynamically activates the corresponding branch, optimizes the detection logic for different types of bolt defect patterns, significantly improves the accuracy and pertinence of bolt defect recognition, and enables the device to efficiently process various types of bolts and defects in industrial scenarios, thereby providing a reliable automated solution for bolt performance detection.

[0055] In an embodiment of the present application, after the third image set is input into the defect recognition network to obtain the detection result of the target bolt, the method further comprises: performing defect quantification calculation on the detection result; the detection result is the third image set with defect labels; the defect quantification calculation is calculating the proportion of the defect area to the area of the target bolt.

[0056] In this embodiment, the detection result to be quantified (the third image set with defect labels) needs to contain two types of key information: the effective area of the target bolt and the defect label data. The defect label data can be cracks, scratches, pits, thread wear, head deformation, or bending.

[0057] In this embodiment, when the defect type is surface wear or loss, or thread wear or loss, the identified defect area is first binarized to obtain a binary image containing only the defect area. By counting the number of pixels in the defect area and the total number of pixels in the bolt area , the defect area proportion calculation formula is:

[0058] In this embodiment, when the defect type is a crack or scratch, the identified crack or scratch area is skeletonized to convert it into a single-pixel-width line. By counting the number of pixels, the pixel length of the crack or scratch is calculated and the pixel length of the bolt , the crack or scratch length proportion calculation formula is:

[0059] As can be seen from the above, by designing a targeted quantification method, the qualitative defect label is converted into a quantifiable numerical indicator, achieving accurate measurement and objective evaluation of bolt performance.

[0060] The specific implementation process of the image-based bolt performance detection method provided in this embodiment is described with reference to Figure 2It can be seen that the first image set includes multiple images of the target bolt at different angles, which are used to completely capture the appearance features of the key regions of the bolt head, threaded segment and tail. The first image set is preprocessed, including generating a binary mask of the target bolt region using a semantic segmentation model, multiplying the first image set with the mask pixel by pixel, keeping the original pixels of the bolt region and clearing the pixels of the background region, to obtain a second image set containing only the effective region of the bolt and invalidating the background. The second image set is input into the trained MobileNetV3 classification network, and the network judges the bolt category through a multi-task classification head combined with an angle weighted voting mechanism, and outputs a third image set with a category label. According to the category label of the third image set, the improved U-Net defect recognition network selects the corresponding branch, and independent detection logic is designed for different types of bolts, improving the defect detection pertinence, focusing on the vulnerable areas of different types of bolts, and accurately identifying defects such as cracks and wear, and outputting the detection results with defect labels. The detection results are quantitatively calculated, and according to the defect type, the pixel number of the defect area and the total pixel number of the bolt area are calculated by means of binarization processing and skeleton extraction, to obtain the proportion of the defect area to the target bolt area, and realize the quantitative evaluation of the defect degree.

[0061] The image-based bolt performance detection method according to the above embodiment, Figure 3 The structure block diagram of the image-based bolt performance detection device provided by an embodiment of the present application is shown. For ease of illustration, only parts related to the embodiments of the present application are shown. For reference Figure 3 The image-based bolt performance detection device 20 includes a first data processing module 21, a second data processing module 22 and a third data processing module 23.

[0062] The first data processing module 21 is configured to obtain a first image set and pre-process the first image set to obtain a second image set. The first image set includes multiple images of a target bolt at different angles. The second data processing module 22 is configured to input the second image set into a trained MobileNetV3 classification network to obtain a third image set, which is a second image set with a category label obtained through classification. The third data processing module 23 is configured to input the third image set into a defect recognition network to obtain a detection result of the target bolt.

[0063] In an embodiment of the present application, the image-based bolt performance detection device 20 further includes a preprocessing module. The preprocessing module is configured to generate a binary mask of the target bolt region through a semantic segmentation model. The first image set is multiplied by the mask to obtain the second image set.

[0064] In an embodiment of the present application, the bolt performance detection device 20 based on images further comprises a classification network training module. The classification network training module is configured to train the MobileNetV3 classification network based on a classification training image set, to obtain a trained MobileNetV3 classification network, wherein the classification training image set is a plurality of images of the standard bolt at different angles after preprocessing.

[0065] In an embodiment of the present application, the bolt performance detection device 20 based on images further comprises a defect recognition network module. The defect recognition network module is configured to train the first defect recognition network based on an identification training image set, to obtain a trained defect recognition network, wherein the identification training image set is the classification training image set with category labels obtained after classification.

[0066] In an embodiment of the present application, the bolt performance detection device 20 based on images further comprises a loss function module. The loss function module is configured to provide a stable optimization direction in the training process, and the loss function of the first defect recognition network is a multi-task perception loss function.

[0067] In an embodiment of the present application, the bolt performance detection device 20 based on images further comprises a defect recognition module. The defect recognition module is configured to detect the bolt performance, and the defect network is an improved U-Net defect recognition network, which comprises a mask perception convolution module and a multi-branch detection module. The mask perception convolution module is used for shielding processing on the invalid area, the invalid area is an area with a pixel value of 0 in the target image, and the target image is an image in the second image set. The multi-branch detection module provides independent detection branches for different types of bolts.

[0068] In an embodiment of the present application, the bolt performance detection device 20 based on images further comprises a quantification module. The quantification module is configured to perform defect quantification calculation on the detection result. The detection result is a third image set with defect labels. The defect quantification calculation is to calculate the proportion of the defect area to the target bolt area.

[0069] Referring to Figure 4 , Figure 4 The schematic block diagram of the electronic device provided in an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the electronic device comprises a processor 10, a memory 20 and a communication interface 30. Figure 4The electronic device 300 in the embodiment shown can include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processor 301, the input device 302, the output device 303, and the memory 304 can communicate with each other through a communication bus 305. The memory 304 is configured to store a computer program including program instructions. The processor 301 is configured to execute the program instructions stored in the memory 304. For example, the processor 301 is configured to invoke the program instructions to implement the functions of the modules / units in the above-described device embodiments. Figure 3 The functions of the first data processing module 21, the second data processing module 22, and the third data processing module 23.

[0070] It should be understood that, in the embodiments of the present application, the processor 301 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0071] The input device 302 can include a touchpad, a fingerprint collection sensor (for collecting fingerprint information and direction information of a fingerprint of a user), a microphone, etc., and the output device 303 can include a display (LCD, etc.), a speaker, etc.

[0072] The memory 304 can include read-only memory and random access memory, and provide instructions and data for the processor 301. A portion of the memory 304 can also include non-volatile random access memory. For example, the memory 304 can also store device type information.

[0073] In specific implementations, the processor 301, the input device 302, and the output device 303 described in the embodiments of the present application can execute the implementation manners described in the image-based bolt performance detection method provided by the embodiments of the present application, and can also execute the implementation manners of the electronic device described in the embodiments of the present application, which will not be described here.

[0074] In another embodiment of the present application, a computer readable storage medium is provided, which stores a computer program. The computer program includes program instructions, which, when executed by a processor, implement all or part of the processes of the above-mentioned embodiment methods. The computer program can also instruct related hardware to complete the implementation. The computer program can be stored in a computer readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate form. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0075] The computer readable storage medium can be an internal storage unit of the electronic device of any of the preceding embodiments, such as a hard disk or a memory of the electronic device. The computer readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the computer readable storage medium can include both the internal storage unit and the external storage device of the electronic device. The computer readable storage medium is used to store the computer program and other programs and data required by the electronic device. The computer readable storage medium can also be used to temporarily store data that has been output or will be output.

[0076] Those skilled in the art can appreciate that the modules / units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0077] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the electronic device and the units described above can refer to the corresponding processes in the above-mentioned method embodiments, which will not be described here.

[0078] In several embodiments provided in the present application, it should be understood that the disclosed electronic device and method can be implemented in other manners. For example, the division of the above-described apparatus embodiments is merely an example, and there can be other division manners. For example, multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, or can be in electrical, mechanical or other forms.

[0079] The modules or units illustrated as separate parts can or can not be physically separate, and the parts illustrated as modules or units can or can not be physical modules or units, i.e., can be located in one place, or can be distributed on multiple network modules or units. Some or all of the modules or units can be selected according to actual needs to achieve the purposes of the embodiments of the present application.

[0080] In addition, each functional module / unit in each embodiment of the present application can be integrated into a processing module / unit, or each module / unit can exist physically separately, or two or more modules / units can be integrated into one module / unit. The integrated module / unit can be implemented in the form of hardware or in the form of a software functional module / unit.

[0081] The above is merely specific embodiments of the present application, and the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An image-based bolt performance detection method, characterized by, The method comprises the following steps: acquiring a first image set, and preprocessing the first image set to obtain a second image set; the first image set comprises multiple images of the target bolt at different angles; inputting the second image set into a trained MobileNetV3 classification network to obtain a third image set, wherein the third image set is the second image set with category labels obtained through classification; inputting the third image set into a defect recognition network to obtain a detection result of the target bolt.

2. The image-based bolt performance detection method of claim 1, wherein, The acquiring of the first image set and the preprocessing of the first image set to obtain the second image set comprises: generating a binary mask of the target bolt region through a semantic segmentation model based on the first image set; multiplying the first image set by the binary mask to obtain the second image set.

3. The image-based bolt performance detection method of claim 1, wherein, Before the inputting of the second image set into the MobileNetV3 classification network to obtain the third image set, the method further comprises: training the MobileNetV3 classification network based on a classification training image set to obtain the trained MobileNetV3 classification network, wherein the classification training image set comprises multiple images of a standard bolt at different angles after preprocessing.

4. The image-based bolt performance detection method of claim 1, wherein, Before the inputting of the third image set into the defect recognition network to obtain the detection result of the target bolt, the method further comprises: training the first defect recognition network based on an identification training image set to obtain the trained defect recognition network, wherein the identification training image set is the classification training image set with category labels obtained through classification.

5. The image-based bolt performance detection method of claim 4, wherein, The loss function of the first defect recognition network is a multi-task perception loss function.

6. The image-based bolt performance detection method of claim 1, wherein, The inputting of the third image set into the defect recognition network to obtain the detection result comprises: the defect recognition network is an improved U-Net defect recognition network, which comprises a mask perception convolution module and a multi-branch detection module; the mask perception convolution module is used for shielding processing of an invalid region, the invalid region is a region with a pixel value of 0 in a target image, and the target image is an image in the second image set; the multi-branch detection module provides independent detection branches for different types of bolts.

7. The image-based bolt performance detection method of claim 1, wherein, After the inputting of the third image set into the defect recognition network to obtain the detection result of the target bolt, the method further comprises: performing defect quantification calculation on the detection result; the detection result is the third image set with defect labels; and the defect quantification calculation is to calculate the proportion of the defect area to the area of the target bolt.

8. An image-based bolt performance detection apparatus, characterized by, The method comprises the following steps: a first data processing module is configured to acquire a first image set, and preprocess the first image set to obtain a second image set; the first image set comprises multiple images of a target bolt at different angles; a second data processing module is configured to input the second image set into a trained MobileNetV3 classification network to obtain a third image set, wherein the third image set is the second image set with category labels obtained through classification; a third data processing module is configured to input the third image set into a defect recognition network to obtain a detection result of the target bolt.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program, which is executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images