Defect detection method based on shared bicycle

By introducing the TBFormer module and a two-layer routing attention mechanism in the YOLOv9 network, the accuracy of shared bicycle detection is improved, and the inefficiency and subjectivity of shared bicycle defect detection in the existing technology is solved, and efficient and scientific vehicle maintenance decisions are achieved.

CN119027736AActive Publication Date: 2024-11-26CHANGZHOU UNIV
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411165288.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-23
Publication Date
2024-11-26
Estimated Expiration
2044-08-23

AI Technical Summary

Technical Problem

The lack of high-precision shared bicycle defect detection methods in the prior art, which leads to time-consuming and subjective manual detection, and is unable to effectively evaluate the damage of shared bicycles.

Method used

Using an improved network model based on YOLOv9, the attention of key information in the image is increased by inserting the TBFormer module, combined with the double-layer routing attention mechanism and depth separation convolution, improve detection accuracy, and calculate the comprehensive receptor through the vehicle component damage index. The degree of loss is implemented with different processing strategies.

Benefits of technology

It improves the accuracy and efficiency of defect detection of shared bicycles, reduces labor costs, and realizes scientific evaluation and reasonable maintenance strategies for shared bicycle damage.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027736A_ABST
    Figure CN119027736A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of shared bicycles, in particular to a shared bicycle-based defect detection method, which comprises the following steps of: acquiring a large number of images containing shared bicycles, and marking detection parts of the shared bicycles; a TBFormer module is inserted between a ninth RepNCSPELAN4 module of a backbone layer of the YOLOv9 and a tenth SPPELAN module of a check layer, so that the attention of key information in an image is increased, and the detection precision is improved; and performing part damage degree index calculation on the identified defects of the shared bicycle, performing comprehensive damage degree evaluation by using the part damage degree index, and performing different processing strategies on the shared bicycle according to the comprehensive damage degree. The problem of detecting the damage of the shared bicycle is solved, manual detection of the damage condition of the shared bicycle is replaced, the working efficiency is improved, and the labor cost is saved; the problem that the damage of the shared bicycle is not effectively evaluated in the prior art is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of shared bicycles, and in particular to a defect detection method based on shared bicycles. Background Art

[0002] Shared bicycles are a means of short-distance travel in cities. They solve the problems of urban traffic congestion and terminal travel, and provide a convenient and environmentally friendly travel option. They are not restricted by public transportation routes, greatly improving the convenience of urban travel.

[0003] However, the development of shared bicycles also faces some challenges, the most important of which is the problem of vehicle damage; for example, the wheels, handlebars, baskets, pedals, kickstands, seats and other vehicle parts may suffer obvious damage.

[0004] The traditional method of detecting shared bicycle defects is mainly to manually inspect the various parts of the shared bicycles, and distinguish which parts are damaged and which parts are normal based on the experience of maintenance technicians. This method requires inspectors to check and test each bicycle one by one, which is particularly time-consuming in the large-scale shared bicycle market. In addition, the results of manual inspections are often affected by the inspectors' experience, skills and attitudes, and are highly subjective.

[0005] The patent with publication number CN118522123A uses the mobilenetv3 network to determine the parking location of shared bicycles, which is used to detect the problem of illegal parking of shared bicycles; however, there is no method for detecting defects of shared bicycles. Since the defect detection of shared bicycles needs to be specific to each vehicle component, the detection accuracy of the model is required to be higher. How to improve the detection accuracy of shared bicycle components is an urgent problem to be solved. Summary of the invention

[0006] In view of the shortcomings of existing methods, the present invention solves the problem of detecting damage to shared bicycles and is used to replace manual detection of damage to shared bicycles, thereby improving work efficiency and saving labor costs; it also solves the problem of the lack of effective evaluation of damage to shared bicycles in the prior art.

[0007] The technical solution adopted by the present invention is: a defect detection method based on a shared bicycle comprises the following steps:

[0008] Step 1: Collect a large number of images containing shared bicycles and mark the detection parts of the shared bicycles;

[0009] As a preferred embodiment of the present invention, the detection components include: wheels, handlebars, a basket, pedals, a footrest, and a seat.

[0010] As a preferred embodiment of the present invention, the defects of the detected components include: scratches, rust, wheel cracks, basket cracks, missing pedals, missing footrests, and damaged seats.

[0011] Step 2: Insert the TBFormer module between the ninth RepNCSPELAN4 module of the backbone layer of YOLOv9 and the tenth SPPELAN module of the neck layer to increase the attention of key information in the image and improve the detection accuracy;

[0012] As a preferred embodiment of the present invention, the TBFormer module includes: DWConv 3x3, LN layer and MLP. The feature map A0 is residually connected with the feature map A0 after passing through DWConv 3x3 to obtain the feature map A1, and then two LN layers and a double-layer routing attention mechanism are respectively used to obtain the feature map A2 and the feature map A3. The feature maps A1, A2 and A3 are concat-operated to obtain the feature map A4, and then the feature map A4 is residually connected after passing through the LN layer and MLP.

[0013] Step 3: Calculate the component damage degree index for the identified shared bicycle defects, use the component damage degree index to evaluate the comprehensive damage degree, and adopt different treatment strategies for the shared bicycles according to the comprehensive damage degree;

[0014] As a preferred embodiment of the present invention, the formula of the component damage degree index is:

[0015]

[0016] in, is the i-th defect size of the j-th defect on the k-th component of the shared bicycle; j is the weight of the j-th defect on the shared bicycle; n is the number of the j-th defect on the k-th component of the shared bicycle; ε is a positive number; It is the maximum value of the j-th defect size on the k-th component of the shared bicycle.

[0017] As a preferred embodiment of the present invention, the formula for the comprehensive damage degree is:

[0018]

[0019] Among them, w i is the weight of each component of the shared bicycle, S i It is the damage degree index value of each component of the shared bicycle.

[0020] As a preferred embodiment of the present invention, different processing strategies are divided according to the damage conditions of the shared bicycles.

[0021] As a preferred embodiment of the present invention, the damage conditions include: normal, slightly damaged, moderately damaged, and severely damaged.

[0022] As a preferred embodiment of the present invention, a defect detection system based on a shared bicycle includes: a memory for storing instructions executable by a processor; and a processor for executing the instructions to implement a defect detection method based on a shared bicycle.

[0023] As a preferred embodiment of the present invention, a computer-readable medium stores a computer program code, and the computer program code implements a defect detection method based on a shared bicycle when executed by a processor.

[0024] Beneficial effects of the present invention:

[0025] 1. Build the TBFormer module to improve the backbone network of YOLOv9. The TBFormer module dynamically selects the key-value pairs most relevant to each query instead of simply interacting with all features, which improves the model's attention to key information in the image, optimizes the model's detection performance, and improves the model's detection accuracy.

[0026] 2. Construct a damage degree index for shared bicycle components, use the damage degree index to calculate the comprehensive damage degree of shared bicycles, implement different treatment strategies for shared bicycles based on the comprehensive damage degree, and improve the work efficiency of shared bicycle maintenance companies. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flow chart of a defect detection method based on a shared bicycle of the present invention;

[0028] Figure 2 It is a structural diagram of the dual-layer routing attention mechanism of the present invention;

[0029] Figure 3 It is the module structure diagram of TBFormer of the present invention;

[0030] Figure 4 It is a structural diagram of the improved YOLOv9 network model of the present invention. DETAILED DESCRIPTION

[0031] The present invention is further described below in conjunction with the accompanying drawings and embodiments. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner, and therefore it only shows the components related to the present invention.

[0032] like Figure 1 As shown, a defect detection method based on a shared bicycle includes the following steps:

[0033] Step 1: Collect a large number of images containing shared bicycles as training samples, and use the labeling tool Labelimg to label the sample images. The image files and labeling files together constitute a labeled data set;

[0034] The inspection parts of shared bicycles include six parts: wheels, handlebars, basket, pedals, footrests, and seats.

[0035] Component defects include scratches, rust, wheel cracks, basket cracks, missing footrests, missing kickstands, and damaged seats.

[0036] The size, coordinate position, and category information of the marked component defects. The image file and the annotation file together constitute the annotated data set, and each image file name and its corresponding annotation file name must be consistent.

[0037] The video of the shared bicycle that needs to be processed is collected in real time through the camera, and the collected video is transmitted to NVIDIA's JetsonNX development board for defect detection; the shared bicycle image is obtained by frame reading the video at intervals of three frames, and CLAHE is used to perform contrast equalization on the image obtained by the camera to avoid the image being affected by local overexposure or too dark and affecting the defect detection of the shared bicycle; CLAHE is used to limit the contrast adaptive histogram equalization.

[0038] Step 2: Construct a TBFormer module (dynamic sparse attention mechanism), use the YOLOv9 network model as the basic network model of the present invention, introduce TBFormer into the backbone network, and construct an improved YOLOv9 network model;

[0039] Improve the YOLOv9 structure such as Figure 4As shown in the figure, it includes four parts: input end, backbone network, neck network, and head network. The image data is sent to the input end, passes through the backbone network and neck network, and is output through the head network. Among them, the Conv convolution layer is used for image feature extraction. RepNCSPELAN4 is the feature extraction fusion module in YOLOv9. The TBFormer module adopts a dynamic sparse attention mechanism to improve the model's attention to key information in the image, optimize the model's detection performance, and improve detection accuracy. The SPPELAN module combines SPP with ELAN to improve the performance of the model in target detection tasks. Ups Ample is upsampling, which is used to improve the resolution of feature maps so that more detailed information can be captured in subsequent convolutional layers; Concat is used to connect feature maps of different layers; Conv-reg is a convolutional layer used for target detection to detect the coordinates of the bounding box; Conv-cls is a convolutional layer used for target classification to classify the bounding box; CBLinear is a convolutional block that combines a convolutional layer, a batch normalization layer, and an activation function. When constructing the basic unit of the network, it can enhance the information flow of the network through reversible connections; the CBFuse module is mainly used for feature fusion; the arrows in the figure indicate the direction of data flow.

[0040] The core part of the TBFormer module is composed of a two-layer routing attention mechanism. The structure of the two-layer routing attention mechanism is as follows: Figure 2 As shown in the figure, an image containing a shared bicycle is input into the two-layer routing attention mechanism, and the feature tensor X of the input image is defined as:

[0041]

[0042] Among them, H is the height of the feature tensor, W is the width of the feature tensor, and C is the number of channels in the feature tensor.

[0043] The image is divided into S×S different regions, each of which contains HW / S 2 feature vectors, the feature tensor X of the input image is transformed into X t :

[0044]

[0045] X t After linear mapping, we can obtain Q, K, and V:

[0046] Q=X t W q ,K=X t W k ,V=X t W v

[0047] Among them, Wq , W k , W v are the projection weights for query, key, and value, respectively.

[0048] The projection weights aim to map the original feature space into another space where queries, keys, and values ​​can be efficiently computed for the subsequent two-layer routing attention mechanism to operate.

[0049] Create a directed graph to find the position where each area should act, and perform average pooling on Q and K, that is, calculate the average value Q of Q and K in each area of ​​S×S areas t , K t .

[0050] Q t =AveragePooling(Q),K t =AveragePooling(K)

[0051]

[0052] The attention weight is calculated on the coarse-grained Token, and then only the Topk regions are taken as relevant regions to participate in fine-grained operations.

[0053] First calculate Q t , K t The critical matrix A of the correlation between regions t :

[0054] A t =Q t (K t ) T

[0055] Secondly, only the top k connections of each region need to be retained to prune the dependency graph. The TopkIndex selection operation is used to retrieve the index of the top k values ​​from the input data, and the routing index matrix I t It is the index set used to select the k elements with the highest attention in the attention weight allocation process, with a routing index matrix I t It can be expressed as:

[0056] I t =topkIndex(A t )

[0057] Among them, I t The i-th row of contains the indices of the top k most relevant regions of the i-th region.

[0058] Finally, the most relevant Topk coarse-grained regions of each Token are used as keys and values ​​to participate in the final operation, that is, the routing index matrix It As the key and value for the final operation; in order to reduce the complexity of the calculation, the gather operation is used to aggregate the key and value tensors K g and V g , the formula is:

[0059] K g =gather(K,I t ),V g =gather(V,I t )

[0060] After aggregation, K g and V g Using the attention operation, attention weights are assigned according to the similarity between the query and the key, and the weighted sum of the values ​​is calculated using these weights to obtain a context-dependent representation of each query; in order to enhance locality, so as to provide richer local context information and enhance the model's understanding of the input image, a local context enhanced deep convolution LCE is introduced. Finally, the output O of the double-layer routing attention mechanism can be expressed as:

[0061] O=Attention(Q,K g ,V g )+LCE(V)

[0062] The TBFormer model is mainly composed of a bi-level routing attention mechanism (Bi-level RoutingAttention), a deep separable convolution (DWConv 3x3), a layer normalization (LN), and a multi-layer perceptron (MLP), such as Figure 3 The figure shows the TBFormer model; the two-layer routing attention mechanism is the core component of the TBFormer model, which dynamically selects the key-value pairs most relevant to each query instead of simply interacting with all features; the depthwise separable convolution (DWConv 3x3) is a lightweight convolution layer used to capture local features and maintain computational efficiency; layer normalization (LN) is used to stabilize the training process of the model; the multi-layer perceptron (MLP) is a fully connected network part for nonlinear feature transformation.

[0063] The TBFormer module is an improvement on the BiFormer module. The input information is firstly connected with the original information through a depthwise separable convolution to avoid information loss. Then, layer regularization training is performed to stabilize the model. After the model is stabilized, it is divided into two paths, each of which enters a two-layer routing attention mechanism with different weights to dynamically select the most relevant key-value pairs for each query. The outputs O1 and O2 of the two paths after the two-layer routing attention mechanism are weighted averaged to obtain the output O TBF The formula is as follows:

[0064] O TBF =w a O1+w b O2

[0065] Among them, w a is the weight of output O1, w b is the weight of output O2.

[0066] It consists of DWConv 3x3, LN layer, and MLP. Feature map A0 is residually connected with feature map A0 after passing through DWConv 3x3 to obtain feature map A1. Then, after the first LN layer and the first double-layer routing attention mechanism, the second LN layer and the second double-layer routing attention mechanism, feature map A2 and feature map A3 are obtained. Feature maps A1, A2, and A3 are concat-operated to obtain feature map A4. After passing through the third LN layer and MLP, a residual connection is performed with feature map A4 to obtain feature map A5.

[0067] The improved YOLOv9 network is trained using a labeled shared bicycle dataset, and the trained model is used to identify shared bicycle images. It detects six types of bicycle parts: wheels, handlebars, baskets, pedals, kickstands, and seats, as well as seven types of defects on shared bicycles: scratches, rust, wheel cracks, basket cracks, missing pedals, missing kickstands, and damaged seats.

[0068] Step 3. Since the seven defects of scratches, rust, wheel cracks, basket cracks, missing pedals, missing kickstands, and damaged seats have different impacts on the normal driving of shared bicycles, different weights are given to the seven defects; the damage degree indicators of the wheels, handlebars, baskets, pedals, kickstands, seats, and frames are calculated based on the number, size, and corresponding weights of the seven defects; and whether the corresponding components need to be replaced is determined based on the damage degree indicators of each component.

[0069] The expression of each defect size g on the shared bicycle is:

[0070] g=wh

[0071] Where w is the length of the defect and h is the width of the defect.

[0072] The wheel damage index S1 is determined by calculating the number, size and corresponding weights of the three defects of scratches, rust and wheel cracks; the handlebar damage index S2 is determined by calculating the number, size and corresponding weights of the two defects of scratches and rust; the basket damage index S3 is determined by calculating the number, size and corresponding weights of the three defects of scratches, rust and basket cracks; the pedal damage index S4 is determined by calculating the number and size of the three defects of scratches, rust and missing pedals; the footrest damage index S5 is determined by calculating the number and size of the three defects of scratches, rust and missing footrest; the seat damage index S6 is determined by calculating the number and size of the two defects of scratches and seat breakage.

[0073] Therefore, the specific expression of the damage degree index of the shared bicycle components is:

[0074]

[0075] in, is the i-th defect size of the j-th defect on the k-th component of a shared bicycle, k∈{1,2,3,4,5,6}, 1-6 correspond to the six components of the shared bicycle, namely, wheel, handlebar, basket, pedal, footrest, and seat; j∈{1,2,3,4,5,6,7}, 1-7 correspond to the seven defects of the shared bicycle, namely, scratches, rust, wheel cracks, basket cracks, pedal missing, footrest missing, and seat damage; λ j is the weight of the j-th defect on the shared bicycle; n is the number of the j-th defect on the k-th component of the shared bicycle; ε is a positive number infinitely close to but not equal to zero, in order to ensure that the denominator is not zero; It is the maximum value of the j-th defect size on the k-th component of the shared bicycle.

[0076] Introduction The main purpose is to control the damage index S of vehicle parts k To prevent numerical explosion, S k satisfy:

[0077]

[0078] Since the two defects of missing pedals and missing footrests can only be identified as presence or absence, when k=4, the network model recognizes that the shared bicycle has pedals: When the network model does not recognize the pedals of a shared bicycle: Equal to the size of the pedal itself; when k≠4: When k=5, the network model recognizes that the shared bicycle has a kickstand: When the network model does not recognize the shared bicycle's kickstand: Equal to the size of the footrest itself; when k≠5,

[0079] The frame damage index S7 cannot be calculated directly, but can be calculated based on the difference between the defects of the entire shared bicycle and the defects of other vehicle parts. The expression of the frame damage index S7 is:

[0080]

[0081] Among them, (g j ) i is the i-th defect size of the j-th defect on the shared bicycle, is the i-th defect size of the j-th defect on the k-th component of the shared bicycle, k∈{1,2,3,4,5,6}, 1-6 correspond to the six components of the shared bicycle, namely, wheel, handlebar, basket, pedal, footrest, and seat, and j∈{1,2,3,4,5,6,7}, 1-7 correspond to the seven defects of the shared bicycle, namely, scratches, rust, wheel cracks, basket cracks, pedal missing, footrest missing, and seat damage; λ j is the weight of the j-th defect on the shared bicycle; m is the number of the j-th defect on the shared bicycle, n is the number of the j-th defect on the k-th component of the shared bicycle; ε is a positive number infinitely close to but not equal to zero, in order to ensure that the denominator is not zero; (g j +ε) max is the maximum value of the j-th defect size on the shared bicycle.

[0082] After calculating the damage degree index of each component of the shared bicycle, different weights are assigned to the seven components of the shared bicycle, namely the wheel, handlebar, basket, pedals, kickstand, seat and frame. The comprehensive damage degree of the shared bicycle is displayed by calculating the weighted sum of the damage degree index of each component of the shared bicycle. Based on the comprehensive damage degree of the shared bicycle, it is judged whether the vehicle is worth repairing.

[0083] The calculation formula for the comprehensive damage degree S of shared bicycles is:

[0084]

[0085] Among them, w i is the weight of each component of the shared bicycle, S i is the damage index value of each component of the shared bicycle; i∈{1,2,3,4,5,6,7} and 1-7 represent the wheels, handlebars, basket, pedals, kickstand, seat and frame of the shared bicycle respectively.

[0086] According to the comprehensive damage degree S of shared bicycles, the damage conditions of shared bicycles are divided into four types, namely normal, slightly damaged, moderately damaged, and severely damaged. Different thresholds are set according to the damage conditions, and the comprehensive damage degree S of shared bicycles is compared with the threshold to determine the damage condition of the shared bicycles.

[0087] Different handling strategies are adopted for damaged situations. For normal types of shared bicycles, it is only necessary to use a high-pressure water gun and detergent to thoroughly clean the body, remove stains, and then repaint; for slightly damaged and moderately damaged shared bicycles, it is necessary to replace the damaged parts of the vehicle, reduce it to normal type, and then clean the body, remove stains, repaint, etc.; and severely damaged shared bicycles have lost their value for repair and can be directly scrapped.

[0088] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. A defect detection method based on shared bicycles, characterized in that: The following steps are involved: Step 1: Collect a large number of images containing shared bicycles and mark the detection parts of the shared bicycles; Step 2: Insert the TBFormer module between the ninth RepNCSPELAN4 module of the backbone layer of YOLOv9 and the tenth SPPELAN module of the neck layer to increase the attention of key information in the image and improve the detection accuracy; Step 3: Calculate the component damage degree index for the identified shared bicycle defects, use the component damage degree index to evaluate the comprehensive damage degree, and adopt different treatment strategies for the shared bicycles according to the comprehensive damage degree.

2. The defect detection method based on shared bicycles according to claim 1 is characterized in that: The TBFormer module includes: DWConv 3x3, LN layer and MLP. Feature map A0 is passed through DWConv 3x3 and then residually connected with feature map A0 to obtain feature map A1. Then, two LN layers and a double-layer routing attention mechanism are used to obtain feature maps A2 and A3. Feature maps A1, A2 and A3 are concat-operated to obtain feature map A4. Feature map A4 is passed through the LN layer and MLP and then residually connected with feature map A4.

3. The defect detection method based on shared bicycles according to claim 1 is characterized in that: The formula for the component damage index is: in, is the i-th defect size of the j-th defect on the k-th component of the shared bicycle; j is the weight of the j-th defect on the shared bicycle; n is the number of the j-th defect on the k-th component of the shared bicycle; ε is a positive number; It is the maximum value of the j-th defect size on the k-th component of the shared bicycle.

4. The defect detection method based on shared bicycles according to claim 3 is characterized in that: The formula for the comprehensive degree of damage is: Among them, w i is the weight of each component of the shared bicycle, S i It is the damage degree index value of each component of the shared bicycle.

5. The defect detection method based on shared bicycles according to claim 1 is characterized in that: The test parts include: wheels, handlebars, basket, pedals, footrests, and seats.

6. The defect detection method based on shared bicycles according to claim 5 is characterized in that: The defects of the inspected parts include: scratches, rust, wheel cracks, basket cracks, missing pedals, missing footrests, and damaged seats.

7. The defect detection method based on shared bicycles according to claim 1 is characterized in that: Different handling strategies are divided according to the damage of the shared bicycles.

8. The defect detection method based on shared bicycles according to claim 7 is characterized in that: The damage conditions include: normal, slightly damaged, moderately damaged, and severely damaged.

9. The defect detection system based on shared bicycles is characterized by: include: a memory for storing instructions executable by a processor; A processor, used to execute instructions to implement the defect detection method based on a shared bicycle as described in any one of claims 1-8.

10. A computer readable medium storing computer program code, characterized in that: When the computer program code is executed by a processor, it implements the defect detection method based on a shared bicycle as described in any one of claims 1 to 8.

Citation Information

Patent Citations

  • Shared bicycle returning and parking management method and system

    CN107464363A

  • Industrial product defect detection method and industrial intelligent camera

    CN112082999A

  • Target detection method based on deployment optimization of shared bicycles

    CN113392799A

  • Shared vehicle damage monitoring system and method thereof

    WO2019085241A1