A method and an electronic device for monitoring the fastening degree of vehicle bottom bolts

Abstract

The present invention relates to a method and an electronic device for monitoring the fastening degree of vehicle bottom bolts. The method includes: for pre-acquired bolt images, identifying all bolts in the images and the regions where the bolts are located. Each bolt in the image includes pre-marked loosening prevention line information on the bolt body. Extracting a sub-image of each bolt from the image, and obtaining the category and mask image of each extracted bolt; for the sub-image of each extracted bolt, according to the loosening prevention line information, the bolt sub-image and the mask image of the bolt, obtaining the loosening prevention line region corresponding to the bolt; checking whether there is a minimum circumscribed rectangle in the loosening prevention line region. If there is one circumscribed rectangle and the area of the circumscribed rectangle is less than half of the area of the bolt sub-image, it is determined that the bolt is not loose; otherwise, according to the loosening judgment strategy, it is determined whether the bolt is loose. It solves the problems of multiple imaging, weak anti-interference ability, slow detection speed, low efficiency and inaccurate judgment when detecting bolts by traditional image methods.

Description

Technical Field

[0001] The present invention relates to the technical field of train infrastructure detection, and particularly to a method and an electronic device for monitoring the fastening degree of bolts at the bottom of a train. Background Art

[0002] With the development of the transportation industry, rail transit has gradually become a very important means of transportation in modern society; in railway tracks, various components at the bottom of a train will use different types of bolts to assist in the installation, fixation, and connection between components at the bottom of the train. Bolts are key components of the train bottom structure and an important guarantee for the safe operation of rail transit. The most important part of track inspection is the detection of the bolt state. Therefore, the positioning of bolts in images and the discrimination of their looseness are particularly crucial.

[0003] Currently, the commonly used method is manual inspection, observing the bolt position with the naked eye and manually discriminating bolt looseness. This method is time-consuming and laborious, largely relying on the experience judgment of workers and technicians, and the accuracy is difficult to guarantee. When using image recognition for detection, the imaging of bolts in images is affected by the diversity of types and uncertain external interference in the bottom environment of the train. The processing method relying on traditional image processing has weak robustness and cannot perfectly solve the above problems. There are defects such as slow speed, low efficiency, and inaccurate judgment. Summary of the Invention

[0004] (I) Technical Problems to be Solved

[0005] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a method and an electronic device for monitoring the fastening degree of bolts at the bottom of a train. It solves the technical problems of multiple imaging, weak anti-interference ability, slow detection speed, low efficiency, and inaccurate judgment when using image recognition to detect bolts.

[0006] (II) Technical Solutions

[0007] To achieve the above object, the main technical solutions adopted by the present invention include:

[0008] In the first aspect, an embodiment of the present invention provides a method for monitoring the fastening degree of bolts at the bottom of a train, which includes:

[0009] S10. For an image pre-collected including at least one bolt, identify all bolts in the image and the regions where the bolts are located. Each bolt in the image includes anti-loosening line information pre-labeled on the bolt body;

[0010] S20. Based on the image, extract a sub-image of each bolt, and obtain the category and mask image of each extracted bolt;

[0011] S30. For each sub - graph of the bolt extracted, according to the anti - loosening line information, the bolt sub - graph and the mask graph of the bolt, obtain the anti - loosening line area corresponding to the bolt;

[0012] S40. Check whether there is a minimum bounding rectangle in the anti - loosening line area. If there is one such bounding rectangle and the area of the bounding rectangle is less than half of the area of the bolt sub - graph, determine that the bolt is not loose;

[0013] Otherwise, according to the loosening judgment strategy, determine whether the bolt is loose.

[0014] Optionally, input the captured image into a pre - trained object detection network to frame the area of each bolt;

[0015] The object detection network is Darknet53 of YOLOv4.

[0016] Optionally, extract each bolt sub - graph from the image according to the framing information;

[0017] For each bolt sub - graph, input the bolt sub - graph into a classification network to obtain the category information of the bolt;

[0018] And input the bolt sub - graph and the category information into a segmentation network to obtain the mask graph of the bolt.

[0019] Optionally, for each bolt sub - graph, perform an erosion operation on the mask graph of the bolt to obtain a binary graph and acquire the anti - loosening line area.

[0020] Optionally, S40 includes: S401. Based on a pre - constructed function, find the anti - loosening line contour in the anti - loosening line area and obtain the minimum bounding rectangle;

[0021] S402. Judge the number and area of the bounding rectangles:

[0022] If the number of the bounding rectangles is 0, judge that the bolt is not loose;

[0023] If the number of the bounding rectangles is 1 and the area of the bounding rectangle is less than half of the area of the bolt sub - graph, judge that the bolt is not loose;

[0024] If the number of the bounding rectangles is not less than 2 and the area of the bounding rectangle is less than half of the area of the bolt sub - graph, use the loosening judgment strategy to judge;

[0025] If the area of any one of the bounding rectangles is greater than half of the area of the bolt sub - graph, judge that the shooting distance is too close to be recognized.

[0026] Optionally, the loosening judgment strategy further includes:

[0027] Identify the side lengths of the circumscribed rectangle, sort the circumscribed rectangles in descending order of the area, and select the first two circumscribed rectangles S1 and S2, where the area of the circumscribed rectangle S1 is greater than the area of S2;

[0028] According to a pre-set square discrimination strategy, if the circumscribed rectangle S2 is a square, calculate the distance between the centers of the two circumscribed rectangles. If this distance is less than a set first threshold, it is determined that the bolt is not loose;

[0029] If the distance is greater than 0.6 of the long side of the circumscribed rectangle S1, it is determined that the bolt is loose.

[0030] If the distance is outside the set first threshold and within 0.6 of the long side of the circumscribed rectangle S1, it is determined that the bolt is about to be loose;

[0031] If the circumscribed rectangle S2 is a rectangle, determine the rotation angles of the circumscribed rectangles S1 and S2 relative to the picture respectively, calculate the included angle between the circumscribed rectangles S1 and S2. If this included angle exceeds a preset included angle threshold, it is determined that the bolt is loose;

[0032] If the included angle is within the preset included angle threshold, calculate the ratio of the area of the anti-loosening line contour to the area of the circumscribed rectangle. If any of the ratios is less than 0.71, it is determined that the bolt is loose;

[0033] If both ratios are greater than 0.71, calculate the straight-line distance between the upper left corners of the two anti-loosening line contours. If the straight-line distance is above a preset second threshold, it is determined that the bolt is not loose. If the straight-line distance is less than the second threshold, determine whether the straight-line distance is above the first threshold. If so, it is determined that the bolt is loose, otherwise, it is determined that the bolt is not loose.

[0034] Optionally, the method further includes:

[0035] Display the information on whether each detected bolt is loose and the identification of the looseness judgment in the bolt areas of the captured image and present it.

[0036] Optionally, the pre-set square discrimination strategy is specifically:

[0037] If the ratio of the area of the anti-loosening line contour in the circumscribed rectangle S2 to the circumscribed rectangle S2 is greater than a pre-set third threshold, and the width-to-height ratio of the circumscribed rectangle S2 is within the open interval (0.71, 1.4), it is determined that the circumscribed rectangle is a square;

[0038] Otherwise, it is determined that the circumscribed rectangle is a rectangle.

[0039] In a second aspect, the present invention further provides an electronic device for monitoring the status of vehicle bottom bolts, including a memory and a processor. Computer instructions are stored in the memory, and the processor executes the computer instructions stored in the memory to specifically execute the method for monitoring the fastening degree of vehicle bottom bolts according to any one of the above first aspects.

[0040] (III) Beneficial effects

[0041] The present invention provides a method and an electronic device for monitoring the fastening degree of vehicle bottom bolts. Based on the methods of image acquisition, image processing, and model deep learning, bolts are located and classified, and then the loosening prevention line area on the bolts is segmented. After calculation, a loosening judgment strategy is used to determine whether the loosening prevention line is misaligned, and then a conclusion on whether the bolt is loose is obtained and displayed, solving the technical problems in the prior art such as multiple images, weak anti-interference ability, slow detection speed, low efficiency, and inaccurate judgment when identifying and detecting bolts by image, greatly improving the detection efficiency and accuracy, saving labor costs, and reducing the errors of manual detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a schematic flowchart of the method for monitoring the fastening degree of vehicle bottom bolts provided by an embodiment of the present invention;

[0043] Figure 2 is a schematic diagram of selecting a bolt area in another embodiment of the present invention

[0044] Figure 3 is a schematic diagram of a bolt sub-image and a binary image in another embodiment of the present invention;

[0045] Figure 4 is a schematic diagram of the minimum bounding rectangle in another embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0046] In order to better explain the present invention for easy understanding, the present invention will be described in detail below in conjunction with the drawings through specific embodiments. It can be understood that the specific embodiments described below are only used to explain the related invention and do not limit the invention. In addition, it should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other; for the convenience of description, only the parts related to the invention are shown in the drawings.

[0047] As Figure 1 shown, this embodiment provides a method for monitoring the fastening degree of vehicle bottom bolts. The monitoring method of this embodiment includes:

[0048] S10. For an image pre-collected including at least one bolt, identify all the bolts in the image and the areas where the bolts are located. Each bolt in the image includes loosening prevention line information pre-labeled on the bolt body;

[0049] In some embodiments, the anti-loosening line information pre-labeled on the bolt body can be manually labeled.

[0050] S20. Based on the image, extract the sub-graph of each bolt, and obtain the category and mask graph of each extracted bolt.

[0051] S30. For the sub-graph of each extracted bolt, according to the anti-loosening line information, the bolt sub-graph and the mask graph of the bolt, obtain the anti-loosening line area corresponding to the bolt.

[0052] S40. Check whether there is a minimum bounding rectangle in the anti-loosening line area. If there is one bounding rectangle and the area of the bounding rectangle is less than half of the area of the bolt sub-graph, it is determined that the bolt is not loose.

[0053] Otherwise, according to the loosening judgment strategy, determine whether the bolt is loose.

[0054] The method for monitoring the tightness of the bolts at the bottom of the vehicle in this embodiment is a method for monitoring whether the bolts at the bottom of the vehicle are tightened. Based on image processing and deep learning methods, the bolts are first located, then the anti-loosening line area on the bolts is segmented, and then it is judged whether the anti-loosening line is misaligned by calculation, and then it is concluded whether the bolts are loose. It can greatly improve the detection efficiency and accuracy, save labor costs and reduce the errors of manual detection.

[0055] In practical applications, for step S10, identifying all the bolts and the areas where the bolts are located in the pre-collected image specifically means: inputting the captured image into a pre-trained object detection network, and bounding the area of each bolt. In this embodiment, the object detection network specifically used is Darknet53 of YOLOv4. Of course, in some other embodiments, other object detection networks that can achieve the purpose of object detection can also be used, which are not limited here.

[0056] As Figure 2 shown, Figure 2 shows a schematic diagram of bounding the bolt area in an embodiment. For step S20, extract each bolt sub-graph from the pre-collected image according to the bounding information; each bolt sub-graph includes a bolt and the anti-loosening line information pre-labeled on the bolt. For each bolt sub-graph, input the bolt sub-graph into a classification network to obtain the category information of the bolt.

[0057] In this embodiment, the classification network is pre-trained to implement the function of the classification network to identify the category information of the bolts. Input the bolt sub-graph and its category information into the segmentation network to obtain the mask graph of the bolt sub-graph.

[0058] For each bolt sub - figure, perform an erosion operation on the mask figure of the bolt sub - figure to obtain a binary image and acquire the lock - prevention line region. As Figure 3 shown, Figure 3 (a) shows a bolt sub - figure in an embodiment, Figure 3 (b) shows the binary image of the bolt sub - figure after passing through the segmentation network and the erosion operation. The white part in the binary image is the lock - prevention line region.

[0059] In this embodiment, the erosion operation includes an erosion - dilation operation, which eliminates the boundary points of the lock - prevention line region, shrinks the target, eliminates the noise points smaller than the structural element, merges all the background points in contact with the lock - prevention line region into the lock - prevention line region, enlarges the target, and fills the holes in the lock - prevention line region, so that there are no holes inside the segmented lock - prevention line.

[0060] Further, for step S40, based on a pre - constructed function, find the contour of the lock - prevention line for the lock - prevention line region and obtain the minimum bounding rectangle; specifically, in this embodiment, through the minAreaRect function of Opencv, obtain the minimum bounding rectangle of each lock - prevention line region.

[0061] In actual use, the lock - prevention lines in the image may be misaligned and divided into multiple segments of lock - prevention lines. Therefore, when obtaining the minimum bounding rectangle, one or more minimum bounding rectangles may occur. As Figure 4 shown, Figure 4 shows a schematic diagram of the minimum bounding rectangle of an embodiment of the present invention. As shown in the figure, there are two lock - prevention line regions for the lock - prevention line, and each lock - prevention line region has a corresponding minimum bounding rectangle.

[0062] Further, judge the number and area of the bounding rectangles:

[0063] If the number of the bounding rectangles is 0, it is judged that the lock - prevention line region is not detected, and it is defaulted that the bolt is not loose;

[0064] If the number of the bounding rectangles is 1 and the area of the bounding rectangle is less than half of the area of the bolt sub - figure, that is, the lock - prevention line is not misaligned, it is judged that the bolt is not loose;

[0065] If the number of the bounding rectangles is not less than 2 and the area of the bounding rectangle is less than half of the area of the bolt sub - figure, use the loosening judgment strategy to judge;

[0066] If the area of any one of the bounding rectangles is greater than half of the area of the bolt sub - figure, it is judged that the shooting distance is too close to be recognized.

[0067] Further, when the number of the circumscribed rectangles is not less than 2 and the area of the circumscribed rectangles is within half of the area of the bolt sub - graph, a loosening judgment strategy is used for judgment, which specifically includes:

[0068] Identify the side lengths of the circumscribed rectangles, sort the circumscribed rectangles in descending order according to the area, and select the first two circumscribed rectangles S1 and S2, where the area of the circumscribed rectangle S1 is greater than the area of S2;

[0069] According to the pre - set square discrimination strategy, if the circumscribed rectangle S2 is a square, calculate the distance between the centers of the two circumscribed rectangles. If this distance is less than the set first threshold, it is judged that the bolt is not loose.

[0070] For the pre - set square discrimination strategy, specifically, if the ratio of the anti - loosening line contour area in the circumscribed rectangle S2 to the circumscribed rectangle S2 is greater than the pre - set third threshold, and the width - to - height ratio of the circumscribed rectangle S2 is within the open interval (0.71, 1.4), then the circumscribed rectangle is judged to be a square; otherwise, the circumscribed rectangle is judged to be a rectangle.

[0071] If the distance is greater than 0.6 of the long side of the circumscribed rectangle S1, it is judged that the bolt is loose.

[0072] If the distance is outside the set first threshold and within 0.6 of the long side of the circumscribed rectangle S1, it is judged that the bolt is about to be loose.

[0073] Specifically, the long side of the circumscribed rectangle S1 is the side with the largest length of the circumscribed rectangle.

[0074] Further, in practical applications, there is also a situation where the circumscribed rectangle S2 is a rectangle. At this time, the rotation angles of the circumscribed rectangles S1 and S2 relative to the picture can be judged respectively. Calculate the included angle between the circumscribed rectangles S1 and S2. If the included angle exceeds the preset included - angle threshold, it is judged that the bolt is loose.

[0075] Specifically, find the rotation angle of each rectangle relative to the picture. The angle calculation method is the angle rotated counter - clockwise from the x - axis to the wide side of the circumscribed rectangle, and calculate the included angle between the circumscribed rectangles S1 and S2. The wide side of the circumscribed rectangle is the side that the x - axis touches the circumscribed rectangle for the first time when rotating counter - clockwise, and the adjacent side of the wide side is the height.

[0076] If the included angle is within the preset included - angle threshold, calculate the ratio of the anti - loosening line contour area to the area of its corresponding circumscribed rectangle. If any of the ratios is less than 0.71, it is judged that the bolt is loose.

[0077] If the above ratios are all greater than 0.71, calculate the straight-line distance between the upper left corners of the two anti-loosening line contours. If the straight-line distance is above a preset second threshold, it is determined that the bolt is not loose. If the straight-line distance is less than the second threshold, determine whether the straight-line distance is above a first threshold. If so, it is determined that the bolt is loose; otherwise, it is determined that the bolt is not loose.

[0078] In this embodiment, the information on whether the detected bolt is loose can be displayed on an image interface, making the judgment result more obvious and readable.

[0079] The embodiment of the present invention also provides an electronic device for monitoring the status of bolts under the vehicle, which may include a memory, a processor, etc. Computer instructions are stored in the memory, and the processor executes the computer instructions stored in the memory to specifically execute the method for detecting the looseness of bolts under the vehicle described in any one of the above.

[0080] In some other embodiments, the electronic device further includes an image acquisition device for acquiring an image of the train underbody.

[0081] To better explain the present invention, the following will be described in conjunction with a specific embodiment.

[0082] In a specific embodiment, an electronic device for monitoring the status of bolts under the vehicle is used to detect the status of bolts under the vehicle. The resolution of its image acquisition device is a 4-million-pixel camera, with a maximum resolution of 2560×1440 and a working distance of 200 - 600 mm. Images of bolts under the vehicle are acquired, and the size of any captured image is scaled to 960x544, as Figure 2 shown. There are multiple bolts with different distances, angles, sizes, etc. in the figure. In this embodiment, the bolts under the vehicle are pre-marked with red anti-loosening line information. Of course, in actual applications, there are no requirements for the image acquisition device, the color of the anti-loosening line area, the image size, etc., which are determined according to the actual situation.

[0083] Regarding step S10, in this embodiment, the captured image is input into Darknet53 of the pre-trained object detection network YOLOv4 to frame each bolt on the image.

[0084] Regarding step S20, each bolt sub-image is cropped according to the framing information in the image. In this embodiment, the size of any bolt sub-image is scaled to 224*224, and each bolt sub-image is input into the classification network mobilenet to classify the bolts in the bolt sub-image. The classified bolt sub-images and category information are input into the segmentation network Unet to segment and extract the red pixel area of the bolts in the rectangular area, obtaining the mask image of the bolt sub-image.

[0085] For step S30, perform erosion and dilation operations on the mask image of the bolt sub - image to reduce the internal holes in the lock - prevention line and obtain a binary image. At this time, only the lock - prevention line area exists on the image.

[0086] For step S401, in this embodiment, after obtaining the lock - prevention line area, based on the findcontours function of Opencv, find the contour of the lock - prevention line area in the binary image. After obtaining the lock - prevention line contour, based on the minAreaRect function of Opencv, obtain the minimum bounding rectangle of each lock - prevention line area. In some practical applications, the coordinates of the minimum bounding rectangle are also obtained here, specifically, it can be the upper - left corner coordinates of the bounding rectangle, and the width and height of the bounding rectangle can also be obtained, as well as the angle of rotation of the bounding rectangle relative to the image. Both the width and height are the side lengths of the bounding rectangle, and the longer side is the long side.

[0087] Of course, in practical applications, other embodiments do not exclude any other technical means that can achieve the same purpose, and no restrictions are imposed here.

[0088] Further, for step S402, after obtaining the minimum bounding rectangle, judge the number and area of the minimum bounding rectangle:

[0089] If there is no such bounding rectangle, it means that the lock - prevention line area is not detected, and it is defaulted that the bolt is not loose, and the identification code is true = 0;

[0090] If there is one minimum bounding rectangle and the area of the bounding rectangle is within half of the area of the bolt sub - image, it is judged that the bolt is not loose, and the identification code is true = 1;

[0091] If there are two or more minimum bounding rectangles and the area of the bounding rectangle is within half of the area of the bolt sub - image, then enter the next loosening judgment strategy;

[0092] If the area of any one of the bounding rectangles is greater than half of the area of the bolt sub - image, it is judged that the shooting distance is too close to be recognized, and the identification code is DL_b2b, and it is defaulted that the bolt is not loose.

[0093] In this embodiment, when there are two or more minimum bounding rectangles and the area of the bounding rectangle is within half of the area of the bolt sub - image, sort the bounding rectangles in descending order of their areas, and select the first two bounding rectangles S1 and S2 with the largest areas, where the area of the bounding rectangle S1 is greater than the area of S2.

[0094] Further, according to the square judgment strategy, if the bounding rectangle S2 is a square, calculate the distance between the centers of the two bounding rectangles:

[0095] In this embodiment, for the square judgment strategy, the third threshold is set to 0.5. If the ratio of the anti-loosening line contour area in the circumscribed rectangle S2 to the circumscribed rectangle S2 is greater than 0.5, and the width-to-height ratio of the circumscribed rectangle S2 is within the open interval (0.71, 1.4), then this circumscribed rectangle is a square; otherwise, it is determined that the circumscribed rectangle is a rectangle.

[0096] If the distance between the centers of the two circumscribed rectangles is less than the set first threshold, it is determined that the bolt is not loose; the identification code is true + actual distance; in this embodiment, the first threshold is 18, and in some other embodiments, the first threshold is set according to the actual situation, and no limitation is made here.

[0097] If the distance between the centers of the two circumscribed rectangles is greater than 0.6 of the long side of the circumscribed rectangle S1, it is determined that the bolt is loose, and the identification code is flase_sq_d2b;

[0098] If the distance is above the set first threshold of 18 and within 0.6 of the long side of the circumscribed rectangle S1, it is determined that the bolt is about to loosen but has not yet loosened, and the identification code is DL_sq.

[0099] Further, if the circumscribed rectangle S2 is a rectangle, calculate the included angle between the two circumscribed rectangles. The angle calculation method is to find the rotation angle of the circumscribed rectangle relative to the screen, that is, the angle rotated counterclockwise from the x-axis until it first touches the side of the circumscribed rectangle. First, take the absolute value of the angle obtained by subtracting the rotation angle of the circumscribed rectangle S2 from the rotation angle of the circumscribed rectangle S1. When the absolute value of the angle is greater than 90 degrees and greater than 180 degrees minus the absolute value of this angle, the included angle θ is taken as 180 degrees minus the absolute value of the angle. When the absolute value of the angle does not belong to the above angle range, the included angle θ is equal to the absolute value of the angle.

[0100] In practical applications, the relative screen can be the x, y coordinate system, and the origin of the coordinate system is determined according to the actual situation, and no limitation is made here.

[0101] In this embodiment, the preset included angle threshold is 20 degrees. If this included angle is greater than or equal to the preset included angle threshold of 20 degrees, it is determined that the bolt is loose, and the identification code is flase + actual included angle degree.

[0102] If the included angle is less than the preset included angle threshold of 20 degrees, then calculate the ratio of the anti-loosening line contour area in S1 to the area of the circumscribed rectangle S1 and the ratio of the anti-loosening line contour area in S2 to the area of the circumscribed rectangle S2, that is, the filling rate. If any of the ratios is below 0.71, it is determined that the bolt is loose, and the identification code is flase + actual included angle;

[0103] If all the ratios are greater than 0.71, calculate the straight-line distance between the upper left corners of the two lockwire contours. In this embodiment, if the straight-line distance is greater than the preset second threshold of 1000, then determine that the distance is equal to 0, and determine that the bolt is not loose. The identification code is true + the actual included angle. If the straight-line distance is less than the second threshold of 1000, then determine whether the straight-line distance is greater than the first threshold of 18. If so, determine that the bolt is loose, and the identification code is flase_d2b. Otherwise, determine that the bolt is not loose, and the identification code is true + the actual included angle.

[0104] Of course, the thresholds in this embodiment are set based on the actual requirements of this embodiment. In some other embodiments, they should be set according to their actual requirements and are not limited hereby.

[0105] After detecting whether the bolt is loose, this embodiment also displays the information on whether each detected bolt is loose and the identification of the looseness judgment in the bolt areas of the captured image and presents it.

[0106] In addition, the types and specific identifications of the judgment identifications are established according to the requirements in specific applications and are not required here.

[0107] A method and an electronic device for monitoring the tightness of vehicle bottom bolts provided by the present invention. For the pre-collected bolt images, all the bolts in the images and the areas where the bolts are located are identified. Each bolt in the image includes the lockwire information manually marked on the bolt body in advance. Sub-images of each bolt are extracted from the image, and the category and mask image of each extracted bolt are obtained. For the sub-image of each extracted bolt, the lockwire area corresponding to the bolt is obtained by using each bolt sub-image and the mask image of the bolt. Check whether there is a minimum bounding rectangle in the lockwire area, and determine whether the bolt is loose according to the looseness judgment strategy. It can accurately detect the tightness of vehicle bottom bolts while saving manpower and material resources, reducing the waste of human resources, and having the advantages of high efficiency, strong anti-interference ability, and high accuracy.

[0108] Since the device described in the above embodiment of the present invention is the device adopted for implementing the method in the above embodiment of the present invention, based on the method described in the above embodiment of the present invention, those skilled in the art can understand the specific structure and deformation of the device, so it will not be elaborated here. All devices adopted by the method in the above embodiment of the present invention belong to the scope protected by the present invention.

[0109] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0110] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions.

[0111] In addition, it should be noted that in the description of this specification, the descriptions of terms such as "one embodiment", "some embodiments", "embodiment", "example", "specific example", or "some examples" mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. Moreover, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0112] Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications to these embodiments after learning the basic creative concept. Therefore, the claims should be construed to include the preferred embodiments and all changes and modifications falling within the scope of the present invention.

Claims

1. A method for monitoring the tightness of vehicle bottom bolts, characterized in that: include: S10. For a pre-collected image including at least one bolt, identifying all the bolts and the areas where the bolts are located in the image, wherein each bolt in the image includes anti-loosening line information pre-marked on the bolt body; S20. Extracting a sub-image of each bolt based on the image, and obtaining a category and a mask image of each extracted bolt; S30. For each extracted bolt subgraph, obtain the anti-loosening line area corresponding to the bolt according to the anti-loosening line information, the bolt subgraph, and the mask graph of the bolt; S40, searching whether there is a minimum circumscribed rectangle in the anti-loosening line area. If there is a circumscribed rectangle and the area of the circumscribed rectangle is less than half of the area of the bolt sub-graph, it is determined that the bolt is not loose; Otherwise, determining whether the bolt is loose according to a looseness judgment strategy; The S40 includes: S401, searching for the anti-loosening line contour of the anti-loosening line area based on a pre-built function, and obtaining a minimum circumscribed rectangle; S402: Determine the number and area of the circumscribed rectangles: If the number of the circumscribed rectangles is 0, it is determined that the bolt is not loose; If the number of the circumscribed rectangles is 1 and the area of the circumscribed rectangle is less than half of the area of the bolt subgraph, it is determined that the bolt is not loose; If the number of the circumscribed rectangles is not less than 2 and the area of the circumscribed rectangle is less than half of the area of the bolt sub-drawing, the looseness judgment strategy is used for judgment; If the area of any of the circumscribed rectangles is larger than half of the area of the bolt sub-image, it is determined that the shooting distance is too close and cannot be recognized; The looseness judgment strategy also includes: Identify the side lengths of the circumscribed rectangles, sort the circumscribed rectangles from largest to smallest according to the areas, select the first two circumscribed rectangles S1 and S2, and the area of the circumscribed rectangle S1 is larger than that of S2; According to a preset square determination strategy, if the circumscribed rectangle S2 is a square, the distance between the centers of the two circumscribed rectangles is calculated. If the distance is less than a set first threshold, it is determined that the bolt is not loose. If the distance is greater than 0.6 of the long side of the circumscribed rectangle S1, it is determined that the bolt is loose. If the distance is outside the set first threshold and within 0.6 of the long side of the circumscribed rectangle S1, it is determined that the bolt is about to loosen; If the circumscribed rectangle S2 is a rectangle, determine the rotation angles of the circumscribed rectangles S1 and S2 relative to the image, calculate the angle between the circumscribed rectangles S1 and S2, and if the angle exceeds a preset angle threshold, determine that the bolt is loose; The included angle of the circumscribed rectangles S1 and S2 is the included angle of the wide sides of the two circumscribed rectangles; If the angle is within a preset angle threshold, calculating the ratio of the anti-loosening line contour area to the circumscribed rectangle area, including: calculating the ratio of the anti-loosening line contour area in S1 to the area of the circumscribed rectangle S1 and the ratio of the anti-loosening line contour area in S2 to the area of the circumscribed rectangle S2; If any of the ratios is below 0.71, it is determined that the bolt is loose; If the ratios are both greater than 0.71, calculate the straight-line distance between the upper left corners of the two anti-loosening line contours. If the straight-line distance is above the preset second threshold, it is judged that the bolt is not loose. If the straight-line distance is less than the second threshold, determine whether the straight-line distance is above the first threshold. If so, it is judged that the bolt is loose. Otherwise, it is judged that the bolt is not loose.

2. The method according to claim 1, characterized in that The S10 includes: The captured image is input into the pre-trained object detection network to select the area of each bolt; The target detection network is Darknet53 of YOLOv4.

3. The method according to claim 1, characterized in that The S20 includes: Extracting each bolt sub-image from the image according to frame selection information; For each bolt subgraph, input the bolt subgraph into the classification network to obtain the category information of the bolt; And the bolt subgraph and the category information are input into a segmentation network to obtain a mask graph of the bolt.

4. The method according to claim 1, wherein The S30 includes: For each bolt sub-graph, the mask graph of the bolt is eroded to obtain a binary graph and the anti-loosening line area.

5. The method according to claim 1, wherein The method further comprises: The information of whether each bolt is loose and the loose judgment mark are displayed in each bolt area of the captured image and displayed.

6. The method according to claim 1, characterized in that The preset square discrimination strategy is specifically as follows: If the ratio of the area of the anti-loosening line contour in the circumscribed rectangle S2 to the area of the circumscribed rectangle S2 is greater than a preset third threshold, and the aspect ratio of the circumscribed rectangle S2 is between the open interval (0.71, 1.4), then the circumscribed rectangle is determined to be a square; Otherwise, it is determined that the circumscribed rectangle is a rectangle.

7. An electronic device for monitoring the status of vehicle bottom bolts, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer instructions, and the processor executes the computer instructions stored in the memory, specifically performing the method for monitoring the tightness of vehicle bottom bolts according to any one of claims 1 to 6.

Images