Fastener loosening identification method, system, terminal and storage medium
Through image acquisition and point cloud data analysis based on grayscale cameras and depth cameras, the fatigue problem of manual inspection of fastener looseness and inaccurate judgment of two-dimensional image are solved, achieving more efficient and accurate fastener looseness recognition .
Patent Information
- Application Number
- CN202011227065.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-11-06
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-11-06
AI Technical Summary
In the prior art, manual inspection of loose fasteners has problems such as fatigue and slackness and inaccurate judgment of two-dimensional images, resulting in missed detection and judgment errors.
Image acquisition based on grayscale cameras and depth cameras is used to extract the mask depth values of the marking lines of the fastener, generate point cloud data, and automatically identify whether the fastener is loose through connectivity analysis and PCA algorithm.
It improves the accuracy and efficiency of fastener looseness judgment, replaces manual inspection, and provides re-testable data.
Smart Images

Figure CN112365461B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to quality inspection technology, and more particularly to a method, system, terminal and storage medium for identifying loose fasteners. Background Art
[0002] In rail vehicles such as high-speed trains and subways, multiple parts are usually connected by fasteners such as bolts. In order to prevent the fasteners from loosening and causing safety accidents, anti-loosening parts such as 8-shaped iron wire are usually passed through and wrapped between two fasteners. The vibration and other factors generated during the operation of rail vehicles cause anti-loosening parts such as iron wire to break easily, which in turn leads to the risk of fasteners such as bolts loosening.
[0003] In the related art, manual inspection is usually used to determine whether a fastener is loose. Specifically, a marking line is drawn on the fastened bolt, and the inspector observes with the naked eye whether the marking line is misaligned. When the marking line is observed to be misaligned, it is determined that the fastener is loose. However, with the manual inspection method, workers are easily tired and slack, resulting in missed inspections. Summary of the invention
[0004] In order to solve one of the above-mentioned technical defects, a method, system, terminal and storage medium for identifying loose fasteners are provided in an embodiment of the present application.
[0005] A first aspect of the present application provides a method for identifying loose fasteners, comprising:
[0006] Expanding and segmenting the acquired fastener detection frame to obtain a mask of the fastener marking line;
[0007] Extracting corresponding depth values on the depth map according to the mask of the marking line of the fastener and obtaining corresponding point cloud data according to the camera intrinsic parameters;
[0008] When there is disconnection and different main directions in the point cloud data, it is determined that the fastener is loose.
[0009] A second aspect of the present application provides a fastener loosening identification system, including:
[0010] A first processing module is used to expand and segment the acquired fastener detection frame to obtain a mask of a marking line of the fastener;
[0011] A second processing module, configured to extract a corresponding depth value on a depth map according to a mask of a marking line of the fastener and obtain corresponding point cloud data according to a camera intrinsic parameter;
[0012] The third processing module is used to determine that the fastener is loose when there is no connectivity and different main directions in the point cloud data.
[0013] A third aspect of the present application provides a terminal, including:
[0014] Memory;
[0015] Processor; and
[0016] Computer programs;
[0017] The computer program is stored in the memory and is configured to be executed by the processor to implement any of the methods described above.
[0018] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon; the computer program is executed by a processor to implement a method as described in any one of the foregoing.
[0019] The embodiments of the present application provide a method, system, terminal and storage medium for identifying loose fasteners. Based on the images captured by the grayscale camera and the depth camera, the depth value corresponding to the mask of the marking line of the fastener is extracted and the corresponding point cloud data is obtained according to the camera internal parameters. The point cloud data is subjected to connectivity analysis and PCA, and the loose fasteners are automatically identified according to the analysis results. In the embodiments of the present application, the spatial information of the fastener is more complete, and the accuracy is higher than that of the judgment result based on the two-dimensional image, and it can replace manual inspection and identification, which is more efficient and the data can be repeated. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0021] Figure 1 A flowchart of a method provided for an exemplary embodiment;
[0022] Figure 2 A flowchart of a method provided for an exemplary embodiment;
[0023] Figure 3 A schematic diagram of the structure of a system provided for an exemplary embodiment;
[0024] Figure 4a A schematic diagram of an exemplary embodiment when a bolt is not loosened;
[0025] Figure 4b A schematic diagram of a bolt being loosened is provided for an exemplary embodiment. DETAILED DESCRIPTION
[0026] In order to make the technical solutions and advantages in the embodiments of the present application more clearly understood, the exemplary embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than an exhaustive list of all the embodiments. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.
[0027] In the related art, manual inspection is usually used to determine whether a fastener is loose. Specifically, a marking line is drawn on the fastened bolt, and the inspector observes with the naked eye whether the marking line is misaligned. When the marking line is observed to be misaligned, it is determined that the fastener is loose. However, with the manual inspection method, workers are easily tired and slack, resulting in missed inspections.
[0028] In addition, there are also some methods that determine whether the marking lines are in a straight line based on the two-dimensional image. However, due to the fact that the direction of the marking lines in the two-dimensional image is not fixed or the camera angle is not good, it is impossible to accurately determine whether the marking lines are misaligned in the two-dimensional image.
[0029] In order to overcome at least one of the above problems, the embodiments of the present application provide a method, system, terminal and storage medium for identifying loose fasteners, which automatically judge whether the fasteners are loose in three-dimensional space based on images captured by a grayscale camera and a depth camera. In this way, the spatial information of the fasteners is more complete, and the accuracy is higher than that of the judgment results based on two-dimensional images. It can also replace manual inspection and identification, which is more efficient and the data can be repeated.
[0030] The functions and implementation process of the fastener loosening identification method, system, terminal and storage medium provided in the embodiments of the present application are illustrated below in conjunction with the accompanying drawings.
[0031] like Figure 1 As shown, the method for identifying loose fasteners provided in this embodiment includes:
[0032] S101, expanding and segmenting the acquired fastener detection frame to obtain a mask of a marking line of the fastener;
[0033] S102, extracting corresponding depth values on the depth map according to the mask of the marking line of the fastener and obtaining corresponding point cloud data according to the camera intrinsic parameters;
[0034] S103: When there is disconnection and different main directions in the point cloud data, it is determined that the fastener is loose.
[0035] Before step S101, Figure 2 As shown, the detection frame of the fastener must be obtained first. Specifically:
[0036] Obtain a grayscale image and a depth image respectively captured by an aligned grayscale camera and a depth camera;
[0037] The fasteners are detected and located according to the grayscale image and the depth image to obtain a fastener detection frame.
[0038] Among them, a camera with grayscale and depth alignment is used for shooting, and the segmentation and positioning of fastener marking lines can be completed based on the grayscale image; the depth camera can obtain three-dimensional point cloud data, which is more accurate than two-dimensional image judgment.
[0039] The fastener position is detected and located based on the grayscale image to obtain the fastener detection frame. Specifically, a convolutional neural network based on deep learning, such as the target detection network such as Faster R-CNN, can be used to output the position information [x1, y1, w, h] of multiple fasteners in an image, where x1, y1 are the coordinates of the upper left corner of the fastener, and w, h are the width and height of the fastener; the detection frame of each fastener is obtained based on the position information of multiple fasteners.
[0040] In step S101, it specifically includes:
[0041] Expand the obtained fastener detection frame and crop it according to the preset area;
[0042] The cropped area is input into a deep learning-based image segmentation network to obtain a mask of the marking line of the fastener.
[0043] The fastener marking line is generally within a certain range of the fastener detection frame. Therefore, the fastener detection frame can be expanded within a certain range, and a preset area such as an area of [x1-w / 2, y1-h / 2, 2w, 2h] size can be cropped and sent to a deep learning-based image segmentation network, such as Mask R-CNN, to obtain a mask of the fastener marking line.
[0044] In step S102, the corresponding depth value is extracted from the depth map according to the mask result and the corresponding three-dimensional point cloud data is obtained according to the camera intrinsic parameters.
[0045] In step S103, connectivity analysis and PCA are performed on the point cloud data, and whether the fastener is loose is determined based on the analysis results.
[0046] When all points in the point cloud data are connected, it means that the marking line is not misplaced and the fastener is not loose.
[0047] When it is determined that there are disconnected points in the point cloud data, PCA is performed on the disconnected points; when the main directions obtained are not on a straight line, it is determined that the fastener is loose, such as Figure 4b As shown, Figure 4b The curved segments L2 and L3 in the figure are used to indicate two bolt identification lines that are not on a straight line.
[0048] When it is determined that there are disconnected points in the point cloud data, if the main directions obtained are on the same straight line, it means that the fastener marking lines are on a straight line and the fasteners are not loose. Figure 4a As shown, Figure 4a The curve segment L1 in FIG. 1 is used to illustrate the bolt identification line.
[0049] In specific implementation, after determining the disconnected points, the disconnected points are respectively passed through the PCA algorithm to find the x, y, and z main directions; if the main directions of the disconnected points are the same, it means that the fastener marking lines are on a straight line and the fastener is not loose; if the main directions of the disconnected points are not on a straight line, the fastener is loose.
[0050] Among them, the nearest neighbor Euclidean distance between each point in the point cloud data and the remaining points is obtained respectively; when the nearest neighbor Euclidean distance is greater than a threshold, it is determined that there are disconnected points.
[0051] Let's take bolts as an example to illustrate.
[0052] like Figure 2 As shown, grayscale images and depth images are collected. Images of bolts are obtained by using a camera with grayscale and depth alignment.
[0053] Convolutional neural network bolt detection model based on grayscale image. Based on the acquired grayscale image, the bolt position detection and positioning can be completed. A convolutional neural network based on deep learning, such as the target detection network such as Faster R-CNN, can be used to output the position information [x1, y1, w, h] of multiple bolts in an image, where x1, y1 are the coordinates of the upper left corner of the bolt, and w, h are the width and height of the bolt.
[0054] Bolt segmentation model based on detection result expansion. The bolt marking line is generally within a certain range of the bolt detection box, so the bolt detection box can be expanded within a certain range, and the area of [x1-w / 2, y1-h / 2, 2w, 2h] can be cropped and sent to a deep learning-based image segmentation network, such as Mask R-CNN, to obtain the mask of the bolt marking line.
[0055] Based on the segmentation results, the point cloud of the marking line is extracted from the depth map. According to the mask result of the previous step, the corresponding depth value is extracted from the depth map and the corresponding 3D point cloud data is obtained according to the camera intrinsic parameters.
[0056] Determine whether the bolt marking line is connected based on the point cloud. Check the connectivity of the point cloud. If it is connected, it means that the marking line is not misaligned and the bolt is not loose. Figure 4a As shown; if it is not connected, the disconnected point cloud is passed through the PCA algorithm to find the x, y, and z main directions. If the main directions are the same, it means that the bolt marking lines are on a straight line, such as Figure 4aThe bolts shown are not loose; if the main directions are not in a straight line, the bolt identification lines are not in a straight line, such as Figure 4b As shown, the bolts are loose.
[0057] The method for identifying loose fasteners provided in the embodiment of the present application extracts the depth value corresponding to the mask of the marking line of the fastener based on the images collected by the grayscale camera and the depth camera, obtains the corresponding point cloud data according to the camera internal parameters, performs connectivity analysis and PCA on the point cloud data, and automatically identifies the loose fasteners according to the analysis results. In the method provided in the present application, the spatial information of the fasteners is more complete, the accuracy is higher than that of the judgment results based on two-dimensional images, and it can replace manual inspection and identification, which is more efficient and the data can be repeated.
[0058] The fastener loosening identification system provided in this embodiment is a product embodiment corresponding to the aforementioned method embodiment. Its functions and implementation process are the same or similar to those of the aforementioned embodiment, and this embodiment will not be repeated here.
[0059] like Figure 3 As shown, the fastener loosening identification system provided in this embodiment includes:
[0060] A first processing module 11 is used to expand and segment the acquired fastener detection frame to obtain a mask of a marking line of the fastener;
[0061] A second processing module 12 is used to extract a corresponding depth value on the depth map according to the mask of the marking line of the fastener and obtain corresponding point cloud data according to the camera intrinsic parameters;
[0062] The third processing module 13 is used to determine that the fastener is loose when there is no connectivity and different main directions in the point cloud data.
[0063] In one possible implementation manner, the first processing module 11 is specifically configured to:
[0064] Expand the obtained fastener detection frame and crop it according to the preset area;
[0065] The cropped area is input into a deep learning-based image segmentation network to obtain a mask of the marking line of the fastener.
[0066] In one possible implementation manner, the third processing module 13 is specifically configured to:
[0067] When it is determined that there are disconnected points in the point cloud data, PCA (Principal Component Analysis) is performed on the disconnected points;
[0068] When the resulting principal directions are not in a straight line, the fastener is determined to be loose.
[0069] In one possible implementation manner, the third processing module 13 is specifically configured to:
[0070] Get the nearest neighbor Euclidean distance between each point in the point cloud data and the other points respectively;
[0071] When the nearest neighbor Euclidean distance is greater than a threshold, it is determined that there are disconnected points.
[0072] In one possible implementation manner, the first processing module 11 is further configured to:
[0073] Obtain a grayscale image and a depth image respectively captured by an aligned grayscale camera and a depth camera;
[0074] The fasteners are detected and located according to the grayscale image and the depth image to obtain a fastener detection frame.
[0075] The loose fastener identification system provided in the embodiment of the present application extracts the depth value corresponding to the mask of the marking line of the fastener based on the images collected by the grayscale camera and the depth camera, obtains the corresponding point cloud data according to the camera internal parameters, performs connectivity analysis and PCA on the point cloud data, and automatically identifies the loose fasteners according to the analysis results. In the system provided by the present application, the spatial information of the fastener is more complete, the accuracy is higher than that of the judgment result based on the two-dimensional image, and it can replace manual inspection and identification, which is more efficient and the data can be repeated.
[0076] This embodiment provides a terminal, including:
[0077] Memory;
[0078] Processor; and
[0079] Computer programs;
[0080] The computer program is stored in the memory and is configured to be executed by the processor to implement the corresponding method.
[0081] The memory is used to store computer programs. After receiving the execution instruction, the processor executes the computer program. The method executed by the device for flow process definition disclosed in the above corresponding embodiments can be applied to the processor or implemented by the processor.
[0082] The memory may include high-speed random access memory (RAM), and may also include non-volatile memory, such as at least one disk memory. The memory can realize the communication connection between the system network element and at least one other network element through at least one communication interface (which can be wired or wireless), and can use the Internet, wide area network, local area network, metropolitan area network, etc.
[0083] The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the method disclosed in Example 1 can be completed by an integrated logic circuit of hardware in the processor or instructions in the form of software. The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The corresponding methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.
[0084] The steps of the method disclosed in the embodiment of the present invention can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software units in the decoding processor. The software unit can be located in a mature storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.
[0085] This embodiment provides a computer-readable storage medium on which a computer program is stored; the computer program is executed by a processor in a corresponding method. The specific implementation thereof can be referred to in the method embodiment, which will not be described in detail here.
[0086] It should be noted that: unless otherwise specifically stated, the relative steps, numerical expressions and numerical values of the parts and steps set forth in these embodiments do not limit the scope of the present invention. In all examples shown and described herein, unless otherwise specified, any specific value should be interpreted as being merely exemplary, rather than as a restriction, and therefore, other examples of exemplary embodiments may have different values.
[0087] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a unit, a program segment or a part of the code, and a part of the unit, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0088] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0089] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0090] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0091] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0092] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.
[0093] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A method for identifying loose fasteners, characterized in that: include: Expanding and segmenting the acquired fastener detection frame to obtain a mask of the fastener marking line; Extracting corresponding depth values on the depth map according to the mask of the marking line of the fastener and obtaining corresponding point cloud data according to the camera intrinsic parameters; When there is a disconnection and a different main direction in the point cloud data, it is determined that the fastener is loose; The step of expanding and segmenting the acquired fastener detection frame to obtain a mask of a marking line of the fastener includes: Expand the obtained fastener detection frame and crop it according to the preset area; The cropped area is input into a deep learning-based image segmentation network to obtain a mask of the marking line of the fastener; When there is disconnection and the main directions are different in the point cloud data, determining that the fastener is loose includes: When it is determined that there are disconnected points in the point cloud data, a principal component analysis PCA is performed on the disconnected points; When the obtained main directions are not in a straight line, it is determined that the fastener is loose.
2. The method according to claim 1, characterized in that Determine the presence of disconnected points in the point cloud data, including: Respectively obtain the nearest neighbor Euclidean distance between each point in the point cloud data and the remaining points; When the nearest neighbor Euclidean distance is greater than a threshold, it is determined that there are disconnected points.
3. The method according to claim 1, characterized in that Before expanding and segmenting the acquired fastener detection frame, the following steps are performed: Obtain a grayscale image and a depth image respectively captured by an aligned grayscale camera and a depth camera; The fastener is detected and located according to the grayscale image and the depth image to obtain a fastener detection frame.
4. A fastener loosening identification system, characterized in that: include: A first processing module is used to expand and segment the acquired fastener detection frame to obtain a mask of a marking line of the fastener; A second processing module, configured to extract a corresponding depth value on a depth map according to a mask of a marking line of the fastener and obtain corresponding point cloud data according to a camera intrinsic parameter; A third processing module is used to determine that the fastener is loose when there is no connectivity and different main directions in the point cloud data; The first processing module is specifically used for: Expand the obtained fastener detection frame and crop it according to the preset area; The cropped area is input into a deep learning-based image segmentation network to obtain a mask of the marking line of the fastener; The third processing module is specifically used for: When it is determined that there are disconnected points in the point cloud data, a principal component analysis PCA is performed on the disconnected points; When the obtained main directions are not in a straight line, it is determined that the fastener is loose.
5. The system according to claim 4, characterized in that The third processing module is specifically used for: Respectively obtain the nearest neighbor Euclidean distance between each point in the point cloud data and the remaining points; When the nearest neighbor Euclidean distance is greater than a threshold, it is determined that there are disconnected points.
6. The system according to claim 4, characterized in that The first processing module is also used for: Obtain a grayscale image and a depth image respectively captured by an aligned grayscale camera and a depth camera; The fastener is detected and located according to the grayscale image and the depth image to obtain a fastener detection frame.
7. A terminal, characterized in that: include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 3.
8. A computer-readable storage medium, characterized in that: A computer program is stored thereon; the computer program is executed by a processor to implement the method according to any one of claims 1 to 3.
