Method, device, terminal and storage medium for identifying loss of rotatable fastening screws

Abstract

The embodiments of the present application provide a method, device, terminal, and storage medium for identifying the loss of rotatable fastening screws, which are related to quality inspection technologies. The method includes: registering the standard point cloud of the rotatable fastening screws at the brake disc or coupling determined in advance to the current point cloud, and obtaining the position information of the registered point cloud in the current point cloud coordinate system; transforming the centroid of the registered point cloud in the standard diagram to the current point cloud coordinate system according to the position information, and obtaining the coordinate information of the centroids of the point clouds of the rotatable fastening screws at the brake disc and coupling in the current point cloud coordinate system; determining the lost rotatable fastening screws according to the coordinate information. Thus, the lost rotatable fastening screws can be automatically identified. Compared with the manual inspection method, the embodiments of the present application have obvious improvements in terms of accuracy, reliability, and efficiency, which is conducive to ensuring the reliability of quality inspection and avoiding production or traffic accidents caused by the loss of rotatable fastening screws.

Description

Technical Field

[0001] The present application relates to quality inspection technology, and in particular, to a method, device, terminal and storage medium for identifying the loss of rotatable fastening screws. Background Art

[0002] In the bottom surface inspection technology of high-speed trains, the existing identification scheme for the loss of rotatable fastening screws adopts the method of manual inspection. Specifically, the inspection worker needs to irradiate the area to be inspected with a flashlight, and the inspection worker observes with the naked eye whether the rotatable fastening screws are missing and records them on a paper document to form an inspection record and a report document.

[0003] However, with the continuous increase in the number of high-speed train vehicles and the number of areas to be inspected, the labor cost of the manual inspection scheme has increased linearly, and the pressure on the workers' inspections has increased greatly. At the same time, the inspection accuracy will decrease significantly when the working hours of the workers are too long. Once bolts on important rotatable devices such as couplings are lost, it is extremely easy to cause major production or traffic accidents. Summary of the Invention

[0004] In order to solve one of the above technical defects, the embodiments of the present application provide a method, device, terminal and storage medium for identifying lost rotatable fastening screws.

[0005] The first aspect of the embodiments of the present application provides a method for identifying lost rotatable fastening screws, including:

[0006] Registering the standard point cloud of the rotatable fastening screws at the brake disc and the coupling determined in advance to the current point cloud, and obtaining the position information of the registered point cloud in the coordinate system of the current point cloud;

[0007] Transforming the centroid of the registered point cloud in the standard map to the coordinate system of the current point cloud according to the position information, and obtaining the coordinate information of the centroids of the respective point clouds of the rotatable fastening screws in the coordinate system of the current point cloud;

[0008] Determining the lost rotatable fastening screws according to the coordinate information.

[0009] The second aspect of the embodiments of the present application provides a device for identifying lost rotatable fastening screws, including:;

[0010] An acquisition module, configured to register the standard point cloud of the rotatable fastening screws at the brake disc and the coupling determined in advance to the current point cloud, and obtain the position information of the registered point cloud in the coordinate system of the current point cloud;

[0011] The first processing module is configured to transform the centroid of the registered point cloud in the standard diagram to the current map point cloud coordinate system according to the position information, so as to obtain the coordinate information of the centroid of the point cloud of each rotatable fastening screw in the current map point cloud coordinate system;

[0012] The second processing module is configured to determine the missing rotatable fastening screws according to the coordinate information.

[0013] An embodiment of the third aspect of the present application provides a terminal, including:

[0014] A memory; capable of supporting the processor to read a device with raw data, and at the same time supporting the processor to store the data processed by the method described in any one of the foregoing;

[0015] A processor; capable of reading raw data from the memory and processing the data according to the method described in any one of the foregoing to obtain the position information of the missing rotatable fastening screws;

[0016] A computer program; stored in the memory, capable of implementing the complete algorithm function of the method described in any one of the foregoing through a computer language, completing compilation, and capable of running quickly in the processor.

[0017] An embodiment of the fourth aspect of the present application provides a computer-readable storage medium, on which a computer program is stored; the computer program is executed by a processor to implement the method described in any one of the foregoing.

[0018] The embodiments of the present application provide a method, device, terminal and storage medium for identifying missing rotatable fastening screws. Through 2D picture coordinate positioning, component 3D point cloud registration, 3D to 2D mapping to locate all components, and component coordinate screening to filter out the detected rotatable fastening screws, the position of the missing rotatable fastening screws can be located, so as to automatically identify the missing rotatable fastening screws. Compared with the manual detection method, the embodiments of the present application have obvious improvements in accuracy, reliability and efficiency, which is conducive to ensuring the reliability of quality inspection and avoiding production or traffic accidents caused by the loss of rotatable fastening screws. Description of the Drawings

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments and descriptions thereof of the present application are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings:

[0020] Figure 1 It is a schematic flowchart of the method provided for an exemplary embodiment;

[0021] Figure 2 It is a schematic flowchart of the method provided for an exemplary embodiment;

[0022] Figure 3 Flow schematic diagram of the device provided for an exemplary embodiment;

[0023] Figure 4a Overall registration schematic diagram of the standard drawing and the current drawing provided for an exemplary embodiment;

[0024] Figure 4b Partial registration schematic diagram of the standard drawing and the current drawing provided for an exemplary embodiment;

[0025] Figure 4c Overall registration schematic diagram of the current rotatable fastening screw to the standard drawing provided for an exemplary embodiment;

[0026] Figure 4d Schematic diagram of the recognition result of the loss of the rotatable fastening screw at the coupling provided for an exemplary embodiment. Detailed implementation manners

[0027] In order to make the technical solutions and advantages in the embodiments of the present application clearer and more understandable, the following further describes the exemplary embodiments of the present application in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than an exhaustive list of all embodiments. It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0028] In the related art, the method of manual detection has relatively low cost in the short term, and multiple workers can perform maintenance at the same time. It can also quickly adapt to changes and enter the working state under the condition of vertex changes. However, as the number of areas to be detected continues to increase, the labor cost of the manual detection scheme rises linearly, and the pressure on workers' patrols increases greatly. At the same time, the detection accuracy of workers will decrease significantly when the working hours are too long. And once a bolt is lost on an important rotatable device such as a coupling but not detected, major production or traffic accidents may occur. In addition, the manual detection scheme is difficult to ensure that workers can detect each point, which also poses a hidden danger to production quality.

[0029] To overcome at least one of the above problems, an embodiment of the present application provides a method, device, terminal, and storage medium for identifying the loss of rotatable fastening screws. By using a deep learning method, the position information of the rotatable fastening screws in a 2D image is located, and the corresponding 3D point cloud is obtained and loaded into the standard map point cloud. After 3D point cloud registration, the coordinates of all rotatable fastening screws on the standard map within the current point cloud are obtained. After mapping to 2D and filtering out the detected rotatable fastening screws, the position of the lost rotatable fastening screw can be located, thereby automatically identifying the lost rotatable fastening screw. Compared with the manual detection method, the embodiment of the present application has obvious improvements in accuracy, reliability, and efficiency, which is conducive to ensuring the reliability of quality inspection and avoiding production or traffic accidents caused by the loss of rotatable fastening screws.

[0030] The following takes the drawings as an example to illustrate the functions and implementation processes of the method, device, terminal, and storage medium for identifying lost rotatable fastening screws provided by the embodiments of the present application.

[0031] As Figure 1 shown, the method for identifying lost rotatable fastening screws provided in this embodiment includes:

[0032] S101. Register the standard map point cloud of the rotatable fastening screws at the brake disc and coupling determined in advance to the current map point cloud, and obtain the position information of the registered point cloud in the current map point cloud coordinate system;

[0033] S102. Transform the centroid of the registered point cloud in the standard map to the current map point cloud coordinate system according to the position information, and obtain the coordinate information of the centroid of each point cloud of the rotatable fastening screws in the current map point cloud coordinate system;

[0034] S103. Determine the lost rotatable fastening screws according to the coordinate information.

[0035] It should be noted first that: this embodiment involves 3D point cloud registration technology, deep learning detection technology, and 3D point cloud stitching technology. Among them, the 3D point cloud registration technology mainly uses the Iterative-Closest-Point Method (ICP), which is mainly used for the pose matching of two similar point clouds; the deep learning detection technology is mainly used to detect the position of the rotatable fastening screws on the 2D image, and detection methods such as Faster R-CNN, Yolo v4, etc.; the 3D point cloud stitching technology is mainly used for the complete modeling of the rotatable fastening screws, and software such as Cloud Compare can be used for stitching.

[0036] In step S101, as Figure 2As shown, the position of the rotatable fastening screw needs to be determined in advance from the 2D image to obtain the 3D point cloud of the rotatable fastening screw. That is, the coordinate positioning of the 2D pictures of some components. In specific implementation, relevant pictures need to be obtained in advance, and high-definition pictures of the positions with rotatable fastening screws are taken and stored; the positions of all rotatable fastening screws are determined from the taken pictures to obtain the point cloud of the rotatable fastening screws. Among them, when determining the rotatable fastening screws from the taken pictures, it can be achieved by manual determination, or by terminal recognition, or by a combination of manual and terminal methods.

[0037] Exemplarily, the terminal can detect and locate the coordinates of some rotatable fastening screws in the 2D picture through deep learning methods. Specifically, Faster R-CNN can be used. Faster R-CNN mainly includes two modules: RPN (Region Proposal Network, a deep fully convolutional network) and Fast R-CNN detector. RPN is used to generate candidate regions; the Fast R-CNN detector is used to perform classification and bounding box regression calculations based on the candidate regions generated by the RPN network. The entire detection process shares the convolutional feature map, that is, this feature map serves as both the input of RPN and the input of Fast R-CNN, and it is a unified network for object detection. The rotatable fastening screws in the detection boxes obtained by the classification and bounding box regression calculations of the Fast R-CNN detector are the detected rotatable fastening screws, that is, the non-lost rotatable fastening screws.

[0038] After obtaining the position information, that is, the coordinates of all rotatable fastening screws in the current figure point cloud coordinate system, component 3D point cloud registration can be performed; specifically, according to the standard figure point cloud of the rotatable fastening screws determined in advance, registration is performed on the current figure (the 2D image corresponding to the current frame) point cloud, and the position information of the registered point cloud in the current figure point cloud coordinate system is obtained.

[0039] Optionally, point-to-point ICP (Iterative Closest Point, closest point matching) is used to register the standard figure point cloud to the current figure point cloud. The specific implementation process can be as follows:

[0040] Step 1: Take a point set pi∈P in the source point cloud P;

[0041] Step 2: Find the corresponding point set qi∈Q in the target point cloud Q such that ||qi - pi|| = min;

[0042] Step 3: Calculate the rotation matrix R and the translation matrix t to minimize the error function;

[0043] Step 4: Rotate and translate pi using the rotation matrix R and the translation matrix t obtained in the previous step to obtain a new corresponding point set

[0044] p'i = {p'i = Rpi + t, pi ∈ P}

[0045] Step 5: Calculate the average distance d between p'i and the corresponding point set qi: d = 1 / n ∑i=1n ||p'i - qi||2;

[0046] Step 6: If d is less than a given threshold or greater than the preset maximum number of iterations, stop the iterative calculation; otherwise, return to Step 2.

[0047] For example, the point cloud in the standard image is the source point cloud P; the point cloud in the current image is the target point cloud Q. The calculated point set p'i is the registered point cloud.

[0048] In the specific implementation, since the shooting angle is not fixed and the ICP registration result is highly dependent on the initial value, to improve the accuracy of the registration result, in this embodiment, the coordinates of all rotatable fastening screws on the standard image within the current point cloud are obtained through the methods of global-global, local-local, and local-global matching.

[0049] Optionally, Step S101 includes:

[0050] Load the point cloud of the standard image; the point cloud of the standard image includes the global point cloud and the local point cloud. The local point cloud includes the point cloud of the rotatable fastening screws, and the global point cloud includes the point cloud of the rotatable fastening screws and their associated components;

[0051] Perform downsampling on the global point cloud;

[0052] Match the global point cloud in the standard image point cloud with the global point cloud in the current image point cloud to obtain the global matching result;

[0053] Taking the global matching result as the initial value, match the local point cloud in the standard image point cloud with the local point cloud in the current image point cloud to obtain the point cloud of the rotatable fastening screws that match;

[0054] Among the obtained point cloud of the rotatable fastening screws that match, register the local point cloud in the current image with the global point cloud of the standard image to obtain the registered point cloud of the rotatable fastening screws;

[0055] Obtain the position information of the registered point cloud of the rotatable fastening screws in the coordinate system of the current image point cloud.

[0056] The loaded point cloud of the standard image can also be referred to as the complete point cloud of the standard image. The standard image contains two point clouds. One point cloud is the complete point cloud of the whole device, such as the complete point cloud including the coupling body and the bolts on the coupling; the other point cloud is the point cloud only including the rotatable fastening screws, such as the point cloud only of the bolts on the coupling. The corresponding point coordinates of the two point clouds are exactly the same.

[0057] Downsample the global point clouds of the standard image and the current image. Since the complete point cloud is too large, in order to accelerate the registration, it is necessary to downsample the complete device point clouds of the current image and the standard image, and then register the downsampled complete point clouds. For the local point cloud that only contains rotatable fastening screws, no downsampling operation is performed.

[0058] The following is an example of the matching process with reference to the attached drawings.

[0059] As Figure 4a shown, global-global matching; Figure 4a The yellow part in Figure 4a is the point cloud of the standard image, and the blue part in

[0060] is the point cloud of the current image. Since there are many features in the global point cloud, the matching results of the complete device point clouds of the current image and the standard image are relatively good. However, due to the shooting angle and some differences in device details, the registration effect of the rotatable fastening screws is not ideal. Therefore, this step can provide a good initial value for the subsequent fine matching. Figure 4b As Figure 4b shown, local-local matching; Figure 4b The yellow part in

[0061] is the point cloud of the standard image, and the blue part in Figure 4c is the point cloud of the current image. This step is used for registering the local point cloud of the rotatable fastening screws in the current frame and the local point cloud of the rotatable fastening screws in the standard image, and the initial value is the result of the global-global matching. This step can finely match the two local point clouds together. Figure 4c As Figure 4c shown, local-global matching;

[0062] In the above matching stages, ICP can be used for configuration.

[0063] In step S102, according to the position information of the final point cloud obtained in step S101, the centroid of the finally obtained point cloud is transformed into the coordinate system of the current image point cloud, and the coordinate information of the centroids of the respective point clouds of the rotatable fastening screws in the coordinate system of the current image point cloud is obtained. Realize the mapping from 3D to 2D to locate all the rotatable fastening screws in the 2D image and obtain the coordinate information of all the rotatable fastening screws. Among them, the matrix (rotation matrix R and translation matrix t) of the transformation from the standard image to the current image obtained in step S101 can be used to transform the centroids of the respective point clouds of the rotatable fastening screws in the standard image into the coordinate system of the current image point cloud.

[0064] In step S103, the coordinate information is filtered to identify the lost rotatable fastening screw. Among all the rotatable fastening screws located in step S102, the detected rotatable fastening screws are filtered out, and the lost rotatable fastening screws are obtained. Figure 4d The rotatable fastening screw corresponding to the bold dashed rectangular frame is the lost rotatable fastening screw identified by the method of this example; Figure 4d The bold solid rectangle in the figure corresponds to the screws that can be turned and tightened detected by Faster R-CNN. This step can be referred to as component coordinate screening.

[0065] Specifically, step S103 includes:

[0066] The camera internal parameters are used to map the coordinate information to the 2D plane of the current image to obtain the mapping coordinates of the rotatable fastening screw;

[0067] A rotatable fastening screw whose mapping coordinates are not located within the predetermined preselection box is determined as a lost rotatable fastening screw.

[0068] The rotatable fastening screws whose mapping coordinates are located within the predetermined pre-selected box are non-lost rotatable fastening screws.

[0069] In a specific implementation, after the lost rotatable fastening screw is identified in step S103, a distinguishing mark may be applied to the lost rotatable fastening screw. The distinguishing mark is mainly used to distinguish the lost rotatable fastening screw from the non-lost rotatable fastening screw. For example, the distinction may be made by color, size, shape, etc. In this way, it is convenient for the operator to intuitively understand the situation of the rotatable fastening screw, and it is also convenient to handle the lost rotatable fastening screw accordingly in the future.

[0070] Compared with the manual detection method in the related art, the embodiments of the present application have significantly improved the detection speed. The embodiments of the present application use a robot equipped with a camera to collect 3D depth information and 2D image information of a rotatable fastening screw, and send the collected data to a server for analysis. The time required to detect the position of a device in the related art is usually around 10 seconds, while the solution of the embodiments of the present application only takes less than 1 second, which is at least 10 times the difference in efficiency. Furthermore, the entire process in the embodiments of the present application can be fully automated and run 24 hours a day, thereby further improving the detection efficiency.

[0071] Compared with the manual detection method in the related art, the embodiment of the present application has significantly improved detection accuracy and reliability. The method of the embodiment of the present application uses depth information and 2D image information to locate the missing parts, which can avoid errors caused by human factors, thereby improving the accuracy and reliability of detecting the missing parts.

[0072] The device for identifying lost rotatable fastening screws provided in this embodiment is a product embodiment corresponding to the foregoing method embodiment. Its functions and implementation processes are the same as or similar to those of the foregoing embodiments, and will not be elaborated here.

[0073] As Figure 3 shown, this embodiment provides a device for identifying lost rotatable fastening screws, including:

[0074] An acquisition module 11, configured to register the standard point cloud of rotatable fastening screws at the brake disc and coupling according to a pre-determined standard point cloud, and obtain the position information of the registered point cloud in the current point cloud coordinate system;

[0075] A first processing module 12, configured to transform the centroid of the registered point cloud in the standard map into the current point cloud coordinate system according to the position information, and obtain the coordinate information of the centroid of each point cloud of the rotatable fastening screws in the current point cloud coordinate system;

[0076] A second processing module 13, configured to determine the lost rotatable fastening screws according to the coordinate information.

[0077] In one possible implementation, the acquisition module 11 is specifically configured to register the standard map point cloud to the current point cloud by using the nearest point matching.

[0078] In one possible implementation, the acquisition module 11 is specifically configured to:

[0079] Load the standard map point cloud; the standard map point cloud includes a global point cloud and a local point cloud. The local point cloud includes the point cloud of the rotatable fastening screws, and the global point cloud includes the point cloud of the rotatable fastening screws and their associated components;

[0080] Perform downsampling processing on the global point cloud;

[0081] Match the global point cloud in the standard map point cloud with the global point cloud in the current point cloud to obtain a global matching result;

[0082] Taking the global matching result as an initial value, match the local point cloud in the standard map point cloud with the local point cloud in the current point cloud to obtain the point cloud of the rotatable fastening screws that match;

[0083] In the obtained point cloud of the rotatable fastening screws that match, register the local point cloud in the current map with the global point cloud in the standard map to obtain the registered point cloud of the rotatable fastening screws;

[0084] Obtain the position information of the registered point cloud of the rotatable fastening screws in the current point cloud coordinate system.

[0085] In one possible implementation, the obtaining module 11 is further configured to:

[0086] Determine the position information of the rotatable fastening screw from the current 2D image to obtain the point cloud of the rotatable fastening screw.

[0087] In one possible implementation, the second processing module 13 is specifically configured to:

[0088] Map the coordinate information to the 2D plane of the current image by using the camera internal parameters to obtain the mapped coordinates of the rotatable fastening screw;

[0089] Determine the rotatable fastening screws whose mapped coordinates are not within the pre-determined preselection box as the lost rotatable fastening screws.

[0090] In one possible implementation, the obtaining module 11 is further configured to:

[0091] Determine candidate regions from the captured images based on a deep fully convolutional network;

[0092] Classify and perform bounding box regression processing on the candidate regions through a Faster R-CNN detector to obtain a preselection box; wherein, the input of the deep fully convolutional network and the input of the Faster R-CNN detector use the same convolutional feature map.

[0093] Compared with the manual detection method in the related art, the embodiment of the present application has a significant improvement in the detection speed. The embodiment of the present application uses the method of installing a camera on a robot to collect the 3D depth information and 2D image information of the rotatable fastening screw, and sends the collected data to the server for analysis. In the related art, the time required to detect the position of a device is usually about 10 seconds, while the solution of the embodiment of the present application only takes less than 1 second, and the efficiency difference is at least 10 times. Further, the entire process in the embodiment of the present application can be fully automated and run 24 hours a day, thereby further improving the detection efficiency.

[0094] Compared with the manual detection method in the related art, the embodiment of the present application has a significant improvement in the detection accuracy and reliability. The method of the embodiment of the present application uses the depth information and 2D image information to locate the lost components, which can avoid errors caused by human factors, thereby improving the accuracy and reliability of detecting the lost components.

[0095] This embodiment provides a terminal, including:

[0096] A memory; capable of supporting the processor to read the device with the original data, and at the same time supporting the processor to store the data processed through the above steps.

[0097] A processor; capable of reading raw data from a memory and processing the data according to the above steps to obtain the position information of the lost rotatable fastening screw.

[0098] A computer program; capable of implementing the complete algorithm function of the above steps through a computer language, completing compilation, and being able to run quickly in a processor.

[0099] Wherein, the computer program is stored in a memory and is configured to be executed by the processor to implement the corresponding method.

[0100] The memory is used to store the computer program. After receiving an execution instruction, the processor executes the computer program. The method executed by the device defined by the flow process disclosed in the foregoing corresponding embodiment can be applied to the processor or implemented by the processor.

[0101] The memory may include a high-speed random access memory (RAM: Random Access Memory), and may also include a non-volatile memory, such as at least one disk memory. The memory can implement a communication connection between the system network element and at least one other network element through at least one communication interface (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used.

[0102] The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the method disclosed in Embodiment 1 can be completed by the integrated logic circuit in the hardware of the processor or by instructions in software form. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU for short), a network processor (NP for short), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the corresponding methods, steps and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0103] The steps of the method disclosed in combination with the embodiments of the present invention can be directly embodied as being completed by a hardware decoding processor, or by a combination of hardware and software units in the decoding processor. The software unit may be located in a mature storage medium in the art such as a random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.

[0104] This embodiment provides a computer-readable storage medium, on which a computer program is stored; the computer program is executed by a processor to implement a corresponding method. For its specific implementation, reference may be made to the method embodiment, which will not be elaborated here.

[0105] It should be noted that: Unless otherwise specifically stated, the relative steps, numerical expressions, and values of the components and steps set forth in these embodiments do not limit the scope of the present invention. In all the examples shown and described here, unless otherwise specified, any specific value should be construed as merely exemplary and not as a limitation. Therefore, other examples of the exemplary embodiments may have different values.

[0106] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to multiple embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a unit, a program segment, or a part of code, and the unit, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0107] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0108] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of the flows and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the processes Figure 1means for the functions specified in one or more processes and / or blocks Figure 1 means for the functions specified in one block or more blocks.

[0109] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction means that implements the functions in the process Figure 1 one or more processes and / or blocks Figure 1 the functions specified in one block or more blocks.

[0110] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus, such that a series of operational steps are performed on the computer or other programmable apparatus to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions in the process Figure 1 one or more processes and / or blocks Figure 1 the functions specified in one block or more blocks.

[0111] Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications once they learn the basic creative concepts. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications falling within the scope of the present application.

[0112] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these modifications and variations.

Claims

1. A method for identifying the loss of a rotatable fastening screw, characterized in that, Including: Register the standard point cloud of the rotatable fastening screws at the brake disc and coupling according to the pre-determined standard point cloud of the rotatable fastening screws, and obtain the position information of the registered point cloud in the coordinate system of the current point cloud; Transform the centroid of the registered point cloud in the standard drawing into the coordinate system of the current point cloud according to the position information, and obtain the coordinate information of the centroids of the respective point clouds of the rotatable fastening screws in the coordinate system of the current point cloud; Determine the missing rotatable fastening screws according to the coordinate information; Determining the missing rotatable fastening screws according to the coordinate information includes: Mapping the coordinate information to the 2D plane of the current drawing by using the camera internal parameters to obtain the mapping coordinates of the rotatable fastening screws; Determine the rotatable fastening screws whose mapping coordinates are not located within the pre-determined pre-selection box as the missing rotatable fastening screws; Among them, the position information of the rotatable fastening screws in the 2D current drawing is located by a deep learning method to obtain a pre-determined pre-selection box.

2. The method according to claim 1, wherein Registering the standard point cloud to the current point cloud includes: Register the standard point cloud to the current point cloud by using the nearest point matching.

3. The method according to claim 1, characterized in that Register the standard point cloud of the rotatable fastening screws at the brake disc and coupling according to the pre-determined standard point cloud of the rotatable fastening screws, and obtain the position information of the registered point cloud in the coordinate system of the current point cloud, including: Load the standard point cloud; the standard point cloud includes a global point cloud and a local point cloud, the local point cloud includes the point cloud of the rotatable fastening screws, and the global point cloud includes the point cloud of the rotatable fastening screws and their associated components; Perform downsampling processing on the global point cloud; Match the global point cloud in the standard point cloud with the global point cloud in the current point cloud to obtain a global matching result; Taking the global matching result as an initial value, match the local point cloud in the standard point cloud with the local point cloud in the current point cloud to obtain the point cloud of the rotatable fastening screws that match; Among the obtained point clouds of the rotatable fastening screws that match, register the local point cloud in the current drawing with the global point cloud of the standard drawing to obtain the point cloud of the registered rotatable fastening screws; Obtain the position information of the point cloud of the registered rotatable fastening screws in the coordinate system of the current point cloud.

4. The method according to claim 1, wherein Before registering the standard point cloud to the current point cloud, it includes: Determine the position information of the rotatable fastening screws from the 2D current drawing to obtain the point cloud of the rotatable fastening screws.

5. The method according to claim 1, wherein Before determining the rotatable fastening screws whose mapping coordinates are not located within the pre-determined pre-selection box as the missing rotatable fastening screws, it also includes: Determine candidate regions from the captured images based on a deep fully convolutional network; Perform classification and bounding box regression processing on the candidate regions through a Faster R-CNN detector to obtain a pre-selection box; among them, the input of the deep fully convolutional network and the input of the Faster R-CNN detector use the same convolutional feature map.

6. A device for identifying the loss of a rotatable fastening screw, characterized in that, Including: An acquisition module for registering the standard point cloud of the rotatable fastening screws at the brake disc and coupling according to the pre-determined standard point cloud of the rotatable fastening screws, and obtaining the position information of the registered point cloud in the coordinate system of the current point cloud; A first processing module is used to transform the centroid of the registered point cloud in the standard image into the point cloud coordinate system of the current image according to the position information, and obtain the coordinate information of the centroid of each point cloud of the rotatable fastening screw in the point cloud coordinate system of the current image; A second processing module, configured to determine the lost rotatable fastening screw according to the coordinate information; The second processing module is specifically used to: map the coordinate information to the 2D plane of the current image using the camera intrinsic parameter to obtain the mapping coordinates of the rotatable fastening screw; determining a rotatable fastening screw whose mapping coordinates are not within a predetermined preselection box as a lost rotatable fastening screw; Among them, the position information of the rotatable fastening screw in the current 2D image is located by a deep learning method to obtain a predetermined pre-selected box.

7. The device according to claim 6, characterized in that, The acquisition module is specifically used to align the standard image point cloud with the current image point cloud by using the nearest point matching.

8. The device according to claim 6, characterized in that, The acquisition module is specifically used for: Loading a standard image point cloud; the standard image point cloud includes a global point cloud and a local point cloud, the local point cloud includes a point cloud of the rotatable fastening screw, and the global point cloud includes a point cloud of the rotatable fastening screw and its associated components; Performing downsampling processing on the global point cloud; Matching the global point cloud in the standard image point cloud with the global point cloud in the current image point cloud to obtain a global matching result; Using the global matching result as an initial value, matching the local point cloud in the standard image point cloud with the local point cloud in the current image point cloud to obtain a matching point cloud of the rotatable fastening screw; In the point cloud of the matched rotatable fastening screw, the local point cloud of the current image is registered with the global point cloud of the standard image to obtain the registered point cloud of the rotatable fastening screw; The position information of the registered point cloud of the rotatable fastening screw in the current image point cloud coordinate system is obtained.

9. The device according to claim 6, characterized in that, The acquisition module is also used for: The position information of the rotatable fastening screw is determined from the current 2D image to obtain a point cloud of the rotatable fastening screw.

10. The device according to claim 6, characterized in that, The acquisition module is also used for: Determine candidate regions from captured images based on a deep fully convolutional network; The candidate area is classified and bounding box regressed by the Faster R-CNN detector to obtain a pre-selected box; wherein the input of the deep full convolutional network and the input of the Faster R-CNN detector use the same convolutional feature map.

11. A terminal, characterized in that, include: Memory; A device capable of supporting a processor to read raw data and supporting the processor to store data processed by the method described in any one of claims 1 to 5; Processor; capable of reading raw data from the memory and processing the data according to the method described in any one of claims 1 to 5 to obtain the position information of the lost rotatable fastening screw; Computer program; stored in the memory, capable of implementing the complete algorithm function of the method described in any one of claims 1 to 5 through a computer language, completing compilation, and being able to run quickly in a processor.

12. A computer-readable storage medium, characterized in that, A computer program is stored thereon; the computer program is executed by a processor to implement the method according to any one of claims 1 to 5.

Images