Device inspection method based on feature point matching, processor and device

Abstract

The application relates to the field of industrial inspection, in particular to a device inspection method based on feature point matching, a processor and an apparatus. The method comprises the following steps: acquiring an inspection picture in an inspection process; selecting pictures similar to the inspection picture from a reference frame set to form a candidate reference frame set; determining a map point set corresponding to the candidate reference frame set; determining an inspection feature point of the inspection picture, and matching the inspection feature point with the map point set to obtain a matching point pair, so as to determine a target map point in the map point set matched with the inspection feature point; in the case that the proportion of the target map point matched in the map point set reaches a first proportion value, determining that a candidate reference frame corresponding to the map point set is a target reference frame; and determining a target spare part corresponding to the target reference frame as a target spare part included in the inspection picture. Through the above scheme, the spare part of the inspection is determined through the inspection picture, so that the integrity of the inspection is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of industrial inspection, in particular, to a device inspection method based on feature point matching, a processor and an apparatus. BACKGROUND

[0002] In the current industrial environment, mechanical equipment needs to be inspected and maintained according to a fixed cycle to ensure normal operation of the equipment and reduce wear and tear. Since the equipment inspection steps are complex, it is difficult for service engineers to accurately grasp the real service quality on site during on-site inspection. At present, the service engineer self-evaluation method is mainly used to evaluate the completion degree of inspection.

[0003] During the inspection process of the service engineer, it is difficult to accurately grasp the real inspection behavior and service quality. At present, the engineer self-evaluation method is used to determine whether the inspection is completed. However, this method completely depends on the self-consciousness of the engineer, cannot locate the inspection behavior of the engineer, lacks objective standards, and is difficult to accurately grasp the on-site environment of the inspection. SUMMARY

[0004] The purpose of the present application is to provide a device inspection method based on feature point matching which can determine whether the equipment to be inspected is included in the inspection picture by using the industrial inspection picture, so as to determine whether the inspection of the equipment is completed.

[0005] In order to achieve the above-mentioned purpose, the present application provides a device inspection method based on feature point matching, which comprises:

[0006] acquiring an inspection picture in an inspection process;

[0007] selecting pictures similar to the inspection picture from a reference frame set to form a candidate reference frame set;

[0008] determining a set of map points corresponding to the candidate reference frame set;

[0009] determining inspection feature points of the inspection picture, and matching the inspection feature points with the set of map points to obtain a matched point pair, so as to determine a target map point in the set of map points matched with the inspection feature points;

[0010] in a case where the proportion of the target map point matched in the set of map points reaches a first proportion value, determining that the candidate reference frame corresponding to the set of map points is a target reference frame;

[0011] determining a target component corresponding to the target reference frame as a target component included in the inspection picture.

[0012] In an embodiment of the present application, selecting the pictures similar to the inspection picture from the reference frame set to form the candidate reference frame set comprises: inputting the inspection picture into the visual bag-of-words model, so that the visual bag-of-words model selects the candidate reference frame set similar to the inspection picture from the reference frame set according to the inspection picture, wherein the candidate reference frame set comprises a plurality of candidate reference frames.

[0013] In an embodiment of the present application, determining the inspection feature points of the inspection picture and matching the inspection feature points with the map point set to obtain the matching point pair to determine the target map point in the map point set matched with the inspection feature points comprises: determining the inspection feature points of the inspection picture, determining the first reference feature points of each candidate reference frame; matching the inspection feature points with the first reference feature points of each candidate reference frame to determine the first matching feature point pair of the inspection picture and each candidate reference frame; determining the pose of the inspection picture according to the first matching feature point pair; re-projecting the map points corresponding to each candidate reference frame to the inspection picture according to the pose to determine the number of inliers of the inspection picture; and determining the map point corresponding to the candidate reference frame as the target map point matched with the inspection feature points in the case that the proportion of the number of inliers in the map points of the candidate frame reaches the second proportion value.

[0014] In an embodiment of the present application, matching the inspection feature points with the first reference feature points of each candidate reference frame to determine the first matching feature point pair of the inspection picture and each candidate reference frame comprises: inputting the inspection feature points and the first reference feature points into the leaf node of the visual bag-of-words model; the visual bag-of-words model sequentially compares the inspection feature points and the first reference feature points in the same leaf node to determine the first feature distance between the inspection feature points and the first reference feature points; and determining the feature point pair with the smallest first feature distance as the first matching feature point pair.

[0015] In an embodiment of the present application, re-projecting the map points corresponding to each candidate frame to the inspection picture according to the pose to determine the number of inliers of the inspection picture comprises: re-projecting the map points corresponding to each candidate reference frame to the inspection picture according to the pose to obtain the projection points; determining the pixel points corresponding to the map points to obtain the error distance between the projection points and the pixel points corresponding to the map points; adjusting the pose to adjust the error distance; determining the final pose in the case that the error distance is less than the preset threshold; and re-projecting the map points corresponding to each candidate reference frame to the inspection picture according to the final pose to determine the number of inliers of the inspection picture.

[0016] In an embodiment of the present application, before obtaining the inspection picture in the inspection process, further comprising: collecting the environment picture in which the inspection process is located; extracting the feature points of the environment picture, and generating the bag-of-words vector corresponding to the environment through the visual bag-of-words model, and characterizing the feature points of the picture through the bag-of-words vector.

[0017] In an embodiment of the present application, before acquiring the inspection pictures in the inspection process, the method further comprises: collecting part pictures of each part to be identified; and determining matched feature point pairs in feature points of the part pictures to determine poses and map points corresponding to the part pictures; adjusting the poses corresponding to the part pictures to determine optimal map points of the part pictures; and determining a reference frame of the part and a map point set of the part according to the feature points of the part pictures and the optimal map points of the part pictures.

[0018] In an embodiment of the present application, the collecting of the part pictures of each part to be identified and the determining of matched feature point pairs in feature points of the part pictures to determine poses and map points corresponding to the part pictures comprise: performing feature extraction on the part pictures to obtain feature points of the part pictures; and matching the feature points of adjacent part pictures to determine second matched feature point pairs.

[0019] In an embodiment of the present application, the matching of the feature points of adjacent part pictures to determine second matched feature point pairs comprises: determining second feature distances between any two feature points belonging to different part pictures in two adjacent part pictures; and determining a feature point pair composed of two feature points with the closest second feature distances as the second matched feature point pairs.

[0020] In an embodiment of the present application, the determining of the poses and map points corresponding to the part pictures comprises: determining a rotation matrix and a translation matrix of each part picture according to the second matched feature point pairs; and determining the poses and map points corresponding to each part picture according to the rotation matrix and the translation matrix.

[0021] In an embodiment of the present application, the adjusting of the poses corresponding to the part pictures to determine optimal map points of the part pictures comprises: re-projecting the map points into the part pictures to obtain projection points; determining pixel points corresponding to the map points to obtain error distances between the projection points and the pixel points corresponding to the map points; adjusting the poses to adjust the error distances; and determining the pose as an optimal pose and the map point as an optimal map point when it is determined that the adjusted error distance is less than a preset threshold.

[0022] The second aspect of the present application provides a processor configured to execute the feature point matching-based equipment inspection method of any one of the above.

[0023] The third aspect of the present application provides a feature point matching-based equipment inspection device, comprising the processor described above.

[0024] In the technical solution, the feature points of the inspection picture are collected and matched, the reference frame matched with the inspection picture is selected from the prepared reference frame set, and the target part included in the inspection picture is determined according to the matched reference frame, so that the inspected part is determined through the inspection picture, and the integrity of the inspection is ensured.

[0025] Other features and advantages of the embodiments of the present application will be described in detail in the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0026] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of the specification, and are used to explain the present application together with the following detailed description, but do not constitute a limitation of the present application. In the drawings:

[0027] Figure 1 The flowchart of the device inspection method based on feature point matching according to an embodiment of the present application is schematically shown;

[0028] Figure 2 The internal structure diagram of the computer device according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0029] The specific embodiments of the present application are described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application.

[0030] It should be noted that if the present application embodiments involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, motion condition, etc. between the components in a certain posture (as shown in the drawings), and if the certain posture changes, the directional indications also change accordingly.

[0031] In addition, if the present application embodiments involve descriptions of "first", "second", etc., the descriptions of "first", "second", etc. are only for description purposes, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first" and "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions of the various embodiments can be combined with each other, but it must be based on the realization of the ordinary skilled in the art, and when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, and is not within the protection scope claimed by the present application.

[0032] As Figure 1A flowchart of a device inspection method based on feature point matching according to an embodiment of the present application is shown schematically. As shown in Figure 1 In an embodiment of the present application, a device inspection method based on feature point matching is provided, comprising the following steps:

[0033] Step 101, obtaining an inspection picture in an inspection process.

[0034] Step 102, selecting pictures similar to the inspection picture from a reference frame set to form a candidate reference frame set.

[0035] The processor can first obtain an inspection picture in an industrial inspection process. The industrial inspection can refer to a patrol inspection of an industrial site environment, for example, to check whether a component has a problem, such as whether an engine of a construction machine is in a normal working state. The inspection picture can be obtained by an engineer wearing a safety helmet equipped with a camera to shoot the industrial site. For example, the engineer can perform an inspection through the safety helmet equipped with the camera to obtain an industrial inspection video, upload the industrial inspection video to the processor, and process the industrial inspection video through the processor to obtain the inspection picture.

[0036] After obtaining the inspection picture, the processor can compare the inspection picture with a reference frame set stored in the processor, so as to select pictures similar to the inspection picture from the reference frame set, and form a candidate reference frame set from the selected similar pictures.

[0037] In an embodiment, selecting pictures similar to the inspection picture from the reference frame set to form a candidate reference frame set comprises: inputting the inspection picture into a visual bag-of-words model, so that the visual bag-of-words model selects a candidate reference frame set similar to the inspection picture from the reference frame set according to the inspection picture, wherein the candidate reference frame set includes multiple candidate reference frames.

[0038] The visual bag-of-words model can determine the frequency of occurrence of each visual word of a picture in the reference frame set according to the feature points of the picture, and form a visual word histogram of the picture to represent the picture. Similarly, after obtaining the inspection picture of the industrial inspection, the processor can input the inspection picture into the visual bag-of-words model, form a visual word histogram of the inspection picture to represent the current inspection picture, and calculate the distance between the picture in the reference frame set and the inspection picture using the visual word histograms of the two, wherein the distance can be L1 norm, L2 norm, etc. The candidate reference frame set is determined according to a certain rule using the calculated distance between the pictures, wherein the candidate reference frame set can include multiple candidate reference frames.

[0039] Step 103, determining a set of map points corresponding to the candidate reference frame set.

[0040] In step 104, the inspection feature points of the inspection picture are determined, and the inspection feature points are matched with the set of map points to obtain a matching point pair, so as to determine the target map points in the set of map points that match the inspection feature points.

[0041] After the processor selects the set of candidate reference frames from the set of reference frames, the processor can determine the set of map points corresponding to the set of candidate reference frames. The processor can determine the map points of each candidate reference frame. The processor can determine the inspection feature points of the inspection picture, and match the determined inspection feature points of the inspection picture with the set of map points corresponding to the set of candidate reference frames, so as to obtain a matching point pair. According to the obtained matching point pair, the target map points in the set of map points corresponding to the candidate reference frames that match the inspection feature points are determined.

[0042] In an embodiment, the determination of the inspection feature points of the inspection picture, and the matching of the inspection feature points with the set of map points to obtain a matching point pair, so as to determine the target map points in the set of map points that match the inspection feature points, includes: determining the inspection feature points of the inspection picture, determining the first reference feature points of each candidate reference frame; matching the inspection feature points with the first reference feature points of each candidate reference frame, so as to determine the first matching feature point pair between the inspection picture and each candidate reference frame; determining the pose of the inspection picture according to the first matching feature point pair; re-projecting the map points corresponding to each candidate reference frame to the inspection picture according to the pose, so as to determine the number of inliers in the inspection picture; in a case where the proportion of the number of inliers in the map points of the candidate frame reaches a second proportion value, determining that the map points corresponding to the candidate reference frame are the target map points that match the inspection feature points.

[0043] The processor can determine the inspection feature points of the inspection picture, and determine the first reference feature points of each candidate reference frame. After the processor obtains the feature points of the pictures that need to be compared, the processor can match the inspection feature points of the inspection picture with the first reference feature points of each candidate reference frame, so as to determine the first matching point pair between the inspection picture and each candidate reference frame. The processor can determine the pose of the inspection picture according to the first matching point pair between each candidate reference frame and the inspection picture. The processor can re-project the map points corresponding to each candidate reference frame to the inspection picture according to the obtained pose of the inspection picture, so as to determine the number of inliers in the inspection picture corresponding to each candidate reference frame, and determine the proportion of the number of inliers. In a case where the map points of the candidate reference frame are re-projected to the inspection picture, and the proportion of the obtained number of inliers reaches a second proportion value, it is determined that the map points corresponding to the candidate reference frame that are re-projected are the target map points that match the inspection feature points of the inspection picture.

[0044] In one embodiment, matching the inspection feature points with the first reference feature points of each candidate reference frame to determine the first matching feature point pair between the inspection picture and each candidate reference frame includes: inputting the inspection feature points and the first reference feature points into a leaf node of a visual bag-of-words model; the visual bag-of-words model sequentially compares the inspection feature points and the first reference feature points in the same leaf node to determine a first feature distance between the inspection feature points and the first reference feature points; and determining the feature point pair with the smallest first feature distance as the first matching feature point pair.

[0045] The processor can input the inspection feature points of the inspection picture and the first reference feature points of each candidate reference frame into a leaf node of a visual bag-of-words model, and after receiving the inspection feature points and the first reference feature points, the visual bag-of-words model compares the inspection feature points of the inspection picture and the first reference feature points of the candidate reference frame in the same leaf node to determine a first feature distance between the inspection feature points and the first reference feature points. The first feature distance can be a Hamming distance between the inspection feature points and the first reference feature points, and the processor can determine the inspection feature points and the first reference feature points with the smallest first feature distance, i.e., the Hamming distance, as the matching first matching feature point pair.

[0046] In one embodiment, re-projecting the map points corresponding to each candidate frame to the inspection picture according to the pose to determine the number of inliers of the inspection picture includes: re-projecting the map points corresponding to each candidate reference frame to the inspection picture according to the pose to obtain projected points; determining the pixel points corresponding to the map points to obtain error distances between the projected points and the pixel points corresponding to the map points; adjusting the pose; determining the pose as a final pose when the error distances are less than a preset threshold; and re-projecting the map points corresponding to each candidate frame to the inspection picture according to the final pose to determine the number of inliers of the inspection picture.

[0047] The processor can determine the pose of the inspection picture according to the determined first matching feature point pair between the inspection feature points and the first reference feature points. According to the determined pose of the inspection picture, the processor can re-project the map points corresponding to each candidate reference frame to the inspection picture to obtain projected points of the map points on the inspection picture, and simultaneously determine the pixel points corresponding to the candidate reference frames. Comparing the projected points and the pixel points to determine error distances between the projected points and the pixel points. The processor can adjust the pose to adjust the error distances. When the error distances between the projected points and the pixel points are less than a preset threshold set by the processor, the processor can determine the pose at this time as a final pose corresponding to each candidate reference frame. The projected points with error distances less than the preset threshold set by the processor are determined as inliers. The map points corresponding to each candidate reference frame are re-projected to the inspection picture again according to the determined final pose to determine the number of inliers of the inspection picture.

[0048] In one embodiment, a patrol feature point of a patrol picture is determined, and first reference feature points of each candidate reference frame are determined; the patrol feature point and the first reference feature points are input into leaf nodes of a visual bag-of-words model; the visual bag-of-words model sequentially compares the patrol feature point and the first reference feature points in the same leaf node to determine a first feature distance between the patrol feature point and the first reference feature points; a feature point pair with the smallest first feature distance is determined as a first matching feature point pair; a pose of the patrol picture is determined according to the first matching feature point pair; a map point corresponding to each candidate reference frame is re-projected onto the patrol picture according to the pose to obtain a projection point; a pixel point corresponding to the map point is determined to obtain an error distance between the projection point and the pixel point corresponding to the map point; the pose is adjusted to adjust the error distance; in a case where the error distance is smaller than a preset threshold, the pose is determined as a final pose; the map point corresponding to each candidate reference frame is re-projected onto the patrol picture according to the final pose to determine an inlier number of the patrol picture; in a case where a proportion of the inlier number in the determined map points corresponding to the candidate frame reaches a second proportion value, the map point corresponding to the candidate reference frame is determined as a target map point matched with the patrol feature point.

[0049] The processor can determine the inspection feature points of the inspection picture and determine the first reference feature points of each candidate reference frame. After the processor obtains the feature points of the picture that needs to be compared, the processor can match the inspection feature points of the inspection picture with the first reference feature points of each candidate reference frame. The processor can input the inspection feature points of the inspection picture and the first reference feature points of each candidate reference frame into the leaf nodes of the visual bag-of-words model. After the visual bag-of-words model receives the inspection feature points and the first reference feature points, the visual bag-of-words model compares the inspection feature points of the inspection picture and the first reference feature points of the candidate reference frame in the same leaf node, so as to determine the first feature distance between the inspection feature points and the first reference feature points. The first feature distance can be the Hamming distance between the inspection feature points and the first reference feature points. The processor can determine the inspection feature points and the first reference feature points with the smallest first feature distance, that is, the Hamming distance, as the matched first matching feature point pair. The processor can determine the pose of the inspection picture according to the first matching point pair between each candidate reference frame and the inspection picture. After the pose of the inspection picture is determined, the processor can re-project the map points corresponding to each candidate reference frame to the inspection picture according to the determined pose of the inspection picture, so as to obtain the projection points of the map points on the inspection picture and determine the pixel points corresponding to the candidate reference frame. The projection points and the pixel points are compared to determine the error distance between the projection points and the pixel points. The processor can adjust the pose, so as to adjust the error distance. When the error distance between the projection points and the pixel points is less than the preset threshold value set by the processor, the processor can determine the pose at this time as the final pose corresponding to each candidate reference frame. The map points corresponding to each candidate reference frame are re-projected to the inspection picture again according to the determined final pose, so as to determine the number of inliers of the inspection picture and determine the proportion of the number of inliers. When it is determined that the map points of the candidate reference frame are re-projected to the inspection picture, and the proportion of the number of inliers obtained reaches a second proportion value, the map points corresponding to the candidate reference frame are determined as the target map points matched with the inspection feature points of the inspection picture.

[0050] Step 105, in a case where the proportion of the matched target map points in the map point set reaches a first proportion value, the candidate reference frame corresponding to the map point set is determined as a target reference frame.

[0051] Step 106, the target component corresponding to the target reference frame is determined as the target component included in the inspection picture.

[0052] After the processor determines the target map points matched with the inspection feature points of the inspection picture, the processor can determine the proportion of the target map points in the map point set. When the proportion of the target map points in the map point set reaches a first proportion value set by the processor, the processor can determine that the candidate reference frame corresponding to the map point set having the first proportion value of target map points is the target reference frame. The processor can determine the target component corresponding to the target reference frame according to the determined target reference frame, and determine the component corresponding to the target reference frame as the target component included in the inspection picture.

[0053] In an embodiment, before obtaining the inspection picture in the inspection process, the method further includes: collecting an environment picture in which the inspection process is located; extracting feature points of the environment picture, and generating a bag-of-words vector corresponding to the environment through a bag-of-words model, and characterizing the feature points of the picture through the bag-of-words vector.

[0054] Before the inspection process, the processor also needs to complete the preparation work. The processor can collect an environment picture in which the inspection process is located, extract feature points of the environment picture, and generate a bag-of-words vector corresponding to the environment through a bag-of-words model. The cluster center is determined through the determined bag-of-words vector, and each picture is characterized by the feature through the cluster center to determine the feature points of the picture.

[0055] In an embodiment, before obtaining the inspection picture in the inspection process, the method further includes: collecting an environment picture in which the inspection process is located; extracting feature points of the environment picture, and generating a bag-of-words vector corresponding to the environment through a bag-of-words model, and characterizing the feature points of the picture through the bag-of-words vector.

[0056] The processor can collect pictures of each component to be inspected in the industrial inspection, and determine the feature points of each component picture obtained, and match the feature point pairs to determine the matched feature point pairs, so as to determine the pose and map points corresponding to the component picture according to the matched feature point pairs. Adjust the pose corresponding to the component picture, so as to determine the optimal map points of the component picture. According to the feature points of the component picture obtained by the processor and the optimal map points of the component determined, the reference frame of the component and the map point set of the component are determined.

[0057] In an embodiment, collecting component pictures of each component to be identified and determining matched feature point pairs in the feature points of the component pictures to determine the pose and map points corresponding to the component pictures includes: performing feature extraction on the component pictures to obtain feature points of the component pictures; for any two adjacent component pictures, matching the feature points of the adjacent component pictures to determine second matched feature point pairs.

[0058] After the processor collects the part pictures of each part to be identified, the processor needs to determine the feature points of the part pictures and the matched feature point pairs, and determine the pose and the map point corresponding to the part pictures according to the matched feature point pairs. The processor can first perform feature extraction on the part pictures to determine the feature points of each part picture. For any two adjacent part pictures of the same part, the feature points of the adjacent part pictures are matched to determine the second matched feature point pairs.

[0059] In an embodiment, matching the feature points of the adjacent part pictures to determine the second matched feature point pairs includes: determining the second feature distances between any two feature points respectively belonging to different part pictures in the two adjacent part pictures; and determining the feature point pairs composed of the two feature points with the closest second feature distances as the second matched feature point pairs.

[0060] After the processor extracts the feature points of the part pictures, the processor determines the second feature distances between the feature points in the two adjacent part pictures of the same part. For example, the processor determines two adjacent part pictures A1 and A2 of the part A, determines the feature points on the part picture A1 and the part picture A2, matches the feature points on the part picture A1 with the feature points on the part picture A2, and determines the second feature distances between any two feature points, wherein the two feature points are respectively from the part picture A1 and the part picture A2. After determining the second feature distances between any two feature points respectively belonging to different part pictures, the processor can select the two feature points with the closest feature distances, and determine the feature point pairs composed of the two feature points with the closest second feature distances as the second matched feature point pairs.

[0061] In an embodiment, determining the pose and the map point corresponding to the part pictures includes: determining the rotation matrix and the translation matrix of each part picture according to the second matched feature point pairs; and determining the pose and the map point corresponding to each part picture according to the rotation matrix and the translation matrix.

[0062] After the processor determines the second feature distances of the two adjacent part pictures and the second matched feature points, the processor can determine the rotation matrix and the translation matrix of each part picture according to the second matched feature points. The processor determines the pose and the map point corresponding to each part picture according to the rotation matrix and the translation matrix.

[0063] In an embodiment, the adjusting the pose corresponding to the part picture to determine the optimal map point of the part picture comprises: re-projecting the map point into the part picture to obtain a projection point; determining a pixel point corresponding to the map point to obtain an error distance between the projection point and the pixel point corresponding to the map point; adjusting the pose to adjust the error distance; and determining the pose as the optimal pose and determining the map point as the optimal map point when the adjusted error distance is less than a preset threshold.

[0064] After the processor determines the pose corresponding to each part picture and the map point according to the second matching feature point pair, the processor re-projects the map point into the part picture according to the pose to obtain a projection point, determines a pixel point corresponding to the map point, compares the pixel point with the projection point, determines an error distance between the projection point and the pixel point, and adjusts the pose to adjust the error distance between the projection point and the pixel point. When the error distance is adjusted to be less than a preset threshold set by the processor, the processor can determine the adjusted pose as the optimal pose and the adjusted map point as the optimal map point.

[0065] In an embodiment, before the inspection pictures in the inspection process are obtained, the method further comprises: collecting an environment picture of an environment in which the inspection process is located; extracting feature points of the environment picture, and generating a bag-of-words vector corresponding to the environment through a bag-of-words model, and characterizing the feature points of the picture through the bag-of-words vector. Collecting part pictures of each part to be recognized; performing feature extraction on the part pictures to obtain feature points of the part pictures; for any two adjacent part pictures, determining a second feature distance between any two feature points belonging to different part pictures in the two adjacent part pictures; determining a feature point pair composed of two feature points with the closest second feature distance as a second matching feature point pair; determining a rotation matrix and a translation matrix of each part picture according to the second matching feature point pair; determining a pose corresponding to each part picture and a map point according to the rotation matrix and the translation matrix; re-projecting the map point into the part picture to obtain a projection point; determining a pixel point corresponding to the map point to obtain an error distance between the projection point and the pixel point corresponding to the map point; adjusting the pose to adjust the error distance; and determining the pose as the optimal pose and determining the map point as the optimal map point when the adjusted error distance is less than a preset threshold. Determining a reference frame of the part and a map point set of the part according to the feature points of the part picture and the optimal map point of the part picture.

[0066] Before the inspection process is performed, the processor also needs to complete the preparation work. The processor can collect the environment pictures of the inspection process, extract the feature points of the environment pictures, and generate the bag-of-words vector corresponding to the environment through the bag-of-words model of the feature points of the environment pictures. The cluster center is determined through the determined bag-of-words vector, and each picture is represented by the feature vector through the cluster center. The processor can collect pictures of each part to be inspected in the industrial inspection, and determine the feature points of each part picture obtained. For any two adjacent part pictures of the same part, the feature points of the adjacent part pictures are matched, and the processor can determine the second feature distance between the feature points of the two adjacent pictures, and determine the second matching feature point pair composed of the two feature points with the closest second feature distance. After the processor determines the second feature distance of the two adjacent part pictures and determines the second matching feature point, the processor can determine the rotation matrix and the translation matrix of each part picture according to the second matching feature point. The pose and the map point corresponding to each part picture are determined according to the rotation matrix and the translation matrix. After the processor determines the pose and the map point corresponding to each part picture according to the second matching feature point pair, the map point is re-projected into the part picture according to the pose, so as to obtain a projection point, determine a pixel point corresponding to the map point, compare the pixel point with the projection point, determine the error distance between the projection point and the pixel point, and adjust the pose, so as to adjust the error distance between the projection point and the pixel point. When the error distance is adjusted to be less than a preset threshold set by the processor, the processor can determine the adjusted pose and determine the adjusted pose as the optimal pose, and determine the map point obtained after the adjustment as the optimal map point. The processor can determine the reference frame of the part and the map point set of the part according to the feature points of the part picture and the optimal map point of the part picture determined. The reference frame set is obtained through the determined reference frame of the part, which serves as the reference basis for the inspection stage.

[0067] In one embodiment, a processor configured to perform the device inspection method based on feature point matching according to the above is provided.

[0068] In the above technical solution, the environment pictures to be inspected and the part pictures to be inspected are first collected, the reference frame of the part and the map point set are established according to the collected part pictures, and the visual dictionary model is determined to determine the feature points of the pictures. After the preparation stage is completed, the inspection pictures of the inspection process are obtained in real time, the feature points of the inspection pictures are collected and matched, the reference frame matching the inspection picture is selected from the prepared reference frame set, the target part in the inspection picture is determined according to the matching reference frame, and the inspected part is determined through the inspection picture, so as to ensure the integrity of the inspection.

[0069] In one embodiment, a device inspection apparatus based on feature point matching is provided, comprising the processor described above.

[0070] The processor comprises a core, and the core retrieves corresponding program units from the memory. The core can be one or more, and the device inspection method based on feature point matching is realized by adjusting core parameters.

[0071] The memory can include non-permanent memory in a computer readable medium, random access memory (RAM) and / or non-volatile memory such as read-only memory (ROM) or flash memory (flash RAM), and the memory comprises at least one memory chip.

[0072] In one embodiment, a computer device is provided, which can be a server, and the internal structure diagram thereof can be as shown in Figure 2 The computer device comprises a processor A01, a network interface A02, a memory (not shown in the figure) and a database (not shown in the figure) connected through a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device comprises an internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02 and a database (not shown in the figure). The internal memory A03 provides an environment for the operating system B01 and the computer program B02 in the non-volatile storage medium A04 to run. The database of the computer device is used to store the data of the field images and part images to be inspected collected by the image acquisition device. The network interface A02 of the computer device is used to communicate with the external terminal through network connection. The computer program B02 is executed by the processor A01 to realize a device inspection method based on feature point matching.

[0073] Figure 1 The flowchart of the device inspection method based on feature point matching in one embodiment is shown. It should be understood that although each step in the flowchart is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise stated in this article, the execution of these steps has no strict order restriction, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps in the flowchart can comprise multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with other steps or sub-steps or stages of other steps. Figure 1

[0074] ​The embodiment of the present application provides a device, the device comprises a processor, a memory and a program stored on the memory and executable on the processor, and the processor implements the following steps when executing the program: obtaining an inspection picture in an inspection process; selecting pictures similar to the inspection picture from a reference frame set to form a candidate reference frame set; determining a map point set corresponding to the candidate reference frame set; determining an inspection feature point of the inspection picture, and matching the inspection feature point with the map point set to obtain a matching point pair, so as to determine a target map point in the map point set that is matched with the inspection feature point; in the case that a proportion of the target map point in the map point set that is matched reaches a first proportion value, determining that a candidate reference frame corresponding to the map point set is a target reference frame; and determining a target spare part corresponding to the target reference frame as a target spare part included in the inspection picture.

[0075] In one embodiment, the selecting of the pictures similar to the inspection picture from the reference frame set to form the candidate reference frame set comprises: inputting the inspection picture into a visual bag-of-words model, so that the visual bag-of-words model selects the candidate reference frame set similar to the inspection picture from the reference frame set according to the inspection picture, wherein the candidate reference frame set comprises a plurality of candidate reference frames.

[0076] In one embodiment, the determining of the inspection feature point of the inspection picture and the matching of the inspection feature point with the map point set to obtain the matching point pair, so as to determine the target map point in the map point set that is matched with the inspection feature point comprises: determining the inspection feature point of the inspection picture, determining a first reference feature point of each candidate reference frame; matching the inspection feature point with the first reference feature point of each candidate reference frame, so as to determine a first matching feature point pair of the inspection picture and each candidate reference frame; determining a pose of the inspection picture according to the first matching feature point pair; re-projecting a map point corresponding to each candidate reference frame to the inspection picture according to the pose, so as to determine an inlier number of the inspection picture; and in the case that a proportion of the inlier number in the map point of the candidate frame reaches a second proportion value, determining that the map point corresponding to the candidate reference frame is the target map point that is matched with the inspection feature point.

[0077] In one embodiment, the matching of the inspection feature point with the first reference feature point of each candidate reference frame, so as to determine the first matching feature point pair of the inspection picture and each candidate reference frame comprises: inputting the inspection feature point and the first reference feature point into a leaf node of a visual bag-of-words model; the visual bag-of-words model sequentially compares the inspection feature point and the first reference feature point in the same leaf node, so as to determine a first feature distance between the inspection feature point and the first reference feature point; and determining a feature point pair with the smallest first feature distance as the first matching feature point pair.

[0078] In one embodiment, the determining the inlier number of the inspection picture according to the poses includes: projecting the map points corresponding to each candidate reference frame to the inspection picture according to the poses to obtain projected points; determining pixel points corresponding to the map points to obtain error distances between the projected points and the pixel points corresponding to the map points; adjusting the poses to adjust the error distances; determining the poses as final poses when the error distances are less than a preset threshold; and projecting the map points corresponding to each candidate reference frame to the inspection picture according to the final poses to determine the inlier number of the inspection picture.

[0079] In one embodiment, before the acquiring the inspection picture in the inspection process, the method further includes: collecting an environment picture of an environment in which the inspection process is located; extracting feature points of the environment picture, and generating a bag-of-words vector corresponding to the environment through a bag-of-words model, and characterizing the feature points of the picture through the bag-of-words vector.

[0080] In one embodiment, before the acquiring the inspection picture in the inspection process, the method further includes: collecting part pictures of each part to be recognized; and determining matched feature point pairs in feature points of the part pictures to determine poses and map points corresponding to the part pictures.

[0081] In one embodiment, the collecting part pictures of each part to be recognized and determining matched feature point pairs in feature points of the part pictures to determine poses and map points corresponding to the part pictures includes: performing feature extraction on the part pictures to obtain feature points of the part pictures; and matching the feature points of any two adjacent part pictures to determine second matched feature point pairs.

[0082] In one embodiment, the matching the feature points of the adjacent part pictures to determine the second matched feature point pairs includes: determining second feature distances between any two feature points belonging to different part pictures in the two adjacent part pictures; and determining a feature point pair composed of two feature points with the closest second feature distances as the second matched feature point pairs.

[0083] In one embodiment, the determining the poses and the map points corresponding to the part pictures includes: determining a rotation matrix and a translation matrix of each part picture according to the second matched feature point pairs; and determining the poses and the map points corresponding to each part picture according to the rotation matrix and the translation matrix.

[0084] In one embodiment, the adjusting the pose corresponding to the part picture to determine the optimal map point of the part picture comprises: re-projecting the map point into the part picture to obtain a projection point; determining a pixel point corresponding to the map point to obtain an error distance between the projection point and the pixel point corresponding to the map point; adjusting the pose to adjust the error distance; and determining the pose as the optimal pose and the map point as the optimal map point when the adjusted error distance is less than a preset threshold.

[0085] Those skilled in the art will understand that embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer usable program code.

[0086] The present application is described with reference to the flowcharts and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions, which are executed via the processor of the computer or other programmable data processing apparatus, generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the functions specified in the flowcharts and / or block diagrams.

[0087] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction means, which implement the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the functions specified in the flowcharts and / or block diagrams.

[0088] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide a process for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the functions specified in the flowcharts and / or block diagrams.

[0089] In one typical arrangement, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0090] Memory can include non-persistent memory and / or volatile memory, such as a random access memory (RAM) including a cache area for the temporary storage of data. A memory can also include non-volatile memory, such as a read only memory (ROM), EPROM, EEPROM, or flash memory. Memory can further include a data storage 110, which can include a disk drive, an optical memory, a solid-state memory, or other storage media. Memory can store computer readable instructions that, when processed by a processor, cause a computing device to perform operations.

[0091] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for the storage of information. Information can be computer readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0092] It should also be noted that the terms "comprising," "including," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements in the list, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus.

[0093] The above merely provides an example of the present application and is not intended to limit the present application. The present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application should be included in the scope of claims of the present application.

Claims

1. A device inspection method based on feature point matching, characterized in that, The method includes: Acquire inspection photos during the inspection process; Images similar to the inspection images are selected from the reference frame set to form a candidate reference frame set; Determine the map point set corresponding to the candidate reference frame set; Determine the inspection feature points of the inspection images, and determine the first reference feature point of each candidate reference frame; The inspection feature points and the first reference feature points are input into the leaf nodes of the visual bag-of-words model; The visual bag-of-words model compares the inspected feature points in the same leaf node with the first reference feature point in turn to determine the first feature distance between the inspected feature point and the first reference feature point. The first feature distance includes the Hamming distance between the inspected feature point and the first reference point. The feature point pair with the smallest first feature distance is determined as the first matching feature point pair; The pose of the inspection image is determined based on the first matching feature point pair; Based on the pose, the map points corresponding to each candidate reference frame are reprojected onto the inspection image to determine the number of interior points in the inspection image. If the proportion of the number of internal points in the map points where candidate reference frames exist reaches a second ratio value, the map point corresponding to the candidate reference frame is determined as the target map point that matches the inspection feature point. If the proportion of the matched target map points in the map point set reaches a first proportion value, the candidate reference frame corresponding to the map point set is determined as the target reference frame. The target component corresponding to the target reference frame is determined as the target component included in the inspection image.

2. The equipment inspection method based on feature point matching according to claim 1, characterized in that, The step of selecting images similar to the inspection images from the reference frame set to form a candidate reference frame set includes: The inspection image is input into the bag-of-visual-words model, so that the bag-of-visual-words model selects a set of candidate reference frames similar to the inspection image from the set of reference frames, wherein the set of candidate reference frames includes multiple candidate reference frames.

3. The equipment inspection method based on feature point matching according to claim 1, characterized in that, The step of reprojecting the map points corresponding to each candidate reference frame onto the inspection image based on the pose, and determining the number of interior points in the inspection image, includes: Based on the pose, the map points corresponding to each candidate reference frame are reprojected onto the inspection image to obtain the projection points; Determine the pixel corresponding to the map point to obtain the error distance between the projection point and the pixel corresponding to the map point; The pose is adjusted to regulate the error distance; If the error distance is less than a preset threshold, the pose is determined to be the final pose; Based on the final pose, the map points corresponding to each candidate reference frame are reprojected onto the inspection image to determine the number of interior points in the inspection image.

4. The equipment inspection method based on feature point matching according to claim 1, characterized in that, Before acquiring inspection photos during the inspection process, the following is also included: Collect environmental images of the inspection process; Feature points of the environment image are extracted, and a bag-of-words vector corresponding to the environment is generated using the visual bag-of-words model. The feature points of the image are then represented using the bag-of-words vector.

5. The equipment inspection method based on feature point matching according to claim 1, characterized in that, Before acquiring inspection photos during the inspection process, the following is also included: Collect images of each component to be identified; And determine the matching feature point pairs in the feature points of the component image to determine the pose and map points corresponding to the component image; Adjust the pose corresponding to the component image to determine the optimal map point of the component image; The reference frame and the map point set of the component are determined based on the feature points of the component image and the optimal map points of the component image.

6. The equipment inspection method based on feature point matching according to claim 5, characterized in that, The process of acquiring images of each component to be identified and determining matching feature point pairs among the feature points of the component images to determine the pose and map points corresponding to the component images includes: Feature extraction is performed on the component image to obtain the feature points of the component image; For any two adjacent component images, feature points of the adjacent component images are matched to determine a second matching feature point pair.

7. The equipment inspection method based on feature point matching according to claim 6, characterized in that, The step of matching feature points of any two adjacent component images to determine a second matching feature point pair includes: Determine the second feature distance between any two feature points belonging to different component images in two adjacent component images; The feature point pair consisting of the two feature points closest to the second feature is determined as the second matching feature point pair.

8. The equipment inspection method based on feature point matching according to claim 7, characterized in that, Determining the pose and map points corresponding to the component image includes: The rotation and translation matrices of each component image are determined based on the second matching feature point pair. The pose and map points corresponding to each component image are determined based on the rotation matrix and the translation matrix.

9. The equipment inspection method based on feature point matching according to claim 5, characterized in that, The step of adjusting the pose corresponding to the component image to determine the optimal map points of the component image includes: The map points are reprojected onto the component image to obtain the projection points; Determine the pixel corresponding to the map point to obtain the error distance between the projection point and the pixel corresponding to the map point; The pose is adjusted to regulate the error distance; If the adjusted error distance is less than a preset threshold, the pose is determined to be the optimal pose and the map point is determined to be the optimal map point.

10. A processor, characterized in that, The processor is configured to execute the device inspection method based on feature point matching according to any one of claims 1 to 9.

11. A device inspection apparatus based on feature point matching, characterized in that, Includes the processor as described in claim 10.

Images